Wagtail Link Audit

Wagtail Link Audit is a Django/Wagtail application for finding and managing broken links across a website. It crawls internal pages, checks both internal and external links, and identifies broken or problematic URLs. Scans can be configured with crawl-depth and page limits and run without additional infrastructure such as Celery, Redis, or a separate task queue.

Rather than simply producing a report, the application provides a review workflow directly within Wagtail Admin. Content managers can review issues by severity, track them as new, under review, fixed, or whitelisted, add notes for other team members, and export results to CSV. Email summaries can also notify designated users when a scan completes and provide direct access to the results.

Known false positives can be excluded from future scans by whitelisting individual URLs, URL patterns, or entire domains—for example, external sites that intentionally reject automated requests. This allows repeated scans to focus on actionable problems while keeping link maintenance integrated with the CMS content managers already use.

Wagtail Link Audit provides automated broken-link scanning for Django and Wagtail websites, with the results integrated directly into the Wagtail administration interface. It checks both internal and external links and gives content managers a central place to review and manage problems without requiring developer access.

Scans can be configured with page and crawl-depth limits and scheduled to run automatically using the hosting platform's existing scheduling tools. When a scan completes, configured recipients receive an email summary and can review the full results in Wagtail.

Broken links are categorized by severity to distinguish clear failures, such as 404 responses and connection errors, from responses that may require further investigation, such as authentication errors, rate limiting, or servers that block automated requests.

The application also includes a review workflow, notes, CSV exports, URL and domain whitelisting, configurable email recipients, and tools for cleaning up historical scan data.

The application uses a synchronous crawler that starts with the site's configured base URL and follows internal links through the website. As pages are processed, discovered links are recorded and checked for HTTP errors or connection failures. Internal links are followed within configurable page and depth limits, while external links are checked without crawling the external site.

Each audit is stored as a Scan, with individual problems recorded as BrokenLink records containing information such as the source page, target URL, HTTP status, review status, notes, and review history. Additional models maintain URL exclusions, domain exclusions, and notification recipients.

The scanner intentionally does not require a background task queue. A scan runs as a standard Django management command, making it easy to execute manually or schedule through cron, systemd, Heroku Scheduler, or another platform scheduler. This keeps the package independent of infrastructure such as Celery, Redis, RabbitMQ, or dedicated worker processes.

The Wagtail admin provides the management layer on top of the scanner. Content managers can review results and maintain URL-level exclusions, while more administrative operations, such as managing globally excluded domains, can remain restricted to superusers.

Scan results are designed to function as a working maintenance queue rather than simply a report of HTTP errors.

Each detected problem records both the URL that failed and the page where the link was found, allowing content managers to identify where a correction needs to be made. Results can be filtered by scan or review status and searched by URL or notes.

Links move through a simple review workflow, including statuses for new issues, items under review, fixed links, and whitelisted results. Notes can be added to document what was investigated or how an issue was resolved.

Not every unsuccessful HTTP response represents a genuinely broken link. Some external services block automated requests, require authentication, or impose rate limits. These results are identified separately for investigation rather than automatically being treated as critical failures.

Known exceptions can be excluded from future reports. Individual URLs can be whitelisted using exact or prefix matching, while entire domains can be excluded when a service consistently prevents automated link checking.

Wagtail Link Audit was originally developed to solve a recurring maintenance problem across multiple Wagtail websites rather than for a single project. The functionality was separated into a reusable Django application so that link auditing, reporting, and the content-manager workflow could be maintained in one codebase and installed across multiple sites.

Site-specific details such as the base URL, site name, administration URL, notification addresses, and scan limits are provided through Django settings or Wagtail configuration rather than being embedded in the application.

Packaging the functionality separately also means improvements and bug fixes can be made once and incorporated into each website through a package version update instead of maintaining independent copies of the same auditing code.

The application deliberately keeps its infrastructure requirements small. Because it does not depend on a particular task queue, hosting provider, or deployment architecture, it can be added to a conventional Django/Wagtail project without introducing additional services solely to support link auditing.