What counts as a confirmed broken link?
Only HTTP 404 and 410 are labelled confirmed broken. HTTP 429 is rate-limited; DNS failures, timeouts and other HTTP failures are labelled unreachable.
Check a public site's internal and external links with robots.txt support, strict crawl limits, SSRF protections, progress and CSV export.
This limited crawler checks up to 10 public pages and 75 unique links, follows at most five redirects, respects robots.txt and reports confirmed 404/410 responses separately from rate limits and network failures.
Input: https://example.com
Output: A bounded list of confirmed broken, unreachable, rate-limited and redirected destinations
Only HTTP 404 and 410 are labelled confirmed broken. HTTP 429 is rate-limited; DNS failures, timeouts and other HTTP failures are labelled unreachable.
At most 10 pages, two link levels and 75 unique links, with four concurrent checks, five redirects per request and a 20-second runtime limit.
Yes. The starting site's robots.txt rules for this crawler are applied before pages are crawled, and blocked URLs are listed as skipped.
No. Localhost, private, loopback, link-local, metadata and reserved IP ranges, non-standard ports and redirects to unsafe destinations are blocked. DNS is pinned per request to reduce rebinding risk.