Screaming Frog SEO Spider

WORKING SLICE
Crawl one authorized site with a fixed page/depth budget, record technical metadata and compare raw HTML with optional rendered output. Export every finding with the affected URL, evidence and a concrete remediation note.

Build this scoped Screaming Frog SEO Spider-inspired workflow with a documented data model and visible failure states.

Architecture
- Python, FastAPI and server-rendered HTML with HTMX for a local interface.
- SQLite for manifests and job state, with an explicit worker process and immutable source files.
- HTTP crawling with HTML parsing and an optional bounded Playwright renderer for user-authorized sites.

Prerequisites and limits
A Python virtual environment, writable input/output directories and sufficient disk for both originals and outputs. Bind the service to localhost.
Outside this release: years of crawl edge cases, JavaScript rendering, integrations, and desktop performance; massive hosted crawl capacity

Data model and correctness
crawl frontiers, fetched/rendered snapshots, redirect chains, link edges, indexability signals and findings
Invariant: Orphan pages require an external URL inventory to establish; robots denial and fetch failure are not equivalent to noindex.
Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.

Security and privacy
Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs. Constrain crawl hosts, block private-network URLs and redirect pivots, and respect crawl delays and access restrictions.

Recovery and export
Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together.

Implementation order
1. Phase 1 — Scope and fixtures. Implement this bounded workflow: Crawl one authorized site with a fixed page/depth budget, record technical metadata and compare raw HTML with optional rendered output. Export every finding with the affected URL, evidence and a concrete remediation note. Record prerequisites, select representative user-owned fixtures and document the unsupported features: years of crawl edge cases, JavaScript rendering, integrations, and desktop performance; massive hosted crawl capacity
2. Phase 2 — Durable model. Model crawl frontiers, fetched/rendered snapshots, redirect chains, link edges, indexability signals and findings Add migrations or a versioned document format, explicit validation, stable IDs and a visible import-error report. Preserve this rule: Orphan pages require an external URL inventory to establish; robots denial and fetch failure are not equivalent to noindex.
3. Phase 3 — Complete the first useful path. Implement the workflow's input, review and output interface, with clear controls and explicit empty/error states. Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.
4. Phase 4 — Permissions and integration failure. Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs. Constrain crawl hosts, block private-network URLs and redirect pivots, and respect crawl delays and access restrictions. Request integration credentials and permissions only for the enabled feature; show a disconnected state instead of mock results.
5. Phase 5 — Portable handoff. Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together. Include setup, operating limits, fixture walkthrough and shutdown/restart instructions in the README.
6. Phase 6 — Acceptance scenarios. A redirect loop terminates within a defined budget; resuming an interrupted crawl retains dated results and identifies pages not yet fetched. Repeat the workflow after restart and with a denied permission or unavailable dependency; show recoverable failure rather than a success placeholder.

Acceptance
A redirect loop terminates within a defined budget; resuming an interrupted crawl retains dated results and identifies pages not yet fetched.
Use real source data or clearly labeled fixtures. Explain unsupported input and provider failures; do not fabricate analytics, delivery receipts, accuracy claims or security guarantees.

Optional agent guidance
Optional external skill: [seo-audit](https://github.com/coreyhaines31/marketingskills/blob/main/skills/seo-audit/SKILL.md) — Investigate crawlability, indexing, page metadata, internal linking and on-page content issues. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Optional external skill: [modern-python](https://github.com/trailofbits/skills/blob/main/plugins/modern-python/skills/modern-python/SKILL.md) — Set up Python projects with pyproject.toml, dependency management, linting, typing and automated checks. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Optional external skill: [agent-browser](https://github.com/vercel-labs/agent-browser/blob/main/skills/agent-browser/SKILL.md) — Automate browser interaction using accessibility snapshots, element references and reproducible navigation workflows. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Project rule — data model: crawl frontiers, fetched/rendered snapshots, redirect chains, link edges, indexability signals and findings
Project rule — preserve this invariant: Orphan pages require an external URL inventory to establish; robots denial and fetch failure are not equivalent to noindex.
Project rule — acceptance evidence: A redirect loop terminates within a defined budget; resuming an interrupted crawl retains dated results and identifies pages not yet fetched.

# AGENTS.md — Build guide for Screaming Frog SEO Spider

## Project scope
Crawl one authorized site with a fixed page/depth budget, record technical metadata and compare raw HTML with optional rendered output. Export every finding with the affected URL, evidence and a concrete remediation note.

Catalogue verdict: yes. The core loop is small enough for a capable coding agent to produce a useful local version in one sitting. For Screaming Frog SEO Spider, crawl an owned site locally, inspect technical signals, and export raw audit data. The hard boundary is years of crawl edge cases, javascript rendering, integrations, and desktop performance, plus crawl scale, rule depth, and operational polish.
Use the implementation prompt below to define the deliverable. Complete each phase's acceptance checks before extending the scope.

## Working agreement
- Inspect the repository and its existing instructions before choosing paths, dependencies or commands. Keep one coherent stack and explain changes to the proposed architecture.
- Plan a vertical slice that accepts a real input and produces the useful output described below. Persist only the state the prompt calls for; respect memory-only and upstream-managed workflows. Use fixtures only when they are clearly labelled.
- After scaffolding, document the actual install, development, check and build commands in README and keep them synchronized with the package or project manifest. Do not report commands as successful unless they ran.
- Work in small steps. At handoff, list implemented flows, checks actually performed, remaining blockers, and any credentials or provider setup the owner must supply.
- Do not publish, spend money, contact customers, delete source data or run irreversible migrations without the project owner's authorization.

## Prerequisites
- A Python virtual environment, writable input/output directories and sufficient disk for both originals and outputs. Bind the service to localhost.
- Implementation components: Python, FastAPI and server-rendered HTML with HTMX for a local interface. SQLite for manifests and job state, with an explicit worker process and immutable source files. HTTP crawling with HTML parsing and an optional bounded Playwright renderer for user-authorized sites.
- Scope boundary: years of crawl edge cases, JavaScript rendering, integrations, and desktop performance; massive hosted crawl capacity

## Stack and architecture
- Python, FastAPI and server-rendered HTML with HTMX for a local interface.
- SQLite for manifests and job state, with an explicit worker process and immutable source files.
- HTTP crawling with HTML parsing and an optional bounded Playwright renderer for user-authorized sites.
- Domain model: crawl frontiers, fetched/rendered snapshots, redirect chains, link edges, indexability signals and findings

## Security and data integrity
- Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs. Constrain crawl hosts, block private-network URLs and redirect pivots, and respect crawl delays and access restrictions.
- Correctness boundary: Orphan pages require an external URL inventory to establish; robots denial and fetch failure are not equivalent to noindex.
- Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.
- Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together.

## Agent implementation rules
- Project rule — data model: crawl frontiers, fetched/rendered snapshots, redirect chains, link edges, indexability signals and findings
- Project rule — preserve this invariant: Orphan pages require an external URL inventory to establish; robots denial and fetch failure are not equivalent to noindex.
- Project rule — acceptance evidence: A redirect loop terminates within a defined budget; resuming an interrupted crawl retains dated results and identifies pages not yet fetched.

## Optional agent skills and references
- Optional external skill: [seo-audit](https://github.com/coreyhaines31/marketingskills/blob/main/skills/seo-audit/SKILL.md) — Investigate crawlability, indexing, page metadata, internal linking and on-page content issues. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
- Optional external skill: [modern-python](https://github.com/trailofbits/skills/blob/main/plugins/modern-python/skills/modern-python/SKILL.md) — Set up Python projects with pyproject.toml, dependency management, linting, typing and automated checks. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
- Optional external skill: [agent-browser](https://github.com/vercel-labs/agent-browser/blob/main/skills/agent-browser/SKILL.md) — Automate browser interaction using accessibility snapshots, element references and reproducible navigation workflows. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.

Read the linked SKILL.md and its dependencies before adding a skill. Select only the skills matching this project's runtime and task; their documentation does not supply API access, credentials or approval to perform external actions. Pin the reviewed revision where the tool supports it. Follow the chosen agent's documented project-level installation mechanism.

## Distribution ideas
These are optional planning notes. Obtain the owner's approval before publishing or contacting anyone.
- Demonstrate the actual Screaming Frog SEO Spider-inspired workflow with owned or clearly labeled sample data: Crawl one authorized site with a fixed page/depth budget, record technical metadata and compare raw HTML with optional rendered output. Export every finding with the affected URL, evidence and a concrete remediation note.
- Publish a reproducible walkthrough with this observable result: A redirect loop terminates within a defined budget; resuming an interrupted crawl retains dated results and identifies pages not yet fetched.
- Explain who can operate this scoped tool, its setup and ongoing costs, and these remaining product gaps: years of crawl edge cases, JavaScript rendering, integrations, and desktop performance; massive hosted crawl capacity Avoid guaranteed savings, performance scores or implied endorsement.

## Engineering roadmap
1. Phase 1 — Scope and fixtures. Implement this bounded workflow: Crawl one authorized site with a fixed page/depth budget, record technical metadata and compare raw HTML with optional rendered output. Export every finding with the affected URL, evidence and a concrete remediation note. Record prerequisites, select representative user-owned fixtures and document the unsupported features: years of crawl edge cases, JavaScript rendering, integrations, and desktop performance; massive hosted crawl capacity
2. Phase 2 — Durable model. Model crawl frontiers, fetched/rendered snapshots, redirect chains, link edges, indexability signals and findings Add migrations or a versioned document format, explicit validation, stable IDs and a visible import-error report. Preserve this rule: Orphan pages require an external URL inventory to establish; robots denial and fetch failure are not equivalent to noindex.
3. Phase 3 — Complete the first useful path. Implement the workflow's input, review and output interface, with clear controls and explicit empty/error states. Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.
4. Phase 4 — Permissions and integration failure. Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs. Constrain crawl hosts, block private-network URLs and redirect pivots, and respect crawl delays and access restrictions. Request integration credentials and permissions only for the enabled feature; show a disconnected state instead of mock results.
5. Phase 5 — Portable handoff. Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together. Include setup, operating limits, fixture walkthrough and shutdown/restart instructions in the README.
6. Phase 6 — Acceptance scenarios. A redirect loop terminates within a defined budget; resuming an interrupted crawl retains dated results and identifies pages not yet fetched. Repeat the workflow after restart and with a denied permission or unavailable dependency; show recoverable failure rather than a success placeholder.

## Paid-product capabilities outside this build
- years of crawl edge cases, JavaScript rendering, integrations, and desktop performance
- massive hosted crawl capacity
- proprietary scoring
- continuous monitoring
- agency reporting and support

## Implementation prompt
WORKING SLICE
Crawl one authorized site with a fixed page/depth budget, record technical metadata and compare raw HTML with optional rendered output. Export every finding with the affected URL, evidence and a concrete remediation note.

Build this scoped Screaming Frog SEO Spider-inspired workflow with a documented data model and visible failure states.

Architecture
- Python, FastAPI and server-rendered HTML with HTMX for a local interface.
- SQLite for manifests and job state, with an explicit worker process and immutable source files.
- HTTP crawling with HTML parsing and an optional bounded Playwright renderer for user-authorized sites.

Prerequisites and limits
A Python virtual environment, writable input/output directories and sufficient disk for both originals and outputs. Bind the service to localhost.
Outside this release: years of crawl edge cases, JavaScript rendering, integrations, and desktop performance; massive hosted crawl capacity

Data model and correctness
crawl frontiers, fetched/rendered snapshots, redirect chains, link edges, indexability signals and findings
Invariant: Orphan pages require an external URL inventory to establish; robots denial and fetch failure are not equivalent to noindex.
Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.

Security and privacy
Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs. Constrain crawl hosts, block private-network URLs and redirect pivots, and respect crawl delays and access restrictions.

Recovery and export
Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together.

Implementation order
1. Phase 1 — Scope and fixtures. Implement this bounded workflow: Crawl one authorized site with a fixed page/depth budget, record technical metadata and compare raw HTML with optional rendered output. Export every finding with the affected URL, evidence and a concrete remediation note. Record prerequisites, select representative user-owned fixtures and document the unsupported features: years of crawl edge cases, JavaScript rendering, integrations, and desktop performance; massive hosted crawl capacity
2. Phase 2 — Durable model. Model crawl frontiers, fetched/rendered snapshots, redirect chains, link edges, indexability signals and findings Add migrations or a versioned document format, explicit validation, stable IDs and a visible import-error report. Preserve this rule: Orphan pages require an external URL inventory to establish; robots denial and fetch failure are not equivalent to noindex.
3. Phase 3 — Complete the first useful path. Implement the workflow's input, review and output interface, with clear controls and explicit empty/error states. Save a job manifest with input hash, parameters and state. Write to temporary outputs, then atomically finalize only successful results; resume unfinished jobs without replacing originals.
4. Phase 4 — Permissions and integration failure. Bound file sizes and processing time, reject path traversal, and use argument arrays for subprocesses. Treat imported text as data and redact confidential source content from logs. Constrain crawl hosts, block private-network URLs and redirect pivots, and respect crawl delays and access restrictions. Request integration credentials and permissions only for the enabled feature; show a disconnected state instead of mock results.
5. Phase 5 — Portable handoff. Export sources, manifests and outputs with checksums. Keep failed-job diagnostics and allow retry into a new output path; restore the database and file directory together. Include setup, operating limits, fixture walkthrough and shutdown/restart instructions in the README.
6. Phase 6 — Acceptance scenarios. A redirect loop terminates within a defined budget; resuming an interrupted crawl retains dated results and identifies pages not yet fetched. Repeat the workflow after restart and with a denied permission or unavailable dependency; show recoverable failure rather than a success placeholder.

Acceptance
A redirect loop terminates within a defined budget; resuming an interrupted crawl retains dated results and identifies pages not yet fetched.
Use real source data or clearly labeled fixtures. Explain unsupported input and provider failures; do not fabricate analytics, delivery receipts, accuracy claims or security guarantees.

Optional agent guidance
Optional external skill: [seo-audit](https://github.com/coreyhaines31/marketingskills/blob/main/skills/seo-audit/SKILL.md) — Investigate crawlability, indexing, page metadata, internal linking and on-page content issues. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Optional external skill: [modern-python](https://github.com/trailofbits/skills/blob/main/plugins/modern-python/skills/modern-python/SKILL.md) — Set up Python projects with pyproject.toml, dependency management, linting, typing and automated checks. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Optional external skill: [agent-browser](https://github.com/vercel-labs/agent-browser/blob/main/skills/agent-browser/SKILL.md) — Automate browser interaction using accessibility snapshots, element references and reproducible navigation workflows. Review its instructions and compatibility before use; it does not grant deployment, data-access or publication permission.
Project rule — data model: crawl frontiers, fetched/rendered snapshots, redirect chains, link edges, indexability signals and findings
Project rule — preserve this invariant: Orphan pages require an external URL inventory to establish; robots denial and fetch failure are not equivalent to noindex.
Project rule — acceptance evidence: A redirect loop terminates within a defined budget; resuming an interrupted crawl retains dated results and identifies pages not yet fetched.

## Completion evidence
Demonstrate the prompt's acceptance scenarios against the scoped workflow. Include setup from a clean checkout and failure recovery. Check persistence across restart and export/restore only for the state the prompt says to store; for memory-only tools, confirm that temporary content is discarded as specified. Record actual results and remaining limitations. A detailed plan alone does not establish a working replacement.