The site audit pipeline orchestrates concurrent crawling, HTML analysis, and status tracking: SiteAuditWorkflow wraps D1 reads in replay-safe steps, delegates crawl execution to runAuditPhases, and marks audit completion or failure with telemetry; AuditService exposes a public API (startAudit, getStatus, getCrawlProgress, getResults, getHistory, remove) that handles redirect resolution, concurrency limits, workflow termination races, and self-heals stale running audits via reconcileRunningAudit. HTML analysis in page-analyzer.ts uses streaming tokenization instead of DOM parsing to avoid 5–10× memory overhead; analyzeHtml extracts title, headings, links (capped at 1,000), images (capped at 1,000), metadata, OG tags, and structured data presence, with anchor text capped at 200 characters and non-content tags (<script>, <style>, <noscript>, <svg>) excluded from word count.
The site audit crawl architecture ADR (specs/0009-site-audit-crawl-architecture.md) has status Accepted.[1] AuditScratchpad stores the crawl frontier, link edges, and a page mirror in the DO's SQLite; it is destroyed at finalize and self-cleans via an alarm if the audit dies, per wrangler.jsonc comments.[2] AuditService.ts exports AuditService, a const object exposing seven methods: resolveAuditLimitTier, startAudit, getStatus, getCrawlProgress, getResults, getHistory, and remove.[3]
SiteAuditWorkflow in src/server/workflows/SiteAuditWorkflow.ts wraps the initial DB read in a pgStep step ("validate-context") so that D1 reads are retried and replay-cached; a bare read outside a step would re-execute on every replay, and a transient failure would kill the workflow instance before the catch handler runs.[4] SiteAuditWorkflow in src/server/workflows/SiteAuditWorkflow.ts validates the workflow context by checking both that the audit record exists for the workflow instance ID and that its projectId matches the parameter — mismatches throw immediately, preventing phantom workflow runs.[4] SiteAuditWorkflow in src/server/workflows/SiteAuditWorkflow.ts delegates the main crawl execution to runAuditPhases, passing auditId, workflowInstanceId, billingCustomer, projectId, startUrl, and config.[4] On audit failure, SiteAuditWorkflow in src/server/workflows/SiteAuditWorkflow.ts reads currentPhase before calling AuditRepository.failAudit (which stamps currentPhase = "failed"), preserving the phase in which the failure occurred as failedPhase.[4] After marking an audit as failed, SiteAuditWorkflow fires a site_audit:complete PostHog event with status: "failed", error_code, failed_phase, pages_crawled, pages_total, and run_lighthouse properties.[4] The error handler in src/server/workflows/SiteAuditWorkflow.ts explicitly skips PostHog capture for "Durable Object reset because its code was updated" errors, treating deploy-time resets as expected churn rather than actionable errors.[4]
startAudit in AuditService.ts resolves redirect chains on the startUrl before storing it and launching the workflow, so that a domain that 301s to another origin does not dead-end after one page at the same-origin crawl boundary.[3] startAudit in AuditService.ts inserts the audit row first and enforces concurrency and capacity limits after the insert, deliberately avoiding a check-then-act race; if the post-insert check fails, the workflow is terminated and the row is deleted via a rollback path.[3] A 'running' audit row with a workflowInstanceId but no live workflow can occur legitimately — if startAudit's rollback delete fails after the workflow was never created, the row persists as stale. remove in AuditService.ts accounts for this by catching a null instance and skipping terminate().[3] getStatus in AuditService.ts self-heals audits stuck in 'running' status by calling reconcileRunningAudit when the workflow died without reaching the mark-failed step (terminated, errored, or expired from retention) — without this, stale running audits would hold capacity forever.[3] getCrawlProgress in AuditService.ts delegates to AuditProgressKV.getCrawledUrls(auditId) to return the list of crawled URLs for an in-progress audit.[3] getHistory in AuditService.ts derives a ranLighthouse boolean for each audit entry by checking whether parsedConfig.lighthouseStrategy !== 'none', avoiding a separate database column for this flag.[3] remove in AuditService.ts handles the race where terminate() throws because the workflow reached a terminal state between the user clicking stop and the termination call: it re-checks live workflow status and only raises AppError('CONFLICT') if the workflow is genuinely still in a running-like state (queued, running, paused, waiting, waitingForPause).[3] remove in AuditService.ts calls getAuditScratchpad(auditId).destroy() as a best-effort cleanup after deleting the audit row; a missed destroy self-cleans via the Durable Object's 7-day alarm.[3]
src/server/lib/audit/page-analyzer.ts uses htmlparser2's streaming tokenizer instead of a DOM parser (previously cheerio) because building a full DOM consumed 5–10× the HTML size per page, which was the dominant OOM cause when 25 pages were parsed concurrently on a 128 MB isolate.[5] analyzeHtml in page-analyzer.ts accepts statusCode, responseTimeMs, and an optional redirectUrl (defaults to null), and extracts: title, meta description, headings (H1–H6 with order), images, internal/external links, canonical URL, OG tags (og:title, og:description, og:image), structured data presence, robots meta, word count, hreflang tags, and redirect URL — returning all fields in the PageAnalysis result.[5] analyzeHtml uses a ??= first-write-wins pattern for metaDescription, canonical, robotsMeta, and OG tags, so only the first occurrence of each in the document is recorded.[5] analyzeHtml caps extracted links at MAX_EXTRACTED_LINKS (1,000) and images at MAX_EXTRACTED_IMAGES (1,000) per page to prevent uncapped collections from causing OOM in the 25-page persist batches of the audit crawl.[5] Anchor text in extracted links is capped at MAX_ANCHOR_CHARS (200 characters) via slice(0, MAX_ANCHOR_CHARS) in page-analyzer.ts.[5] Links with javascript:, mailto:, tel:, or # protocols are skipped during extraction in page-analyzer.ts via the SKIPPED_LINK_PROTOCOLS regex.[5] Text inside <script>, <style>, <noscript>, and <svg> subtrees is excluded from visible-content word-count accumulation in page-analyzer.ts via NON_CONTENT_TAGS suppression.[5] analyzeHtml preferentially collects visible text from inside <body>; for HTML fragments that never open a <body>, it falls back to all non-<head> text.[5] Word count in analyzeHtml is computed by splitting the normalized visible-body text on whitespace (/\s+/); an empty body returns 0.[5] analyzeHtml handles implicit nested <a> tags by closing any open anchor before opening a new one, mirroring browser behavior since HTML forbids nested anchors and the streaming tokenizer has no tree correction.[5] analyzeHtml ignores <title> elements inside <svg> (tracked by suppressDepth) and only records the first document-level <title> text via a titleDone flag.[5] analyzeHtml detects structured data by checking for <script type="application/ld+json"> and returns a boolean hasStructuredData field rather than parsing the JSON content.[5] analyzeHtml treats an <img> tag without an alt attribute differently from one with alt="": the alt field is null when the attribute is absent, and an empty string when it is explicitly set.[5]
Sources