Trawlspace runs the whole crawl-to-warehouse pipeline for you: on-page and AEO data, LLM enrichment, and change tracking. Clean, complete, and queryable through one API, so your team spends its time on decisions, not plumbing.
You run Screaming Frog, export the CSVs, and wire them into your warehouse by hand. It works until the crawl hangs overnight, an export changes a column and breaks your loader, and every client report is written from scratch. The £199 license was never the cost. Your engineers' and analysts' hours are.
Screaming Frog
A desktop app doing server work. It hangs and loses progress.
One license doing the work of a team
CSV exports feeding brittle ETL you maintain forever
Point-in-time only. No monitoring, no alerts.
Raw grids of "300 issues," no prioritization
Every fix, from schema to titles to meta, written by hand
Trawlspace
Start a crawl with an API call. It runs in the cloud.
Usage-based, multi-site, no per-seat license games
Clean structured results with JSON/CSV/Parquet export
Continuous monitoring & change alerts (on the roadmap)
Prioritized, severity-rated issues out of the box
Optional AI fix drafts, you keep the judgment
What it does
One API to crawl, audit, and fix.
The API is the product. A dashboard sits on top of it, not the other way around.
🕷️
Crawl any site by API
Start a crawl of any client site with one call. It runs in the cloud and scales from a few hundred pages to millions, rendering JavaScript when a page needs it. Nothing to install, nothing to babysit.
🧩
Structured on-page data
Titles, meta, headings, canonicals, hreflang, robots directives, redirect chains, broken links, Core Web Vitals, structured data, and the internal link graph. Clean JSON, not a desktop export.
🗂️
Page categorization · default + custom
Every page mapped to a type and segment, sensible defaults (PDP, PLP, blog, hub) out of the box, plus your own rules by URL pattern, template, or extracted field. Every issue, score, export, and alert rolls up by your categories.
✍️
LLM enrichment
Every page enriched for you: semantic categories, entities, summaries, answer-readiness scores, and schema validation. Want a fix drafted? Trawlspace will draft the JSON-LD, titles, meta, and FAQ too, as an optional output your team takes or leaves.
🔌
Delivered into your stack
Clean tables through the API, plus a JSON, CSV, or Parquet export to a signed URL, and a bulk export for big result sets. Provider-agnostic, no lock-in: pipe it into your warehouse, your dashboards, or your own AI agent. Native BigQuery and Snowflake connectors are on the roadmap.
🔎
Search your whole crawl
Query the entire indexed site, by field, issue, or category, and full-text across the crawled content. "Every blog post that mentions SOC 2 and is missing FAQ schema" is a single call.
See it on a real site
Point Trawlspace at ramp.com. Here's what comes back.
Not a feature list. The real page, the data we pull off it, the issues we rank, and the enrichment we layer on. (Homepage crawled live with Trawlspace; enrichment values illustrative.)
1 · EXTRACT
Every on-page signal, as clean JSON
One fetch of the live page, parsed into the fields your audit and your warehouse actually use, with issues flagged inline.
the live page
https://ramp.com/what Trawlspace returns
{
"url": "https://ramp.com/",
"status": 200,
"title": "Spend Management, Corporate Cards & Accounts Payable ... | Ramp",
"title_len": 69,
"meta_description_len": 192,
"h1": "Time is money. Save both.",
"h2": ["Corporate cards","Bill pay","Accounting"],
"canonical": "self", "lang": "en-US",
"structured_data": ["Organization"],
"links": { "internal": 82, "external": 8, "broken": 0 },
"core_web_vitals": { "lcp_ms": 2900, "cls": 0.04 },
"body_excerpt": "Ramp is the corporate card and spend platform that helps teams control spend...",
"ai_summary": "Strong brand; thin on Product / FAQ schema for AEO. 214 pages need schema.",
"related_pages": [
{ "url": "/pricing", "score": 91 },
{ "url": "/blog/soc-2", "score": 64 }
],
"issues": ["title_too_long","meta_too_long",
"missing_product_schema"]
}
2 · AUDIT
Prioritized issues, not a wall of 300 rows
Every finding normalized, typed, and severity-ranked across the whole crawl, filterable and roll-up-able by your own page segments. The critical one first.
criticalCDN blocks Claude-SearchBot & PerplexityBotYou're excluded from those AI answers, a bot rule, not robots.txtall pages
warningTitle over length (69 chars)Truncates in the SERP, real, on the homepage1 page
warningMeta description over 160 chars (192)Gets cut off in results88 pages
warningMissing Product / FAQ schemaRich-result and answer-engine eligibility left on the table214 pages
opportunityLCP > 2.5s from the hero imageCore Web Vitals "needs improvement" on high-traffic pages340 pages
opportunity22 JS-only links on the homepageContent that only appears after render, weaker for AI crawlers1 page
3 · OPTIONAL FIX DRAFT
Want the fix drafted? Trawlspace does that too.
For ramp.com/pricing, if you ask, Trawlspace drafts the rewritten title and meta, the schema, and the FAQ block. Your analyst takes it, edits it, or ignores it. You keep the judgment.
Can the AI crawlers reach the page, and is the content answer-ready? We report and score both. This is the part you control, not just a visibility score.
61/100
answer-readiness
6% pages with valid schema71% server-rendered
AI bot access 4 of 6 retrieval bots can reach you
GPTBot allowed
OAI-SearchBot allowed
ClaudeBot allowed
Claude-SearchBot blocked
PerplexityBot blocked
Google-Extended allowed
Top fix: your CDN bot rule blocks Claude-SearchBot and PerplexityBot, the retrieval bots that feed AI answers. Allow them and you become eligible for those citations.
Snapshot & diff
Every crawl is a snapshot. See exactly what changed.
Trawlspace keeps a timestamped snapshot of every crawl, so any two can be diffed field by field, with a plain-language read of what moved and why it matters, an audit log of what changed between publishes. Regressions fire an alert the moment they ship.
Jun 19
70
Jun 26
71
Jul 3
73
Jul 10 · now
72
GET /jobs/job_9f2a/diff?against=job_prev · Jul 10 ⟷ Jul 3
▲ 27 improvements▼ 3 regressionsnet health −1.4
field
Jul 3 · was
Jul 10 · now
/pricing canonical
self
staging.ramp.comREGRESSION
/blog robots
index,follow
noindexREGRESSION
pages w/ product schema
6%
21%▲
8 PDPs · lcp
2.9s
4.1s
site answer-readiness
44
61▲
✨ AI summaryWhat changed: Since Jul 3, 27 pages gained the Product schema you shipped, site answer-readiness rose 44 → 61. But three regressions need attention: the /pricing canonical now points to a staging URL, the blog hub went noindex, and 8 product pages slowed past LCP 4s. The noindex on the blog hub is the urgent one, an alert already went to Slack.
The whole product, as an API
Every feature is one call.
Crawl, audit, fix, track change, analyze, and deliver, a request in, a structured result out. Click through the endpoints.
Audit one URL, instantly. No crawl, no setup. Point at any live page and get the on-page facts, the enriched fields, and the ranked issues back in a single call. AI enrichment
GET Request
/v1/page?url=https://ramp.com/pricing
# any live URL, no job required.
# audit + AI recommendation in one call
Categorize every page, intelligently. Trawlspace reads each page and classifies it semantically with an LLM, no rules to write. Pin or override with your own rules or regex when you want precision. Every issue, score, export, and alert rolls up by category. AI classification
{
"matched": 37,
"results": [
{ "url": "/blog/soc-2-compliance",
"title": "Ramp is SOC 2 Type II ..." },
{ "url": "/blog/security-at-ramp",
"title": "Security at Ramp" }
]
}
# the whole crawl is queryable,
# find the pages that match, then fix them
Query the issues. Normalized, typed, severity-ranked, filter by severity, type, or your segment.
Query the link graph. Internal edges, anchors, redirect chains, broken links, and orphans, as data. The graph Screaming Frog makes you export and re-join by hand.
GET Request
/v1/jobs/job_9f2a/links?type=broken
# also: redirects, orphans, anchors.
# ?to=/pricing = who links to a page
Draft a fix, on request. Ask and Trawlspace drafts the artifact (new meta, schema, FAQ) and shows the answer-readiness lift. Your team edits it, ships it, or ignores it. You keep the judgment. AI draft
POST Request
/v1/jobs/job_9f2a/enrich
{
"url": "/pricing",
"tasks": ["schema","meta","faq"],
"brand_voice": "direct, no fluff"
}
POST /v1/sites then POST /v1/sites/{id}/jobs. Import your existing Screaming Frog scope. We crawl in the cloud and report progress.
We audit & enrich
We compute every SEO signal, rank the issues, and run the enrichment you asked for across the pages that need it, schema, rewrites, FAQ, answer-readiness.
Results land in your stack
The moment a job finishes, the results are queryable through the API and exportable as JSON, CSV, or Parquet. Your dashboards keep working; your analysts review, not rebuild.
Built for the AEO era
Answer-engine readiness, the part you can fix.
AEO is part of the job now. Trawlspace audits the site-side of it, the part you can actually fix, and drafts the recommended fix.
🤖
AI-bot crawlability
We check whether GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, and PerplexityBot can actually reach each page, at robots.txt and the CDN, and flag the common mistake of blocking the retrieval bots that feed AI answers.
📐
Answer-readiness scoring
A per-page score for how liftable your content is into an AI answer, schema, server-rendered content, clear headings, extractable Q&A, with the specific fixes to raise it, written for you.
To be clear: Trawlspace is not a measure-only AI-visibility dashboard and not a keyword question-bank. Measuring how often you're mentioned across AI answers is table stakes; the value is what to do about it. We work the site-side of perception: what your pages convey, what's wrong, and the fix, so you're eligible to be described the way you intend. Keep your Semrush or Ahrefs seat for research and rank tracking; Trawlspace replaces the crawl pipeline and the manual fixing.
Honest scope
What Trawlspace is not.
Not a measure-only dashboard.
Most AI-visibility tools stop at a score. Trawlspace does the part that moves it, the site-side fix. Measurement finds the gap; we close it.
Not a keyword/rank suite.
No keyword research or rank tracking. Keep your Semrush/Ahrefs seat; feed Trawlspace into your stack alongside it.
Not a desktop app.
No local install, no per-seat license games, no single machine that hangs at 2am on a big crawl.
Brand perception
Convey the perception you intend, on every page.
Trawlspace learns your brand voice and positioning from your best pages, scores every page for how well it conveys them, and flags the ones drifting off-message with the exact passages that break. Give AI the right material to describe you the way you mean, not the way it guesses. You decide what to rewrite.
Brand voice profileinferred from 12 exemplar pages
tonedirect, confident, no jargon
formality
reading levelgrade 9
avoid"synergy", "leverage", "best-in-class"
Pages trending off-voicethis crawl
/blog/enterprise-solutions48 ▼
/product/platform67
/pricing91
drift on /blog/enterprise-solutions: "Leverage our best-in-class synergistic solution..."jargon; your voice is plain and direct
1 · Identify
Auto-infer the voice rubric from your best pages, or define it: tone, formality, reading level, sentence structure, vocabulary, and a do/don't lexicon.
2 · Track deviation
Score every page, and watch the Human Edit Rate, the share of drafts your team still has to fix. Alerts fire the moment a page drifts after a CMS edit.
3 · Trainpremium
Fine-tune a per-brand model (LoRA adapters) on a curated golden set, so agencies get a cheap, swappable voice model per client.
Pricing
Transparent and usage-based.
You pay per page by render tier, with enrichment as a clear add-on, so cost tracks value. No five-figure quotes, no per-URL surprises. Prices are indicative.
Model your book: how many client sites, how big, and how often you crawl. Crawl is cheap. Enrichment is the value. Everything is usage-based, bundled into a plan allowance with overage.
Estimated
$0/mo
Pro / Team
Pro from $49/mo · Agency from $499/mo. Re-crawls billed at ~35% (conditional GET), so weekly costs far less than 4x monthly.
FAQ
Questions.
Do I replace Screaming Frog entirely?
You replace the brittle export-and-ETL pipeline you built around it. Run Trawlspace in parallel against the same seed lists for a cycle, diff the output, and cut over when you trust the data. Keep Screaming Frog on a desktop for one-off manual work if you like.
Do I keep my Semrush or Ahrefs seat?
Yes. Trawlspace doesn't do keyword research or rank tracking. Keep those tools for research; use Trawlspace to crawl, audit, fix, and deliver into your own stack alongside them.
Is this a measure-only AI-visibility tool?
No. Tracking your citations across AI answers is where most tools stop. Trawlspace works the fixable, site-side part: can the AI crawlers reach your pages, is the content answer-ready, and we draft the fix. Measurement finds the gap; moving it is the product.
How does my data get into my warehouse?
Today, clean structured results come through the API plus a JSON, CSV, or Parquet export, with a bulk export for large result sets, a far simpler load than parsing Screaming Frog CSVs. Native BigQuery and Snowflake connectors that auto-load each crawl are on the near roadmap.
What does "done-for-you" actually mean?
For flagged pages we generate the real artifact, the JSON-LD, the rewritten title and meta, the FAQ block, the internal-linking suggestion, as reviewable output. Your analyst approves and ships it instead of writing each one from scratch.
Get started
Bring one client site.
We'll run it in parallel with your current pipeline so you can see the difference on your own data: the crawl, the enriched data, and the prioritized issues.
the live page