POST/api/v1/scrape

Scrape

Fetch a URL and extract structured data using AI. The API handles fetching, rendering to markdown, and extraction in a single request.

Try it without a key

A small set of sample URLs run without an API key, in JSON format only, and don't count against any quota. Each one is a recorded run of the service, so it answers the exact request it was recorded with: its extract prompt and, where one is marked, its JSON Schema (key order does not matter). Send the request below with no Authorization header. A different prompt or schema needs a key like any other live extraction.

https://news.ycombinator.com

“Get the top 5 stories with title, URL, and points”

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com",
    "extract": "Get the top 5 stories with title, URL, and points"
  }'

https://example.com

“Extract the page title, heading, and all paragraph text”

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "extract": "Extract the page title, heading, and all paragraph text"
  }'

https://github.com/trending

“Get the top 5 trending repositories with name, description, language, and stars”

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://github.com/trending",
    "extract": "Get the top 5 trending repositories with name, description, language, and stars"
  }'

https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html

“The product name and its current price”, with its recorded JSON Schema

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "extract": "The product name and its current price",
    "schema": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "offers": {
          "type": "object",
          "properties": { "price": { "type": "number" }, "priceCurrency": { "type": "string" } },
          "required": ["price", "priceCurrency"],
          "additionalProperties": false
        }
      },
      "required": ["name", "offers"],
      "additionalProperties": false
    }
  }'

https://quotes.toscrape.com/

“Only the first 3 quotes on the page”, with its recorded JSON Schema

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://quotes.toscrape.com/",
    "extract": "Only the first 3 quotes on the page",
    "schema": {
      "type": "object",
      "properties": {
        "quotes": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "text": { "type": "string" },
              "author": { "type": "string" },
              "tags": { "type": "array", "items": { "type": "string" } }
            },
            "required": ["text", "author", "tags"],
            "additionalProperties": false
          }
        }
      },
      "required": ["quotes"],
      "additionalProperties": false
    }
  }'

https://www.contrie.com/pricing

“Each plan with its monthly price in USD and included monthly credits”

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.contrie.com/pricing",
    "extract": "Each plan with its monthly price in USD and included monthly credits"
  }'

https://github.com/trending/rust?since=weekly

“The top 10 trending repositories this week, in page order, with total stars, forks and stars gained this week”, with its recorded JSON Schema

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://github.com/trending/rust?since=weekly",
    "extract": "The top 10 trending repositories this week, in page order, with total stars, forks and stars gained this week",
    "schema": {
      "type": "object",
      "required": ["repositories"],
      "properties": {
        "repositories": {
          "type": "array",
          "items": {
            "type": "object",
            "required": [
              "rank",
              "repository",
              "description",
              "language",
              "stars",
              "forks",
              "stars_this_week"
            ],
            "properties": {
              "rank": { "type": "integer" },
              "repository": { "type": "string" },
              "description": { "type": ["string", "null"] },
              "language": { "type": ["string", "null"] },
              "stars": { "type": "integer" },
              "forks": { "type": "integer" },
              "stars_this_week": { "type": "integer" }
            }
          }
        }
      }
    }
  }'

https://en.wikipedia.org/wiki/List_of_tallest_buildings

“The five tallest buildings in rank order, with city, country, height in metres, floors and year completed”, with its recorded JSON Schema

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://en.wikipedia.org/wiki/List_of_tallest_buildings",
    "extract": "The five tallest buildings in rank order, with city, country, height in metres, floors and year completed",
    "schema": {
      "type": "object",
      "required": ["buildings"],
      "properties": {
        "buildings": {
          "type": "array",
          "items": {
            "type": "object",
            "required": ["rank", "name", "city", "country", "height_m", "floors", "year"],
            "properties": {
              "rank": { "type": "integer" },
              "name": { "type": "string" },
              "city": { "type": "string" },
              "country": { "type": "string" },
              "height_m": { "type": "number" },
              "floors": { "type": "integer" },
              "year": { "type": "integer" }
            }
          }
        }
      }
    }
  }'

https://arxiv.org/abs/1602.03837

“The citation details of this paper: title, authors, submission date, journal reference, DOI and primary subject”, with its recorded JSON Schema

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://arxiv.org/abs/1602.03837",
    "extract": "The citation details of this paper: title, authors, submission date, journal reference, DOI and primary subject",
    "schema": {
      "type": "object",
      "required": [
        "title",
        "authors",
        "submitted",
        "journal_reference",
        "doi",
        "primary_subject"
      ],
      "properties": {
        "title": { "type": "string" },
        "authors": { "type": "array", "items": { "type": "string" } },
        "submitted": { "type": "string" },
        "journal_reference": { "type": ["string", "null"] },
        "doi": { "type": ["string", "null"] },
        "primary_subject": { "type": "string" }
      }
    }
  }'

https://pypi.org/project/requests/

“The package name, latest version, release date, license, required Python version and one-line summary”, with its recorded JSON Schema

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://pypi.org/project/requests/",
    "extract": "The package name, latest version, release date, license, required Python version and one-line summary",
    "schema": {
      "type": "object",
      "required": ["name", "version", "released", "license", "requires_python", "summary"],
      "properties": {
        "name": { "type": "string" },
        "version": { "type": "string" },
        "released": { "type": "string" },
        "license": { "type": ["string", "null"] },
        "requires_python": { "type": ["string", "null"] },
        "summary": { "type": "string" }
      }
    }
  }'

https://en.wikipedia.org/wiki/Stripe,_Inc.

“The company profile from the infobox: name, industries, year founded, founders, headquarters, revenue and number of employees”, with its recorded JSON Schema

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://en.wikipedia.org/wiki/Stripe,_Inc.",
    "extract": "The company profile from the infobox: name, industries, year founded, founders, headquarters, revenue and number of employees",
    "schema": {
      "type": "object",
      "required": [
        "name",
        "industries",
        "founded",
        "founders",
        "headquarters",
        "revenue",
        "employees"
      ],
      "properties": {
        "name": { "type": "string" },
        "industries": { "type": "array", "items": { "type": "string" } },
        "founded": { "type": "integer" },
        "founders": { "type": "array", "items": { "type": "string" } },
        "headquarters": { "type": "string" },
        "revenue": { "type": ["string", "null"] },
        "employees": { "type": ["integer", "null"] }
      }
    }
  }'

https://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/index.en.html

“Every euro foreign exchange reference rate on the page, with the date they apply to”, with its recorded JSON Schema

Keyless request

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/index.en.html",
    "extract": "Every euro foreign exchange reference rate on the page, with the date they apply to",
    "schema": {
      "type": "object",
      "required": ["date", "rates"],
      "properties": {
        "date": { "type": "string" },
        "rates": {
          "type": "array",
          "items": {
            "type": "object",
            "required": ["code", "currency", "rate"],
            "properties": {
              "code": { "type": "string" },
              "currency": { "type": "string" },
              "rate": { "type": "number" }
            }
          }
        }
      }
    }
  }'

Any other URL requires a key from /dashboard/keys. The home page replays recorded examples with their evidence.

Parameters

Parameter	Type	Required	Description
url	string	Required	A public HTTP/HTTPS URL. Private and internal addresses are rejected, including when reached via a redirect.
extract	string	Optional	Natural language description of what to extract. Max 4,000 characters.
schema	object	Optional	A JSON Schema subset to enforce the output shape: types `object`, `array`, `string`, `number`, `integer`, `boolean`, `null`; keywords `properties`, `required`, `items`, `enum`, `description`. Max 10 KB. A local `$ref` is inlined, and an `anyOf` or `oneOf` of a type and `null` becomes that type made nullable, so Pydantic and zod output reads as this subset. Other keywords, such as value constraints like `minLength` or `pattern` and other combinators, are carried but not checked. A schema is refused with `INVALID_REQUEST` only when it cannot be read, such as `required` or `enum` given as something other than a list; the message names the keyword.
format	string	Optional	`json` (default), `markdown`, or `both`. `markdown` returns `data: null`.
render	string	Optional	`auto` (default) renders the page in a browser only when the plain fetch returns a JavaScript shell; `always` renders first; `never` keeps the plain fetch and may return `RENDER_REQUIRED`. An accepted rendered answer costs 2 extra credits, within the plan's per-request credit cap.
options	object	Optional	`timeout` — page fetch timeout in ms, 1000–30000 (default 15000). `maxAge` — maximum age of page bytes reused within your account, in seconds, 0–86400 (default 0 skips cache reads and writes). `waitFor` — a CSS selector, up to 200 characters, that browser rendering must find before capturing the source snapshot. Without it, network quiet is a snapshot heuristic and does not prove the application is ready.

Any other field, at the top level or inside options, is refused with INVALID_REQUEST naming it. Another API's name for one of these fields, such as prompt, or an MCP tool's top-level maxAge, also gets the name to send instead.

How Contrie fetches a page

A request fetches the one URL it names, following up to 10 redirects. It does not crawl a site, follow links or look for other pages. A monitor fetches its one URL on the schedule you set, and /api/v1/extract fetches nothing: you send the HTML.
Before fetching a page, Contrie reads that site's robots.txt (kept for up to an hour) and refuses, at 0 credits, any page the site disallows for User-agent: Contrie or marks Content-Signal: ai-input=no; each redirect hop is checked against its own site. Rules for every crawler (User-agent: *) and for other named agents govern crawling, and a one-page request is not a crawl, so they do not apply to it. A robots.txt that cannot be read (a server error or a challenge) says nothing, and the fetch goes ahead. To opt out, add User-agent: Contrie / Disallow: /.
Contrie's own discovery crawler is a crawler, and follows crawler rules. Once a day it reads a small, reviewed list of public reference sites: only pages the list names or the site's own sitemaps and feeds list, never links found in pages. It obeys the group naming Contrie, else User-agent: *, waits at least a second between requests to a site (longer when Crawl-delay asks), and skips pages marked Content-Signal: ai-input=no or search=no. Its requests carry the User-Agent below and are not signed. The same User-agent: Contrie / Disallow: / opts a site out of both.
Every request says it is Contrie: its User-Agent is a Chrome browser string followed by (compatible; Contrie/1.0; +https://www.contrie.com/docs/scrape#how-contrie-fetches-a-page). A plain fetch of a page on sec.gov or bls.gov, including their subdomains, sends a User-Agent naming Contrie with a contact address instead, because those sites ask automated clients to identify themselves.
Every customer request, plain fetch or browser render, is signed with Web Bot Auth (HTTP Message Signatures, Ed25519): every hop carries Signature-Agent: "https://www.contrie.com" and a fresh signature for that host, and the public key is published at /.well-known/http-message-signatures-directory. In a browser render, that includes each redirect hop and every request the page makes. A signature proves a request came from Contrie; each site still decides what it allows.
A request that names a reddit.com URL is never fetched as a page or rendered in a browser. It is read only through Reddit's own API, and refused where that API is not available. A plain fetch that is redirected to reddit.com is refused the same way, and a browser render never loads anything from reddit.com.
A PyPI project page, an npm package page and, where the operator has an api.data.gov key, a congress.gov bill page are read through that site's own JSON API with a User-Agent naming Contrie and a contact address, and never rendered in a browser. The site's robots.txt is still honored, the URL you sent stays the source URL, and a refusal from the API is returned as it is, never retried.
On a plain fetch (render set to auto or never), when the site answers 401, 403, 429 or 451, the request ends with SOURCE_REFUSED at 0 credits. So does a 2xx that is an anti-bot challenge Contrie recognizes (today, Fastly's Client Challenge, AWS WAF's challenge, and a nearly empty Cloudflare, PerimeterX, Distil, DataDome or Akamai challenge or denial page) instead of the page. Contrie does not retry it or render it in a browser instead. A browser render also refuses those HTTP statuses. With an explicit always render, the page's own scripts may resolve a challenge; a recognized challenge still present when rendering completes stops with SOURCE_REFUSED at 0 credits. A 429 or a challenge page means “not now”, and the message says to retry later. A 429 with a usable Retry-After passes that wait back to you in seconds; see Error Codes. A 404 or 410, or a short 2xx page whose title or heading says it was not found, ends with FETCH_FAILED at 0 credits; retrying the same URL will not help, so check it instead.
Every hop, redirects included, is resolved and checked before connecting, and a URL that reaches a private or internal address, or a port other than 80, 443, 8080 or 8443, is refused with INVALID_URL. The connection goes to the address that passed the check.
One deadline covers the whole fetch, redirects included: options.timeout, 15 seconds by default. A page may be up to 10 MB, and Contrie reads at most the first 100,000 characters of its markdown, for extraction and markdown alike; metadata.evidenceCoverage.sourceTruncated is true when it cut. A file is refused with FETCH_FAILED at 0 credits when it is served as a PDF, an image other than SVG, audio, video, a font, an archive or an office document, or when its first bytes are a PDF, PNG, JPEG, GIF, TIFF, WebP, ZIP, legacy Office (OLE), gzip, bzip2, xz, MP3, WAV, AVI, Ogg, FLAC, WebM, MP4/MOV or Windows executable file. On a plain fetch, other binaries under a generic label such as application/octet-stream are read as text.
With render set to auto, the default, a headless Chromium browser loads the page only when the plain fetch returned a JavaScript shell; always skips the plain fetch and goes straight to the browser. The browser loads the page with the scripts and data requests it makes, each checked against private addresses like the page itself, then captures the HTML and a screenshot. Browser responses of 401, 403, 429 or 451 end with SOURCE_REFUSED at 0 credits; a usable 429 Retry-After is preserved for REST. Other HTTP errors, startup failures and readiness timeouts retain the existing bounded rendering fallback; an explicit always render that cannot complete returns RENDER_REQUIRED at 0 credits. The original deadline still applies while waiting for scripts or a requested selector. The browser refuses a file with the same FETCH_FAILED when it is served with one of the media types above, or when the browser shows it and the type the browser detects or its first bytes are one of those; any other file the browser downloads instead of showing ends with RENDER_REQUIRED at 0 credits.

Choose freshness for repeat reads

Use options.maxAge to avoid fetching the same reference page on every call. Page reuse stays within your account; extraction still runs on the selected bytes and credit prices stay the same. A cache hit reports metadata.cached and the original metadata.fetchedAt.

Reference docs: accept an hour with maxAge: 3600.
Changing pages: choose a short window that fits your refresh schedule.
Final check before acting: use maxAge: 0 for a live fetch.

Origin expiry can shorten your window. URLs with query strings, fragments or user information, and responses with restrictive cache headers or cookies, bypass caching. This setting controls page reuse, not extraction-log retention. A fresh fetch shows what the page returned; it does not prove that a listed price, job or other fact is still valid.

Safe retries

On an authenticated JSON request, send an optional Idempotency-Key header of 8–200 visible ASCII characters. The key binds your account, this operation and every behavior-affecting input. An identical retry replays the encrypted response for 24 hours without fetching, inference or another credit charge. Changed inputs return IDEMPOTENCY_CONFLICT; after the response expires the durable key returns IDEMPOTENCY_EXPIRED. If work expires without a confirmed response, the key returns IDEMPOTENCY_OUTCOME_UNKNOWN (409). That outcome is terminal: the key starts no more work and has no retry hint. Contact support to reconcile the request. An active request can still return REQUEST_IN_PROGRESS with a short retry hint. Use a new key only when you intend new work. Idempotency keys are not accepted for keyless samples or NDJSON streaming.

Generate one key per distinct request and keep it with that request. A fixed key copied into every call returns IDEMPOTENCY_CONFLICT as soon as any input changes.

shell

KEY=$(uuidgen)   # new work: a new key
curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ck_live_your_key_here" \
  -H "Idempotency-Key: $KEY" \
  -d @request.json
# Connection dropped? Run the same curl again with the same $KEY and request.json.

Request Example

curl

curl -X POST https://www.contrie.com/api/v1/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ck_live_your_key_here" \
  -d '{
    "url": "https://news.ycombinator.com",
    "extract": "Get the top 5 stories with title, URL, and points",
    "options": {
      "timeout": 20000
    }
  }'

Response

Inspect success before using the answer. Acceptance requires actual validation, requested-field completeness and the quality threshold together; a high score alone is insufficient. Validation includes finding at least half of all returned values in the page text. It also requires more than 30% of the fields in the answer to be populated: metadata.completeness above 0.3. Null and empty values are not populated; zero and false are. An answer at or below that share is refused, so request fewer fields. A refused answer has no data, signed receipt or customer credit charge. Credit, budget and Free-capacity limits return HTTP 429 (PAGE_LIMIT_REACHED or INFERENCE_BUDGET_EXCEEDED); a temporarily unavailable service returns 503. Check the error code and Retry-After to decide whether to retry. The example below is abridged from a real run of this request recorded on 2026-09-22 (one of five items, one of fifteen grounding entries). A natural-language request like this one scores at most 80: without a schema the requested fields cannot be counted, so coverageMeasured is false and completeness earns half credit. Send a JSON Schema to have coverage measured.

200 OK

{
  "success": true,
  "data": [
    {
      "title": "AMD's random number generator can't generate a 0?",
      "url": "https://board.flatassembler.net/topic.php?t=24261",
      "points": 119
    }
  ],
  "metadata": {
    "url": "https://news.ycombinator.com",
    "qualityScore": 80,
    "qualityScoreKind": "structural_and_textual",
    "valid": true,
    "coverageMeasured": false,
    "completeness": 1,
    "grounding": 1,
    "latencyMs": 1666,
    "trace": [
      { "stage": "loading", "detail": "Loading source" },
      { "stage": "processing", "detail": "Processing request" },
      { "stage": "complete", "detail": "Request complete" }
    ],
    "groundingFields": [
      {
        "path": "$[0].points",
        "value": "119",
        "grounded": true,
        "excerpt": "…ator can't generate a 0? (flatassembler.net) 119 points by BruceEel 3 hours ago \\| hide \\| 64…"
      }
    ],
    "groundingCoverage": { "total": 15, "checkable": 15, "grounded": 15 },
    "cached": false,
    "credits": 1,
    "evidenceStatus": "signed"
  }
}

Key Response Fields

Field	Meaning
qualityScore	0–100. A scoped indicator of structure, field coverage and text support, not a probability of truth. Threshold for `success` is 70.
grounding	0–1, the fraction of checkable extracted string/number values whose text was found in the page text: strings case-insensitive, with whitespace runs collapsed and the page's bold, italic and inline-code markers read as formatting (a value's own * or ` must be on the page); links as the same URL; numbers by value (so 31.0 matches 31); an ISO date also as the page writes it in words, inside an ISO timestamp, or as a numeric date in the day/month order the page's own dates establish; and GBP, EUR or INR as its symbol beside an amount (£53.74). Undefined when nothing was checkable. An absent ratio is unmeasured. It shows the text is present, not that it belongs to the requested field or entity, or that it is correct.
groundingFields	Per-value grounding verdicts backing `grounding`: `path` (JSONPath into `data`), `value`, `grounded`, and (when grounded) an `excerpt` of surrounding page text, roughly 45 characters either side, with case preserved and whitespace runs collapsed. Ordered ungrounded-first and capped at 50 entries; the ratio in `grounding` is computed over every checked value before the cap, so on a large extraction it can disagree with what's visible in this array. Booleans, nulls, and strings under 3 characters get no entry; absent entirely when nothing was checkable. Booleans and short strings still count in `groundingCoverage.total` as values that were not found.
binding	Present on accepted model answers, structured or natural-language, from current deployments; absent on markdown, page-published structured data and rejected results, and on recordings and responses stored before 2026-09-28. Absence means no check ran. `{ judged, conflicts: [{ path }], truncated }`. It judges numbers, ISO dates and short digit-bearing strings such as versions. A conflict means the value is on the page but the passages about that field state a different value: treat the field at that `path` as unconfirmed and inspect it. Not a truth claim, and it changes neither acceptance nor credits. `judged: 0` means no field could be checked; `truncated` means some values were not judged. Paths only: no values or page text.
groundingCoverage	How much of the data the ratio speaks for: `total` values that assert something (nulls excluded), `checkable` of those with enough text to look for, and `grounded` found in the page. Counted over every value, so unlike `groundingFields` it is unaffected by the cap. Read it before trusting an absent `grounding`: nothing checkable is not the same as nothing wrong. Acceptance requires `grounded` to be at least half of `total`, so an answer made mostly of yes/no values or one- or two-letter codes is refused: ask for such a value as the text the page shows (“the stock status as shown”).
coverageMeasured	True when a JSON Schema let Contrie count the requested fields, so `completeness` measures them. False for a natural-language-only request: coverage was not measured, completeness earns half credit and `qualityScore` is at most 80.
latencyMs	Processing time in milliseconds after the source was acquired. It excludes the fetch and any browser render, so it is not the request's total time.
credits	Credits charged for the call. 0 unless it delivered an answer.
trace	High-level request progress: loading, processing or complete. The final acceptance decision determines whether data is usable.
cached	True only when your account's permitted page reuse (`maxAge`) supplied the source; `fetchedAt` is then the original fetch time. Keyless recorded examples replay their original response, which reports false; its `source.observedAt` is the recording time, and it carries no `fetchedAt`.

Streaming

Send Accept: application/x-ndjson to receive high-level request progress instead of waiting for the final result. See Streaming for the line format and examples.

Error Responses

Errors return { success: false, error: { code, message } }. Full descriptions are on the Error Codes page.

Status	Code
400	INVALID_REQUEST, INVALID_URL
401	UNAUTHORIZED
403	INSUFFICIENT_SCOPE
409	IDEMPOTENCY_CONFLICT, IDEMPOTENCY_EXPIRED, IDEMPOTENCY_OUTCOME_UNKNOWN, REQUEST_IN_PROGRESS
422	RESULT_NOT_ACCEPTED, FETCH_FAILED, SOURCE_REFUSED, RENDER_REQUIRED
429	RATE_LIMITED, PAGE_LIMIT_REACHED, INFERENCE_BUDGET_EXCEEDED
503	SERVICE_UNAVAILABLE, MODELS_UNAVAILABLE, INFERENCE_UNAVAILABLE
500	EXTRACTION_FAILED