Content Use directive enforced by Cloudflare Crawl API

The Cloudflare /crawl endpoint now reads the Content Signals use directive from a target site’s robots.txt and introduces a new contentUse request field. This change forces AI‑driven crawling jobs to declare the maximum reuse level they intend and to respect any stricter limits imposed by the site owner.

New contentUse parameter

Calls to the crawl API can include a JSON key named contentUse. The accepted values are reference (least permissive) and full (most permissive), with full as the default when the field is omitted. If the target’s robots.txt contains a use level that is more restrictive than the supplied contentUse, Cloudflare returns a 400 error and aborts the request.

curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \
  -H 'Authorization: Bearer ' \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "contentUse": "reference",
    "formats": ["markdown"]
  }'

Why the change matters to engineers

AI engineers and platform teams that rely on automated content extraction now have an explicit compliance checkpoint baked into the API. A mismatch between the declared contentUse and the site’s policy will surface as a client‑side error, preventing silent over‑use of copyrighted or restricted material. Security and DevSecOps practitioners gain a clearer audit trail: the request payload records the intended reuse level, and the 400 response signals a policy violation.

Operational and architectural implications

• Automation scripts must be updated to set contentUse deliberately, rather than relying on the previous default behavior.
• Failure handling should be expanded to detect 400 responses and either downgrade the requested level or abort the workflow.
• Monitoring dashboards can surface the rate of 400 rejections as an indicator of policy mismatches across crawled domains.
• Teams should review existing crawl jobs to ensure they do not unintentionally request full when only reference is permissible, reducing the risk of compliance incidents.

Related CloudNinjas coverage: hands-on guides.

What This Means For Practitioners

Update your crawl client to include contentUse with the least permissive level that satisfies your downstream processing. Add logic to catch 400 errors, log the offending target, and optionally retry with a more restrictive setting. Treat the new field as a compliance control rather than an optional tweak, and incorporate its status into your operational health checks.