The Browser Run /crawl response has been extended to surface detailed diagnostics. Each URL entry now carries an outcome value (e.g., success, robots_disallowed, timeout, page_error), and the overall job includes a robots array that records the robots.txt state for up to 50 origins, including the origin URL, rule availability, and HTTP status.
New Per‑URL Outcome Field
Previously the crawl payload only indicated whether a request completed. The added outcome enumerates the exact result, allowing scripts to differentiate between a legitimate block by robots.txt and a network timeout. Practitioners can now filter or retry only the relevant subset of URLs, reducing noise in downstream processing.
Robots.txt State Array
At the job level, the robots array provides a concise view of robots.txt handling across distinct origins. Each entry reports:
- origin: the base URL where robots.txt was fetched.
- state: whether usable rules were found (e.g.,
available). - status: the HTTP response code received.
{
"robots": [
{"origin": "https://example.com", "state": "available", "status": 200}
],
"records": [
{"url": "https://example.com/docs/", "status": "completed", "outcome": "success"}
]
}
Operational and Security Considerations
With explicit outcomes, monitoring can be tuned to alert on unexpected timeout or page_error values, which may indicate network instability or site‑level issues. The robots.txt state helps enforce compliance policies; a robots_disallowed outcome signals that the crawler correctly obeyed a disallow rule, reducing the risk of unintended scraping. Conversely, a missing or non‑200 robots.txt response may warrant a review of site configuration before scaling the crawl.
Related CloudNinjas coverage: hands-on guides.
What This Means For Practitioners
Update any post‑processing logic to consume the new outcome field and to parse the robots array. Use the outcome values to drive selective retries or to flag sites that consistently return errors. Incorporate robots.txt status checks into compliance dashboards to ensure that automated crawls remain within allowed boundaries. Monitoring pipelines should now include alerts for non‑success outcomes and for robots.txt fetch failures, enabling faster remediation and more reliable crawl operations.
