Cloudflare now lets you push Browser Run crawl lifecycle events directly into a Cloudflare Queue. By creating a subscription you can receive crawl.started, crawl.updated and crawl.finished notifications without having to poll the Browser Run API.
Creating a Queue Subscription
The subscription is defined at the account level with the wrangler CLI. Replace <QUEUE_NAME> with the name of an existing queue and run the command below:
npx wrangler queues subscription create <QUEUE_NAME> --source browserRun --events crawl.started,crawl.updated,crawl.finished
The same syntax works with yarn or pnpm if you prefer those package managers. Once the subscription is active, each crawl job will emit the three events into the specified queue.
Architectural Implications
Moving from a poll‑based model to an event‑driven flow changes the way you compose downstream processing. Instead of a scheduled job that checks the Browser Run status, you can attach a worker, serverless function, or any consumer that reads from the queue. This decouples the crawl execution from the processing logic, enabling:
- Real‑time reaction to crawl progress, such as updating dashboards or triggering alerts.
- Parallel handling of
crawl.updatedpayloads for incremental indexing. - Automatic cleanup or archiving once
crawl.finishedarrives.
Because the events are delivered through a queue, ordering is preserved per queue but not guaranteed across multiple queues. If ordering matters for your workflow, you should design the consumer to handle out‑of‑order messages or use a single queue.
Operational and Security Considerations
Adopting this pattern introduces a few operational responsibilities:
- Queue scaling: Monitor message throughput and configure appropriate queue size or rate limits to avoid back‑pressure on the Browser Run service.
- Retry and dead‑letter handling: Ensure your consumer acknowledges messages only after successful processing; configure dead‑letter queues for messages that repeatedly fail.
- Access control: The subscription uses the account’s credentials to write to the queue. Restrict who can create or modify subscriptions and who can read from the queue to limit exposure of crawl metadata.
- Observability: Instrument both the queue and the consumer to capture latency, error rates, and event loss.
From a security perspective, the events themselves do not contain raw page content, but they may reveal crawl identifiers and timestamps. Treat the queue as a sensitive data store and apply the same secret‑management and audit‑logging practices you would for any other internal messaging system.
Related CloudNinjas coverage: hands-on guides.
What This Means For Practitioners
Practitioners can replace periodic status checks with a push‑based model, reducing API load and latency. The primary action is to provision a queue, create the subscription with the CLI command above, and build a consumer that processes the three event types. Keep an eye on queue health, enforce strict access policies, and design idempotent processing to handle possible duplicate deliveries.