Amazon Bedrock Knowledge Bases introduced a new ingestion option that leverages the open‑source Crawl4AI tool to prepare web pages as Markdown files with accompanying metadata before they are loaded from Amazon S3. Practitioners can now choose between the built‑in Bedrock Web Crawler and a custom Crawl4AI preprocessing step, depending on the level of control required over the extracted content.
When to use Crawl4AI preprocessing vs. the built‑in Web Crawler
The native Bedrock Web Crawler is sufficient when the target site can be crawled directly, respects robots.txt, and the default extraction meets quality expectations. It performs incremental syncs after the initial full crawl, capturing only added, updated, or deleted pages.
Use the Crawl4AI path when any of the following conditions apply:
- Structured Markdown preparation: You need a clean Markdown representation that removes navigation, ads, or other boilerplate while preserving headings, lists, and tables.
- Targeted region extraction: Only a specific page region identified by a CSS selector should be indexed.
- Custom metadata: Additional attributes such as crawl timestamps, source URLs, or domain‑specific tags must be attached to each document for downstream filtering.
Architecture overview
The workflow consists of four AWS services orchestrated by an EventBridge schedule:
- EventBridge triggers an Amazon ECS task running on AWS Fargate at the configured interval.
- The Fargate container executes Crawl4AI against the list of seed URLs, applying the chosen Markdown generator and optional CSS selector.
- For each page, Crawl4AI writes a Markdown file and a JSON sidecar containing the custom metadata to an Amazon S3 bucket that backs the Knowledge Base.
- Amazon Bedrock Knowledge Bases monitors the S3 bucket and ingests the files, handling parsing, chunking, embedding, and indexing automatically.
Both managed and customer‑managed Knowledge Bases accept S3 as a data source, so the same preprocessing pipeline works for either deployment model. The sample implementation uses a managed Knowledge Base and runs the Fargate task in a public subnet, allowing outbound internet access to the target sites.
Implementation highlights
Key steps to replicate the sample flow are:
- Define an EventBridge rule with a cron expression that matches the desired refresh cadence.
- Create an ECS task definition that references a container image containing Crawl4AI and any required Python dependencies.
- Configure the container entry point to accept seed URLs, optional CSS selectors, and the S3 destination bucket.
- Grant the task role permissions to write objects to the target S3 bucket (
s3:PutObject) and to read the bucket if needed for incremental logic. - Enable the Knowledge Base S3 connector and point it at the same bucket; Bedrock will automatically pick up new or changed files.
When using a managed Knowledge Base, no additional vector store provisioning is required; Bedrock provisions and scales the underlying store internally. If a customer‑managed Knowledge Base is preferred, the same S3 ingestion path applies, but the operator must manage the vector store lifecycle separately.
Operational and security considerations
Running a custom crawler introduces several operational responsibilities:
- Network exposure: The Fargate task must have outbound internet access. Placing it in a public subnet simplifies connectivity but also expands the attack surface; consider using a NAT gateway with restricted egress rules.
- IAM scope: The task role should follow the principle of least privilege, limiting S3 actions to the specific bucket and prefix used for ingestion.
- Content validation: Because Crawl4AI’s pruning logic is heuristic, sample pages should be reviewed regularly to ensure that essential context is not removed.
- Metadata consistency: Downstream retrieval filters rely on the sidecar JSON. Inconsistent attribute naming can lead to missed results or extra post‑processing.
- Cost monitoring: Fargate charges accrue per vCPU‑second and memory‑second; schedule frequency and page volume should be balanced against budget constraints.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Adopting the Crawl4AI preprocessing path gives engineers fine‑grained control over what content reaches a Bedrock Knowledge Base and how that content is described. The trade‑off is added operational complexity: you must manage a scheduled Fargate task, secure its network placement, and maintain metadata hygiene. Evaluate the need for custom extraction against the simplicity of the native web crawler, and monitor both cost and data quality as the pipeline runs in production.
