Agents SDK adds web_fetch tool for on‑the‑fly markdown conversion

The Agents SDK v0.28.0 introduces a web_fetch tool that lets an agent request a URL, receive the page content as markdown, and get metadata such as the final URL, HTTP status, and content type. Long documents are split into chunks with offsets so the model can continue reading. The change matters because it removes the need for custom scraping or third‑party conversion services when building agents that need to ingest external text.

What Changed

Previously agents could only call external services or rely on manual preprocessing to obtain readable text. The new tool integrates Workers AI markdown conversion and, when a site advertises “Markdown for Agents”, the SDK can request markdown directly. The response is wrapped in JSON that includes:

• the extracted markdown content (or a chunk of it),
• the resolved URL after redirects,
• the HTTP status code, and
• the content‑type header.

If the source exceeds the configured response size, the SDK returns a segment and an offset that the model can use to request the next segment.

Why It Matters for AI, Cloud, and DevOps Practitioners

Agents that need up‑to‑date information—such as knowledge‑base assistants, compliance checkers, or monitoring bots—can now fetch and parse documents without writing separate parsers. The markdown output aligns with the model’s native text handling, reducing token waste and improving prompt reliability. Controlling fetch parameters (allowed hosts, size limits, timeouts, redirects) lets operators enforce cost and security boundaries at the SDK level rather than in downstream code.

Architectural and Operational Considerations

Integrating web_fetch requires adding the tool to a Pi Durable extension registry before opening the harness. The typical pattern is to import the tool, install it with a name, and bind it to the AI worker instance:

import { webFetchTool } from "agents/webfetch/pi";

this.registry.install({
  name: "web-tools",
  tools: [webFetchTool({ binding: this.env.AI })]
});

Key operational knobs include:

• Host whitelist: limit which domains the agent may contact.
• Response size: cap the amount of markdown returned per call to control token usage.
• Timeouts and redirects: prevent hanging requests and unintended navigation.

When combined with the existing web_search tool, the model can first discover URLs and then immediately read them via web_fetch, creating a streamlined search‑and‑read loop.

Related CloudNinjas coverage: hands-on guides.

What This Means For Practitioners

To adopt the new capability, ensure your project is on Agents SDK v0.28.0 or later and update your harness registration to include web_fetch. Review your host allowlist and size limits to match your cost and security posture. Test the chunked response flow with a long document to verify that offset handling works as expected. Finally, consider whether the built‑in markdown conversion satisfies your downstream processing needs, or if you still require custom post‑processing for specialized formats.