Engineers building agents on Amazon Bedrock AgentCore discovered that passing multi‑megabyte JSON objects through every tool call inflated the model’s context window, driving up token consumption and response times. By externalizing the heavy payloads and applying three targeted strategies, they reduced token counts by up to 71% and cut average latency by as much as 80% on a 23 MB benchmark.
Background and Pain Points
The Flightscape crew‑scheduling optimizer exposes three JSON documents – a request, a response, and a validation report – each ranging from several to tens of megabytes. The agent, built with LangGraph and running on Bedrock AgentCore, stores these objects directly in its conversational state. Every tool invocation serializes the full payload back into the LLM prompt, so a single turn can consume thousands of tokens and exceed the model’s context limit. The result was >15 seconds latency per request and escalating costs as conversations grew longer.
Bedrock AgentCore JSON Strategies
Three composable tactics were introduced. The first runs at the beginning and end of each turn, extracting the large JSON artifacts from the in‑memory state before the model is called and re‑hydrating them afterward. This prevents the model from repeatedly seeing the same bulk data. The second and third tactics trigger after each tool returns a result; they keep the incremental output lightweight and avoid re‑injecting the full payload on subsequent steps. While the source does not enumerate the exact mechanisms, the overall effect is to keep the model’s context focused on the minimal information needed for reasoning.
class GraphState(TypedDict):
"""LangGraph state that combines messages with application data."""
messages: Annotated[List[BaseMessage], add_messages]
request: Optional[TOPTRequest]
responses: List[TOPTResponse]
validation_report: Optional[Dict[str, Any]]
basic_stats: Optional[Dict[str, Any]]
cache_data: Optional[Dict[str, Any]]
Originally the request, responses, and validation_report fields held the raw JSON inline. After applying the first strategy, those fields store references or lightweight summaries, while the full documents reside in external storage accessed by the tools as needed.
Operational and Architectural Implications
- State Management: Moving large artifacts out of the conversational state changes the persistence model. Engineers must ensure that external storage (e.g., S3 or a database) is highly available and that the agent can retrieve data with low latency.
- Tool Design: Each tool now needs to accept a reference (such as an object key) rather than a full JSON blob. This may require updating function signatures and adding error handling for missing or stale references.
- Cost Monitoring: Token usage drops dramatically, but the external storage introduces its own cost profile. Practitioners should instrument both LLM token metrics and storage I/O to maintain a balanced cost view.
- Latency Profile: By avoiding repeated serialization of megabyte payloads, network round‑trips shrink, and the model can focus on reasoning. However, the added fetch step adds a deterministic latency component that should be measured and, if needed, cached.
- Security Considerations: Storing raw crew‑scheduling data outside the model context shifts the attack surface to the storage layer. Access controls, encryption at rest, and audit logging become the primary safeguards for sensitive operational data.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
When building AgentCore‑based agents that ingest large structured inputs, treat the model context as a premium resource. Externalize bulk data, keep the conversational state lean, and design tools to operate on references rather than full payloads. Monitor both token consumption and storage access patterns to verify that cost and latency improvements hold in production. Finally, apply appropriate security controls to any external repository that now holds the raw JSON, as it becomes the de‑facto source of truth for the agent’s reasoning.


