Creating a Continuous Model‑Improvement Loop Using SageMaker HyperPod Data Capture

SageMaker HyperPod Data Capture adds native, multi‑tier recording of inference requests and responses directly to Amazon S3. The change eliminates the need for custom proxy layers or external logging pipelines, giving teams immediate access to production data for monitoring, debugging, and model refinement.

Three‑Tier Capture Architecture

The feature offers three independent capture points:

• Tier 1 – Endpoint: Stores full request and response payloads via the native SageMaker data capture configuration.
• Tier 2 – Load balancer: Persists Application Load Balancer access‑log fields such as request line, status codes, timings, client IP, and user‑agent.
• Tier 3 – Model pod: Records full payloads plus event metadata using sidecar containers (NGINX, OpenTelemetry collector, S3 uploader) inside the inference pod.

Each tier can be enabled independently. Tier 1 and Tier 3 both store full payloads, so enabling both duplicates records. Practitioners typically use Tier 1 for compliance and Model Monitor, Tier 2 for low‑cost traffic analytics, and Tier 3 when the payloads are needed for downstream training.

Implementation Steps on a Single HyperPod Cluster

The walkthrough assumes an EKS‑orchestrated HyperPod cluster with both the inference operator and the amazon-sagemaker-hyperpod-training-operator add‑on installed. Two NVIDIA GPUs are required: one dedicated to the serving pod and another for the training job. After confirming the custom resource definitions with

kubectl get crd | grep -E 'inferenceendpointconfigs|hyperpodpytorchjobs'

, the workflow proceeds as follows:

1. Enable Tier 3 capture in the endpoint dataCapture block, configuring sampling percentage, payload limits, and optional KMS encryption.
2. All request/response pairs are streamed to an S3 bucket in the same region.
3. Use Amazon Athena to query the captured objects, surface patterns such as unseen product names, and generate a SQL‑based labeling policy.
4. Apply the policy to tag records that represent a coverage gap.
5. Launch a HyperPod PyTorch training job on the second GPU, feeding the labeled captures as fine‑tuning data.
6. Deploy the newly fine‑tuned model alongside the original endpoint, keeping capture active for the next iteration.

The example uses synthetic data and provides scripts, manifests, and expected outputs in a public GitHub repository.

Operational and Security Considerations

Because Tier 3 captures full payloads, the data may contain personally identifiable information (PII) such as names, addresses, or order numbers. Practitioners should treat the captured S3 objects as sensitive and apply the following controls, all of which are mentioned in the source:

• Enable sampling to limit the volume of captured records.
• Encrypt objects at rest using an AWS KMS key via the kmsKeyId setting.
• Define S3 lifecycle rules to expire or transition old captures.
• Redact PII before using the data for training, for example with Amazon Comprehend.

From an operations perspective, the separation of serving and training GPUs prevents resource contention; a single GPU would leave the training job pending. Monitoring the S3 bucket size and Athena query costs is also advisable, as continuous capture can generate large volumes of data.

Related CloudNinjas coverage: AWS.

What This Means For Practitioners

Enabling HyperPod Data Capture turns production traffic into a feedback loop that can be queried, labeled, and fed back into the model without downtime. Teams should evaluate which tier(s) meet their compliance, cost, and debugging needs, apply encryption and PII redaction to protect captured data, and provision separate GPU resources for serving and fine‑tuning. Ongoing assessment of capture volume, query performance, and lifecycle policies will keep the flywheel efficient and secure.