Amazon SageMaker AI bidirectional streaming now includes a usage‑based metering capability that reports consumption to AWS Marketplace. Engineers building or operating real‑time speech models need to adapt because the billing model shifts from static endpoint fees to per‑unit usage, and a companion WebSocket stream must be handled alongside the existing inference channel.
Usage‑Based Metering Overview
Previously, SageMaker Marketplace metering relied on a single HTTP response header that conveyed a unit count for request‑response APIs. That approach cannot capture the variable length of a bidirectional audio session, where usage accrues over time. The new mechanism introduces a second WebSocket connection dedicated to metering data. Model providers send JSON frames that contain a Metering object with a Dimension (e.g., inference.count) and a ConsumedUnits value representing the amount of work performed, such as seconds of audio processed.
Architectural Impact
When a streaming session starts, SageMaker AI establishes two parallel connections to the model server:
- Data connection: carries the audio payload and transcription results.
- Metadata connection: carries metering records.
The model server signals support by returning the header X-Amzn-SageMaker-Metadata-Stream-Supported: true during the WebSocket upgrade. SageMaker then opens the metadata channel at the path /invocations-bidirectional-stream-metadata. Both channels include the same X-Amzn-SageMaker-Request-Id header, allowing the model server to correlate metering frames with the active inference session.
Operational Considerations
Metering frames must be sent as individual JSON text messages, limited to one per second per connection and a maximum size of 512 bytes. The recommended practice is to emit incremental records as usage accrues—either per unit or in small batches—so that unexpected session termination leaves minimal unreported consumption. A final record should be transmitted immediately when the session ends. Operators should monitor the rate quota, verify that each frame contains exactly one record, and reconcile the reported units against AWS Marketplace billing reports to detect gaps.
Security and Compliance Notes
The metadata stream does not introduce a new authentication mechanism; it reuses the same request‑ID and underlying authorization that protects the data connection. Practitioners should ensure that any custom handling of the metadata channel respects the existing IAM permissions granted to the SageMaker endpoint. Because the metering payload is small and text‑based, standard network security controls (TLS, VPC isolation) remain sufficient, but logging of metering traffic can aid in auditability and dispute resolution.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
- Update model servers to detect the
X-Amzn-SageMaker-Metadata-Stream-Supportedheader and open the companion WebSocket at/invocations-bidirectional-stream-metadata. - Implement logic to emit
MeteringJSON frames at a maximum of one per second, staying within the 512‑byte limit. - Correlate metering records with the
X-Amzn-SageMaker-Request-Idto ensure accurate attribution. - Validate reported
ConsumedUnitsagainst Marketplace charge reports to confirm billing integrity. - Monitor the health of both data and metadata connections, and design graceful shutdown handling to guarantee the final metering record is sent.