Cost anomaly detection at BMW Group moved from static dashboards to an automated, serverless pipeline that forecasts daily spend for each account‑service pair and emails owners when actual costs fall outside the predicted confidence interval. Practitioners care because the pattern shows how to scale fine‑grained FinOps monitoring across thousands of cloud accounts while keeping compute spend under $50 per month.
Forecasting baseline with Prophet
CLEA pulls daily billing rows from AWS Cost and Usage Reports (CUR) and equivalent exports from other providers, then aggregates the roughly 3 billion rows per month to a single grain: daily cost per account per service. Each account‑service combination—hundreds of thousands in total—gets a 365‑day history that feeds an open‑source Prophet model. The model produces a 12‑month rolling forecast and a per‑day expected value with confidence bounds, allowing the system to adapt to each service’s usage pattern rather than applying a one‑size‑fits‑all threshold.
Serverless orchestration with Step Functions and Lambda
A daily Step Functions state machine kicks off the run. A first Lambda discovers active accounts, writes the list as JSON to Amazon S3, and hands it to a Distributed Map state. The map fans out to as many as 500 concurrent Lambda invocations, each responsible for forecasting all services of a single account. This parallelism completes the full 14 000‑account workload in roughly 20 minutes, and the entire compute footprint costs about $50 per month.
From forecast to actionable alert
After a forecast is generated, CLEA computes impact as actual spend – expected spend. When the actual value lies outside Prophet’s confidence interval, the day is marked as a potential anomaly. A downstream filter evaluates the magnitude and business relevance of the impact before triggering an email to the account owner. This two‑step approach—statistical flag followed by business‑logic filter—reduces noise compared to a static dollar‑threshold rule.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Implementing a similar pipeline requires:
- Reliable ingestion of cost data (e.g., CUR) and a consistent aggregation step.
- A forecasting component that can be swapped without touching detection logic.
- Serverless orchestration capable of fan‑out parallelism (Step Functions + Lambda) to keep runtime and cost low.
- Alert‑filtering logic that balances sensitivity with alert fatigue.
Key considerations for future evaluation include monitoring model drift as spending patterns evolve, extending the approach to other cloud providers, and integrating alerts with existing incident‑response tooling. The architecture demonstrates that large‑scale, per‑service cost anomaly detection is feasible with modest operational overhead.
