At Atlassian we moved from single‑node GPU jobs to true multi‑node training, and the platform had to evolve. The change was the introduction of a managed stack that couples RDMA‑enabled networking with a Lustre parallel filesystem, and adds automated validation of placement, topology, and storage paths. Practitioners care because without these controls a distributed AI training job can silently lose half its performance, hide failures, and force engineers to troubleshoot low‑level infra instead of model code.
Predictable Inter‑node Communication
GPU workers need to exchange gradients and model state quickly. The default socket/TCP path incurs a large overhead; on current GPUs it can cut throughput by roughly 50 % compared to a direct‑memory‑access (RDMA) link. The platform now provisions RDMA‑capable networks for any job that requests the fabric, ensuring that collective operations run over the low‑latency path.
- Implementation note: Node pools are created with multi‑network support and the RDMA device plugin is installed on each fabric‑capable node.
- Operational implication: Jobs that do not explicitly request RDMA fall back to sockets without error, so teams must verify that the fabric is advertised and that the plugin is healthy.
High‑throughput Shared Storage
Training data, checkpoints, and intermediate artifacts are accessed concurrently by many workers. A conventional storage path becomes a bottleneck, causing uneven step times and low GPU utilization. By integrating Lustre, the platform offers a parallel distributed filesystem that can sustain the required I/O rates for large models.
- Implementation note: Lustre is provisioned as a shared volume attached to the training cluster, abstracted away from the ML team.
- Operational implication: Monitoring must include I/O latency metrics; a slowdown may indicate a mis‑mounted Lustre client rather than a model issue.
Automated Validation and Visibility
Before the redesign, failures were hard to detect. A mis‑matched container image caused the RDMA device plugin to crash in CrashLoopBackOff on every production node, but the problem went unnoticed for 271 days because no alert existed for a DaemonSet that never became ready. Additionally, pod restart counters gave a false impression of recent failures when the underlying condition was long‑standing.
- Consideration: Deploy synthetic health checks that deliberately request the RDMA fabric; a pending job is loud, while a silent fallback to sockets is not.
- Consideration: Add alerts for DaemonSet readiness and for mismatches between staging and production configurations.
Related CloudNinjas coverage: hands-on guides.
What This Means For Practitioners
When building or operating a distributed AI training platform, treat the network, storage, and scheduler as a single, managed capability rather than independent layers. Ensure that:
- RDMA device plugins are verified as running on every node before exposing the fabric to workloads.
- Lustre or an equivalent high‑throughput filesystem is the default shared volume for training data.
- Automated validation jobs exercise the full stack regularly, and alerts fire on any component that never reports ready.
- Observability includes both GPU utilization and infra‑level metrics (network latency, storage I/O) to catch silent degradations.
By embedding these controls in the platform, ML teams can focus on model development while the underlying infrastructure remains reliable, repeatable, and secure from accidental misconfiguration.
