Amazon Nova’s supervised fine‑tuning (SFT) workflow now includes a set of post‑cleaning techniques that let you decide how much data to use, which examples to keep, and how to extend the set without blowing up compute costs. The shift from “just clean the data” to “measure, prune, and augment” matters because it directly influences model quality, training expense, and the risk of erasing the model’s baseline capabilities.
Learning curve evaluation
Before adding more samples, run a single training job on the full dataset and checkpoint the model at regular intervals (for example every 10–20 % of the total steps). Evaluate each checkpoint against a held‑out benchmark that mirrors production traffic. Plot the primary downstream metric against the number of training tokens consumed. The key indicator is the saturation point: when doubling the data yields less than a 1–2 % gain on the metric, additional data of the same type is unlikely to help.
- Train once, save checkpoints. Use the same run to approximate a scaling curve without multiple experiments.
- Chart performance versus token count. Look for the flattening region where improvements shrink.
- Stop or diversify. If the curve flattens, either halt training to save compute or inject qualitatively different examples that target remaining failure modes.
Training‑token accuracy can serve as a practical stop condition because gains typically plateau once the model reaches near‑perfect accuracy on the training set.
Subset selection and filtering
When the learning curve indicates diminishing returns, selective sampling can outperform training on the entire collection. Techniques such as DEITA (as referenced in the source) illustrate that intelligent pruning—based on relevance, difficulty, or redundancy—can reduce the dataset while preserving or improving performance. Practitioners should consider:
- Removing low‑signal or malformed examples that survived earlier quality checks.
- Prioritizing examples that cover edge cases the model still fails on.
- Balancing domain coverage to avoid over‑fitting to a narrow instruction set.
These steps reduce storage, lower I/O pressure, and cut training time, which is especially relevant for cloud‑based training pipelines that bill per compute hour.
Data augmentation and mixing
When raw examples are scarce, generating synthetic variants can increase diversity without manual annotation. The source notes that repeated exposure to a small, high‑quality set (e.g., 400 reasoning examples run for 128 epochs) can outperform a single pass over a much larger set (51,200 examples) by 12–26 percentage points on benchmark tasks. This suggests two practical patterns:
- Epoch‑heavy training on curated data. Re‑use the same high‑quality examples many times to let the model fully internalize the instruction patterns.
- Mixing synthetic and real data. Blend augmented examples with the original set to broaden coverage while keeping the signal strong.
Both patterns require monitoring for over‑fitting, which can be detected by a divergence between training‑token accuracy and evaluation metrics.
Related CloudNinjas coverage: AWS.
What This Means For Practitioners
Engineers should first establish a reliable evaluation benchmark before scaling data. Use a single‑run learning‑curve analysis to locate the saturation point, then decide whether to stop, add diverse data, or increase epoch count on a curated subset. Filtering and augmentation are not optional add‑ons; they become cost‑saving levers when the curve flattens. Continuous monitoring of token‑level accuracy versus downstream metrics will alert you to over‑fitting or under‑utilization of the dataset, guiding operational decisions around compute allocation and data pipeline updates.
