FROST: Smart Framework for Effective Synthetic Data Training

Paper: Let Training Guide Selection: Online Synthetic Data Filtering via Real-Anchored Utility

Authors: Yanran Wu, Sana Lakdawala, Renzo Tassara Miller, Chongyang Bai, Sharath Ciddu, Shivendra Pratap Singh, Kungang Li, Sandeep Pandey, Chunwei Liu

Executive Summary

The research introduces FROST, an innovative framework designed to enhance the use of synthetic data in training machine learning models, particularly when access to real-world data is limited. Synthetic data can be beneficial for scaling training efforts, but issues such as noise and differences in data distribution can diminish its effectiveness. Traditional methods for selecting synthetic data tend to focus on either the fidelity of the data (how closely it resembles real data) or its diversity (variety within the data) without considering the specific needs of the learning model as it develops.

FROST addresses this gap by utilizing a method that continuously evaluates the utility of synthetic data based on direct feedback from the training process itself. By using gradient feedback from real training data, FROST dynamically assesses the effectiveness of synthetic data batches against recent training performance. This allows the framework to determine when to filter data and to do so in a manner that is both efficient and effective, avoiding the need for external validation processes or separate verification datasets.

The results from testing FROST on two public benchmarks demonstrate its potential. In experiments focused on image classification and fine-tuning large language models for text-to-SQL tasks, FROST was able to filter out about 20% to 30% of synthetic data while simultaneously improving the performance of models trained on the remaining samples compared to models using the entire synthetic dataset.

Beyond benchmarks, FROST was also applied in a real-world scenario within a large-scale advertising re-ranking system. Here, it achieved notable performance improvements over an existing, highly optimized production baseline. This suggests that FROST could be effectively implemented in various industrial applications where synthetic data plays a critical role, potentially enhancing training efficiency and model performance.

This research highlights the importance of adapting synthetic data selection to the evolving needs of machine learning models, offering a new approach that could lead to better outcomes in environments where real data is scarce. The ability to filter synthetic data dynamically based on its utility may offer organizations a more effective strategy for leveraging synthetic data, ultimately driving improvements in their machine learning initiatives.

Academic Abstract

Synthetic data can scale training supervision when real-world data are limited, but noise and distribution mismatch can reduce its value. Existing synthetic data selection methods often emphasize fidelity or diversity rather than the learner's evolving needs. We propose FROST, an online framework that estimates synthetic-data utility through gradient feedback anchored in real training data. It calibrates batch utility against recent history to determine when filtering is needed and filters samples only in out-of-band batches to determine what to retain, without an external verifier or held-out validation set. Experiments on two public benchmarks for image classification and LLM fine-tuning for text-to-SQL show that FROST filters out around 20--30% of the synthetic data while improving real-task performance compared with training on the full synthetic data pool. We further apply FROST during training in a large-scale industrial ads re-ranking system, achieving significant performance gains over a highly optimized production baseline, demonstrating its effectiveness and generalizability.