UniPipe: Efficient Inference and Fine-Tuning for DNN Workflows

Paper: UniPipe: Unified Inference Serving and Fine-Tuning for Deadline-Constrained DNN Surrogate Workflows

Authors: Jaiaid Mobin, Avinash Maurya, M. Mustafa Rafique, Bogdan Nicolae

Executive Summary

Recent advancements in deep neural network (DNN) surrogates have significantly improved the efficiency of scientific workflows, particularly in areas requiring rapid processing of high-frequency sensor data, such as Ptychographic imaging at next-generation X-ray light sources. These surrogates can achieve throughput rates that are 100 to 1000 times greater than traditional iterative solvers, which accelerates the pace at which experiments can be conducted.

However, maintaining the accuracy of these surrogates during experiments poses a challenge. Current methods typically involve either freezing the pretrained surrogate model or running a separate context that fine-tunes the model while it serves data. This approach, while functional, can lead to increased competition for GPU resources, resulting in higher inference latency and missed deadlines when processing sensor data that must meet strict timing requirements.

To tackle this issue, researchers have developed UniPipe, a streamlined system that consolidates both the inference serving and fine-tuning of a DNN surrogate into a single context on an edge GPU. This design is based on three critical insights: the processes of serving and fine-tuning can be combined into a single forward pass, the latency associated with modern GPUs varies non-linearly with batch size, and the repetitive nature of workload allows for offline scheduling.

UniPipe operates by merging the training and inference data into one forward pass, supplemented by a selective backward pass for the training subset, which enhances efficiency. Additionally, it employs a polynomial-time scheduler to optimize batch composition based on the GPU's latency characteristics.

In evaluations conducted on a real-world Ptychographic reconstruction workflow with sensor rates of up to 5000 patterns per second, UniPipe demonstrated notable improvements. It reduced the mean squared error (MSE) in reconstruction by up to 30% and in inference by up to 50% compared to a baseline that relied solely on pretrained models. Furthermore, UniPipe effectively decreased the rate of missed deadlines by 13% to 22% relative to the multi-context approach. The unified deployment not only minimized the variability in latency—reducing it from 106% to 57% of the mean—but also increased overall throughput when compared to the multi-context baseline.

This research highlights the potential for UniPipe to enhance workflows in environments that require rapid processing of sensor data. By improving accuracy and reducing latency, it could help organizations in scientific fields optimize their operations and achieve more reliable outcomes in real-time applications. While the results stem from a controlled evaluation rather than widespread real-world deployment, the implications for future applications are promising.

Academic Abstract

DNN surrogates are increasingly used to accelerate scientific workflows that must process high-rate sensor streams in near real time, exemplified by Ptychographic imaging at next-generation X-ray light sources. Such surrogates deliver 100-1000x higher throughput than iterative solvers, enabling faster steering of experiments. To preserve surrogate accuracy during an experiment, as the input distribution evolves, state-of-the-art deployments freeze the pretrained surrogate or run a separate CUDA context that fine-tunes concurrently with serving. However, multi-context concurrency causes significant contention on the GPU, inflating both the mean and the variance of inference latency, triggering deadline misses in workflows where sensor data has to be processed within a hard per-unit deadline. To address this challenge, we propose UniPipe, a single-context system that unifies inference serving and fine-tuning of a DNN surrogate on a single edge GPU. UniPipe is driven by three key insights: both serving and fine-tuning require a forward pass and can be fused into one; the per-sample latency of modern GPUs is non-linear in batch size; and the workload is repetitive across intervals, enabling offline scheduling. Specifically, UniPipe fuses training and inference data in a single forward pass with a selective backward pass on the training subset, and uses a polynomial-time scheduler that exploits the latency curve to compute the batch composition. We evaluate UniPipe on a real-world Ptychographic reconstruction workflow with sensor rates up to 5000 patterns/sec.; UniPipe lowers reconstruction MSE by up to 30% and inference MSE by up to 50% over a pretrained baseline, and reduces the deadline-miss rate by 13%-22% over a multi-context baseline. UniPipe's unified deployment reduces the latency spread from 106% to 57% of the mean, and increases total throughput compared to multi-context baseline.