Fine-Tuning Time Prediction: Beyond FLOP Estimates13 min read2,619 words

Fine-Tuning Time Prediction: Beyond FLOP Estimates

Learn why FLOP estimates miss real fine-tuning times and how overhead-aware prediction can improve GPU scheduling, capacity planning, and delivery dates.

fine-tuning time predictionLLM training durationGPU schedulingCatBoost regressionmodel fine-tuning
Fine-Tuning Time Prediction: Beyond FLOP Estimates

Why LLM Fine-Tuning Takes Longer Than the FLOPs Suggest

A fine-tuning estimate can look very exact on paper and still land off by half a working day. The common back-of-the-envelope calculation - model size, token count, GPU count, and an assumed utilization rate - captures the training loop. It doesn't capture everything that happens after you allocate GPUs and before those same GPUs are available again.

On a shared cluster, that gap matters. One job running late holds up the queue behind it. Another that finishes early leaves expensive hardware sitting idle. If you're trying to plan experiments, deadlines, or GPU capacity, you need an estimate that reflects the whole job, not just the theoretical compute.

Dharma-AI tested this idea using 272 production supervised fine-tuning runs on NVIDIA H200 GPUs. The results explain where analytical estimates fall short, which job settings contain the most useful signals, and how much a learned predictor improved in this specific environment. The findings look promising - but they come with limits: the data is for one GPU type and one training objective, and the longest jobs show up too rarely.

Why do FLOP estimates miss real fine-tuning time?

A typical estimate for dense Transformer training starts with the approximate compute cost of six times the parameter count times the number of training tokens. You divide by GPU count, assumed peak GPU throughput, and an expected utilization factor (MFU):

Theoretical loop time ≈ (6 × parameters × training tokens) ÷ (GPU count × peak FLOPS × MFU)

This helps you reason about compute demand. It's not a full estimate of wall-clock duration. GPUs don't sustain peak throughput during real training: memory movement, communication overhead, and kernel efficiency all reduce the achieved performance. MFU tries to cover that gap, but its value depends on the workload and system.

Even a good MFU only adjusts the training-loop part of the story. A production fine-tuning job also spends time on setup, downloading and loading the model, and preparing data. It may run evaluations on a schedule, save checkpoints, upload artifacts, and write the final model to object storage. On a shared cluster, network and storage work can compete with other jobs.

The hidden costs are configuration-dependent

Those extra steps don't scale neatly with token count or GPU allocation. Two jobs with similar theoretical compute can still differ because one evaluates more often, uses a bigger evaluation set, or writes more checkpoint data. I/O contention can make the same configuration behave differently across runs.

This matters for scheduling. If you only estimate the optimization loop, the number might be fine for comparing compute requirements, but it won't tell you when a worker will actually be free. For a scheduler, the question isn't just "How many FLOPs?" - it's "How long will this configured job occupy the allocated resources in this environment?"

Dharma-AI included evaluation tokens in the FLOP baseline and tested different MFU assumptions. Even with those tweaks, the best baseline had a mean absolute error of 12.41 hours. With an average job duration of 23.43 hours, that's too much uncertainty to plan around tightly.

What should an end-to-end estimate account for?

A useful prediction starts with defining what you're predicting. In this study, "duration" meant wall-clock time from the moment a job received its GPUs until it finished. Queue time was excluded. That choice makes sense if your goal is to estimate resource occupancy - but it's not the same as predicting when a submitted job will start or finish from a user's perspective.

The predictor uses only information available before dispatch: configuration and model metadata. That's intentional. A scheduler can't rely on runtime logs or partial progress to decide what to run next; it needs an estimate before assigning the hardware.

Add features that represent work outside training steps

Dharma-AI derived features from the submitted job configuration, including:

  • Evaluation volume: num_eval_data covered samples processed across evaluation cycles.
  • Checkpoint activity: num_saves and num_params_saved described how often and how much model data would be persisted.
  • Workload scale: training/evaluation token volumes, parameter counts, and batch settings describe the job itself.
  • Allocation and regime: GPU count, fine-tuning approach, and the software stack capture differences between run configurations.
  • A compute-only starting point: an MFU-based estimate with MFU fixed at 0.45 (baseline_time) so the model could learn adjustments to the familiar FLOP calculation.

Not every team needs exactly these 24 features. The main idea is simple: the model should reflect the operational steps that actually consume time. A feature is only useful if you know it before dispatch and it connects meaningfully to what the job will do.

Let the data capture interactions

Configuration-to-duration relationships aren't always linear. For instance, evaluation overhead can matter more when evaluation is frequent and the dataset is large. Checkpoint cost might depend on model size and save policy together. A single multiplier on FLOPs can't represent those interactions well.

The team compared linear regression and Ridge against XGBoost and CatBoost. They used correlation filtering, mutual information, and permutation importance to narrow feature sets, then picked model settings by mean absolute error. That's a practical recipe: start with a simple baseline, add operational features, and check whether the model can learn nonlinear combinations that a FLOP-only baseline misses.

How much did the learned predictor improve?

The dataset included 272 production SFT runs totaling about 7,382 GPU-hours. Jobs covered several model families up to 15 billion parameters, LoRA and full fine-tuning, and a range of dataset sizes and hyperparameters. Runs used one, two, or four H200 GPUs drawn from a shared pool of 8 to 32 GPUs across one to four nodes. Other training and inference workloads were running alongside them, so the observed durations reflect real infrastructure contention.

Durations ranged from 1.6 to 172.4 hours. The mean was 23.43 hours and the median was 15.29 hours. Dharma-AI used stratified five-fold cross-validation and evaluated predictions on held-out folds rather than measuring performance only on the same data used to fit each model.

The gap between analytical and learned estimates

The FLOP baseline with MFU set to 30% achieved an R² of 0.31 and a mean absolute error (MAE) of 12.41 hours. Trying different MFU values didn't fix the problem: at 45%, MAE was 13.01 hours; at 60%, it was 14.62 hours. Even sweeping MFU from 25% to 65% left substantial error.

A more detailed analytical model, Calculon, also didn't do well on end-to-end duration in this comparison. Its MAE was 20.36 hours. That result should be read carefully: Calculon was built as a performance model for training steps, not as a full predictor for setup, evaluation, checkpointing, and storage work in production.

Linear regression and Ridge improved the FLOP baseline, with MAEs of 7.52 and 7.23 hours, respectively. Gradient-boosted models performed much better. XGBoost reached 1.95 hours, while CatBoost reached 1.71 hours, with an R² of 0.98. Across all out-of-fold CatBoost predictions, the median absolute error was 0.51 hours. Eighty percent of predictions were within 2.20 hours, and 90% were within 4.31 hours.

Good aggregate accuracy does not mean uniform accuracy

Where the errors land matters as much as the average. For jobs up to 60 hours, median absolute error stayed around an hour or less across duration bands. For the 13 runs above 60 hours, median absolute error rose to 5.19 hours, and the 90th-percentile error reached 18.51 hours.

That's a big limitation if you schedule long-running work. The study had few examples in that range, so the model had limited evidence to learn from. The overall MAE is strong for this dataset, but it shouldn't be treated as a guarantee that every job will land within two hours of its estimate.

The strongest signals also make operational sense. Evaluation volume ranked first in feature importance, followed by trainable parameter count, evaluation batch size relative to training, total parameter count, a GPU-count and training-batch interaction, and checkpoint volume. Feature importance isn't proof of causality - but it does point to configuration choices you should pay attention to when estimating job duration.

How can teams use duration prediction in production?

A duration estimate becomes useful when it changes a decision. On a shared cluster, a scheduler can use it to compare jobs, decide what can run together, forecast when GPUs may become free, or flag runs that likely won't finish before a deadline. For teams planning experiments, it can also help estimate how many runs fit into a week or within a fixed GPU budget.

Just don't treat the estimate like a promise. A production system should track what the prediction includes, when it was generated, and how uncertain it is. In this setup, the model predicted execution duration after resources were allocated - it didn't predict queue delay. If you mash queue and execution into one number, it becomes harder to interpret.

A practical rollout sequence

If you're building something like this for your own cluster, a cautious rollout beats a quick transplant from another environment:

  1. Define the target. Decide whether you need time from GPU allocation to completion, submission to completion, or a different interval. Keep queue time separate if another scheduler process controls it.
  2. Build a consistent run record. Capture the configuration available before launch and the actual end-to-end duration. Make sure timestamps and job boundaries mean the same thing across tools and teams.
  3. Establish a baseline. Treat the FLOP estimate as a reference, not the ground truth. Measure its error on completed jobs and report both average and tail behavior.
  4. Add overhead-aware inputs. Record evaluation cadence and volume, checkpoint policy, model and dataset scale, GPU allocation, and relevant software/hardware configuration.
  5. Evaluate on held-out runs. Use a validation scheme that matches deployment. If the model will predict future jobs, test on later runs, not only randomly mixed historical examples.
  6. Expose uncertainty. Show a prediction with a range or error band based on local validation. Use wider buffers for uncommon configurations and long jobs.
  7. Monitor drift. Recheck error when the GPU pool, software stack, job mix, or storage environment changes. Retrain only after you confirm the change is real and consistent with the new data.

Use the estimate to reduce operational waste

With decent duration predictions, a scheduler can make better trade-offs between throughput, fairness, and deadlines. But a more accurate estimate doesn't automatically make a better scheduler. Someone still has to decide what gets priority: short jobs, capacity for deadline-bound work, or avoiding indefinite delays for large jobs.

Start from the decision your estimate is supposed to improve. If your goal is fewer idle gaps, look at idle GPU time and queue throughput. If your goal is deadline reliability, measure the fraction of jobs that finish before their deadline. Model accuracy is necessary for those outcomes, but it's not the outcome itself.

Where does this result stop generalizing?

The reported performance is evidence that an overhead-aware predictor can work well on this specific production dataset. It doesn't automatically mean a model trained on these runs will match accuracy across every organization's jobs.

This study looked at supervised fine-tuning, not other training or alignment approaches like RLHF, DPO, or RLVR. Those workloads can have different phases, different resource patterns, and different overheads. The study also used only H200 GPUs. Even if two clusters are "similar," you shouldn't expect a model calibrated to one hardware pool to transfer numerically to another without validation - and likely retraining.

Infrastructure and data coverage set the boundary

A predictor trained on a shared H200 environment learns patterns from that environment, including the software stack and the contention profile. A mixed hardware pool needs descriptors such as compute throughput, memory capacity/bandwidth, and interconnect topology. Even then, hardware specs alone won't capture differences in kernels, storage behavior, or job placement.

Long jobs are another issue. Only 13 observations exceeded 60 hours, and errors widened a lot there. The training set also didn't show a clear plateau as more training data was added: on a fixed 20% holdout, MAE fell from 1.63 hours at 85% of the training partition to 1.50 hours when using all the data. That suggests more coverage could help for the sparse cases - but it's still not a proof that more data will produce dramatic gains.

There are limits to interpreting feature importance too. For example, GPU count showing up as positively associated with predicted duration doesn't mean "more GPUs makes a job slower." In this dataset, larger allocations tended to come with larger jobs, so GPU count can act as a proxy for job scale. Operational patterns and correlations have to be understood before you turn them into policy.

What transfers is the method: define the duration boundary, use inputs available before dispatch, include features that reflect overhead, compare against simple baselines, and validate on data that resembles the jobs the scheduler will actually see. That's a more portable lesson than any specific accuracy number.

Conclusion

FLOP calculations are still useful for understanding the compute scale of fine-tuning. But they're a weak substitute for end-to-end duration estimates when evaluation, checkpointing, data movement, and shared infrastructure determine how long GPUs stay occupied.

In the Dharma-AI production study, adding operational signals mattered: across 272 H200 SFT runs, CatBoost substantially outperformed the analytical baselines. The takeaway isn't to copy someone else's model. Instead, build a consistent record of your own jobs, predict the duration your scheduling decision actually needs, and validate accuracy - especially when your workload is sparse or changing. Better estimates help you plan better, but only when you're clear about the scope and the uncertainty.

Frequently Asked Questions

FLOPs estimate compute for the training loop, but a real job also spends time setting up, loading data, evaluating, saving checkpoints, and transferring artifacts. Shared network and storage contention add further variation. These costs depend on job configuration and infrastructure, so one MFU assumption cannot reliably account for them.
It means the CatBoost predictions differed from actual duration by an average of 1.71 hours in absolute terms across the study’s cross-validation predictions. It does not mean every run was within 1.71 hours. The median absolute error was 0.51 hours, while errors were notably larger for the small set of jobs lasting more than 60 hours.
Not without checking and likely retraining it. All 272 runs used H200 GPUs, so the fitted model reflects that hardware and its operating environment. Other GPU types differ in throughput, memory, and interconnect behavior. Validate against local runs before using predictions to schedule another hardware pool.
No. Its target was wall-clock time from GPU allocation until job completion. Queue time before a job received resources was excluded. A system planning user-visible completion dates needs to estimate queue delay separately or combine it with execution duration while keeping the two components explicit.

Related articles