3D Model Fine-Tuning ⚡ TRELLIS.2: LoRA Guide (2026)
Learn 3D model fine-tuning for TRELLIS.2 with LoRA fine-tuning: prep mesh data, train smart stages, and boost 3D asset generation with measured gains.

Fine-Tuning a 3D Generator: A Practical TRELLIS.2 Workflow
A general-purpose 3D generator can still produce something that looks right at a glance - while getting the part that actually matters wrong. A shoe might look recognizable from the top, for instance, but have a distorted sole. For product design, simulation, robotics, or asset production, that isn't a cosmetic issue: the geometry can make or break what happens downstream.
Fine-tuning can help, but the first decision isn't "how long should we train?" It's which part of the pipeline needs to change - and whether your data can teach that part reliably. In a TRELLIS.2 footwear experiment, LoRA adapters were added to the stages responsible for structure and shape, while the material stage stayed frozen. The training set had 179 objects, with nine held out for evaluation.
This guide walks through the reasoning behind that setup, how the 3D pipeline is broken into stages, how to prepare and train adapters, and how to tell whether the results are actually meaningful. The example is meant to be practical - not a promise that the same configuration will work for every category or dataset.
Why fine-tune a 3D generation model?
A pretrained model is built to handle a broad range of inputs. That generality is useful when you want varied outputs, but it becomes a problem when the job depends on a category-specific detail. The model may recognize "this is a shoe," but still fail to produce a consistently accurate sole, heel, or tread pattern.
So the real question isn't whether it can generate 3D. It's whether the errors are acceptable for your use case. A visual mock-up might tolerate approximate geometry. A pipeline for collision testing, manufacturing review, or a digital product catalogue might not.
Fine-tune the failure, not the whole model
In the footwear experiment, the base TRELLIS.2 model produced shoes that looked broadly right, but the soles were often off. The goal was specific: improve footwear geometry, not make the model better at every object type or change how materials behave.
That affects your training plan. If the problem is mostly in shape, you don't want to spend compute updating the parts that are already working. There's also a risk with small, narrow datasets: they can overwrite useful general knowledge. That's one reason to look at parameter-efficient fine-tuning instead of updating every weight.
Why LoRA fits a small dataset
LoRA (low-rank adaptation) trains small adapter matrices while keeping the original model weights frozen. That cuts down the number of trainable parameters, and it reduces the memory needed for gradients and optimizer state. It also makes it easier to keep the base model intact - and reuse or remove the adaptation later.
For this experiment, the footwear dataset came from Google Scanned Objects: 188 shoes drawn from a collection of 1,030 objects. The two adapters together trained about 9.3 million parameters. That's a more controlled change than full fine-tuning, but it's not risk-free. An adapter can still pick up quirks from the training sample, and a small dataset won't represent every shoe design.
Use LoRA when you have a focused adaptation task and limited data or compute, and you want to preserve the base model. Consider broader fine-tuning only if you have evidence the adapter's capacity isn't enough - and you have enough good data to justify updating more than just the adapters.
Which parts of TRELLIS.2 should you fine-tune?
TRELLIS.2 is staged, not one single "do-everything" model. It removes the input image background, encodes the image with DINOv3, predicts coarse structure, generates detailed geometry, and then generates materials. A final step combines geometry and material attributes into a textured mesh, exported as a GLB file.
This decomposition makes it easier to target your failure mode. It also helps you reason about what your dataset can actually supervise. If you have good geometry supervision but mismatched material maps, you might adapt shape and leave texture generation alone.
Coarse structure and detailed shape
The sparse-structure flow DiT predicts a coarse representation of the object. That output then guides the later geometry stage. The shape flow DiT uses that structure, along with image features, to produce a more detailed shape representation.
For the footwear task, LoRA adapters were added to both components. The sparse-structure model is a dense transformer; the shape model uses structured sparse latents. TRELLIS.2's O-Voxel representation focuses on surface-relevant voxels instead of spreading compute over empty space, which is why sparse processing helps for detailed 3D data.
The adapter setup was different for each model. The sparse-structure flow used rank 8 adapters in attention and feed-forward layers. The shape flow used rank 16 adapters in attention layers. In practice, rank controls how much capacity the adapter has: higher rank can adapt more, but it also increases the number of trainable parameters. Treat rank as a capacity choice to test, not something you automatically maximize.
Why the material stage stayed frozen
The texture flow DiT generates material properties like base colour, metallic, roughness, and alpha. In this dataset, the colour maps had diffuse colour with baked-in lighting, rather than separate material maps you'd need to train the material stage cleanly. When lighting is mixed into the colour target, the model can learn illumination as if it were a surface property - which is exactly the kind of mismatch you want to avoid.
There was also a practical product reason to leave the texture stage alone: the base model already produced acceptable colours and textures. The real problem was geometry. Training the texture stage would add risk without fixing the main failure. It also increases the chance of catastrophic forgetting, where the new training degrades what the model learned during pretraining.
A simple selection rule is to ask three questions before you train: Which output is wrong? Which stage controls it? Does your dataset provide a valid target for that stage? If the answers don't line up, changing the training plan is usually better than just adding more training steps.
How do you prepare and train a LoRA adapter?
The workflow has four parts: prepare the mesh data, render conditioning views, convert the meshes into model-specific latents, and train an adapter for each selected DiT stage. Keeping preprocessing separate is useful operationally. Once the latent data is saved, you can run multiple training experiments without redoing the expensive encoding.
The provided workflow uses container images for preprocessing and training. You can run them on Hugging Face Jobs or locally with Docker and a compatible GPU setup. Cloud jobs reduce the headache of provisioning hardware, but they add cost and depend on job limits and available configurations. Local runs give you more control, but you own the environment setup and GPU management.
Prepare a dataset the model can use
The preprocessing container uses Blender to render 16 views of each object. During training, one view is selected as the conditioning image. The mesh also goes through TRELLIS.2's frozen encoders to create the latent representations used during training.
In the reported run, nine of the 188 objects were held out, and 179 were used for training. Holdout objects should be excluded from training, not just from the final metric. If you include the evaluation objects during training, the results can look better simply because the model has already seen what you're asking it to reconstruct.
Before you start, make sure the meshes are available in the dataset location the workflow expects. Provide your Hugging Face access token securely, and request access to DINOv3 (it's gated). Also check you have enough storage for rendered views and saved latents. For this example dataset, preprocessing took about an hour on an A100 large job.
Run training in separate stages
The workflow has two training modes: ss for the sparse-structure flow and shape for the shape flow. Each mode uses its own configuration. The example runs used an A100 with 80 GB of memory. The sparse-structure run took about 7.2 hours; the shape run took about 3.6 hours. Those numbers reflect this setup and dataset, not a universal training-time estimate.
A simplified Jobs command for the structure stage looks like this:
export HF_TOKEN=hf_xxx
hf jobs run -d --flavor a100-large --timeout 12h --secrets HF_TOKEN \
cr.hf.co/suvadityamuk/trellis2-finetuning-train:latest \
trellis2-train --config ss \
--latents your-username/my-meshes-trellis2 \
--push-to your-username/my-trellis2-lora \
--run-name ss_lora
Use --config shape for the shape stage. The same training container can run locally through Docker as well. Keeping everything containerized helps ensure the training settings stay consistent between cloud and local execution. If you change the dataset, model configuration, or hardware, validate the settings - don't assume the example command is a production-ready template.
The training code uses PEFT to inject adapters into the selected model layers while keeping TRELLIS.2's custom model interface. Only parameters whose names identify them as LoRA weights are trainable. The adapter weights are stored in 32-bit floating point, while the forward pass uses bfloat16 autocasting to reduce memory use. Gradient checkpointing saves GPU memory by recomputing some activations during backpropagation, at the cost of extra compute.
How should you evaluate 3D fine-tuning results?
Training loss is a diagnostic, not a verdict. In this experiment, the sparse-structure loss dropped quickly at first and then more slowly. The shape-flow loss stayed about flat. Neither curve, by itself, tells you whether a generated sole is actually closer to the reference scan.
The training objective uses rectified flow matching, aligned with the model's pretraining approach. For each batch, the loop samples noise and a timestep, creates a noisy version of the target latent, predicts a velocity, and computes mean squared error against the target velocity. The optimizer updates adapter weights while the base model stays frozen. Gradient accumulation combines several smaller batches before each optimizer update.
Evaluate on objects the adapter did not train on
The held-out set is the real test. For each object, compare the input image, the base model output, and the fine-tuned output against the reference scan. Visual checks are important because a single averaged score can hide real problems - for example, you might see improvement on the sole but damage around the toe shape, or you might get better results on some views while staying inconsistent overall.
This experiment also reported geometry metrics:
- Chamfer distance measures how far points on one surface are from the nearest points on the other, in both directions. Lower is better.
- F-score at 0.02 combines precision and recall using a matching threshold set to 2% of the scan's bounding-box diagonal. Higher is better.
- Normal consistency measures agreement between surface directions. Higher is better.
On the nine held-out objects, the base model had a mean F-score of 0.276, versus 0.803 with the footwear adapters. Chamfer distance dropped from 0.174 to 0.033, while normal consistency rose from 0.530 to 0.804. The fine-tuned model improved F-score on all nine objects in this comparison.
Check the evaluation design, not only the headline score
Nine test objects can show something is working - but it doesn't prove the improvement generalizes across footwear. Category coverage, scan quality, object orientation, and rendering conditions can all influence the results. Also, the base model output can change depending on generation settings, so comparisons need consistent conditions.
Keep the split fixed across experiments, inspect per-object results, and log the generation settings alongside your metrics. A convincing workflow doesn't just report numbers; it also includes visual inspection and tests for regressions in areas outside your target feature. If the adaptation is for business use, validate it against the downstream tolerance - not against whatever metric is easiest to compute.
What can this footwear experiment tell you?
The clearest takeaway is matching the intervention to both the problem and the data you have. The team adapted structure and shape because those stages drive the geometry defect. They froze the material stage because the dataset didn't provide clean material targets, and the existing texture quality was already good enough.
That's not a one-size-fits-all TRELLIS.2 recipe. A furniture dataset with accurate geometry and well-separated material maps might justify a different plan. A dataset with only a handful of objects may be too small even for LoRA. If the model can't get the broad structure right, you may need a different conditioning approach or more representative data - not just a larger adapter rank.
Practical checklist before committing compute
- Define the failure precisely. Spell out which part of the output is wrong and what "acceptable geometry" means for your intended use.
- Map the failure to a pipeline stage. Don't update components that don't control the defect.
- Audit the targets. Check that meshes, views, and any material maps match what you want the model to learn.
- Hold out representative objects. Keep a test set before training, and inspect individual outputs - not only averages.
- Start with a controlled adapter. Freeze the pretrained weights and adjust rank or target layers only when the results justify it.
- Measure the downstream task. Geometry metrics matter, but they should connect to the requirements of simulation, design, or production.
This operational approach is also reusable: preprocess once, save latents, train stages separately, track runs, and keep checkpoints. Separate runs make it easier to see which adaptation helped. They also make failures easier to diagnose than a single "change everything" training job.
The results support a focused conclusion: LoRA fine-tuning improved measured footwear geometry on this nine-object holdout. They don't prove the adapters will generalize to other categories, that a larger rank is better, or that the outputs are ready for manufacturing or simulation without additional validation.
Conclusion
Fine-tuning a 3D generator works best as a targeted engineering change: identify the output defect, find the stage responsible, and train only where your data provides a real learning signal. In the TRELLIS.2 footwear example, LoRA adapters improved geometry metrics on held-out shoes while the material stage remained frozen.
The practical next step is to audit your own failure cases and dataset before spending GPU time. If the target, model component, and evaluation method line up, a small adapter experiment can tell you whether specialization is worth expanding. If they don't, more training probably won't fix the underlying mismatch.