AI glossary · Advanced · Language models
Model merging
Model merging is a technique that mixes the weights of several fine-tuned models branched from the same base model to create a new model that combines the strengths of each, without additional training.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Infrastructure & using models
- Fields
- Language models
- Added
- 2026-10-12
How it works
The simplest method is to average the weight values at the same positions. Going further, methods include Task Arithmetic, which extracts the difference between the base model and a fine-tuned model (the part that changed as it gained a capability) and adds or subtracts it; TIES, which resolves conflicting changes before combining them; and SLERP, which smoothly interpolates between two models. Because no GPU retraining is needed, many combinations can be tried at low cost. Open-source tools such as mergekit perform these merges from a single configuration file.
Where you'll see it
On Hugging Face, you encounter them as models with 'merge' in the name, or whose model card includes a configuration listing the merge method, the source models and their ratios. Community-made merged models often rank near the top of open leaderboards. Companies experiment with things like combining a fine-tune strong at coding with one strong at Korean. The result is the same size as the originals, so serving costs do not increase.
Good to know
Usually only models with the same architecture and base model can be merged. Mixing does not always combine only strengths: one capability may weaken or strange outputs may appear, so always evaluate after merging. Repeated merging to push up leaderboard scores can also produce models overfitted to the test questions. Merged models also inherit the license terms of their source models, so if even one has non-commercial terms, the result is hard to use commercially.
Example
If a team has built two fine-tunes from the same open-weight base model, one for customer support and one for summarizing internal documents, they test whether merging the two with mergekit lets a single model handle both tasks. Reducing to one model to serve keeps the GPU setup simple, but they deploy only after confirming that the merged model passes the evaluation sets for both tasks.