A new study from researchers at Llama 3.1 challenges the effectiveness of recycling LoRA modules for adaptive model merging. The team examined nearly 1,000 user-contributed LoRAs from the Hugging Face Hub, testing whether adaptive merging methods could outperform training new LoRAs on the same data.

Their findings reveal that while adaptive merging can improve performance over base models, the benefits are limited compared to training fresh LoRAs. Surprisingly, the specific choice of LoRAs to merge matters little - even randomly initialized parameters yield similar results. This suggests adaptive merging may work primarily through regularization rather than cross-task transfer.

The researchers developed a new merging method through extensive design space exploration and confirmed that positive transfer is possible when highly relevant LoRAs are available in the pool. They've released their model checkpoints and code online for further investigation.