# LoRA (machine learning)

> Mediated Wiki article. Canonical URL: https://mediated.wiki/source/LoRA_(machine_learning)
> Markdown URL: https://mediated.wiki/source/LoRA_(machine_learning).md
> Source: https://en.wikipedia.org/wiki/LoRA_(machine_learning)
> Source revision: 1346529752
> License: Creative Commons Attribution-ShareAlike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/)

{{Short description|Parameter-efficient fine-tuning technique for large language models}}
{{Distinguish|LoRa}}{{one source|date=January 2026}}
'''LoRA''' ('''Low-Rank Adaptation''') is a parameter-efficient [fine-tuning](/source/fine-tuning_(deep_learning)) technique for [large language model](/source/large_language_model)s and other [deep neural network](/source/deep_neural_network)s. Introduced in 2021 by researchers at [Microsoft](/source/Microsoft), LoRA enables adaptation of pre-trained models to specific tasks while requiring significantly fewer computational resources and trainable parameters than traditional full model fine-tuning.<ref name="hu2021">{{cite conference |last1=Hu |first1=Edward J. |last2=Shen |first2=Yelong |last3=Wallis |first3=Phillip |last4=Allen-Zhu |first4=Zeyuan |last5=Li |first5=Yuanzhi |last6=Wang |first6=Shean |last7=Wang |first7=Lu |last8=Chen |first8=Weizhu |title=LoRA: Low-Rank Adaptation of Large Language Models |conference=International Conference on Learning Representations |date=2022 |url=https://openreview.net/forum?id=nZeVKeeFYf9}}</ref>

==Background==

The development of increasingly [large language model](/source/large_language_model)s in the late 2010s and early 2020s created substantial computational challenges. [GPT-1](/source/GPT-1), released in 2018 with 117 million parameters, cost less than $50,000 to train.<ref name="pymnts">{{cite web |title=AI Cheat Sheet: Large Language Foundation Model Training Costs |url=https://www.pymnts.com/artificial-intelligence-2/2025/ai-cheat-sheet-large-language-foundation-model-training-costs/ |publisher=PYMNTS |access-date=2026-01-22 |date=2025-02-10}}</ref> [GPT-2](/source/GPT-2), released in 2019 with 1.5 billion parameters, required $40,000 to train.<ref name="pymnts"/>

By 2020, [GPT-3](/source/GPT-3) scaled to 175 billion parameters, with training costs estimated between $500,000 and $4.6 million.<ref name="cudocompute">{{cite web |title=What is the cost of training large language models? |url=https://www.cudocompute.com/blog/what-is-the-cost-of-training-large-language-models |publisher=CUDO Compute |access-date=2026-01-22 |date=2025-05-12}}</ref> Training consumed approximately 1,287 megawatt-hours of electricity.<ref name="umich2023">{{cite web |title=Optimization could cut the carbon footprint of AI training by up to 75% |url=https://news.umich.edu/optimization-could-cut-the-carbon-footprint-of-ai-training-by-up-to-75/ |publisher=University of Michigan |access-date=2026-01-22 |date=2023-04-19}}</ref> [GPT-4](/source/GPT-4), released in 2023, required over $100 million to train and consumed approximately 50 gigawatt-hours of energy using 25,000 Nvidia A100 [GPU](/source/Graphics_processing_unit)s running for 90 to 100 days.<ref name="forwardfuture">{{cite web |title=The Cost of AI: Breakdown of Investments in Training, Infrastructure and More |url=https://www.forwardfuture.ai/p/the-cost-of-ai-breakdown-of-investments-in-training-infrastructure-and-more |publisher=Forward Future |access-date=2026-01-22 |date=2025-05-05}}</ref> [GPT-5](/source/GPT-5), released in August 2025, required individual training runs costing over $500 million each, with total training costs estimated between $1.25 billion and $2.5 billion.<ref name="griffingpt5">{{cite web |title=OpenAI GPT-5 is costing $500 Million per training run and still failing |url=https://www.fanaticalfuturist.com/2025/05/openai-gpt-5-is-costing-500-million-per-training-run-and-still-failing/ |publisher=Fanatical Futurist |access-date=2026-01-22 |date=2025-05-30}}</ref><ref name="felloai">{{cite web |title=All You Need to Know About GPT-5 & OpenAI's 2025 Roadmap |url=https://felloai.com/2025/02/all-you-need-to-know-about-gpt-5-openais-2025-roadmap/ |publisher=Fello AI |access-date=2026-01-22 |date=2025-02-13}}</ref> This created a barrier where adapting such models to specific tasks through traditional fine-tuning became prohibitively expensive for most researchers and organizations.

==Purpose==

LoRA works by decomposing weight update matrices into lower-rank representations. Rather than updating all parameters in a neural network during fine-tuning, LoRA freezes the pre-trained model weights and injects trainable [rank](/source/rank_(linear_algebra)) decomposition matrices into each layer of the [Transformer](/source/Transformer_(machine_learning_model)) architecture.<ref name="hu2021"/> This approach is grounded in [linear algebra](/source/linear_algebra) and exploits the hypothesis that weight updates during fine-tuning have low "intrinsic rank," meaning the changes can be effectively represented with fewer [parameters](/source/Parameter_(machine_learning)) than the full weight matrix.<ref name="hu2021"/>

When applied to [GPT-3](/source/GPT-3), LoRA reduced trainable parameters by approximately 10,000 times (from 175 billion to roughly 18 million) and [GPU](/source/Graphics_processing_unit) memory requirements during training by 3 times (from 1.2 terabytes to 350 gigabytes).<ref name="hu2021"/><ref name="ibm2024">{{cite web |url=https://www.ibm.com/think/topics/lora |title=What is LoRA (Low-Rank Adaption)? |publisher=IBM |access-date=2026-01-22 |date=2024-11-17}}</ref> The technique applies broadly to any [dense layers](/source/Matrix_(mathematics)) in deep learning models, though it has been most extensively studied in the context of large language models.<ref name="hu2021"/> After training, LoRA adapter weights can be merged with the base model weights, resulting in no additional [inference](/source/inference) latency during deployment.<ref name="hu2021"/>

LoRA has been widely adopted in post-training workflows, including integration with preference optimization methods such as [direct preference optimization](/source/Reinforcement_learning_from_human_feedback) (DPO). Its parameter-efficient variations, such as QLoRA, enable fine-tuning of 30-billion-parameter models on a single 24 GB consumer GPU.<ref>{{cite book |last=von Csefalvay |first=Chris |title=Post-Training: A Practical Guide for AI Engineers and Developers |year=2026 |publisher=No Starch Press |isbn=978-1-7185-0520-9 |pages=23, 149–152}}</ref>

==Uses==

A primary use of LoRA is creating customized versions of large models at dramatically reduced cost. The adapter weights trained through LoRA can be folded back into the original base model, producing a new full-scale specialized model for a far lower cost than retraining the entire model.<ref name="hu2021"/> This allows organizations to create domain-specific versions of models like GPT-3 (175 billion parameters) while only bearing the computational cost of training a small adapter (18 million parameters), rather than the prohibitive expense of full model retraining. Once merged, the resulting model can achieve performance comparable to traditional fine-tuning while requiring a fraction of the resources to create.

Alternatively, organizations can maintain a single base model alongside multiple small LoRA adapters, each specialized for different tasks or domains. For example, a 175 billion parameter base model could be paired with separate 18 million parameter adapters for customer service, legal analysis, and medical applications. This approach dramatically reduces storage requirements compared to maintaining multiple full-scale fine-tuned models, as each adapter requires less than one percent of the storage space of a complete model.<ref name="hu2021"/>

LoRA also enables dynamic adapter swapping, where different adapters can be loaded and applied to the same base model without reloading the entire model into memory. This allows systems to switch between specialized tasks efficiently. Multiple adapters can also be combined by merging their weight updates, either with each other or with the base model, to create models with blended capabilities.<ref name="ibm2024"/>

==References==
<references/>

Category:Deep learning
Category:Machine learning
Category:Artificial neural networks
Category:Machine learning algorithms
Category:Mathematical optimization
Category:Linear algebra
Category:Computational linguistics

---
Adapted from the Wikipedia article [LoRA (machine learning)](https://en.wikipedia.org/wiki/LoRA_(machine_learning)) by Wikipedia contributors ([contributor history](https://en.wikipedia.org/wiki/LoRA_(machine_learning)?action=history)). Available under [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/). Changes may have been made.