{{Short description|Parameter-efficient fine-tuning technique for large language models}} {{Distinguish|LoRa}}{{one source|date=January 2026}} '''LoRA''' ('''Low-Rank Adaptation''') is a parameter-efficient fine-tuning technique for large language models and other deep neural networks. Introduced in 2021 by researchers at Microsoft, LoRA enables adaptation of pre-trained models to specific tasks while requiring significantly fewer computational resources and trainable parameters than traditional full model fine-tuning.<ref name="hu2021">{{cite conference |last1=Hu |first1=Edward J. |last2=Shen |first2=Yelong |last3=Wallis |first3=Phillip |last4=Allen-Zhu |first4=Zeyuan |last5=Li |first5=Yuanzhi |last6=Wang |first6=Shean |last7=Wang |first7=Lu |last8=Chen |first8=Weizhu |title=LoRA: Low-Rank Adaptation of Large Language Models |conference=International Conference on Learning Representations |date=2022 |url=https://openreview.net/forum?id=nZeVKeeFYf9}}</ref>

==Background==

The development of increasingly large language models in the late 2010s and early 2020s created substantial computational challenges. GPT-1, released in 2018 with 117 million parameters, cost less than $50,000 to train.<ref name="pymnts">{{cite web |title=AI Cheat Sheet: Large Language Foundation Model Training Costs |url=https://www.pymnts.com/artificial-intelligence-2/2025/ai-cheat-sheet-large-language-foundation-model-training-costs/ |publisher=PYMNTS |access-date=2026-01-22 |date=2025-02-10}}</ref> GPT-2, released in 2019 with 1.5 billion parameters, required $40,000 to train.<ref name="pymnts"/>

By 2020, GPT-3 scaled to 175 billion parameters, with training costs estimated between $500,000 and $4.6 million.<ref name="cudocompute">{{cite web |title=What is the cost of training large language models? |url=https://www.cudocompute.com/blog/what-is-the-cost-of-training-large-language-models |publisher=CUDO Compute |access-date=2026-01-22 |date=2025-05-12}}</ref> Training consumed approximately 1,287 megawatt-hours of electricity.<ref name="umich2023">{{cite web |title=Optimization could cut the carbon footprint of AI training by up to 75% |url=https://news.umich.edu/optimization-could-cut-the-carbon-footprint-of-ai-training-by-up-to-75/ |publisher=University of Michigan |access-date=2026-01-22 |date=2023-04-19}}</ref> GPT-4, released in 2023, required over $100 million to train and consumed approximately 50 gigawatt-hours of energy using 25,000 Nvidia A100 GPUs running for 90 to 100 days.<ref name="forwardfuture">{{cite web |title=The Cost of AI: Breakdown of Investments in Training, Infrastructure and More |url=https://www.forwardfuture.ai/p/the-cost-of-ai-breakdown-of-investments-in-training-infrastructure-and-more |publisher=Forward Future |access-date=2026-01-22 |date=2025-05-05}}</ref> GPT-5, released in August 2025, required individual training runs costing over $500 million each, with total training costs estimated between $1.25 billion and $2.5 billion.<ref name="griffingpt5">{{cite web |title=OpenAI GPT-5 is costing $500 Million per training run and still failing |url=https://www.fanaticalfuturist.com/2025/05/openai-gpt-5-is-costing-500-million-per-training-run-and-still-failing/ |publisher=Fanatical Futurist |access-date=2026-01-22 |date=2025-05-30}}</ref><ref name="felloai">{{cite web |title=All You Need to Know About GPT-5 & OpenAI's 2025 Roadmap |url=https://felloai.com/2025/02/all-you-need-to-know-about-gpt-5-openais-2025-roadmap/ |publisher=Fello AI |access-date=2026-01-22 |date=2025-02-13}}</ref> This created a barrier where adapting such models to specific tasks through traditional fine-tuning became prohibitively expensive for most researchers and organizations.

==Purpose==

LoRA works by decomposing weight update matrices into lower-rank representations. Rather than updating all parameters in a neural network during fine-tuning, LoRA freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture.<ref name="hu2021"/> This approach is grounded in linear algebra and exploits the hypothesis that weight updates during fine-tuning have low "intrinsic rank," meaning the changes can be effectively represented with fewer parameters than the full weight matrix.<ref name="hu2021"/>

When applied to GPT-3, LoRA reduced trainable parameters by approximately 10,000 times (from 175 billion to roughly 18 million) and GPU memory requirements during training by 3 times (from 1.2 terabytes to 350 gigabytes).<ref name="hu2021"/><ref name="ibm2024">{{cite web |url=https://www.ibm.com/think/topics/lora |title=What is LoRA (Low-Rank Adaption)? |publisher=IBM |access-date=2026-01-22 |date=2024-11-17}}</ref> The technique applies broadly to any dense layers in deep learning models, though it has been most extensively studied in the context of large language models.<ref name="hu2021"/> After training, LoRA adapter weights can be merged with the base model weights, resulting in no additional inference latency during deployment.<ref name="hu2021"/>

LoRA has been widely adopted in post-training workflows, including integration with preference optimization methods such as direct preference optimization (DPO). Its parameter-efficient variations, such as QLoRA, enable fine-tuning of 30-billion-parameter models on a single 24 GB consumer GPU.<ref>{{cite book |last=von Csefalvay |first=Chris |title=Post-Training: A Practical Guide for AI Engineers and Developers |year=2026 |publisher=No Starch Press |isbn=978-1-7185-0520-9 |pages=23, 149–152}}</ref>

==Uses==

A primary use of LoRA is creating customized versions of large models at dramatically reduced cost. The adapter weights trained through LoRA can be folded back into the original base model, producing a new full-scale specialized model for a far lower cost than retraining the entire model.<ref name="hu2021"/> This allows organizations to create domain-specific versions of models like GPT-3 (175 billion parameters) while only bearing the computational cost of training a small adapter (18 million parameters), rather than the prohibitive expense of full model retraining. Once merged, the resulting model can achieve performance comparable to traditional fine-tuning while requiring a fraction of the resources to create.

Alternatively, organizations can maintain a single base model alongside multiple small LoRA adapters, each specialized for different tasks or domains. For example, a 175 billion parameter base model could be paired with separate 18 million parameter adapters for customer service, legal analysis, and medical applications. This approach dramatically reduces storage requirements compared to maintaining multiple full-scale fine-tuned models, as each adapter requires less than one percent of the storage space of a complete model.<ref name="hu2021"/>

LoRA also enables dynamic adapter swapping, where different adapters can be loaded and applied to the same base model without reloading the entire model into memory. This allows systems to switch between specialized tasks efficiently. Multiple adapters can also be combined by merging their weight updates, either with each other or with the base model, to create models with blended capabilities.<ref name="ibm2024"/>

==References== <references/>

Category:Deep learning Category:Machine learning Category:Artificial neural networks Category:Machine learning algorithms Category:Mathematical optimization Category:Linear algebra Category:Computational linguistics