# Language model

> Mediated Wiki article. Canonical URL: https://mediated.wiki/source/Language_model
> Markdown URL: https://mediated.wiki/source/Language_model.md
> Source: https://en.wikipedia.org/wiki/Language_model
> Source revision: 1354163461
> License: Creative Commons Attribution-ShareAlike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/)

{{Short description|Statistical model of language}}
{{Use dmy dates|date=July 2022}}

A '''language model''' is a computational [model](/source/Model)  that predicts sequences in [natural language](/source/natural_language).<ref>{{cite journal |last1=Blank |first1=Idan A. |title=What are large language models supposed to model? |journal=Trends in Cognitive Sciences |date=November 2023 |volume=27 |issue=11 |pages=987–989 |doi=10.1016/j.tics.2023.08.006|pmid=37659920 |doi-access=free }}"LLMs are supposed to model how utterances behave." </ref><ref>{{cite book |last1=Jurafsky |first1=Dan |last2=Martin |first2=James H. |title=Speech and Language Processing |date=2021 |edition=3rd |url=https://web.stanford.edu/~jurafsky/slp3/ |access-date=24 May 2022 |chapter=N-gram Language Models |chapter-url= https://web.stanford.edu/~jurafsky/slp3/3.pdf |archive-date=22 May 2022 |archive-url=https://web.archive.org/web/20220522005855/https://web.stanford.edu/~jurafsky/slp3/ |url-status=live }}</ref> Language models are useful for a variety of tasks, including [speech recognition](/source/speech_recognition),<ref>Kuhn, Roland, and Renato De Mori (1990). [https://www.researchgate.net/profile/Roland_Kuhn2/publication/3191800_Cache-based_natural_language_model_for_speech_recognition/links/004635184ee5b2c24f000000.pdf "A cache-based natural language model for speech recognition"]. ''IEEE transactions on pattern analysis and machine intelligence'' 12.6: 570–583.</ref> [machine translation](/source/machine_translation),<ref name="Semantic parsing as machine translation">Andreas, Jacob, Andreas Vlachos, and Stephen Clark (2013). [https://www.aclweb.org/anthology/P13-2009 "Semantic parsing as machine translation"] {{Webarchive|url=https://web.archive.org/web/20200815080932/https://www.aclweb.org/anthology/P13-2009/ |date=15 August 2020 }}. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).</ref> [natural language generation](/source/natural_language_generation) (generating more human-like text), [optical character recognition](/source/optical_character_recognition), [route optimization](/source/route_optimization),<ref>{{cite journal |last1=Liu |first1=Yang |last2=Wu |first2=Fanyou |last3=Liu |first3=Zhiyuan |last4=Wang |first4=Kai |last5=Wang |first5=Feiyue |last6=Qu |first6=Xiaobo |title=Can language models be used for real-world urban-delivery route optimization? |journal=The Innovation |date=2023 |volume=4 |issue=6 |article-number=100520 |doi=10.1016/j.xinn.2023.100520 |doi-access=free|pmid=37869471 |pmc=10587631 |bibcode=2023Innov...400520L }}</ref> [handwriting recognition](/source/handwriting_recognition),<ref>Pham, Vu, et al (2014). [https://arxiv.org/abs/1312.4569 "Dropout improves recurrent neural networks for handwriting recognition"] {{Webarchive|url=https://web.archive.org/web/20201111170554/https://arxiv.org/abs/1312.4569 |date=11 November 2020 }}. 14th International Conference on Frontiers in Handwriting Recognition. IEEE.</ref> [grammar induction](/source/grammar_induction),<ref>Htut, Phu Mon, Kyunghyun Cho, and Samuel R. Bowman (2018). [https://arxiv.org/pdf/1808.10000.pdf?source=post_page--------------------------- "Grammar induction with neural language models: An unusual replication"] {{Webarchive|url=https://web.archive.org/web/20220814010528/https://arxiv.org/pdf/1808.10000.pdf?source=post_page--------------------------- |date=14 August 2022 }}. {{arXiv|1808.10000}}.</ref> [information retrieval](/source/information_retrieval)<ref name="ponte1998">{{cite conference |first1=Jay M. |last1=Ponte |first2= W. Bruce |last2=Croft | title= A language modeling approach to information retrieval |conference=Proceedings of the 21st ACM SIGIR Conference |year=1998 |publisher=ACM |place=Melbourne, Australia | pages = 275–281| doi=10.1145/290941.291008}}</ref><ref name="hiemstra1998">{{cite conference | first=Djoerd | last=Hiemstra | year = 1998 | title = A linguistically motivated probabilistically model of information retrieval | conference = Proceedings of the 2nd European conference on Research and Advanced Technology for Digital Libraries | publisher = LNCS, Springer | pages=569–584 | doi= 10.1007/3-540-49653-X_34}}</ref> and [disaster response](/source/disaster_response).<ref>{{cite journal |last1=Maity |first1=Abhishek |title=CrisisSense: Transforming Social Signals into Real-Time Disaster Awareness via Deep Neural Intelligence |journal=2026 IEEE Madhya Pradesh Section Conference (MPCON) |date=March 2026 |pages=1501–1506 |doi=10.1109/MPCON69668.2026.11508516 |url=https://ieeexplore.ieee.org/document/11508516}}</ref>

[Large language model](/source/Large_language_model)s (LLMs), currently their most advanced form as of 2026, are predominantly based on [transformers](/source/Transformer_(machine_learning)) trained on larger [datasets](/source/Data_set) (frequently using texts [scraped](/source/Web_scraping) from the public [internet](/source/internet)). They have superseded [recurrent neural network](/source/recurrent_neural_network)-based models, which had previously superseded the purely [statistical models](/source/Statistical_model), such as the [word ''n''-gram language model](/source/Word_n-gram_language_model).

== History ==
[Noam Chomsky](/source/Noam_Chomsky) did pioneering work on language models in the 1950s by developing a theory of [formal grammar](/source/formal_grammar)s.<ref>{{Cite journal |last=Chomsky |first=N. |date=September 1956 |title=Three models for the description of language |journal=IRE Transactions on Information Theory |volume=2 |issue=3 |pages=113–124 |doi=10.1109/TIT.1956.1056813 |bibcode=1956IRTIT...2..113C |issn=2168-2712}}</ref>

In 1980, statistical approaches were explored and found to be more useful for many purposes than rule-based formal grammars. Discrete representations like [word ''n''-gram language models](/source/Word_n-gram_language_model), with [probabilities](/source/Probability) for discrete combinations of words, made significant advances.

In the 2000s, continuous representations for words, such as [word embeddings](/source/Word2vec), began to replace discrete representations.<ref>{{Cite news |date=2022-02-22 |title=The Nature Of Life, The Nature Of Thinking: Looking Back On Eugene Charniak's Work And Life |url=https://cs.brown.edu/news/2022/02/22/the-nature-of-life-the-nature-of-thinking-looking-back-on-eugene-charniaks-work-and-life/ |archive-url=https://web.archive.org/web/20241103134558/https://cs.brown.edu/news/2022/02/22/the-nature-of-life-the-nature-of-thinking-looking-back-on-eugene-charniaks-work-and-life/ |archive-date=3 November 2024 |access-date=2025-02-05 |language=en |url-status=live }}</ref> Typically, the representation is a [real-valued vector](/source/Vector-valued_function) that encodes a word’s meaning such that words closer in [vector space](/source/vector_space) are similar in meaning and common relationships between words, such as plurality or gender, are preserved.

== Pure statistical models ==
In 1980, the first significant statistical language model was proposed, and during the decade [IBM](/source/IBM) performed '[Shannon](/source/Claude_Shannon)-style' experiments, in which potential sources for language modeling improvement were identified by observing and analyzing the performance of human subjects in predicting or correcting text.<ref>{{cite journal |last1=Rosenfeld |first1=Ronald |year=2000 |title=Two decades of statistical language modeling: Where do we go from here? |url=https://figshare.com/articles/journal_contribution/6611138 |journal=Proceedings of the IEEE |volume=88 |issue=8 |pages=1270–1278 |doi=10.1109/5.880083 |bibcode=2000IEEEP..88.1270R |s2cid=10959945}}</ref>

=== Models based on word ''n''-grams ===
{{excerpt|Word n-gram language model}}

=== Exponential ===

[Maximum entropy](/source/Principle_of_maximum_entropy) language models encode the relationship between a word and the ''n''-gram history using feature functions. The [equation](/source/equation) is

<math display="block"> P(w_m \mid w_1,\ldots,w_{m-1}) = \frac{1}{Z(w_1,\ldots,w_{m-1})} \exp (a^T f(w_1,\ldots,w_m))</math>

where <math>Z(w_1,\ldots,w_{m-1})</math> is the [partition function](/source/Partition_function_(mathematics)), <math>a</math> is the parameter vector, and <math>f(w_1,\ldots,w_m)</math> is the feature function. In the simplest case, the feature function is just an indicator of the presence of a certain ''n''-gram. It is helpful to use a prior on <math>a</math> or some form of [regularization](/source/Regularization_(mathematics)).

The log-bilinear model is another example of an exponential language model.

=== Skip-gram model ===
{{excerpt|word n-gram language model#Skip-gram language model}}

== Neural models ==
=== Recurrent neural network ===
Continuous representations or [embeddings of words](/source/Word_embedding) are produced in [recurrent neural network](/source/recurrent_neural_network)-based language models (known also as ''continuous space language models'').<ref>{{cite web |last1=Karpathy |first1=Andrej |title=The Unreasonable Effectiveness of Recurrent Neural Networks |url=https://karpathy.github.io/2015/05/21/rnn-effectiveness/ |access-date=27 January 2019 |archive-date=1 November 2020 |archive-url=https://web.archive.org/web/20201101215448/http://karpathy.github.io/2015/05/21/rnn-effectiveness/ |url-status=live }}</ref> Such continuous space embeddings help to alleviate the [curse of dimensionality](/source/curse_of_dimensionality), which is the consequence of the number of possible sequences of words increasing [exponentially](/source/Exponential_growth) with the size of the vocabulary, further causing a data sparsity problem. Neural networks avoid this problem by representing words as non-linear combinations of weights in a neural net.<ref name="bengio">{{cite encyclopedia|title=Neural net language models|first=Yoshua|last=Bengio|year=2008|encyclopedia=[Scholarpedia](/source/Scholarpedia)|volume=3|issue=1|page=3881|url=http://www.scholarpedia.org/article/Neural_net_language_models|doi=10.4249/scholarpedia.3881|bibcode=2008SchpJ...3.3881B|doi-access=free|access-date=28 August 2015|archive-date=26 October 2020|archive-url=https://web.archive.org/web/20201026161505/http://www.scholarpedia.org/article/Neural_net_language_models|url-status=live}}</ref>

=== Large language models ===
{{excerpt|Large language model}}

Although sometimes matching human performance, it is not clear whether they are plausible [cognitive model](/source/cognitive_model)s. At least for recurrent neural networks, it has been shown that they sometimes learn patterns that humans do not, but fail to learn patterns that humans typically do.<ref>{{Cite book|last1=Hornstein|first1=Norbert|url=https://books.google.com/books?id=XoxsDwAAQBAJ&dq=adger+%22goldilocks%22&pg=PA153|title=Syntactic Structures after 60 Years: The Impact of the Chomskyan Revolution in Linguistics|last2=Lasnik|first2=Howard|last3=Patel-Grosz|first3=Pritty|last4=Yang|first4=Charles|date=2018-01-09|publisher=Walter de Gruyter GmbH & Co KG|isbn=978-1-5015-0692-5|language=en|access-date=11 December 2021|archive-date=16 April 2023|archive-url=https://web.archive.org/web/20230416160343/https://books.google.com/books?id=XoxsDwAAQBAJ&dq=adger+%22goldilocks%22&pg=PA153|url-status=live}}</ref>

== Evaluation and benchmarks ==

Evaluation of the quality of language models is mostly done by comparison to human created sample [benchmarks](/source/Benchmark_(computing)) created from typical language-oriented tasks. Other, less established, quality tests examine the intrinsic character of a language model or compare two such models. Since language models are typically intended to be dynamic and to learn from data they see, some proposed models investigate the rate of learning, e.g., through inspection of learning curves.<ref>{{Citation|last1=Karlgren|first1=Jussi|last2=Schutze|first2=Hinrich|chapter=Evaluating Learning Language Representations|date=2015|pages=254–260|publisher=Springer International Publishing|isbn=978-3-319-64205-5|doi=10.1007/978-3-319-64206-2_8|title=International Conference of the Cross-Language Evaluation Forum|series=Lecture Notes in Computer Science}}</ref>

Various data sets have been developed for use in evaluating language processing systems.<ref name=":0">{{cite arXiv|last1=Devlin|first1=Jacob|last2=Chang|first2=Ming-Wei|last3=Lee|first3=Kenton|last4=Toutanova|first4=Kristina|date=2018-10-10|title=BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding|eprint=1810.04805|class=cs.CL}}</ref> These include:

* [Massive Multitask Language Understanding](/source/Massive_Multitask_Language_Understanding) (MMLU)<ref>{{Citation |last=Hendrycks |first=Dan |title=Measuring Massive Multitask Language Understanding |date=2023-03-14 |url=https://github.com/hendrycks/test |archive-url=https://web.archive.org/web/20230315011614/https://github.com/hendrycks/test |archive-date=15 March 2023 |url-status=live |access-date=2023-03-15}}</ref>
* [Corpus of Linguistic Acceptability](/source/Corpus_of_Linguistic_Acceptability)<ref>{{Cite web|url=https://nyu-mll.github.io/CoLA/|title=The Corpus of Linguistic Acceptability (CoLA)|website=nyu-mll.github.io|access-date=2019-02-25|archive-date=7 December 2020|archive-url=https://web.archive.org/web/20201207081834/https://nyu-mll.github.io/CoLA/|url-status=live}}</ref>
* GLUE benchmark<ref>{{Cite web|url=https://gluebenchmark.com/|title=GLUE Benchmark|website=gluebenchmark.com|language=en|access-date=2019-02-25|archive-date=4 November 2020|archive-url=https://web.archive.org/web/20201104161928/https://gluebenchmark.com/|url-status=live}}</ref>
* Microsoft Research Paraphrase Corpus<ref>{{Cite web|url=https://www.microsoft.com/en-us/download/details.aspx?id=52398|title=Microsoft Research Paraphrase Corpus|website=Microsoft Download Center|language=en-us|access-date=2019-02-25|archive-date=25 October 2020|archive-url=https://web.archive.org/web/20201025121243/https://www.microsoft.com/en-us/download/details.aspx?id=52398|url-status=live}}</ref>
* Multi-Genre Natural Language Inference<ref>{{Cite web |title=MultiNLI |url=https://cims.nyu.edu/~sbowman/multinli/ |access-date=2026-05-09 |website=cims.nyu.edu}}</ref>
* Question Natural Language Inference
* Quora Question Pairs<ref>{{Citation|last=Aghaebrahimian|first=Ahmad|chapter=Quora Question Answer Dataset|date=2017|pages=66–73|publisher=Springer International Publishing|isbn=978-3-319-64205-5|doi=10.1007/978-3-319-64206-2_8|title=Text, Speech, and Dialogue|volume=10415|series=Lecture Notes in Computer Science}}</ref>
* Recognizing Textual Entailment<ref>{{Cite web|url=http://l2r.cs.uiuc.edu/~danr/Teaching/CS546-12/TeChapter.pdf|title=Recognizing Textual Entailment|last1=Sammons, V.G.Vinod Vydiswaran, Dan Roth|first1=Mark|last2=Vydiswaran|first2=V.G.|access-date=February 24, 2019|last3=Roth|first3=Dan|archive-date=9 August 2017|archive-url=https://web.archive.org/web/20170809113438/http://l2r.cs.uiuc.edu/~danr/Teaching/CS546-12/TeChapter.pdf}}</ref>
* Semantic Textual Similarity Benchmark
* SQuAD question answering Test<ref>{{Cite web|url=https://rajpurkar.github.io/SQuAD-explorer/|title=The Stanford Question Answering Dataset|website=rajpurkar.github.io|access-date=2019-02-25|archive-date=30 October 2020|archive-url=https://web.archive.org/web/20201030072130/https://rajpurkar.github.io/SQuAD-explorer/|url-status=live}}</ref>
* Stanford Sentiment [Treebank](/source/Treebank)<ref>{{Cite web|url=https://nlp.stanford.edu/sentiment/treebank.html|title=Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank|website=nlp.stanford.edu|access-date=2019-02-25|archive-date=27 October 2020|archive-url=https://web.archive.org/web/20201027125825/https://nlp.stanford.edu/sentiment/treebank.html|url-status=live}}</ref>
* Winograd NLI
* BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC, OpenBookQA, NaturalQuestions, TriviaQA, RACE, BIG-bench hard, GSM8k, RealToxicityPrompts, WinoGender, CrowS-Pairs<ref>{{Cite web |title=llama/MODEL_CARD.md at main · meta-llama/llama |url=https://github.com/meta-llama/llama/blob/main/MODEL_CARD.md |access-date=2024-12-28 |website=GitHub |language=en}}</ref>

== See also ==
{{portal |Linguistics |Mathematics |Technology}}
{{div col|colwidth=15em}}
* {{Annotated link|Artificial intelligence and elections}}
* [Cache language model](/source/Cache_language_model)
* [Deep linguistic processing](/source/Deep_linguistic_processing)
* [Ethics of artificial intelligence](/source/Ethics_of_artificial_intelligence)
* [Factored language model](/source/Factored_language_model)
* [Generative pre-trained transformer](/source/Generative_pre-trained_transformer)
* [Katz's back-off model](/source/Katz's_back-off_model)
* [Language technology](/source/Language_technology)
* [Semantic similarity network](/source/Semantic_similarity_network)
* [Statistical model](/source/Statistical_model)

{{div col end}}

== References ==
{{Reflist|refs=
<!-- <ref name=Manning-2022>{{cite journal
|last=Manning
|first=Christopher D.
|author-link=Christopher D. Manning
|title=Human Language Understanding & Reasoning
|journal=Daedalus
|year=2022
|volume=151
|issue=2
|pages=127–138
|doi=10.1162/daed_a_01905
|s2cid=248377870
|url=https://www.amacad.org/publication/human-language-understanding-reasoning
|access-date=10 March 2023
|archive-date=9 March 2023
|archive-url=https://web.archive.org/web/20230309154322/https://www.amacad.org/publication/human-language-understanding-reasoning
|url-status=live
}}</ref> -->
}}

== Further reading ==
{{refbegin}}

* {{cite conference |author1=Jay M. Ponte |author2=W. Bruce Croft | citeseerx=10.1.1.117.4237 |doi=10.1145/290941.291008 |doi-access=free | title = A Language Modeling Approach to Information Retrieval | book-title=Research and Development in Information Retrieval | year=1998 | pages=275–281 }}
* {{cite conference |author1=Fei Song |author2=W. Bruce Croft | citeseerx=10.1.1.21.6467 |doi=10.1145/319950.320022 |doi-access=free |title=A General Language Model for Information Retrieval | book-title=Research and Development in Information Retrieval |year=1999 | pages=279–280 }}
* {{cite tech report |first=Stanley F. |last=Chen |author2=Joshua Goodman |title=An Empirical Study of Smoothing Techniques for Language Modeling |institution=Harvard University |year=1998 |citeseerx=10.1.1.131.5458 |url=https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=273adbdb43097636aa9260d9ecd60d0787b0ef4d }}

{{refend}}
{{Template:Large language models}}
{{Natural language processing}}
{{Artificial intelligence navbox}}

*
Category:Statistical natural language processing
Category:Markov models

---
Adapted from the Wikipedia article [Language model](https://en.wikipedia.org/wiki/Language_model) by Wikipedia contributors ([contributor history](https://en.wikipedia.org/wiki/Language_model?action=history)). Available under [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/). Changes may have been made.