# Lex (software)

> Mediated Wiki article. Canonical URL: https://mediated.wiki/source/Lex_(software)
> Markdown URL: https://mediated.wiki/source/Lex_(software).md
> Source: https://en.wikipedia.org/wiki/Lex_(software)
> Source revision: 1349005771
> License: Creative Commons Attribution-ShareAlike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/)

{{Short description|Lexical analyzer generator}}
{{Infobox software
| name                   = Lex
| logo                   = 
| screenshot             = 
| screenshot size        = 
| caption                = 
| author                 = [Mike Lesk](/source/Mike_Lesk), [Eric Schmidt](/source/Eric_Schmidt)
| developer              = 
| released               = {{Start date and age|1975}}
| latest release version = 
| latest release date    = 
| programming language   = [C](/source/C_(programming_language))
| operating system       = [Unix](/source/Unix), [Unix-like](/source/Unix-like), [Plan 9](/source/Plan_9_from_Bell_Labs)
| platform               = [Cross-platform](/source/Cross-platform)
| genre                  = [Command](/source/Command_(computing))
| license                = Plan 9: [MIT License](/source/MIT_License)
| website                = 
}}

'''Lex''' is a [computer program](/source/computer_program) that generates [lexical analyzer](/source/lexical_analysis)s ("scanners" or "lexers").<ref>{{cite book |last1=Levine |first1=John R. | author-link1=John R. Levine |last2=Mason |first2=Tony |last3=Brown |first3=Doug |title=lex & yacc |pages=[https://archive.org/details/lexyacc00levi/page/1 1]–2 |publisher=[O'Reilly](/source/O'Reilly_Media) |year=1992 |edition=2 |isbn=1-56592-000-7 |url=https://archive.org/details/lexyacc00levi|url-access=registration }}</ref><ref>{{Cite book | last=Levine | first=John | author-link=John Levine | title=flex & bison | publisher=O'Reilly Media | date=August 2009 | page=304 | url=http://oreilly.com/catalog/9780596155988 | isbn=978-0-596-15597-1 | archive-date=2011-08-16 | access-date=2010-01-08 | archive-url=https://web.archive.org/web/20110816025643/http://oreilly.com/catalog/9780596155988 | url-status=live }}</ref> It is commonly used with the [yacc](/source/yacc) [parser generator](/source/parser_generator) and is the standard lexical analyzer generator on many [Unix](/source/Unix) and [Unix-like](/source/Unix-like) systems. An equivalent tool is specified as part of the [POSIX](/source/POSIX) standard.<ref>[https://pubs.opengroup.org/onlinepubs/9699919799/utilities/lex.html The Open Group Base Specifications Issue 7, 2018 edition § Shell & Utilities § Utilities § lex]</ref>

Lex reads an input [stream](/source/stream_(computing)) specifying the lexical analyzer and writes [source code](/source/source_code) which implements the lexical analyzer in the [C programming language](/source/C_(programming_language)).

In addition to C, some old versions of Lex could generate a lexer in [Ratfor](/source/Ratfor).<ref>{{cite book|title=Lex & Yacc |url=https://archive.org/details/lexyacc00levi |url-access=registration |author1=John R. Levine |author2=John Mason |author3=Doug Brown |publisher=O'Reilly |date=1992 |isbn=9781565920002 }}</ref>

==History==
Lex was originally written by [Mike Lesk](/source/Mike_Lesk) and [Eric Schmidt](/source/Eric_Schmidt)<ref>{{cite web|first1=M.E. |last1=Lesk |first2=E. |last2=Schmidt |title=Lex – A Lexical Analyzer Generator|archive-url=https://archive.today/20120728112736/http://dinosaur.compilertools.net/lex/index.html|archive-date=2012-07-28|url-status=dead|url=http://dinosaur.compilertools.net/lex/index.html|access-date=August 16, 2010}}</ref> and described in 1975.<ref>{{cite web |url=http://epaperpress.com/lexandyacc/download/lex.pdf |title=Lex – A Lexical Analyzer Generator |first1=M.E. |last1=Lesk |first2=E. |last2=Schmidt |date=July 21, 1975 |work=UNIX TIME-SHARING SYSTEM:UNIX PROGRAMMER’S MANUAL, Seventh Edition, Volume 2B |publisher=bell-labs.com |access-date=Dec 20, 2011 |archive-date=March 16, 2012 |archive-url=https://web.archive.org/web/20120316125600/http://epaperpress.com/lexandyacc/download/lex.pdf |url-status=live }}</ref><ref>{{cite journal |last1=Lesk |first1=M.E. |date=October 1975 |title=Lex – A Lexical Analyzer Generator |journal=Comp. Sci. Tech. Rep. No. 39 |location=Murray Hill, New Jersey |publisher=Bell Laboratories}}</ref>
In the following years, Lex became the standard lexical analyzer generator on many [Unix](/source/Unix) and [Unix-like](/source/Unix-like) systems. In 1983, Lex was one of several UNIX tools available for Charles River Data Systems' [UNOS](/source/UNOS_(operating_system)) operating system under the [Bell Laboratories](/source/Bell_Laboratories) license.<ref>{{Cite book|year=1983|title=The Insider's Guide To The Universe|publisher=Charles River Data Systems, Inc.|url=https://www.1000bit.it/ad/bro/charles/CharlesRiverSystem-Universe.pdf|page=13|archive-date=2024-02-25|access-date=2024-02-10|archive-url=https://web.archive.org/web/20240225234735/https://www.1000bit.it/ad/bro/charles/CharlesRiverSystem-Universe.pdf|url-status=live}}</ref>
Although originally distributed as proprietary software, some versions of Lex are now [open-source](/source/open-source_software). Open-source versions of Lex, based on the original proprietary code, are now distributed with open-source operating systems such as [OpenSolaris](/source/OpenSolaris) and [Plan 9 from Bell Labs](/source/Plan_9_from_Bell_Labs). One popular open-source version of Lex, called [flex](/source/flex_lexical_analyser), or the "fast lexical analyzer", is not derived from proprietary coding.

==Structure of a Lex file==
The structure of a Lex file is intentionally similar to that of a yacc file: files are divided into three sections, separated by lines that contain only two percent signs, as follows:
*The '''definitions''' section defines [macros](/source/Macro_(computer_science)) and imports [header file](/source/header_file)s written in [C](/source/C_(programming_language)). It is also possible to write any C code here, which will be copied verbatim into the generated source file.
*The '''rules''' section associates [regular expression](/source/regular_expression) patterns with C [statement](/source/statement_(programming))s. When the lexer sees text in the input matching a given pattern, it will execute the associated C code.
*The '''C code''' section contains C statements and [function](/source/function_(programming))s that are copied verbatim to the generated source file. These statements presumably contain code called by the rules in the rules section. In large programs it is more convenient to place this code in a separate file linked in at [compile](/source/compiler) time.

==Example of a Lex file==
The following is an example Lex file for the [flex](/source/Flex_lexical_analyser) version of Lex. It recognizes strings of numbers (positive integers) in the input, and simply prints them out.

<syntaxhighlight lang="c">/*** Definition section ***/

%{
/* C code to be copied verbatim */
#include <stdio.h>
%}

%%
    /*** Rules section ***/

    /* [0-9]+ matches a string of one or more digits */
[0-9]+  {
            /* yytext is a string containing the matched text. */
            printf("Saw an integer: %s\n", yytext);
        }

.|\n    {   /* Ignore all other characters. */   }

%%
/*** C Code section ***/

int main(void)
{
    /* Call the lexer, then quit. */
    yylex();
    return 0;
}</syntaxhighlight>

If this input is given to <code>flex</code>, it will be converted into a C file, {{nowrap | <code>lex.yy.c</code>}}. This can be compiled into an executable which matches and outputs strings of integers. For example, given the input:
 abc123z.!&*2gj6
the program will print:
 Saw an integer: 123
 Saw an integer: 2
 Saw an integer: 6

==Using Lex with other programming tools==

===Using Lex with parser generators===

Lex, as with other lexical analyzers, limits rules to those which can be described by [regular expressions](/source/regular_expressions). Due to this, Lex can be implemented by a [finite-state automata](/source/finite-state_automata) as shown by the [Chomsky hierarchy](/source/Chomsky_hierarchy) of languages. To recognize more complex languages, Lex is often used with parser generators such as [Yacc](/source/Yacc) or [Bison](/source/GNU_bison). Parser generators use a [formal grammar](/source/formal_grammar) to parse an input stream.

It is typically preferable to have a parser, one generated by Yacc for instance, accept a stream of tokens (a "token-stream") as input, rather than having to process a stream of characters (a "character-stream") directly. Lex is often used to produce such a token-stream.

[Scannerless parsing](/source/Scannerless_parsing) refers to parsing the input character-stream directly, without a distinct lexer.

===Lex and make===
[make](/source/Make_(software)) is a utility that can be used to maintain programs involving Lex. Make assumes that a file that has an extension of <code>.l</code> is a Lex source file. The make internal macro <code>LFLAGS</code> can be used to specify Lex options to be invoked automatically by make.<ref>{{Cite journal
 |url=http://www.opengroup.org/onlinepubs/009695399/utilities/make.html
 |title=make
 |journal=The Open Group Base Specifications
 |issue=6
 |publisher=The IEEE and The Open Group
 |year=2004
 |id=IEEE Std 1003.1, 2004 Edition
 |archive-date=2010-10-31
 |access-date=2007-09-18
 |archive-url=https://web.archive.org/web/20101031131616/http://www.opengroup.org/onlinepubs/009695399/utilities/make.html
 |url-status=live
 }}</ref>

==See also==
*[Flex lexical analyser](/source/Flex_lexical_analyser)
*[Yacc](/source/Yacc)
*[Ragel](/source/Ragel)
*[re2c](/source/re2c)
*[PLY (Python Lex-Yacc)](/source/PLY_(Python_Lex-Yacc))
*[Comparison of parser generators](/source/Comparison_of_parser_generators)

==References==
{{Reflist}}

==External links==
{{Wikibooks|Guide to Unix|Commands}}
*[http://www.mactech.com/articles/mactech/Vol.16/16.07/UsingFlexandBison/ Using Flex and Bison at Macworld.com]
* {{man|1|lex|Solaris}}
* {{man|1|lex|Plan 9}}

{{Unix commands}}
{{Plan 9 commands}}

Category:Compiling tools
Category:Unix programming tools
Category:Unix SUS2008 utilities
Category:Plan 9 commands
Category:Finite-state machines
Category:Lexical analysis

---
Adapted from the Wikipedia article [Lex (software)](https://en.wikipedia.org/wiki/Lex_(software)) by Wikipedia contributors ([contributor history](https://en.wikipedia.org/wiki/Lex_(software)?action=history)). Available under [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/). Changes may have been made.