{{Short description|Programming method}} {{Distinguish|Overload (programming)|Exec (system call)|Overlapping code|Shared code}} {{Use dmy dates|date=July 2019|cs1-dates=y}} {{Use list-defined references|date=July 2022}} thumb|Schematic In a general computing sense, '''overlaying''' means "the process of transferring a block of program code or other data into main memory, replacing what is already stored".<ref name="OED_2015"/> Overlaying is a programming method that allows programs to be larger than the computer's main memory.<ref name="Butterfield_1986"/> An embedded system would normally use overlays because of the limitation of physical memory, which is internal memory for a system-on-chip, and the lack of virtual memory facilities.
==Usage== Constructing an overlay program involves manually dividing a program into self-contained object code blocks called overlays or links, generally laid out in a tree structure.{{efn|In OS/360 and successors, there may be multiple ''regions''{{efn|This has nothing to do with the term ''region'' in MVT storage management.}} each containing a complete overlay tree.}} ''Sibling'' segments, those at the same depth level, share the same memory, called ''overlay region''{{efn|The nomenclature varies depending on the system, e.g., in OS/360 ''region'' refers to an entire overlay tree.}} or ''destination region''. An overlay manager, either part of the operating system or part of the overlay program, loads the required overlay from external memory into its destination region when it is needed; this may be automatic or via explicit code. Often linkers provide support for overlays.<ref name="GNULink_2008"/>
==Example== The following example shows the control statements that instruct the OS/360 Linkage Editor to link an overlay program containing a single region, indented to show structure (segment names are arbitrary): <pre> INCLUDE SYSLIB(MOD1) INCLUDE SYSLIB(MOD2) OVERLAY A INCLUDE SYSLIB(MOD3) OVERLAY AA INCLUDE SYSLIB(MOD4) INCLUDE SYSLIB(MOD5) OVERLAY AB INCLUDE SYSLIB(MOD6) OVERLAY B INCLUDE SYSLIB(MOD7) </pre> <pre> +--------------+ | Root Segment | | MOD1, MOD2 | +--------------+ | +----------+----------+ | | +-------------+ +-------------+ | Overlay A | | Overlay B | | MOD3 | | MOD7 | +-------------+ +-------------+ | +--------+--------+ | | +-------------+ +-------------+ | Overlay AA | | Overlay AB | | MOD4, MOD5 | | MOD6 | +-------------+ +-------------+ </pre> These statements define a tree consisting of the permanently resident segment, called the ''root'', and two overlays A and B which will be loaded following the end of MOD2. Overlay A itself consists of two overlay segments, AA, and AB. At execution time overlays A and B will both utilize the same memory locations; AA and AB will both utilize the same locations following the end of MOD3.
All the segments between the root and a given overlay segment are called a ''path''.
==Applications== {{As of|2015}}, most business applications are intended to run on platforms with virtual memory.{{Citation needed|date=September 2025}} A developer on such a platform can design a program as if the memory constraint does not exist unless the program's working set exceeds the available physical memory. Most importantly, the architect can focus on the problem being solved without the added design difficulty of forcing the processing into steps constrained by the overlay size. Thus, the designer can use higher-level programming languages that do not allow the programmer much control over size (e.g. Java, C++, Smalltalk). On the other hand, failure to consider interactions between program modules might lead to thrashing and a significant performance penalty.
Still, overlays remain useful in embedded systems.<ref name="Levine_2000"/> Some low-cost processors used in embedded systems do not provide a memory management unit (MMU). In addition many embedded systems are real-time systems and overlays provide more determinate response-time than paging. For example, the Space Shuttle ''Primary Avionics System Software (PASS)'' uses programmed overlays.<ref name="NRC_1993"/>
Even on platforms with virtual memory, software components such as codecs may be decoupled to the point where they can be loaded in and out as needed.
==Historical use== IBM introduced the concept of a ''chain job''<ref name="IBM_1963_FORTRAN"/> in FORTRAN II. The program had to explicitly call the CHAIN subroutine to load a new link, and the new link replaced all of the old link's storage except for the Fortran COMMON area.
IBM introduced more general overlay handling<ref name="IBM_1963_IBJOB"/> in IBSYS/IBJOB, including a tree structure and automatic loading of links as part of CALL processing.
In OS/360, IBM extended the overlay facility of IBM 7090/94 IBSYS IBLDR<ref>{{cite book |title=IBM 7090/7094 Programming Systems IBJOB Processor Overlay Feature of IBLDR |date=1963 |publisher=IBM Corporation |url=http://www.bitsavers.org/pdf/ibm/7090/C28-6331_IBLDR_overlay_1963.pdf |access-date=September 1, 2025}}</ref> by allowing an overlay program to have independent overlay regions, each with its own overlay tree. Another ''dynamic'' system of overlays could be constructed using the <code>LOAD</code> or <code>LINK</code> system macros. Separately-compiled modules were loaded anywhere in available memory and retained until a <code>DELETE</code> macro was issued for them. Loaded programs could, in turn, load other programs and so on.<ref>{{cite book |title=IBM System/360 Operating System Supervisor Services and Macro Instructions |date=1974 |publisher=IBM Corporation |page=9 |url=https://bitsavers.org/pdf/ibm/360/os/R21.7_Apr73/GC28-6646-7_Supervisor_Services_and_Macro_Instructions_Rel_21.7_Sep74.pdf |access-date=September 1, 2025}}</ref> OS/360 also had a simpler overlay system for transient operating system SVC routines, using 1024-byte SVC transient areas. <!-- Need information on other vendors here. -->
Systems using Memory segmentation typically do not require overlay facilities. Programs on Burroughs Large Systems are composed of segments, which are natural divisions of the program such as COBOL paragraphs, ALGOL procedures, data structures, etc. Each segment is dynamically loaded as needed, and then possibly removed to swapped out to free storage. Specific overlay structures are not necessary because the MCP overlays automatically.<ref>{{cite book |title=Master Control Program Reference Manual |date=1969 |publisher=Burroughs Corporation |pages=3-1—3-7 |url=https://bitsavers.org/pdf/burroughs/LargeSystems/B5000_5500_5700/1042462_B5500_MCP_Reference_196906.pdf |access-date=September 1, 2025}}</ref> Multics, likewise, does not require specific overlays, because each module is a separate segment, which the supervisor loads, deletes, or swaps out as needed.
In the home computer era overlays were popular because the operating system and many of the computer systems it ran on lacked virtual memory and had very little RAM by current standards: the original IBM PC had between 16K and 64K, depending on configuration. Overlays were a popular technique in Commodore BASIC to load graphics screens.<ref name="Butterfield_1986"/>
"Several PC/MS-DOS linkers in the 1980s supported [overlays] in a form nearly identical to that used 25 years earlier on mainframe computers."<ref name="Levine_2000"/><ref name="Elliot_2012"/> Binary files containing memory overlays had de facto standard extensions '''.OVL'''<ref name="Elliot_2012"/> or '''.OVR'''<ref name="Dohmen_1990"/> (but also used numerical file extensions like '''.000''', '''.001''', etc. for subsequent files<ref name="Gavin"/>). This file type was used among others by WordStar<ref name="Mabett_1985"/> (consisting of the main executable <code>WS.COM</code> and the overlay modules <code>WSMSGS.OVR</code>, <code>WSOVLY1.OVR</code>, <code>MAILMERGE.OVR</code> and <code>SPELSTAR.OVR</code>, where the "fat" overlay files were even binary identical in their ports for CP/M-86 and MS-DOS<ref name="Necasek_2018_WordStar"/>), dBase,<ref name="Sidnam-Wright-Stevens_1990"/> and the ''Enable'' DOS office automation software package from Enable Software. Borland's Turbo Pascal<ref name="Herschel-Dieterich_2000"/><ref name="Eßer_2009"/> and the GFA BASIC compiler were able to produce .OVL files.
== See also == * Bank switching * Chain loading * Dynamic linker * Expanded memory (EMS) * Paging * Recursion * Virtual memory
==Notes== {{notelist}}
==References== <references> <ref name="OED_2015">{{cite web |title=Oxford Dictionaries |date=2015-11-26 |url=https://www.oxforddictionaries.com/definition/american_english/overlay |access-date=2022-07-10 |url-status=dead |archive-url=https://web.archive.org/web/20220710211317/https://www.lexico.com/en/definition/overlay |archive-date=2022-07-10}}</ref> <ref name="Butterfield_1986">{{cite book |title=Loading And Linking Commodore Programs |chapter=Part 4: Overlaying |issue=73 |date=June 1986 |page=74 |magazine=Compute! |editor-first=James "Jim" |editor-last=Butterfield |editor-link=Jim Butterfield |url=http://www.atarimagazines.com/compute/issue73/loading_and_linking.php |access-date=2022-07-10 |url-status=live |archive-url=https://web.archive.org/web/20220710211439/https://www.atarimagazines.com/compute/issue73/loading_and_linking.php |archive-date=2022-07-10 |quote=This lets you run programs which are, in effect, much larger than the amount of memory in your computer.}}</ref> <ref name="GNULink_2008">{{cite web |title=The GNU Linker documentation: Overlay Description |date=2008-06-03 |url=http://sourceware.org/binutils/docs/ld/Overlay-Description.html |access-date=2022-07-10 |url-status=live |archive-url=https://web.archive.org/web/20220623060507/https://sourceware.org/binutils/docs/ld/Overlay-Description.html |archive-date=2022-06-23 }} [https://web.archive.org/web/20080319060220/http://www.gnu.org/software/binutils/manual/ld-2.9.1/html_node/ld_22.html]</ref> <ref name="Levine_2000">{{cite book |title=Linkers & Loaders |author-last=Levine |author-first=John R. |author-link=John R. Levine |date=2000 |publisher=Morgan Kaufmann Publishers |isbn=1-55860-496-0 |page=177 |url=http://linker.iecc.com/ |access-date=2022-07-10 |url-status=live |archive-url=https://web.archive.org/web/20220406090302/https://linker.iecc.com/ |archive-date=2022-04-06}} [http://www.iecc.com/linker/<!-- https://web.archive.org/web/20220705125042/https://www.iecc.com/linker/ -->]</ref> <ref name="NRC_1993">{{cite book |title=An Assessment of Space Shuttle Flight Software Development Processes |date=November 1993 |orig-date=June 1993 |edition=2 |publisher=National Academy of Sciences, The National Academies Press |author=National Research Council |publication-place=Washington, DC, USA |isbn=978-0-309-04880-4 |lccn=93-84549 |doi=10.17226/2222 |hdl=2060/19930019745 |url=http://www.nap.edu/openbook.php?record_id=2222&page=39 |access-date=2012-10-29 }} (208 pages)</ref> <ref name="IBM_1963_FORTRAN">{{cite manual |title=IBM 7090/7094 Programming Systems – FORTRAN II Programming |id=Form C28-6054-4 File No. 7090-25 |date=August 1963 |work=Systems Reference Library |publisher=IBM Corporation |publication-place=Poughkeepsie, New York, USA |pages=34–35 |section=Chapter 12: The Chain Job |section-url=http://bitsavers.org/pdf/ibm/7090/C28-6054-4_7090_FORTRANII.pdf#page=34 |url=http://bitsavers.org/pdf/ibm/7090/C28-6054-4_7090_FORTRANII.pdf |access-date=2022-07-10 |url-status=live |archive-url=https://web.archive.org/web/20220315144942/http://bitsavers.org/pdf/ibm/7090/C28-6054-4_7090_FORTRANII.pdf |archive-date=2022-03-15}} (52 pages)</ref> <ref name="IBM_1963_IBJOB">{{cite manual |title=IBM 7090/7094 Programming Systems – IBJOB Processor – Overlay feature of IBLDR |id=Form C28-6331 File No. 7090-27 |edition=1 |date=May 1963 |work=Systems Reference Library |publisher=IBM Corporation |publication-place=Poughkeepsie, New York, USA |url=http://bitsavers.org/pdf/ibm/7090/C28-6331_IBLDR_overlay_1963.pdf |access-date=2021-12-26 |url-status=live |archive-url=https://web.archive.org/web/20220315161341/http://bitsavers.org/pdf/ibm/7090/C28-6331_IBLDR_overlay_1963.pdf |archive-date=2022-03-15}} (8 pages)</ref> <ref name="Mabett_1985">{{cite book |title=Getting started with WordStar, MailMerge + SpellStar |author-first=Alan |author-last=Mabbett |publisher=Cambridge University Press |date=1985 |isbn=0-521-31805-X}}</ref> <ref name="Sidnam-Wright-Stevens_1990">{{cite web |title=Ashton-Tate ships dBASE IV Version 1.1 |editor-first1=Liz |editor-last1=Sidnam-Wright |editor-first2=Brad |editor-last2=Stevens |date=1990-07-31 |location=Torrance, California, USA |publisher=Ashton Tate |page=2-2-2 |url=http://corphist.computerhistory.org/corphist/documents/doc-4434569d4e6a0.pdf |access-date=2014-02-13 |url-status=dead |archive-url=https://web.archive.org/web/20170404043038/http://corphist.computerhistory.org/corphist/documents/doc-4434569d4e6a0.pdf |archive-date=2017-04-04 |quote=Version 1.1 has a new dynamic Memory Management System (dMMS) that handles overlays more efficiently: the product requires less memory, which results in more applications space availability. […] The product's lower memory requirements of only 450K of RAM provide improved network support because supplemental hardware memory is no longer required to support networks. […] By speeding up areas of dBASE IV that are overlay-dependent, the new dMMS improves performance when working at the Control Center and in programs that use menus and windows.}} (5 pages)</ref> <ref name="Herschel-Dieterich_2000">{{cite book |title=Turbo Pascal 7.0 |language=de |author-first1=Rudolf |author-last1=Herschel |author-first2=Ernst-Wolfgang |author-last2=Dieterich |edition=2 |publisher={{ill|R. Oldenbourg Verlag|de}} |date=2000 |isbn=3-486-25499-5 |page=249}}</ref> <ref name="Eßer_2009">{{cite book |title=Betriebssysteme I |chapter=Chapter 6. Speicherverwaltung und Dateisysteme - Teil 5: Nicht-zusammenhängende Speicherzuordnung |language=de |author-first=Hans-Georg |author-last=Eßer |date=June 2009 |publisher=Hochschule München |location=Munich, Germany |url=http://hm.hgesser.de/bs-ss2009/folien/bs-ss2009-esser-06e-4up.pdf |access-date=2014-02-13 |url-status=live |archive-url=https://web.archive.org/web/20220508155626/http://hm.hgesser.de/bs-ss2009/folien/bs-ss2009-esser-06e-4up.pdf |archive-date=2022-05-08}} (9 pages)</ref> <ref name="Elliot_2012">{{cite web |author-first=John C. |author-last=Elliott |title=PRL file format |date=2012-06-05 |orig-date=2000-01-02 |website=seasip.info |url=https://www.seasip.info/Cpm/prl.html |access-date=2020-01-26 |url-status=live |archive-url=https://web.archive.org/web/20200126182027/https://www.seasip.info/Cpm/prl.html |archive-date=2020-01-26 |quote=[…] A PRL file is a relocatable binary file, used by MP/M and CP/M Plus for various modules other than .COM files. The file format is also used for FID files on the Amstrad PCW. There are several file formats which use versions of PRL: SPR (System PRL), RSP (Resident System Process). LINK-80 can also produce OVL (overlay) files, which have a PRL header but are not relocatable. GSX drivers are in PRL format; so are Resident System Extensions (.RSX). […]}} [https://www.seasip.info/Cpm/amsfid.html#PRL%20file%20format]</ref> <ref name="Dohmen_1990">{{cite magazine |title=Platz schaffen durch Überlagern - Overlay-Strukturen in Turbo Pascal |language=de |author-first=Norbert |author-last=Dohmen |magazine=mc |issn= |publisher= |issue=12 |volume=90 |date=1990 |pages=124–130 |url=https://guru-home.dyndns.org/Overlay.html |access-date=2022-08-04 |url-status=live |archive-url=https://web.archive.org/web/20220804163634/https://guru-home.dyndns.org/Overlay.html |archive-date=2022-08-04}} [https://web.archive.org/web/20220804163948/https://guru-home.dyndns.org/Overlay/pdf/Overlay.pdf]</ref> <ref name="Gavin">{{cite book |chapter=Create Program Overlays |title=Turbo Pascal - Norton Guide |version=v3 |author-first=Bruce |author-last=Gavin |editor-first=Dave |editor-last=Pearson |page=149 |url=http://www.x-hacker.org/ng/pascal/ng19519.html |access-date=2022-08-04 |url-status=live |archive-url=https://web.archive.org/web/20220804163703/http://www.x-hacker.org/ng/pascal/ng19519.html |archive-date=2022-08-04}}</ref> <ref name="Necasek_2018_WordStar">{{cite web |title=WordStar Again |date=2018-01-30 |orig-date=2018-01-28, 2018-01-26 |author-first=Michal |author-last=Necasek |work=OS/2 Museum |url=http://www.os2museum.com/wp/wordstar-again/comment-page-1/ |access-date=2019-07-28 |url-status=live |archive-url=https://web.archive.org/web/20190728011415/http://www.os2museum.com/wp/wordstar-again/comment-page-1/ |archive-date=2019-07-28 |quote=[…] The reason to suspect such difference is that version 3.2x also supported CP/M-86 (the overlays are identical between DOS and CP/M-86, only the main executable is different) […] the .OVR files are 100% identical between DOS and CP/M-86, with a flag (clearly shown in the WordStar 3.20 manual) switching between them at runtime […] the OS interface in WordStar is quite narrow and well abstracted […] the WordStar 3.2x overlays are 100% identical between the DOS and CP/M-86 versions. There is a runtime switch which chooses between calling INT 21h (DOS) and INT E0h (CP/M-86). WS.COM is not the same between DOS and CP/M-86, although it's probably not very different either. […]}}</ref> </references>
==Further reading== * {{cite book |title=IBM OS Linkage Editor and Loader - Program Numbers 360S-ED-510, 360S-ED-521, 360S-LD-547 |id=Order No. GC28-6538-9, File No. S360-31 |version=Release 21 |date=March 1972 |orig-date=January 1972 |edition=10 |publisher=IBM Corporation |publication-place=White Plains, New York, USA |work=Systems Reference Library |url=http://www.bitsavers.org/pdf/ibm/360/os/R21.0_Mar72/GC28-6538-9_OS_Linkage_Editor_and_Loader_Release_21_Jan72.pdf |url-status=live |archive-url=https://web.archive.org/web/20220710205449/http://www.bitsavers.org/pdf/ibm/360/os/R21.0_Mar72/GC28-6538-9_OS_Linkage_Editor_and_Loader_Release_21_Jan72.pdf |archive-date=2022-07-10}} (2+244+4 pages) * {{cite newsgroup |title=GEOS/NDO info for RBIL62? |author-first1=Marcus |author-last1=Groeber |author-first2=Edward "Ed" |author-last2=Di Geronimo, Jr. |author-first3=Matthias R. |author-last3=Paul |date=2002-03-02 |orig-date=2002-02-24 |newsgroup=comp.os.geos.programmer |url=https://groups.google.com/d/msg/comp.os.geos.programmer/8NNPJ4VU23A/cucVV95kf9oJ |access-date=2019-04-20 |url-status=live |archive-url=https://archive.today/20190420111421/https://groups.google.com/forum/%23!msg/comp.os.geos.programmer/8NNPJ4VU23A/cucVV95kf9oJ |archive-date=2019-04-20 |quote=[…] The reason Geos needs 16 interrupts is because the scheme is used to convert inter-segment ("far") function calls into interrupts, without changing the size of the code. The reason this is done so that "something" (the kernel) can hook itself into every inter-segment call made by a Geos application and make sure that the proper code segments are loaded from virtual memory and locked down. In DOS terms, this would be comparable to an overlay loader, but one that can be added without requiring explicit support from the compiler or the application. What happens is something like this: […] 1. The real mode compiler generates an instruction like this: CALL <segment>:<offset> -> 9A <offlow><offhigh><seglow><seghigh> with <seglow><seghigh> normally being defined as an address that must be fixed up at load time depending on the address where the code has been placed. […] 2. The Geos linker turns this into something else: INT 8xh -> CD 8x […] DB <seghigh>,<offlow>,<offhigh> […] Note that this is again five bytes, so it can be fixed up "in place". Now the problem is that an interrupt requires two bytes, while a CALL FAR instruction only needs one. As a result, the 32-bit vector (<seg><ofs>) must be compressed into 24 bits. […] This is achieved by two things: First, the <seg> address is encoded as a "handle" to the segment, whose lowest nibble is always zero. This saves four bits. In addition […] the remaining four bits go into the low nibble of the interrupt vector, thus creating anything from INT 80h to 8Fh. […] The interrupt handler for all those vectors is the same. It will "unpack" the address from the three-and-a-half byte notation, look up the absolute address of the segment, and forward the call, after having done its virtual memory loading thing... Return from the call will also pass through the corresponding unlocking code. […] The low nibble of the interrupt vector (80h–8Fh) holds bit 4 through 7 of the segment handle. Bit 0 to 3 of a segment handle are (by definition of a Geos handle) always 0. […] all Geos API run through the "overlay" scheme […]: when a Geos application is loaded into memory, the loader will automatically replace calls to functions in the system libraries by the corresponding INT-based calls. Anyway, these are not constant, but depend on the handle assigned to the library's code segment. […] Geos was originally intended to be converted to protected mode very early on […], with real mode only being a "legacy option" […] almost every single line of assembly code is ready for it […]}}
==External links== * http://computer-programming-forum.com/29-pascal/45df7513ab22cf7e.htm<!-- https://web.archive.org/web/20220804165731/http://computer-programming-forum.com/29-pascal/45df7513ab22cf7e.htm -->
Category:Virtual memory