Disk compression

{{Short description| Automatic, disk-wide data compression}} {{more citations needed|date=April 2019}} {{use dmy dates|date=April 2019|cs1-dates=y}} {{use list-defined references|date=December 2021}}

'''Disk compression''' increases the amount of information that can be stored on a disk by automatically compressing and decompressing data as it is written to and read from it. A disk compression utility is software that provides disk compression by overriding the system's file access layer so that all programs participate in compression. When a program writes a file, the utility compresses the data before storage. When a program reads a file, the utility decompresses the stored data before providing it to the program.

Disk compression uses a single, normal file for storing the logical, user-visible files of a disk volume. This file tends to be relatively large and acts as a virtual drive. Typically, access to the compressed storage is via device driver. Similar utilities automatically compress and decompress each file instead of using a single, large file, but such a utility is not classified as disk compression.

Unlike a file compression utility which is generally interactive (user specifies which files to compress and when to compress them), a disk compression utility is noninteractive (files are processed automatically as part of normal file use). This automatic behavior is sometimes described as ''on-the-fly'', ''transparent'', ''real-time'' and ''online''.

Disk compression was popular especially in the early 1990s, when microcomputer hard disks were relatively small (20 to 80 megabytes) and rather expensive (roughly 10 USD per megabyte).<ref>{{cite web | url=http://www.pcguide.com/ref/hdd/file/compr-c.html | title=Disk Compression | work=The PC Guide | date=17 April 2001 | first=Charles M. | last=Kozierok}}</ref> Such utilities tended to be a more economic means of having more disk space as opposed to buying a disk with more space. A good disk compression utility could, on average, double the available space with negligible speed loss. Disk compression fell into disuse by the late 1990s, as advances in hard drive technology and manufacturing led to lower cost per unit of space.

==Use== To start using disk compression on an existing drive, the utility would follow steps like: # Create an empty, compressed, virtual drive (a normal file on the host drive). # Transfer files from the host drive to the compressed drive except for certain system files such as the swap file. # Increase the size of the compressed drive to accommodate files and allow empty space when complete.

Since a device driver is required to access a compressed drive, a compressed boot drive (C:) required changes to the boot process as follows:

# The System BIOS loads sector 0 of the first physical hard drive (Master Boot Record). # The partition sector loads sector 0 of the bootable partition (Volume Boot Record). In this case, it's the host drive. # Host drive sector 0 loads (in the case of DOS) the DOS BIOS (IO.SYS, IBMBIO.COM etc.) Depending on the version, only the first few sectors of the file or the whole file may be loaded into memory. In the first case, the loaded stub contains another loader loading the remainder of the file from disk. In the case of DR-DOS, the BIOS is packed and will decompress itself through an embedded decompression routine.<ref name="Paul_1997_OD-A3"/><ref name="Caldera_1998_NEW703"/> # The DOS BIOS then loads the DOS kernel (MSDOS.SYS, IBMDOS.COM etc.) # If the DOS supports the preload API, the DOS BIOS attempts to load a preloadable compression driver<ref name="Schulman_1994_Undocumented-DOS"/><ref name="Chappell_1994_DOS-Internals"/> (DBLSPACE.BIN, DRVSPACE.BIN or STACKER.BIN) from the root of the host drive and attempts to communicate with it through the DOS preload API.<ref name="Paul_1997_OD-A3"/><ref name="Paul_1997_NWDOSTIP"/> If the driver responds as expected, the DOS BIOS will keep the driver, if not, it will get discarded.<ref name="Slivka_1993"/> If kept and configured accordingly, drive letters may get swapped, so that the compressed drive becomes C:. # The DOS BIOS starts CONFIG.SYS processing by looking for the file in the root of the boot drive. In the case of DR-DOS, DCONFIG.SYS takes precedence over CONFIG.SYS, if present.<ref name="Paul_1997_NWDOSTIP"/><ref name="Paul_1997_OD-A3"/> # Compression drivers not loaded through the preload API may be loaded through DEVICE statements. # Depending on configuration and if this hasn't happened already, drive letters may get swapped, so that the compressed drive becomes C:. # Processing continues from compressed drive.

==Considerations== ; Performance: If a system is generally CPU bound, disk compression tends to decrease overall performance because data compression requires significant CPU time. But for a system that is more I/O bound, frequently waiting for storage accesses to complete, then compressing a drive can increase overall performance. With compressed data, there is less data to read from or write to a drive.<ref name="Paul_1997_NWDOSTIP"/> ; Data loss due to crosslinked files: Not all compression utilities would confirm the absence of errors in the FAT file system before compressing a disk in place. Some errors, such as crosslinked files, could result in data loss during the transfer process. ; Boot drive accessibility may depend on another drive: The compressed drive is only visible if the device driver is loaded and the compressed drive is mounted. A boot disk, for example, might not contain the driver. This was not normally an issue for solutions that came bundled with the operation systems unless booting foreign operating systems, as system utilities like SYS were modified to copy the compression drivers to boot disks by default,<ref name="Paul_1997_NWDOSTIP"/><ref name="Caldera_1998_NEW703"/> and the corresponding operation systems would also fall back to look for them on drives other than the boot disk if they couldn't be found there. ; Uninstall limited by host drive size: Uninstallation was problematic if the expanded size of data on the compressed disk was larger than the normal disk capacity. ; Accidentally deleting virtual drive file: Users did not always realize that the large file on the host drive contained the compressed drive. While it was usually hidden by the compression utility, users who did find the large file curious or suspicious were able to delete it which resulted in the loss of all the files stored in it.

==Examples==

===Hardware-assisted=== Some of the initial disk compression solutions used a coprocessor via driver software. Examples include:

; Stacker: From Stac Electronics.<ref name="IW_1992_Comparison"/><ref name="PCMAG_1992_Comp"/> ; {{anchor| Expanz}}Expanz!: From InfoChip Systems.<ref name="IW_1992_Comparison"/><ref name="PCMAG_1992_Comp"/><ref name="PCMAG_1990_Expanz"/> ; DiskDoubler dd2000: From Datran Corporation.

===Software only=== As processor power increased over time, software-only solutions began to reach or even outperform the performance of hardware-assisted solutions. These utilities were third-party components, either bundled in an OS distribution or installed separately.  ; {{visible anchor| Squish}}: From Sundog Software.<ref name="IW_1987_Squish"/><ref name="PCMAG_1987_Squish"/><ref name="PP_1989_Squish"/> ; Stacker: From Stac Electronics.<ref name="PCMAG_1992_Comp"/> ; {{visible anchor| XtraDrive}}: From Integrated Information Technology. ; SuperStor: From AddStor.<ref name="IW_1992_Comparison"/><ref name="PCMAG_1992_Comp"/> ; {{visible anchor| DoubleDisk}}: From Vertisoft Systems.<ref name="IW_1992_Comparison"/><ref name="PCMAG_1992_Comp"/> ; DoubleSpace: Based, in part, on Vertisoft Systems' DoubleDisk.<ref name="Slivka_1993"/> ; DriveSpace: DoubleSpace with a different compression algorithm (DRVSPACE.BIN). ; {{visible anchor| JAM}}: (JAM.SYS) from JAM Software, Kiev, Ukraine.<ref>{{Cite web |title=Metropoli BBS - readme.doc |url=https://files.mpoli.fi/unpacked/software/dos/compress/jam125sw.zip/readme.doc |access-date=2026-05-25 |website=files.mpoli.fi}}</ref> ; DiskDoubler: From Salient Software. ; {{visible anchor| Double Density}}: From Data Becker.

===Bundled=== Bundling disk compression with a new computer appealed to both resellers and users. Resellers could claim more storage space, and users liked that they did not have to install or configure the software. Operating systems pre-installed on new computers with bundled disk compression included:

; DR DOS 6.0 (1991): Bundled with a custom version of SuperStor. ; PalmDOS 1.0 (1992): Bundled with a DPMS-enabled version of SuperStor. ; MS-DOS 6.0 (1993): Bundled with DoubleSpace. (MS-DOS 6.0 introduced the so-called DOS preload API, an undocumented<ref name="Schulman_1994_Undocumented-DOS"/><ref name="Chappell_1994_DOS-Internals"/> challenge–response type interface which allowed DoubleSpace to be loaded before CONFIG.SYS was processed.) ; PC DOS 6.1 (1993): While the original release did not include a disk compressor, a custom version of {{visible anchor| SuperStor/DS}} was included in a later release, also introducing the preload API in the PC DOS line. ; MS-DOS 6.2 (1993): Bundled with an improved version of DoubleSpace. ; Novell DOS 7 (1993): Bundled with a DPMS-enabled version of Stacker (3.12), which also supported compressed data exchange with remote NetWare and Personal NetWare servers, also running Stacker, in order to avoid unnecessary recompression when copying files.<ref name="Paul_1997_NWDOSTIP"/> It also introduced support for the DOS 6 preload API in order to maintain compatibility with Microsoft's DoubleSpace in multi-boot scenarios.<ref name="Schulman_1994_Undocumented-DOS"/><ref name="Paul_1997_NWDOSTIP"/> A similar mechanism was used to load the optional single- or multi-user SECURITY component. ; PTS-DOS Extended 6.4 (1994): Bundled with a disk compression component named {{visible anchor| Folder}} (FOLDER.SYS).<ref name="CW_1994_PTS"/> ; PC DOS 6.3 (1994): Bundled with SuperStor/DS. ; MS-DOS 6.22 (1994): Bundled with DriveSpace. (Note that MS-DOS 6.21 shipped with no disk compression software for legal reasons.) ; PC DOS 7.0 (1995): Bundled with Stacker 4.02 (DPMS-enabled). ; Windows 95 (1995): Bundled with DriveSpace 2. ; Windows 95 OSR2 (1996): Bundled with a stripped-down version of DriveSpace 3. ; Windows 95 Plus! (1996): Bundled with the full version of DriveSpace 3. ; OpenDOS 7.01 (1997): Bundled with Stacker 3.12 (DPMS-enabled).<ref name="Paul_1997_NWDOSTIP"/> ; DR-DOS 7.02 (1998): Bundled with Stacker 3.12 (DPMS-enabled).<ref name="Caldera_1998_USER"/> DR-DOS 7.02 also added support for DriveSpace<ref name="Paul_1997_OD-A3"/> ; PC DOS 2000 (1998): Bundled with Stacker 4.02 (DPMS-enabled). ; Windows 98 (1998): Bundled with DriveSpace 3. ; DR-DOS 7.03 (1999): Bundled with Stacker 3.12 (DPMS-enabled). ; Windows 98 SE (1999): Bundled with DriveSpace 3. ; PTS-DOS Pro 2000 (1999): Bundled with {{visible anchor| Datacomp}}.

===Related=== Notable software adjacent to disk compression includes: ; Squeeze: Squeeze (from Turner Hall Publishing) was a resident DOS file compressor for Lotus 1-2-3 and Lotus Symphony files.<ref name="Hall_1986"/> ; Squish Plus ROM {{anchor| Squish ROM}}: This special ROM chip (from Sundog Software and distributed through Personalized Software) contained a custom version of Squish Plus with only 36 KB footprint adapted for the HP Portable Plus.<ref name="PP_1989_Squish"/> ; Double Tools for DoubleSpace: Double Tools (from Addstor) enhances the functionality of the DoubleSpace utility bundled with MS-DOS 6.0. ; Multimedia Stacker {{anchor| Multimedia Stacker}}: Multimedia Stacker (from Helix Software) bundled Stac's DPMS-enabled Stacker 4.01 with Helix Cloaking and utilities, utilizing Cloaking's built-in DPMS server to relocate and run in protected mode. ; THS filesystem {{anchor| THS}}: THS (from Thomas Scheuermann; 1994–1996) was a compressed file system driver for Linux which could read some compressed volume files.<ref name="Linux_DOSFS"/><ref name="Debian_1996"/><ref name="THSFS"/> ; DMSDOS {{anchor| DMSDOS}}: DMSDOS (1996) was a Linux kernel driver to support the reading and writing of DoubleSpace, DriveSpace 2 & 3, and Stacker 3 & 4 volumes.<ref name="DMSDOS_2005"/><ref name="CMSDOS_1998"/> ; NTFS compression: Windows includes support for compressing files and folders in an NTFS partition. As such, it has aspects of both file and disk compression.

==References== {{Reflist|refs= <ref name="IW_1987_Squish">{{cite journal |title=Squish - File compressor crams data into smaller space, but slowly |author-first=John |author-last=Walkenbach |date=1987-02-02 |journal=InfoWorld |volume=9 |issue=6 |page=55 |url=https://books.google.com/books?id=8jsEAAAAMBAJ&pg=PA55 |access-date=2015-04-05}}</ref> <ref name="PCMAG_1987_Squish">{{cite journal |title=Squish Puts a Squeeze on Files Before They Are Written to Disk |author-first=Stephen Randy |author-last=Davis |date=1987-02-24 |journal=PC Magazine |volume=6 |issue=4 |page=46 |url=https://books.google.com/books?id=phxlBt4dX3oC&pg=PA46 |access-date=2015-04-06}}</ref> <ref name="PP_1989_Squish">{{cite journal |title=50% more file space for as little as $89.95 |author-first=Hal |author-last=Goldstein |date=1989-07-08 |journal=The Portable Paper |volume=4 |issue=4 |page=46 |publisher=Personalized Software, now Thaddeus Computing |url=http://www.hpmuseum.net/pdf/ThePortablePaperV4N4_48pages_Jul-Aug%2089_OCR.pdf |access-date=2015-04-05}}</ref> <ref name="Paul_1997_OD-A3">{{cite web |author-first=Matthias R. |author-last=Paul |title=Caldera OpenDOS 7.01/7.02 Update Alpha 3 IBMBIO.COM README.TXT |url=http://www.uni-bonn.de/~uzs180/download/ibmbioa3.zip |date=1997-10-02 |access-date=2009-03-29 |url-status=dead |archive-url=https://web.archive.org/web/20031004074600/http://www-student.informatik.uni-bonn.de/~frinke/ibmbioa3.zip |archive-date=2003-10-04}} [https://web.archive.org/web/20181225154705/http://mirror.macintosharchive.org/max1zzz.co.uk/+Windows%20&%20DOS/DOS/System/Novell/Support/Bins/Op702src.zip]</ref> <ref name="Schulman_1994_Undocumented-DOS">{{cite book |author-first1=Andrew |author-last1=Schulman |author-first2=Ralf D. |author-last2=Brown |author-link2=Ralf D. Brown |author-first3=David |author-last3=Maxey |author-first4=Raymond J. |author-last4=Michels |author-first5=Jim |author-last5=Kyle |title=Undocumented DOS: A programmer's guide to reserved MS-DOS functions and data structures - expanded to include MS-DOS 6, Novell DOS and Windows 3.1 |publisher=Addison Wesley |edition=2 |date=1994 |orig-year=November 1993 |isbn=0-201-63287-X |url-access=registration |url=https://archive.org/details/undocumenteddosp00andr_0 }} (xviii+856+vi pages, 3.5"-floppy) Errata: [https://web.archive.org/web/20190417215556/http://www.cs.cmu.edu/afs/cs/user/ralf/pub/books/UndocumentedDOS/errata.ud2][https://web.archive.org/web/20190417212906/https://www.pcjs.org/pubs/pc/programming/Undocumented_DOS/#errata-2nd-edition]</ref> <ref name="Paul_1997_NWDOSTIP">{{cite book |title=NWDOS-TIPs — Tips & Tricks rund um Novell DOS 7, mit Blick auf undokumentierte Details, Bugs und Workarounds |series=MPDOSTIP |author-first=Matthias R. |author-last=Paul |date=1997-07-30 |edition=3 |language=de |url=http://www.antonis.de/dos/dos-tuts/mpdostip/html/nwdostip.htm |access-date=2015-04-05 |url-status=live |archive-url=https://web.archive.org/web/20170910194752/http://www.antonis.de/dos/dos-tuts/mpdostip/html/nwdostip.htm |archive-date=2017-09-10}} NWDOSTIP.TXT is a comprehensive work on Novell DOS 7 and OpenDOS 7.01, including the description of many undocumented features and internals. It is part of the author's yet larger MPDOSTIP.ZIP collection maintained up to 2001 and distributed on many sites at the time. The provided link points to a HTML-converted older version of the NWDOSTIP.TXT file.</ref> <ref name="Caldera_1998_USER">{{cite book |publisher=Caldera, Inc. |date=1998 |orig-year=1993, 1997 |title=Caldera DR-DOS 7.02 User Guide |url=http://www.drdos.net/documentation/usergeng/uglontoc.htm |access-date=2013-08-10 |url-status=dead |archive-url=https://web.archive.org/web/20161105165004/http://www.drdos.net/documentation/usergeng/uglontoc.htm |archive-date=2016-11-05}}</ref> <ref name="IW_1992_Comparison">{{cite journal |title=The big squeeze |author-first=Linda |author-last=Slovick |date=1992-02-17 |journal=InfoWorld |volume=14 |issue=7 |pages=57–63 |url=https://books.google.com/books?id=_z0EAAAAMBAJ&pg=PA62 |access-date=2015-04-06}}</ref> <ref name="PCMAG_1990_Expanz">{{cite journal |title=Expanz! board quickly compresses files on all disk types but forfeits some DOS compatibilities |author-first=Edward |author-last=Mendelson |author-link=Edward Mendelson |date=1990-11-13 |journal=PC Magazine |volume=9 |issue=19 |page=13 |url=https://books.google.com/books?id=WIeyaksLI8gC&pg=PT44 |access-date=2015-04-06}}</ref> <ref name="PCMAG_1992_Comp">{{cite journal |title=Double - no - triple your hard disk space with on-the-fly data compression utilities |author-first=Greg |author-last=Pastrick |date=1992-01-28 |journal=PC Magazine |volume=11 |issue=2 |page=261 |url=https://books.google.com/books?id=uEkrL23rU98C&pg=PT280 |access-date=2015-04-06}}</ref> <ref name="CW_1994_PTS">{{cite journal |title=MS-DOS-Clone made in Russia |date=1994-05-06 |journal=Computerwoche |url=http://www.computerwoche.de/a/ms-dos-clone-made-in-russia,1121205 |access-date=2015-04-12}}</ref> <ref name="Chappell_1994_DOS-Internals">{{cite book |author-first=Geoff |author-last=Chappell |date=1994 |title=DOS Internals |publisher=Addison Wesley |isbn=978-0-201-60835-9 }} [https://gopher.tildeverse.org/gopher.viste.fr/9/programming/PC/DOS/DOS%20Internals/DOS_Internals.zip][https://www.pcjs.org/pubs/pc/programming/DOS_Internals/] Errata: [http://www.geoffchappell.com/notes/dos/internals/][https://gopher.tildeverse.org/gopher.viste.fr/1/programming/PC/DOS/DOS%20Internals/crtdrvr][https://gopher.tildeverse.org/gopher.viste.fr/1/programming/PC/DOS/DOS%20Internals/xmswatch]</ref> <ref name="Hall_1986">{{cite journal |title=Squeeze 10 times as many spreadsheets on a disk - SQZ! - The data squeezer for Lotus |journal=PC Magazine |date=1986-10-14 |page=155 |publisher=Turner Hall Publishing |url=https://books.google.com/books?id=nuXmVNll5JEC&pg=PA155 |access-date=2016-11-18}}</ref> <ref name="Linux_DOSFS">{{Cite FTP |url=ftp://sunsite.unc.edu/pub/Linux/system/Filesystems/dosfs/thsfs.tgz |server=sunsite.unc.edu |url-status=dead |title=FTP link }}</ref> <ref name="Debian_1996">{{Cite web|url=https://lists.debian.org/debian-user/1996/09/msg01042.html|title = Re: Disk Compression - Stacker - Drivespace, etc}}</ref> <ref name="THSFS">http://ftp.nluug.nl/ftp/ibiblio/system/filesystems/thsfs.lsm {{Bare URL plain text|date=March 2022}}</ref> <ref name="DMSDOS_2005">{{cite web |url=http://cmp.felk.cvut.cz/~pisa/dmsdos/ |title=DMSDOS - Linux kernel driver |date=2005-03-18 |access-date=2016-05-25 |url-status=live |archive-url=https://web.archive.org/web/20161111124320/http://cmp.felk.cvut.cz/~pisa/dmsdos/ |archive-date=2016-11-11}}</ref> <ref name="CMSDOS_1998">{{cite web |title=DMSDOS CVF module |type=dmsdoc.doc |version=0.9.2.0 |date=1998-11-19 |url=http://cmp.felk.cvut.cz/~pisa/dmsdos/dmsdos.html |access-date=2016-11-01 |url-status=live |archive-url=https://web.archive.org/web/20161102123812/http://cmp.felk.cvut.cz/~pisa/dmsdos/dmsdos.html |archive-date=2016-11-02}}</ref> <ref name="Slivka_1993">{{cite journal |title=Hands on - Under the Hood: Inside MS-DOS 6 - The developers of MS-DOS describe the operating system's memory optimization and disk-compression technology |pages=197–202 |author-first1=Benjamin W. |author-last1=Slivka |author-first2=Eric |author-last2=Straub |author-first3=Richard |author-last3=Freedman |journal=BYTE Magazine |volume=18 |number=8 |date=July 1993 |url=https://archive.org/stream/BYTEVol1808199307PentiumPCs/BYTE%20Vol%2018-08%201993-07%20Pentium%20PCs#page/n203/mode/1up |access-date=2016-11-01}} [https://archive.org/stream/BYTEVol1808199307PentiumPCs/BYTE%20Vol%2018-08%201993-07%20Pentium%20PCs_djvu.txt]</ref> <ref name="Caldera_1998_NEW703">{{cite book |title=DR-DOS 7.03 WHATSNEW.TXT — Changes from DR-DOS 7.02 to DR-DOS 7.03 |publisher=Caldera, Inc. |date=1998-12-24 |url=http://www.lookas.net/ftp/incoming/darbui/Justas/DRDOS/WHATSNEW.TXT |access-date=2019-04-08 |url-status=dead |archive-url=https://web.archive.org/web/20190408142232/http://www.lookas.net/ftp/incoming/darbui/Justas/DRDOS/WHATSNEW.TXT |archive-date=2019-04-08}}</ref> }}

==External links== * {{cite web |url=http://www.microsoft.com/technet/archive/msdos/01_intro.mspx?mfr=true |title=MS-DOS 6 Technical Reference|website=Microsoft}} * {{cite web |url=http://www.techhelpmanual.com/805-doublespace_compressed_volume_file_layout.html |title=DoubleSpace CVF file layout}} ff. * {{cite journal|journal=BYTE Magazine |url=http://www.byte.com/art/9402/sec6/art1.htm |title=How Safe is Disk Compression? |date=February 1994 |url-status=dead |archiveurl=https://web.archive.org/web/20080619060529/http://www.byte.com/art/9402/sec6/art1.htm |archivedate=2008-06-19 }} * {{cite journal |journal=Smart Computing |url=http://www.smartcomputing.com/editorial/article.asp?article=articles/archive/l0504/9834/9834.asp&guid= |title=How To... Compress Hard Drives |date=April 1999}}

Compression Compression Category:Utility software types Category:Compression file systems