# Data loading > Mediated Wiki article. Canonical URL: https://mediated.wiki/source/Data_loading > Markdown URL: https://mediated.wiki/source/Data_loading.md > Source: https://en.wikipedia.org/wiki/Data_loading > Source revision: 1340544454 > License: Creative Commons Attribution-ShareAlike 4.0 International (https://creativecommons.org/licenses/by-sa/4.0/) Part of data processing **Data loading**, or simply **loading**, is a part of [data processing](/source/Data_processing) where [data](/source/Data) is moved between two systems so that it ends up in a [staging area](/source/Staging_(data)) on the target system. With the traditional [extract, transform and load](/source/Extract%2C_transform_and_load) (ETL) method, the load job is the last step, and the data that is loaded has already been transformed. With the alternative method [extract, load and transform](/source/Extract%2C_load_and_transform) (ELT), the loading job is the middle step, and the transformed data is loaded in its original format for [data transformation](/source/Data_transformation_(computing)) in the target system. Traditionally, loading jobs on large systems have taken a long time, and have typically been run at night outside a company's opening hours. ## Purpose Two main goals of data loading are to obtain fresher data in the systems after loading, and that the loading is fast so that the data can be updated frequently. For full data refresh, faster loading can be achieved by turning off [referential integrity](/source/Referential_integrity), secondary indexes and [logging](/source/Logging_(computing)), but this is usually not allowed with incremental update or trickle feed. ## Types Data loading can be done either by complete update (immediate), incremental loading and updating (immediate), or trickle feed (deferred). The choice of technique may depend on the amount of data that is updated, changed or added, and how up-to-date the data must be. The type of data delivered by the source system, and whether historical data delivered by the source system can be trusted are also important factors. ### Full refresh Full data refresh means that existing data in the target table is deleted first. All data from the source is then loaded into the target table, new indexes are created in the target table, and new [measures](/source/Measure_(data_warehouse)) are calculated for the updated table. Full refresh is easy to implement, but involves moving of much data which can take a long time, and can make it challenging to keep historical data.[1] ### Incremental update Main article: [Change data capture](/source/Change_data_capture) Incremental update or incremental refresh means that only new or updated data is retrieved from the source system.[2][3] The updated data is then added to the existing data in the target system, and the existing data in the target system is updated. The indices and statistics are updated accordingly. Incremental update can make loading faster and make it easier to keep track of history, but can be demanding to set up and maintain.[1] ### Tricle feed Tricle feed or trickle loading means that when the source system is updated, the changes in the target system will occur almost immediately.[4][5] ## Loading to systems that are in use Main article: [Real-time computing](/source/Real-time_computing) When loading data into a system that is currently in use by users or other systems, one must decide when the system should be updated and what will happen to tables that are in use at the same time as the system is to be updated. One possible solution is to make use of [shadow tables](/source/Shadow_table).[6][7] ## See also - [Database](/source/Database) - [Data warehouse](/source/Data_warehouse) ## References 1. ^ [***a***](#cite_ref-:0_1-0) [***b***](#cite_ref-:0_1-1) ["Incremental Data Load vs Full Load ETL: 4 Critical Differences - Learn | Hevo"](https://hevodata.com/learn/incremental-data-load-vs-full-load/). 2022-04-14. Retrieved 2023-02-18. 1. **[^](#cite_ref-2)** ["Incremental Loading"](https://help.pyramidanalytics.com/content/root/MainClient/apps/Model/Model%20Pro/Data%20Flow/IncrementalLoading.htm). Retrieved 2023-02-18. 1. **[^](#cite_ref-3)** Mitchell, Tim (2020-07-23). ["The What, Why, When, and How of Incremental Loads"](https://www.timmitchell.net/post/2020/07/23/incremental-loads/). Retrieved 2023-02-18. 1. **[^](#cite_ref-4)** Zuters, Janis (2011). ["Near Real-Time Data Warehousing with Multi-stage Trickle and Flip"](http://link.springer.com/10.1007/978-3-642-24511-4_6). In Grabis, Janis; Kirikova, Marite (eds.). *Perspectives in Business Informatics Research*. Lecture Notes in Business Information Processing. Vol. 90. Springer Berlin Heidelberg. pp. 73–82. [doi](/source/Doi_(identifier)):[10.1007/978-3-642-24511-4_6](https://doi.org/10.1007%2F978-3-642-24511-4_6). [ISBN](/source/ISBN_(identifier)) [978-3-642-24510-7](https://en.wikipedia.org/wiki/Special:BookSources/978-3-642-24510-7). a data warehouse typically is a collection of historical data designed for decision support, so it is updated from the sources periodically, mostly on a daily basis. today's business however asks for fresher data. real-time warehousing is one of the trends to accomplish this, but there are a number of challenges to move towards true real-time. this paper proposes 'multi-stage trickle and flip' methodology for data warehouse refreshment. it is based on the 'trickle and flip' principle and extended in order to further insulate loading and querying activities, thus enabling both of them to be more efficient. 1. **[^](#cite_ref-5)** ["Trickle Loading Data"](https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/AdministratorsGuide/TrickleLoading/TrickleLoadingData.htm). Retrieved 2023-02-18. 1. **[^](#cite_ref-6)** ["Create shadow tables for synchronization - Data Management - Alibaba Cloud Documentation Center"](https://www.alibabacloud.com/help/en/data-management-service/latest/synchronize-shadow-tables). Retrieved 2023-02-18. 1. **[^](#cite_ref-7)** ["Shadow tables"](https://www.ibm.com/docs/en/db2/10.5?topic=tables-shadow). *[IBM](/source/IBM)*. 2015-08-10. Retrieved 2023-02-18. v t e Data warehouses Creating a data warehouse Concepts Database Dimension Dimensional modeling Fact OLAP Star schema Snowflake schema Reverse star schema Aggregate Single version of the truth Variants Column-oriented DBMS Data hub Data mesh Ensemble modeling patterns Anchor modeling Data vault modeling Focal point modeling HOLAP MOLAP ROLAP Operational data store Elements Data dictionary/Metadata Data mart Sixth normal form Surrogate key Fact Fact table Early-arriving fact Measure Dimension Dimension table Degenerate Slowly changing Filling Extract, transform, load (ETL) Extract, load, transform (ELT) Extract Transform Load Using a data warehouse Concepts Business intelligence Dashboard Data mining Decision support system (DSS) OLAP cube Data warehouse automation Languages Data Mining Extensions (DMX) MultiDimensional eXpressions (MDX) XML for Analysis (XMLA) Tools Business intelligence software Reporting software Spreadsheet Related People Bill Inmon Information factory Ralph Kimball Enterprise bus Dan Linstedt Products Comparison of OLAP servers Data warehousing products and their producers --- Adapted from the Wikipedia article [Data loading](https://en.wikipedia.org/wiki/Data_loading) by Wikipedia contributors ([contributor history](https://en.wikipedia.org/wiki/Data_loading?action=history)). Available under [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/). Changes may have been made.