AI glossary · Basic · Data engineering
Data pipeline
A data pipeline is a processing flow that automatically chains together collecting data scattered across many places, refining it for use and moving it to the storage where it is needed. Depending on the order of steps, it is divided into ETL (extract→transform→load) and ELT (extract→load→transform).
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Data, ops & platforms
- Fields
- Data engineering
- Added
- 2026-10-08
How it works
First, data is pulled from various sources such as app databases, web logs, external service APIs and Excel files (Extract). Next, formats are aligned, duplicates and missing values are cleaned up, and needed calculations are added (Transform). Finally, the cleaned data is placed into storage such as a data warehouse (Load). This process runs either in batch mode, processing everything at set intervals, or in streaming mode, sending data through as soon as it is created.
How do ETL and ELT differ?
ETL finishes transformation before loading into storage. It was common when storage space and compute were expensive, and has the advantage of storing only cleaned data. ELT loads the raw data into storage as is, then transforms it inside the storage. It became widely used as the compute power of cloud data warehouses grew, and because the raw data remains, it is easy to reprocess it in a different way later.
ETL vs. ELT
ETLExtract → Transform → Load
- Stores only cleaned-up data
- Better when storage is expensive
- Reprocessing means pulling from the source again
ELTExtract → Load → Transform
- Stores the raw data as is
- Transforms using the store's own compute
- Easy to reprocess differently later
공통 Both are ways to gather scattered data into one place where it can be analyzed
Why it matters
Dashboards, reports, machine learning models and RAG chatbots all ultimately use data delivered by pipelines. If a pipeline stops or sends through wrong values, every result downstream becomes wrong. That is why, in AI projects, building data pipelines often takes more time than building the model.
Things to know
If the format of a source system changes without notice, pipelines break easily, so monitoring that reports failures is needed. Workflow management tools such as Apache Airflow are often used alongside to manage the order of steps and reruns. It is also good to build data quality checks and personal data handling into the pipeline.
Example
An online store, for example, pulls data from its order DB, its member DB and its ad platforms' performance data every morning before dawn and gathers it into a single data warehouse. Marketers arriving at work check an updated sales dashboard built on that data, and data scientists retrain the recommendation model with the same data. When building an internal document search chatbot, the embedding pipeline that collects documents, chunks and embeds them, and loads them into a vector DB runs on the same principle.