AI glossary · Beginner · Data engineering
Batch processing
Batch processing is a method of collecting data over a period of time and processing it all at once at a set time. Totaling a day's sales every night is a typical example.
About this term
- Level
- Beginner · Core ideas for first-time readers
- Category
- Basics
- Fields
- Data engineering
- Added
- 2026-10-09
How it works
Data within a defined range, such as order records or log files accumulated over a day, is collected. When the set time arrives (for example, early every morning), a scheduler starts the job. The processing engine reads the collected data all at once, cleans, transforms and aggregates it, and saves the results to a data warehouse or files. When the data is large, it is split across multiple computers and processed simultaneously to save time.
Steps of a batch job
- 1Collect dataOrder records, log files
- 2Start at a set timeRun by scheduler
- 3Process all at onceClean, transform, aggregate
- 4Save resultsData warehouse, files
Why it matters
Because it processes large volumes of data at once, it is often more efficient and cheaper than doing the same work continuously in small units. Its simple structure makes it easy to build and operate, and recovery is straightforward since a failed job can simply be rerun. Monthly reports, settlements and filling a data warehouse through ETL are mostly handled in batches. In AI, it is also widely used for large jobs that don't need an immediate response, such as cleaning training data, embedding large numbers of documents and evaluating models.
Things to know
Results reflect the time of the last run, so data created in between is not included. That makes it unsuitable for fraud detection or real-time recommendations that must respond within seconds; stream processing is used for those. When jobs depend on each other's order, dependencies must be managed so that later jobs don't use bad data when an earlier job fails. Some teams shorten the run interval with 'micro-batches' to balance timeliness and efficiency.
Batch processing vs stream processing
Batch processingCollect, then process all at once
- Efficient for large volumes
- Simple structure, easy to rerun
- Data after the last run not reflected
Stream processingAs soon as data arrives
- Responds within seconds
- Handling late-arriving data is tricky
- Must run constantly, adding operational burden
공통 Both are processing methods for data pipelines
Example
An online store runs a batch job early every morning that collects the previous day's order, return and visit logs, totals sales and inventory, and updates dashboards in time for the start of the workday. The same batch job also organizes each customer's purchase history to prepare features for the recommendation system. LLM service companies also often use batch processing to collect and handle large volumes of document summarization or classification that don't need immediate answers.