AI glossary · Basic · Data engineering
Stream processing
Stream processing is a way of handling data that continuously receives data the moment it is generated, computes on it right away and produces results. It contrasts with batch processing, which collects data and processes it all at once later.
About this term
- Level
- Basic · Technical terms you often meet in AI news
- Category
- Data, ops & platforms
- Fields
- Data engineering
- Added
- 2026-10-09
How it works
When events such as clicks, payments, sensor readings or logs occur, they go into a message broker (a system that queues and delivers events). A stream processing engine keeps reading this flow and immediately performs filtering, transformation and calculations such as sums and averages. To compute over a flow that never ends, a common approach is to define time intervals (windows) such as 'the last few minutes' and aggregate within them. The results are sent straight to dashboards, alerts, databases or other services.
Why it matters
It is essential for tasks that lose value if they are a few seconds late, such as fraud detection, real-time recommendations and equipment anomaly detection. In AI services, it is used to turn users' latest actions into features and feed them to a model right away, or to monitor LLM response logs in real time. Receiving database change events emitted through CDC and updating other systems immediately is another typical use of stream processing. The key point is that it shifts data from 'stored, then examined' to 'examined while flowing.'
Things to know
Events often arrive late or out of order because of network delays and similar issues, so you need to decide in advance how to handle late data. The design must save and recover state so that the same event is not counted twice or missed even when a failure occurs. Because the system must always be running, its operational burden and cost tend to be higher than batch processing. That is why a mixed setup is common: only the parts that truly need real-time processing use streams, and the rest use batches.
How fast is data processed?
- Nightly batch
- Short-interval batch
- Stream processing
Example
A card company sends an event to its stream processing engine every time a payment occurs and immediately calculates whether payments on the same card are piling up from unfamiliar locations within a short recent period. If it sees an unusual pattern, it holds the payment, together with the score from an anomaly detection model, and sends the customer an alert. The same data is also stored separately and used overnight by batch processing to produce monthly reports.