A guide to real-time data pipelines: streaming, CDC and event-driven architecture

What it is and why it is needed: batch vs. stream

Data pipelines (flows that move data from a source to a destination while processing it) have long defaulted to batch processing. According to an AWS explainer source, batch processing periodically handles data collected over a set period or a fixed volume all at once. It suits deep analysis that scans the full dataset multiple times, but latency is measured in minutes, hours or days. Stream processing takes in continuously arriving data and updates analysis results incrementally each time a record arrives, aiming for latency in seconds or milliseconds.

This difference matters because the value of data often drops over time. The AWS article source gives the example of recommending restaurants based on a user's current location. Location data loses its meaning if it is not used right away. The official Kafka introduction source also lists real-time payment processing, vehicle and shipment tracking, factory sensor analysis and patient monitoring as typical uses of event streaming.

You do not have to choose only one. The AWS article source explains that many organizations use a hybrid model with both a real-time layer and a batch layer. Streams produce the results needed immediately, while the same data is stored and later analyzed in depth in batch.

Components: event log, processing engine, CDC

A real-time pipeline consists of three main parts. The first is the event log. According to the official Kafka introduction source, Kafka is a platform that combines writing and reading events, storing events durably for as long as needed, and processing them as they occur or later. An event consists of a key, a value and a timestamp, and is stored in a topic (a grouping similar to a folder). Unlike traditional messaging systems, events are not deleted after consumption; they are discarded only after the retention period set for each topic. Topics are split into multiple partitions, and events with the same key (e.g. a customer ID) go to the same partition and are read in the order they were written. In production, a replication factor of 3, meaning three copies of the data, is said to be common.

The second is the stream processing engine. The Flink documentation source treats stateful and timely stream processing as its core concepts. It offers several levels of abstraction, from the lowest-level Process Function to the DataStream API, the table-centric Table API, and the highest-level SQL. According to a Databricks article source, Spark's Structured Streaming lets you use the same structured APIs from batch for streaming, so batch jobs can be turned into streaming jobs with few changes. The Google Cloud Dataflow documentation source explains how windows, watermarks and triggers are handled with the Apache Beam SDK. Kafka itself also has the Kafka Streams API, which supports aggregations, joins and windows source.

The third is CDC (Change Data Capture). A Confluent article source defines CDC as the process of tracking every change in a data source and reflecting it in destination systems. According to the Debezium documentation source, Debezium is a set of source connectors for Kafka Connect that uses database-specific features to read changes and send them to Kafka. Kafka Connect has hundreds of community-built connectors, so you usually do not need to build your own source.

Design principles: how to handle time and events

The first problem you run into in stream design is time. The Dataflow documentation source points out that there is no guarantee data will arrive in time order or at predictable intervals. So endlessly incoming data is split into windows (time intervals) for aggregation. There are tumbling windows, which are fixed, non-overlapping intervals; hopping windows, which start at regular intervals and overlap (suited to moving averages); and session windows, which are split by gaps in activity. A watermark is the threshold at which all data for a window is considered to have arrived. Data that arrives after this threshold is handled separately as "late data." Triggers determine when aggregated results are emitted, and distinguish between event time and processing time.

The second principle is decoupling producers and consumers. The official Kafka introduction source states that fully separating the two so that producers never wait for consumers is a key design element behind its high scalability. The AWS article source also describes the three-tier structure of producers, brokers and consumers, adding that consumers can write processed results back to the broker to create new streams.

The third is being clear about what "event-driven" means. Martin Fowler's article source splits it into four patterns. Event notification only announces that something changed; coupling is low, but the flow spanning multiple notifications is not visible in the code. Event-carried state transfer includes the changed data in the event so the receiver can maintain its own copy. It keeps working even if the source system goes down, but copies multiply. Event sourcing records every state change as an event so the state can be restored at any time by replaying them. Git's commit history is a typical example. CQRS uses different data structures for reads and writes. Fowler source warns that conflating these patterns leads to misdiagnosing the causes of failures.

Where it is used in AI services

Real-time features: recommendation and anomaly detection models become more accurate when they use "what just happened" as input. The AWS article source explains that stream consumers perform filtering, aggregation, pattern matching and machine learning. If click and order events are aggregated by window (e.g. a moving average over a recent interval, w7) and continuously written to a feature store, model serving can make predictions with the latest features.

RAG index updates: The Confluent article source cites propagating INSERTs, UPDATEs and DELETEs from operational databases to caches, search indexes, data warehouses and data lakes as a basic use of CDC. The vector databases used by RAG are a kind of search index, so the same structure applies. If changes to a document table are received via CDC and only the changed documents are re-chunked and re-embedded, there is no need to periodically re-index everything. Log-based CDC in particular also captures deletes source, which helps prevent deleted documents from remaining in search results.

Monitoring: The AWS article source explains that server logs preserve time order, making them useful for spotting anomalies by watching changes over time. If model request and response logs are streamed as events and error rates or latency are aggregated per window, you can see signs of data drift or outages much sooner than with batch reports.

Step-by-step adoption

For a first adoption, the following order is recommended. First, pick one result that truly needs to be real-time. If a task can tolerate a delay of minutes or hours, batch is enough source. The AWS article source also explains that companies usually start with simple uses such as collecting system logs and computing rolling minimums and maximums, then gradually expand to more sophisticated real-time processing.

Second, set up the event log and design topics and keys. Order is guaranteed within the same partition only if the unit where order matters (customer or vehicle ID) is used as the key source. Decide retention periods and replication settings at this point as well.

Third, connect operational database changes via CDC. Debezium source can take an initial snapshot of the current state when a connector first starts if older transaction logs have already been deleted, and can also run incremental snapshots during operation. Apply filters so you receive only the schemas, tables and columns you need, and mask sensitive columns.

Fourth, choose a processing engine. If you already use Spark batch, the shortest path is to prototype batch jobs with Structured Streaming and then switch them to streaming source. If you need fine-grained control over state management and time handling, use Flink's DataStream API or Process Function; if you want to build quickly and declaratively, use the Table API or SQL source. Fifth, decide on window, watermark and late-data policies, and connect the results to destinations (feature store, vector DB, warehouse). Finally, monitor latency and errors. Most Debezium connectors can be monitored via JMX source.

Checklist

At the setup stage, check the following. Is the unit that requires ordering set as the event key source? Is each topic's retention period long enough for reprocessing and failure recovery source? Is replication configured so data survives if one broker goes down source? Check whether CDC is log-based and whether it captures deletes and previous values sourcesource.

At the processing stage, check whether you have decided to aggregate by event time or processing time source. Also check whether there are watermark and late-data policies, and whether the processing engine's state is restored consistently after a failure source. Since data may arrive in mixed formats or with parts missing, check that validation logic is in place source.

At the operations and security stage, check whether columns containing personal information are masked or excluded at the CDC stage source, and whether connector and consumer lag is being monitored source. Also confirm that changes from rolled-back transactions are not wrongly applied to destinations source.

Common mistakes and limitations

First, using events like commands. Fowler source calls dressing up a message as an event while expecting the receiver to take a specific action a "passive-aggressive command." When event notifications are chained across multiple steps, the overall flow is not visible in any code and can only be understood by observing the running system.

Second, mixing up patterns. Fowler source points out that projects described as "event sourcing was a disaster" may actually have had problems with CQRS or excessive asynchronous communication. He also notes that CQRS adds complexity by keeping separate read and write models, so many of his colleagues are wary of it. Event sourcing makes it hard to replay events that involve external systems, and requires dealing with event schemas that change over time.

Third, choosing the wrong CDC method. According to the Confluent article source, the timestamp column method is simple but cannot capture actual DELETEs and requires changing the source schema. The trigger method captures every change but adds an extra write for each write, degrading source database performance. The log-based method puts little load on the source, but transaction log formats differ by vendor and may change in later versions. Rolled-back changes also have to be filtered out.

Fourth, thinking about scaling too late. The Dataflow documentation source notes that streaming jobs that do not use Streaming Engine cannot scale beyond the maximum number of workers initially specified. Each processing engine has operational constraints like this, so check the documentation before adopting one.

Further study

If you are new, start with the official Kafka introduction source. It briefly covers the concepts of events, topics, partitions and replication, which form the basis for understanding the other articles. Then check the batch vs. stream comparison table and the producer–broker–consumer structure in the AWS explainer source.

If you plan to connect operational databases, compare the pros and cons of the three CDC methods with Confluent's CDC article source, and look at the snapshot, filter and masking options in the Debezium features documentation source. For time handling, the explanations of windows, watermarks and triggers in the Dataflow documentation source are easy to follow with their diagrams.

When choosing a processing engine, read the Flink concepts documentation source alongside the Databricks introduction to Structured Streaming source. Once you reach the system design stage, use Martin Fowler's article source to pin down which of the four patterns your team means by "event-driven."