AI glossary · Advanced · Data engineering
CDC
CDC (Change Data Capture) is a technique that picks out only the data that has been newly added, modified or deleted in a database and sends it straight to other systems. Instead of copying the entire dataset every time, it lets you follow only 'what changed.'
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Data, ops & platforms
- Fields
- Data engineering
- Added
- 2026-10-09
How it works
Most databases keep a log that records every change in order (the transaction log) in case of failure. Log-based CDC reads this log and turns it into change events such as 'this row was added, this value changed, this row was deleted,' then sends them out. The events usually pass through a message queue or event stream and are delivered to several places, such as a data warehouse, a search index or a cache. When the log cannot be used, other approaches include periodically querying a 'last modified time' column or setting up triggers, but these tend to miss deletions or put load on the source DB.
Why it matters
To reflect the contents of an operational database quickly in analytics, search or AI systems, copying everything every time is slow and costly. With CDC, only the changed parts are moved, so the load on the source DB is small and analytics data follows in near real time instead of lagging by hours. The same principle is used in RAG systems to re-embed only the modified parts of a document and update the vector DB when the document changes. In the end, it acts as the link that 'lets multiple systems see the same up-to-date data.'
Things to know
When the structure (schema) of a source table changes, the receiving side must also be adjusted, so agreements such as data contracts are needed. Events can arrive duplicated or out of order, so the receiving side must be designed so that processing the same event twice gives the same result. When first connecting, you need to copy the existing data in full once (a snapshot) and then append the changes made after that. Changes containing personal information flow through as they are, so de-identification and access control should be considered at the delivery stage.
Comparison of CDC methods
Log-basedRead the transaction log
- Low load on the source DB
- Captures everything, including deletions
- Requires DB-specific setup
Query-basedPeriodically query a modified-time column
- Simple to set up
- Easily misses deletions
- Frequent queries load the DB
Trigger-basedRun a trigger on each change
- Captures changes right inside the DB
- Burdens write performance
공통 All aim to move 'only what changed'
Example
If you attach CDC to an online store's order database, every time a customer places an order or a shipping status changes, that change flows into an event stream. These events are used to update the sales dashboard in the data warehouse in near real time, while also keeping the product search index and the data referenced by the customer service chatbot up to date. Open-source connectors and cloud data replication services are widely used for this purpose.