A practical guide to data pipelines and governance for AI

Why AI needs data pipelines and governance

Every machine learning model is trained on data and evaluated on data. Datasheets for Datasets paper points out that the nature of a dataset fundamentally shapes a model's behavior. If the real-world deployment environment does not match the training and evaluation data, or if the data contains unwanted social biases, the model will not work properly in practice. The paper warns that in high-stakes areas such as hiring, finance and criminal justice, these mismatches can lead to serious harm.

Generative AI adds another layer to the problem. A Google Cloud article source lists the risks of lacking governance: data trapped in separate systems (silos), AI agent hallucinations caused by inaccurate data, sensitive data leaking in answers to people without permission, and violations of regulations such as GDPR and CCPA. An IBM article source likewise explains that governance now has to cover the structured and unstructured data that serves as input and output for RAG (retrieval-augmented generation) systems, vector databases and AI agents.

So data work for AI splits into two parts. One is the pipeline that actually moves and refines data; the other is governance, which decides who is responsible for that data, how its quality is measured and who can see it. If the first is the road, the second is the traffic rules. The IBM article source compares governance to an "air traffic control tower" that makes sure verified data flows through secure pipelines to trusted users.

Components: collection → cleaning → storage → indexing

The collection stage pulls data from many sources. An AWS article source gives examples such as payment and CRM (customer relationship management) systems, inventory systems, IoT sensors, social media and internal HR systems. The raw data is first placed in a temporary storage area called a staging area (landing zone). There are three ways to pull data: "update notification," where the source system reports changed records; "incremental extraction," which pulls only what changed at regular intervals; and "full extraction," which pulls everything again when changes cannot be detected. AWS recommends using full extraction only for small tables because of the large transfer volume.

The cleaning (transformation) stage removes errors, fills blanks with set values, removes duplicate records and standardizes dates, units and character sets into one format. As in the AWS article's source example, recipe data that mixes kilograms and pounds is converted to a single unit. On top of this come advanced transformations such as "derivation," which uses business rules to compute new values from existing ones.

ETL (Extract, Transform, Load) is the approach that finishes this transformation before data goes into storage. The AWS article source defines ETL as the process of combining data from multiple sources into a central repository called a data warehouse, using business rules to clean the raw data and prepare it for analytics and machine learning. ELT reverses the order: raw data is loaded into storage first, and transformation happens inside the storage system when needed. Because the raw data is kept as is, it is easier to reprocess later for other purposes, but you have to manage the storage so that unorganized data does not pile up.

Pipelines for AI add one more stage: indexing. According to an Anthropic engineering article source, RAG splits documents into chunks of a few hundred tokens or fewer (chunking), uses an embedding model to convert each chunk into a vector that captures its meaning, and stores the vectors in a vector database that can be searched by semantic similarity. When a question comes in, the closest chunks are retrieved, added to the prompt and sent to the model.

Storage and organization: warehouses, lakes, lakehouses and data mesh

A data warehouse is a central repository that holds data organized into tables and columns source. A Databricks article source explains that warehouses have long been used for decision support and BI (business intelligence: reports and dashboards), but are ill-suited or expensive for unstructured and semi-structured data, or for data that is large and changes quickly. A data lake is a repository that holds structured and unstructured data as is, without defining a structure in advance source. Because raw data can be piled up in cheap storage, lakes are used for data science and machine learning, but according to the Databricks article source, they do not support transactions or enforce quality, so without management they turn into a "data swamp."

This is why many organizations have used a two-tier architecture with both a lake and a warehouse: one ETL step from operational databases into the lake, then another ETL step that moves only important data into the warehouse. The Databricks article source points out that this architecture leads to data duplication, extra infrastructure costs, security issues and data that is not up to date (staleness). A lakehouse is a design that aims to merge the two: it adds a metadata layer on top of cheap lake storage (for example, Delta Lake on top of Parquet files) to provide ACID transactions, schema enforcement and "time travel" for querying past versions. Databricks states that time travel and audit logs also help with machine learning reproducibility. However, this is the description of a company that sells lakehouses, so you should assess separately whether it fits your organization's scale and existing systems.

Data mesh is an organizational principle, not a storage technology. Zhamak Dehghani's article source argues that technical progress has solved scale in data volume and compute, but not other dimensions of scale: the growing number of sources and users, the diversity of use cases and the speed of change. As a solution, it puts forward four principles: domain-oriented decentralized data ownership, data as a product, a self-serve data platform and federated computational governance. In other words, the business teams that know the data best take responsibility for it as a "product," while a central function provides common platforms and policies as code.

Design principles: governance and dataset documentation

Governance is a way of managing data according to principles across its entire lifecycle, from collection and loading to analysis and secure disposal source. The IBM article source treats it as part of data management and summarizes it as focusing on quality, security and availability. The components the sources share can be grouped into four. The first is quality. The Google Cloud article source views quality in six dimensions: accuracy, completeness, consistency, timeliness, validity and uniqueness. The second is lineage (the record of where data came from and how it changed) and metadata. The IBM article source lists metadata management, data catalogs, lineage tracking and business glossaries as common features of governance tools.

The third is access permissions. The IBM article source describes a division of labor in which the governance team sets policies for sensitive data such as personally identifiable information, and the management team implements them with role-based access control (RBAC). It also recommends defining a "data scope" so that users and apps can access only the data they need. IBM source and Google Cloud source both note that access design has become more important as AI agents, not just people, increasingly read data. The fourth is the accountability structure. The IBM article source divides roles into a steering committee, data owners (responsible for accuracy and quality in each domain), data stewards (day-to-day management) and stakeholders.

Dataset documentation records governance in a form people can read. Datasheets for Datasets paper proposes that, just as every electronic component comes with a datasheet, every dataset should come with a document describing its motivation, composition, collection process, recommended uses and so on. The goal is to make creators reflect carefully on how the dataset is created, distributed and maintained, and to help users choose the right dataset and avoid unintended misuse. The paper says its list of questions is not a fixed template and should be adapted to the field and organization, and it stresses that documentation should not be automated: automation is convenient, but it takes away the creators' chance to think.

Data Cards paper adapted this idea for large-scale industry settings. A Data Card is a structured summary of key facts across a dataset's lifecycle, capturing context that cannot be inferred from the data itself, such as its sources, collection and labeling methods and intended uses. The researchers explain that they designed it as a "boundary object" read by audiences from many different roles.

Step-by-step application

When starting out, the following order is recommended. First, define the purpose and scope. The storage and transformation approach differ depending on whether the data is for analytics, model training or RAG retrieval. The Anthropic article source states that if a knowledge base is smaller than 200,000 tokens (about 500 pages), it is simpler to put the whole thing in the prompt without RAG, so first check whether you really need RAG. Second, list the sources and assign owners. Write down which systems you pull what from and who is responsible for each dataset. Even if you do not adopt data mesh's domain ownership principle source in full, it is best to decide from the start "who fixes this data."

Third, build collection and cleaning. Choose the method based on whether the source provides update notifications and whether incremental extraction is possible source. Set rules for deduplication, format standardization and missing values, and use the quality dimensions (out of the six in w4) that matter for your use case as metrics. Fourth, choose storage. For mostly structured analytics, consider a warehouse; if keeping raw data and processing it in many ways matters, consider a lake or lakehouse. For training data, check for versioning features so you can find again which version a model was trained on source.

Fifth, if you use RAG, build the index. The Anthropic article source describes combining embedding search with BM25, which finds exact word matches, and merging the results. This is because embeddings capture meaning well but can miss strings that must match exactly, such as error codes like 'TS-999'. To reduce the problem of chunks losing document context, it also proposes Contextual Retrieval, which uses a model to generate a 50–100 token context sentence for each chunk describing its place in the whole document and prepends it to the chunk. Anthropic states that this method reduced retrieval failures by 49%, and by 67% when combined with a reranker. Sixth, add access controls and documentation. Apply role-based access control to sensitive data, and write a Datasheet or Data Card for each dataset. Finally, put audits and improvements on a regular schedule source.

Checklist

At the ingestion stage, check that each source has an assigned owner, that the extraction method (change notifications, incremental, or full) suits the nature of the source, and that you have decided how much of the raw data in the staging area to keep for tracing problems source.

At the cleaning and storage stage, check whether the rules for deduplication and format standardization are documented, whether you are actually measuring quality metrics (accuracy, completeness, consistency, timeliness, validity, uniqueness, w4), whether copying the same data into several stores is producing conflicting values source, and whether past versions can be queried again.

At the indexing and RAG stage, check that you are not relying on semantic search alone, that you have separately tested exact-match queries such as error codes and product numbers, and that chunks have not lost their document context source. At the governance stage, check that sensitive data has role-based access controls, that AI agents cannot read data beyond the user's permissions and expose it in answers sourcesource, that lineage and catalogs are up to date, and that each dataset has a document describing its motivation, composition, collection process and recommended uses paper.

Common mistakes and limitations

The most common mistake is to keep bolting on more pipelines. Dehghani's article source describes a scene familiar to many organizations: ETL jobs that keep failing as they try to connect operational and analytical data, and a maze of pipelines that grows ever more complex. The Databricks article source likewise points out that a two-tier architecture passing through multiple ETL steps requires constant maintenance and leaves data stale. Before adding stages, first check whether you are moving the same data twice.

The second mistake is using the lake as a place to "just put everything for now." A lake that does not enforce quality and has inconsistent permissions becomes a data swamp source. The more an approach loads raw data first, as ELT does, the more metadata and permission management need to go along with it.

The third is turning documentation into form-filling. The Datasheets researchers paper refined their questions over about two years, rewriting them so product teams would not answer with just yes or no, and found that legal and ethical questions were answered better when placed at each stage of the lifecycle rather than grouped separately. In the Data Cards paper case studies, too, the writing process surfaced opportunities to improve data design, such as prompting teams to examine why there were so many "unknown" values. Documentation is valuable when it is a tool for drawing out thinking.

There are also limitations. Governance frameworks differ by organization, so there is no single right answer, and older frameworks become outdated quickly source. Many of the web sources are written by companies introducing their own platforms, so use their feature descriptions as reference but validate product choices separately in your own environment. Also, the Contextual Retrieval figures are values Anthropic reported from its own experiments source.

Further study

If you are getting the concepts for the first time, learn the basic terms of extract, transform and load from AWS's explanation of ETL source, then read Databricks' lakehouse article source to see what problems the warehouse, lake and lakehouse each emerged to solve. If organizational structure is your concern, read Dehghani's article on data mesh principles source. It focuses on establishing a common language rather than prescribing tools, so it is a good read before choosing technology.

For governance, the IBM article source breaks down roles, policies, auditing and tools most systematically, and the Google Cloud article source briefly summarizes the risks of the AI era and six dimensions of quality. For dataset documentation, start with the list of questions in Datasheets for Datasets paper, and if you want to know about experience applying it in a large organization, continue with Data Cards paper. If you are building a RAG document pipeline yourself, Anthropic's Contextual Retrieval article source shows chunking, embedding, BM25 and reranking as a single flow.