Lak Lakshmanan, the author of the article, will deliver a keynote at the "Data Engineering in the Age of AI" conference on November 13, 2025. He discusses why agentic artificial intelligence (AI) is shaking up data engineering and how the work of data engineers is changing. This text expands on his ideas and explains the technological shifts driving this transformation. Imagine that data is no longer just raw numbers in tables, but something AI can directly understand and use. Lak Lakshmanan describes how the way people work with data is changing. In the past, software engineers and data scientists built applications and dashboards for non-technical users. Those users then merely used them passively, following fixed procedures. Data engineers supplied data in precise formats for these applications.
Now non-technical users are gaining direct access to data. They can "vibe code" their own tools tailored precisely to their needs. Applications are moving beyond a simple chat window—they embed natural language using frameworks such as CopilotKit. Developers are integrating AI agents into applications, where each backend API serves as a tool for the agent. Another change concerns the center of gravity. In the past, data was enormous, so computation was moved to the data to avoid transferring large volumes. Now frontier AI models (such as large language models, LLMs) are the new center. Applications are built around them. Instead of moving their own computation to the data, AI applications use LLMs as an engine for understanding user intent, reasoning about the task, and calling tools to perform actions. All of this is transforming the work of data engineers. Lak Lakshmanan outlines five principles for preparing data for AI.
1. Rethink ETL/ELT
Today, data engineers devote a great deal of effort to normalizing data, creating clean schemas, and building transformation pipelines. The goal is to make the data understandable to analysts and downstream applications. This does not mean that ETL/ELT (extract, transform, load) is becoming irrelevant—data availability remains critical. But agents can interpret schemas, understand relationships, and work with data in different formats without extensive preprocessing.
Lak Lakshmanan warns against simply adding a data catalog and MCP servers on top of existing tables—this underestimates the capabilities of agentic technology and makes the work of AI agents more difficult. AI agents understand data in context. They do not need everything normalized into rigid schemas in advance. As the number of tables grows, today’s agents struggle to interpret them correctly and write SQL joins. Moreover, as data is divided into more narrow slices, the risk of conflicts and ambiguity increases. For example, a "loan amount" column in one table may mean the amount requested by the borrower, while in another it may mean the principal amount disbursed by the lender. The more data is manipulated, normalized, and expanded, the more context is lost.
Maintain workflows for data availability, but question every normalization step. Can an agent understand the data with sufficient context instead of requiring a transformation? Instead of a single number for the principal amount, it could be a paragraph from the original term sheet or financing memorandum explaining the disbursement tranches. Do not limit AI agents to unstructured data—it is easy to apply RAG (retrieval-augmented generation) to PDFs, emails, and so on, but the truly actionable data in an organization is often structured.
2. Prioritize data curation over collection
In-context learning makes curation more important than collection. In the era of big data, the goal was to collect as much as possible because machine learning required enormous datasets, where more data meant better models.
But AI agents are built on in-context learning, where you include one or two examples in the prompt. An LLM can imitate them, whether they demonstrate a process (Chain of Thought) or a format/style (Few-shot Learning). With the advent of in-context learning, the quality of examples matters more than volume. The data you show an agent as an example shapes how it understands similar data. You may create an "Example Store" and select examples for specific types of user queries. As curation becomes more important, data engineers must build tools for: identifying the highest-quality data and examples that are complete, accurate, and representative; regularly updating these examples as standards change; verifying that curated data serves as effective examples for agent learning.
3. Build agent-ready infrastructure
AI agents need infrastructure that supports two fundamental capabilities: perceiving data and acting on it. Not all data formats are equally accessible to agents based on language models. Consider how easily an agent can parse, understand, and extract meaning from your formats. Formats that preserve semantic meaning and require minimal preprocessing reduce friction.
AI agents act by calling tools—functions, APIs, and services that allow them to operate on data. The infrastructure must make these tools discoverable and usable by agents. This means clear interfaces, good documentation, and reliable execution. Examine data access patterns and tools from an AI agent’s perspective. What would an autonomous system need to know to use them effectively? Where is the friction preventing smooth operation?
4. Manage agent artifacts as first-class data
AI agents not only consume data, but also produce it. Lak Lakshmanan notes that AI-generated content will begin to significantly outweigh "raw" data in systems. When an agent generates output, makes decisions, writes code, or logs its reasoning process, that becomes data.
Whether content is created by people, collected from software systems, or generated by AI agents, it must meet common industry expectations and regulations. Beyond regulatory compliance, this data has value for debugging, auditing, training future agents, and understanding system behavior. Treat agent artifacts with the same care as other data: store outputs systematically; preserve decision logs and reasoning traces; manage generated code as versioned artifacts; make this data accessible for analysis and future training. These artifacts become part of the data ecosystem. Design storage and access patterns accordingly.
5. Connect observability with training
The fastest path to better agents is to close the loop between observability and training. AI agent infrastructure needs bidirectional pipelines that connect model performance and observability back to continuous training.
First, you need an observability platform that tracks data quality metrics and detects data drift (changes in the characteristics of input data) and concept drift (changes in the relationship between input and output). At the same time, it monitors key model performance indicators such as accuracy, latency, and hallucination rate. Set up automated triggers at predefined thresholds. The observability platform should also incorporate human feedback. Every correction users make to generated content is logged and used to improve the AI models.
Next, you need a retraining pipeline that is activated by a monitoring trigger. It must be fully automated, retrieve the latest version of the curated training data, launch a model retraining or fine-tuning job, and run the new model through a comprehensive suite of evaluations and regression tests. Building this closed-loop system, which directly connects performance monitoring with automated deployment, is a key task for ML/data engineers in the agentic era.
