Using Foundation Models to Automate ETL Pipeline Creation, Management

Foundation models, particularly large language models (LLMs), are transforming how data engineering tasks are automated across domains. This paper explores the use of LLMs to automate the creation, management, and optimization of Extract-Transform-Load (ETL) pipelines in a domain-agnostic manner. We provide conceptual frameworks and practical strategies for integrating foundation models into the ETL lifecycle, and we highlight use cases where such models (via platforms like Amazon Bedrock) generate pipeline code, enhance data transformation quality, and adapt pipeline execution. Through a review of related work and an illustrative implementation, we demonstrate that LLMs can translate natural language specifications into pipeline workflows, assist in data harmonization and quality control, and optimize pipeline performance with minimal human intervention. We discuss evaluation metrics for these LLM-driven pipelines and examine the challenges of reliability, scalability, and governance. The results indicate that foundational models offer a promising avenue to drastically reduce development time and improve the robustness of data pipelines, suggesting a paradigm shift in AI-assisted data engineering.