Stop Building a Data Pipeline for Every New Data Source

There is a pattern I have seen repeatedly in data engineering.

Someone asks:

It sounds simple.

Then the real process begins.

We create a repository.
We write some ingestion code.
We create an Airflow DAG.
We configure permissions.
We deploy it.
We monitor it.
And eventually, someone changes the source structure and the pipeline breaks.

A few months later, we have another request.

So we do it again.

And again.

And again.

At some point, I started asking myself a simple question:

Why are we building a new data pipeline every time we need to ingest data?

The traditional approach #

A typical ingestion looks something like this:

New Data Source → Source-specific Code → Repository → Airflow/DAG → BigQuery

This isn’t necessarily a bad architecture.

In fact, it makes perfect sense when the ingestion is complex, highly customized, or requires significant business logic.

The problem appears when most of our ingestions are actually doing fairly simple things:

• Read data from a source
• Understand its structure
• Transform/map it
• Put it into BigQuery
• Run it on a schedule
• Monitor whether it succeeded

Yet we create a new implementation for each one.

The result is predictable.

More repositories.
More DAGs.
More deployments.
More permissions.
More monitoring.
More maintenance.

And, perhaps most importantly, a lot of duplicated code.

The hidden cost of “just one more ingestion” #

Imagine we have ten different data sources.

If each source requires its own implementation, we might end up with something like:

Source A → Repo A → DAG A
Source B → Repo B → DAG B
Source C → Repo C → DAG C
Source D → Repo D → DAG D
...
Source J → Repo J → DAG J

Each individual pipeline may be perfectly manageable.

But now imagine we have 100.

Or 500.

The problem isn’t that any individual pipeline is particularly complicated.

The problem is that we are multiplying the same operational pattern over and over again.

And the engineering effort starts shifting from building useful data capabilities to maintaining plumbing.

What happens when the source changes? #

This is where the model becomes even more expensive.

Suppose an Excel file originally looks like this:

customer_id
customer_name
country
revenue

Everything works.

Then the source owner adds:

customer_segment

Or renames:

customer_name → name

Or changes a data type.

With a traditional implementation, we may need to:

1. Identify the change.
2. Find the relevant repository.
3. Understand the existing code.
4. Modify the ingestion logic.
5. Update the schema.
6. Test it.
7. Deploy it.
8. Monitor the next execution.

None of those steps are inherently difficult.

But multiply that process by hundreds of ingestions and it becomes a significant operational burden.

What if ingestion was configuration instead of code? #

This is the idea behind what I call a Data Ingestion Layer (DIL).

Instead of creating a new codebase for every ingestion, we create the ingestion capability once and make individual ingestions configuration-driven.

The experience could look something like:

DATA INGESTION LAYER

       ┌────────────────────────────────────┐
       │            DIL Dashboard           │
       │                                    │
       │  Source                            │
       │  Credentials                       │
       │  Schema / Mapping                  │
       │  Frequency                         │
       │  APPEND / REPLACE                  │
       │  Documentation                     │
       │  Status / Logs                     │
       └─────────────────┬──────────────────┘
                         │
                         ▼
                ┌─────────────────┐
                │ Ingestion Engine│
                └───────┬─────────┘
                        │
             ┌──────────┴──────────┐
             ▼                     ▼
        AI Assistance          Connectors
        Schema / Mapping       Excel / API
        Change Detection       MCP / Files
             │                     │
             └──────────┬──────────┘
                        ▼
                   ┌──────────┐
                   │ BigQuery │
                   └──────────┘

The important change is not the dashboard itself.

The important change is the abstraction underneath it.

Instead of thinking:

We start thinking:

That is a very different model.

The DIL becomes the single entry point #

A user should be able to open the DIL and select:

Add New Ingestion

Then provide the basic information.

For example:

• Where is the data coming from?
• What type of source is it?
• What credentials are required?
• How frequently should it run?
• Where should the data go?
• Should the data be appended or replaced?
• What is the expected structure?

The platform takes care of the rest.

This also gives us something we don’t have today in a consistent way:

one place to see all our ingestions.

We could see:

• What is being ingested
• Where it comes from
• Where it goes
• How frequently it runs
• Whether it is currently healthy
• When it last ran
• What failed
• What schema it expects
• Who owns it
• Documentation about the source

Instead of searching through repositories and Airflow DAGs, the DIL becomes the obvious starting point.

Where does AI fit? #

This is where I think AI can make the approach particularly interesting.

But I don’t think AI should be responsible for actually running our data pipelines.

The ingestion engine should remain deterministic.

AI should assist with the parts that currently require human interpretation.

For example, when a new source is added, we don’t necessarily need to send the entire dataset to an LLM.

We could provide:

• Column names
• Data types
• Basic metadata
• A small sample of rows — perhaps the first three rows
• Existing schema information, if available

From that, AI could help us understand the structure and suggest:

• BigQuery schema
• Field mappings
• Data types
• Potential transformations
• Schema changes
• Newly introduced fields
• Potentially renamed fields

This is a much more controlled use of AI.

We are not asking:

We are asking:

The platform still owns the actual execution.

Different sources, same problem #

One of the other challenges with ingestion is that sources are not all the same.

Today we might have:

Excel

Excel → Custom Code → DAG → BigQuery

Tomorrow:

API

API → Custom Code → DAG → BigQuery

Then:

MCP

MCP → Custom Code → DAG → BigQuery

Then perhaps a file, database, SaaS application, or some other source.

The implementation details are different, but conceptually the problem is still the same:

The DIL can hide those differences behind reusable connectors.

Data Ingestion Layer

       ┌─────────┐
       │  Excel  │──┐
       └─────────┘  │
       ┌─────────┐  │
       │   API   │──┤
       └─────────┘  │
       ┌─────────┐  │      ┌──────────┐
       │   MCP   │──┼─────►│ BigQuery │
       └─────────┘  │      └──────────┘
       ┌─────────┐  │
       │  Files  │──┤
       └─────────┘  │
       ┌─────────┐  │
       │  Other  │──┘
       └─────────┘

The source changes.

The ingestion platform doesn’t.

This is not about eliminating engineering #

I think this distinction is important.

The goal isn’t:

There will always be sources and use cases that require custom engineering.

The goal is:

If ten pipelines are essentially doing:

Read → Validate → Map → Load → Schedule → Monitor

then perhaps that functionality should exist once as a platform capability rather than ten times as ten separate implementations.

Engineering effort should be spent where engineering actually adds value.

The real architectural shift #

For me, the most interesting part of this idea is not AI.

It is the shift from code-driven ingestion to metadata-driven ingestion.

Today #

New Source
    ↓
Write Code
    ↓
Create Repo
    ↓
Create DAG
    ↓
Deploy
    ↓
Maintain

With DIL #

New Source
    ↓
Configure
    ↓
Run
    ↓
Monitor

The complexity hasn’t magically disappeared.

We’ve simply centralized the complexity instead of recreating it for every source.

That is the key idea.

What happens to our existing repositories? #

They don’t necessarily disappear overnight.

The DIL can be introduced incrementally.

Existing complex pipelines can continue running.

New standard ingestion requirements can start using the DIL.

Over time, as the platform becomes more capable, some existing ingestion implementations can potentially be migrated into it.

This makes the idea much more practical than attempting a big-bang migration.

Does this really make ingestion an “O(1)” problem? #

I initially thought about describing this as O(1), because whether we have 10 ingestions or 1,000, the platform itself doesn’t need to be rebuilt.

Strictly speaking, though, the data processing itself obviously isn’t O(1).

Processing one row and processing a billion rows are very different problems.

What becomes close to constant is the onboarding effort.

The platform already exists.

The connectors already exist.

The orchestration already exists.

The monitoring already exists.

The schema-management capability already exists.

So adding another standard ingestion should largely mean adding another configuration.

And that is the important metric.

We are trying to make the marginal engineering cost of adding an ingestion very small.

From weeks to minutes #

This is ultimately what makes the idea attractive to me.

Today, a seemingly simple ingestion can turn into a multi-day or multi-week engineering exercise once we include development, repositories, permissions, orchestration, testing, deployment and maintenance.

With a mature Data Ingestion Layer, the same request could potentially become:

Perhaps five minutes.

Perhaps ten.

Maybe longer for a more complicated source.

But the fundamental difference is that we aren’t starting another engineering project every time.

Build once. Configure many times. #

I don’t think every data ingestion should become a software project.

If the pattern is generic, we should build the capability once and let people configure it repeatedly.

The Data Ingestion Layer is essentially an attempt to turn ingestion from:

into:

That small change in thinking can have a significant impact as the number of data sources grows.

Fewer repositories.

Fewer duplicated pipelines.

Less maintenance.

Less platform dependency.

Better visibility.

Better documentation.

And, most importantly, engineers can spend their time solving actual data problems rather than repeatedly rebuilding the plumbing required to move the data.

Build the ingestion capability once. Configure it many times. #