Connecting AI agents to data sources has become one of the most discussed capabilities in performance marketing over the past year. The pattern is consistent across the teams we speak to, which is: connect the agent to the tools it needs, point it at the data, and let it answer questions directly. For paid search and paid social teams that usually means the Google Ads or Meta APIs, a BigQuery warehouse, or both.

The opportunity is real. An agent that can answer reporting questions on demand removes a large amount of manual analysis. What very few teams have measured is whether the answers are correct at any meaningful depth. Asking an agent what you spent last week is a different problem from asking what your CPA was by campaign type, whether you were pacing against your monthly target, or which segment drove a week-on-week drop. The questions that actually drive decisions are harder than simple lookups, and there is a wide gap between an agent that can attempt an answer and one that returns the right number.

Most teams have treated reliability as a model question. The assumption is that a more capable model handles complexity better, reasons more accurately, and makes fewer mistakes, so attention has gone to model selection rather than data preparation.

We think the more important variable sits earlier in the chain. Context engineering is the practice of curating what an agent can access: which tables it can query, which metric definitions it has been given, which business rules it knows about, and what fallback strategies exist for known gaps in the source data. Our working hypothesis was that well-structured data produces more reliable outputs than a more capable model working from poorly prepared data. To test it, we ran a structured study against a live Google Ads account using the same data access patterns we deploy for clients.

Two points before the results.

We tested four access patterns that represent a real progression from raw to structured. These are not arbitrary configurations. They map to the path most teams actually follow when building AI on top of performance data, from the quickest setup to the most invested.

We asked 18 questions across four tiers of increasing complexity.

Each source was tested in a fresh agent session, with identical question wording across all four. Every source ran twice to separate systematic failures from random variance. An answer was counted as correct if it fell within 1% of a reference value taken from a Tableau report currently treated as the source of truth. Every failure was tagged with a root cause.

The results form two distinct clusters rather than a gradient. The Google Ads MCP sat in the mid-thirties on both runs. The modelled sources occupied a different band entirely. Within that upper band, the semantic layer added a further step, most visible at Tier 4.

* BigQuery Raw excluded after Run 1.

 
We dropped BigQuery Raw after the first run. It scored one question higher than the Google Ads API across the 18 questions, required more tool calls than any other source, ran up significant BigQuery job cost from processing unnecessary data, and in Tier 4 returned conversion rates above 100%. Without semantic guidance to filter to the right conversion action, the agent took whatever it could find and produced numbers that cannot exist. A confident wrong answer is harder to catch than a declared gap, because nothing signals that anything went wrong. The cost-to-signal ratio did not justify a second run.
 
A note on the BigQuery Curated variance. The 28-point jump from Run 1 to Run 2 is partly an artefact. One question had ambiguous wording that we corrected between runs, and one returned different granularity from run to run because of non-deterministic query behaviour. The true reliability of BigQuery Curated sits somewhere inside that range, not consistently at either end. As we explain below, that instability is itself the finding.

The Google Ads API returns conversions based on whatever is marked as primary in the account. The problem is that the conversion action a team actually reports on is often a custom action, a secondary conversion, or one that aggregates across attribution paths that only exist in the modelled data layer. In our account, the API returned roughly 521 conversions. The figure in our curated layer, combining online-attributed and offline-attributed paths, was 665.

Every metric with conversions in the formula is wrong as a result. CPA was inflated by 28%. That is a magnitude error, and most practitioners would catch it.

Returning the wrong number is one thing. Returning the wrong direction of a week-on-week shift is worse, because it feeds directly into recommended actions. Spend gets reallocated away from what is working. Campaigns get flagged for review that do not need it. A real performance drop goes unaddressed while the team responds to a trend that does not exist.

The instability compounds the problem. The agent chose a different query strategy on each run. Run 1 pulled primary conversions only. Run 2 attempted to sum all conversions using data-driven attribution fractional splits. Neither matched the reference. With no anchor defining which conversion action to use or how it is attributed, the agent guesses differently each time.

It is not all downside. The Google Ads MCP approach is reliable on attribution-independent metrics: spend, impressions, clicks, and CPC. For any account where the conversion action used in reporting differs from what the platform surfaces by default, that is its reliable ceiling.

Client Semantics outperformed BigQuery Curated because three specific data engineering decisions were written into the configuration file that the curated layer alone could not make consistently. A simplified version looks like this.

metrics:
  conversion_rate:
    formula: "conversions / clicks"
  cpa:
    formula: "cost / conversions"
segments:
  campaign_type:
    values: [Brand, Generic, Competitor Type A, Competitor Type B, PMax]
pacing:
  strategy: bq_derived
  fallback: derive from MAX(daily_budget) per campaign_type in performance table

Pacing against target. The curated datasets returned no answer on both runs because the account was absent from where the agent assumed it would find the information. The bq_derived fallback in the config bypassed this entirely, deriving the pacing figure from the performance table instead. Same underlying data, completely different outcome.

Highest-CPA campaign sub-type. The curated approach collapsed two sub-types of a campaign segment into a single aggregated row in Run 1, masking the correct answer. In Run 2, by chance, the agent ran a schema inspection step first and surfaced the distinct values. Client Semantics returned the correct answer on both runs because the segment definitions in the config prevent that collapse.

Lowest conversion rate by campaign type. Here the curated version returned a different wrong answer each run. Client Semantics returned the correct answer both times because an automated campaign type is named as a distinct segment and the formula is anchored explicitly.

None of these are reasoning failures. The curated tables contained the data needed to answer all three correctly. The errors came from how the agent interpreted that data, not from the data itself.

BigQuery Curated’s two-run average of 75% looks acceptable in isolation. The problem is what drives the variance.

On Tier 4 questions, whether the agent gets the right answer depends on whether it runs a schema inspection step at the start of the session. With that step, it surfaces the distinct campaign sub-type values and queries correctly. Without it, it collapses them into aggregated rows and returns the wrong answer. Same data, same model, different result, with no way to predict which one you get unless you explicitly tell the agent to inspect the schema in the prompt.

One of the most notable findings of the study was that the data was right, the modelling was right, but the output was still unreliable.

For a monitoring or reporting use case, non-deterministic accuracy is worse than a consistently lower score. A consistent score is predictable. You know what the agent can and cannot answer, and you build around it. Non-determinism means the same question might be answered correctly on Tuesday and incorrectly on Thursday, with no signal that anything has changed.

Client Semantics returned the same CVR figure across both full runs. The dashboard showed a different number. The config defined CVR using one formula; the reference dashboard used a different but equally valid definition of the same metric, applied to a different stage of the funnel. The agent followed the config, correctly and consistently.

The semantic layer did not produce a wrong answer. It produced a consistent answer to a question where two valid metric definitions coexisted and had never been reconciled. Without the config, different runs returned different CVR figures depending on which formula the agent happened to choose. With the config, the divergence was stable and traceable to a specific piece of business logic defined in the YAML.

This is one of the more useful things a semantic layer can do: it surfaces metric definition disagreements that already exist rather than hiding them in variance. The fix is straightforward, because two calculations measuring two different things should not share the same metric name.

When evaluating AI capabilities it is easy to overlook the operational realities of running these models at scale. A poorly connected agent does not just produce bad reports and questionable recommendations. It also carries significant infrastructure overhead.

Token inflation. The Google Ads API approach required 24 tool calls and multiple retries. In an agentic loop, every retry re-sends large payloads of history back into the model’s context window. More tool calls scale up inference costs directly and create more opportunities for compounding logic errors.

Warehouse compute overhead. BigQuery Raw ran up significant job costs from processing entirely unnecessary rows and columns. Because the agent had no semantic guide telling it where to look, it executed broad data sweeps just to answer basic questions.

The latency penalty. A 12-minute wait for a single reporting lookup is functionally useless in production. Injecting a 400-line YAML config up front lets the agent map the exact columns it needs on the first call, bypassing the loop that slows down raw database setups and dropping latency to a production-grade 3 minutes.

A modern data stack reduces the friction here significantly. The semantic layer in this study is a YAML config file that defines metric formulas, names segment values, and specifies fallback strategies for known data gaps. It sits alongside dbt models in a Git repository, which means every change to a metric definition or a segment value goes through the same review process as a change to the data models themselves.

For a modern data team, that is the same workflow they already use for the rest of their infrastructure. The decisions encoded in the config are the same decisions required when building a curated reporting layer for a dashboard. Context engineering is not a new discipline bolted on top of data work. It is the same data work applied to a new endpoint.

The teams that struggle with AI reporting reliability are, in most cases, the teams that skipped the modelling step. A team that has invested in modelling its data correctly, defining its metrics explicitly, and maintaining those definitions in version control already has most of what an AI reporting agent needs to work reliably.

If you are building or evaluating an AI reporting agent today, the minimum viable setup for anything beyond simple spend and traffic metrics is a modelled data layer with explicit metric formulas, named segment values, and fallback strategies for known data gaps. Direct API access alone will not get you there, and that holds whether the channel is paid search, paid social, or any other reporting domain.

This is the work we do for clients: building the modelled data layers, metric definitions, and semantic configuration that make agentic reporting trustworthy rather than merely impressive in a demo.

If you are weighing up an AI reporting agent on top of your performance data, we are happy to walk through what the setup actually requires. Get in touch.