You Can't Regression-Test a Language Model
The Data Report: Weekly market signals on modern data platform shifts | Week ending July 6, 2026
The 30-second version
The month’s advice was “ground your analyst agent better.” One team’s answer this week: stop grounding, take the model off the decision path, and let deterministic rules make the calls the model just narrates. That’s what makes a recommendation reproducible enough to regression-test.
dltHub put a number on the build-vs-buy connector question: about $5 to scaffold, about $100 a year to keep alive. If it holds, the maintenance premium you’ve been buying your way out of mostly evaporated. They sell the alternative, so price it against your own bill.
In the Radar: capping agent query load at the database, the where-do-your-metrics-live question, plain Markdown over vector search, and an AI-written release that caught its own data-loss bug.
The Analyst Agent That Isn’t Allowed to Decide
You did what the last two months of advice told you to do. You grounded your analyst agent: certified models underneath, a semantic layer in front, the language model pointed at clean, governed inputs. And the answers still come back fluent, confident, and impossible to reproduce. Ask the same question twice and you can get two different recommendations, which means you can’t put the thing under a regression test, and the people relying on it can feel the ground move.
The team behind a build-log that surfaced this week hit that exact wall and decided the problem was the premise, not the tuning. They were building an agentic business-intelligence layer for ops triage, and their first design did the obvious thing: it put the language model in charge of the decision. What came back were recommendations that were often wrong, and worse, irreproducible, which broke both testing and user trust at once. So they rebuilt it inside out. Now deterministic, named rules make the calls, with thresholds, fixtures, full execution traces, and regression tests, and the model is demoted to the two things it’s actually good at: assembling context and writing the human-facing explanation. Roughly a 60/40 split, rules to model, laid out in a post on Dataconomy.
What makes this worth your attention isn’t the specific split. It’s that it rejects the premise the rest of the month has been operating on. Every other agentic-analytics story crossing the wire keeps the model on the decision path and tries to fence it in with better inputs: certified models, semantic-layer grounding, observability wrappers. This one says that for anything that has to be reproducible and testable, grounding is the wrong lever. You don’t make a guess auditable by handing it better source material. You replace the guess with a rule.
That reframes the design question you actually face when you scope one of these. It stops being “how do I constrain the model” and becomes “which decisions belong to rules at all, and which genuinely need judgment no rule can encode.” Draw that line deliberately and the reproducible half turns into code you can test and version like anything else, while the model only touches the parts where its irreproducibility is a price you’ve chosen on purpose. The honest caveat is that this is one team’s build-log with no independent voice behind it yet, so weigh it as an architecture to consider, not a proven pattern.1
Bottom line: The teams that drew the rules-versus-model line before shipping their analyst agent this week can hand you a regression test for every call it makes. The ones who grounded the model harder and hoped are still explaining why yesterday’s answer doesn’t reproduce today.
A Connector Now Costs About $100 a Year
Somewhere in your stack there’s a row-metered ingestion bill, Fivetran or Airbyte, and a renewal date coming. You’ve run the build-vs-buy math before, and it always came out the same way: connectors are cheap to build and expensive to maintain, so you buy. dltHub’s argument this week is that the second half of that sentence stopped being true, and it brought numbers.
In a post aimed squarely at the buyer, dltHub claims a language model can scaffold a working dlt connector in under two hours for about $5 in tokens, and that its open-source run telemetry, a 99.83% success rate and roughly 0.12% non-transient errors, works out to about two maintenance incidents in five years. Price that at engineer time and you land near $100 a year per connector. If the numbers hold, the line item you’ve been renewing specifically to avoid maintenance is guarding against a cost that mostly went away.
The interesting part isn’t the $100. It’s what moves once maintenance stops being the expensive half. Owning the connector code on compute-based billing takes the row-metered overage risk off the table, so a source tripling in volume doesn’t triple a bill, and it removes the lock-in, and it lets an analyst author and review a pipeline instead of filing a ticket and waiting. dltHub points to one migration that saw a 182x cost reduction and a 10x speed-up on a single high-volume sync after moving off a default connector, which is the kind of number you hold up against your own worst line item.
Here’s the part to keep hold of: these are dltHub’s own numbers, and dltHub sells the alternative.2 The telemetry is real but self-reported, and there’s no third-party cost comparison behind it yet. That doesn’t make the argument wrong. It makes it a claim you can check cheaply, because the whole point of a $5, two-hour experiment is that you don’t have to take the vendor’s word for the $100.
Bottom line: The teams that pulled one high-volume source’s overage bill this week and priced a self-owned connector against it walked into the renewal with an actual number. The ones renewing on last year’s build-vs-buy verdict are paying a maintenance premium that may no longer exist.
The Radar
🛢️ If you run Postgres under agent load. PlanetScale’s Database Traffic Control lets you cap CPU, connection concurrency, and per-query time by query fingerprint, app, or comment tag, so one runaway agent workload can’t starve your transactional path. If you’re pointing AI or agent traffic at the same Postgres that serves your app, this is the knob to evaluate before the contention shows up in production.
🧩 If you’re deciding where your metrics live. Snowflake is pitching define-once semantics in the warehouse, consumed in Power BI, with a Semantic View Autopilot that auto-converts existing Power BI models so you stop maintaining metric logic in two places. Worth a look if you’re tired of reconciling definitions across tools, but confirm the portability story (an open interchange standard, still early) is real before you let the warehouse own the semantics outright.
📚 If you’re building agent access to internal knowledge. A widely-debated essay, Knowledge Should Not Be Gated, argues you should reach for plain, versioned Markdown before a vector database: keep raw sources immutable, let the model generate a wiki over them, and save retrieval-augmented search for corpora too big to read directly. The useful question it forces is whether your internal-knowledge stack actually needs embeddings, or picked them up by reflex.
🔧 If you’re weighing AI in the maintenance loop. Simon Willison shipped sqlite-utils 4.0rc2, mostly written by Claude Fable for about $150, and the agent’s review caught a silent data-loss bug in delete_where() that a human pass had missed. Two takeaways: review the new auto-commit-per-write transaction model before you upgrade, and file the costed data point in the “can I trust agent review” debate (the thread itself stayed skeptical).
🚚 If you’re migrating off Synapse. Microsoft shipped an AI-assisted command-line tool that runs a read-only assessment of your Synapse Spark and pipeline workloads and surfaces migration blockers before you commit to a Fabric move. A clean Tuesday action if that migration is on your roadmap: run the assessment, read the blocker list, and decide with data instead of a vendor estimate.
When you scoped your analyst or business-intelligence agent, which calls did you hand to deterministic rules and which did you leave to the model, and where has that line turned out to be in the wrong place? Reply and tell me where you drew it.
Published by RepublicOfData.io. Curated by Olivier Dupuis.
The source is a single practitioner build-log (published on Dataconomy, surfaced via Google Alerts) with no community discussion and no independent team reporting the same architecture yet. Treat the 60/40 split and the “rules decide, model explains” design as one team’s considered decision, not a benchmarked or corroborated pattern. What travels regardless of the specifics is the reframe: decide which calls belong to deterministic rules before you decide how to ground the model.
The cost figures (about $5 to scaffold, ~$100/year upkeep, the 99.83% success rate, the 182x/10x migration numbers) are dltHub’s own telemetry and case data, published on the dltHub blog, and dltHub sells the managed runners and patterns that are the alternative to buying a connector. There’s no third-party cost comparison behind the numbers yet. They’re a claim to test against your own bill, not a verified benchmark. The self-owned-connector experiment is cheap enough ($5, two hours) that you can generate your own number rather than adopt theirs.