The test is the interesting part. The write-up afterwards is not. How a CRO agency lets Claude draft it from data and screenshots, and refuses without real numbers.
October 2026
6 min read
You can automate A/B test reports with AI if your test data already lives in one place. At CRO agency Lyfto a consultant ticks a checkbox in Airtable, a script sends hypothesis, KPIs, conversions, uplift, significance and up to four screenshots to Claude, and the conclusion, learnings and advice are written back. No measured data, no report.
Yes, AI A/B test analysis works, as long as the AI writes the report and people run the test. At Lyfto, a Dutch CRO agency, a consultant ticks one checkbox on an experiment in Airtable. A script sends the hypothesis, KPIs, users, conversions, uplift, significance and up to four screenshots of control and variation to Claude. Claude writes a conclusion, learnings and advice back into the record, in the client's language. If the test has no measured data yet, it refuses to write anything.
That's the whole idea. Here is how it's built, and what sits around it.
A CRO agency lives on experiments. Every finished A/B test needs a report for the client: what we tested, what happened, why we think it happened, what to do next. The thinking is valuable. Typing it out for the twentieth time is not.
At Lyfto those reports were written by hand. The data was already structured, in one Airtable base with one record per experiment. That's the condition that makes this work. If your numbers live in five dashboards and a spreadsheet, start there, not with AI.
There is no Make or Zapier in between. It's an Airtable automation with a script, calling the Claude API directly. Fewer moving parts, and the data passes through fewer places.
When the checkbox is ticked, the script reads the experiment record:
The screenshots go along as images, so Claude can see what changed and use that in its explanation. A small technical note if you build this yourself: send them as base64, not as attachment links. Airtable attachment URLs can't be fetched by the API, and you get an error instead of a report.
Claude returns structured JSON, the script writes each part into its own field, sets the AI status to done and unticks the box. If anything fails, the status goes to error and the box unticks too, so nobody waits for a report that isn't coming.
The output is only as good as the template you give it. This is the A/B test report template that Lyfto's instructions ask for:
The style rules are written in too: numbers first, interpretation second, no marketing fluff, no AI clichés, talk like a CRO specialist talks to a client. The agency's own rules for when a variation counts as a winner are in there as well. That matters. You don't want the model inventing its own idea of significance.
This is my favourite line in the script. Before anything goes to Claude, it checks that there are users in both control and variation. If not, the test isn't finished, and the script stops.
Why bother? Because a language model will happily write a confident conclusion about a test that never ran. Someone ticks the box too early, and you get a polished report built on nothing. A report a client reads should never come from empty fields. So: no data, no write-up, status set to error. And even with good data, I'd always have a consultant read the report before a client does.
Client data in an AI model is a fair question, so here is the setup:
The report is the visible part. Around it I built the plumbing that keeps the agency's lead engine and client communication running:
None of this is AI. That's the point: CRO reporting automation only helps if the data and the communication around it are already in order. You can see more builds like this on our work page.
The pattern is "structured record plus screenshots in, written report out". It works for any agency or consultancy that reports on projects with numbers.
Want to try the Airtable Claude idea on something smaller first? I wrote a practical guide to automating one recurring task with AI.
The hard part isn't calling Claude. It's deciding what goes in the record, writing the template, and making the script say no when the data isn't there. If your team writes the same kind of report over and over, let's talk for 30 minutes.
Yes, if you give it the measured numbers and the context: hypothesis, KPIs, users, conversions, uplift and significance, ideally with screenshots of control and variation. Let it write the conclusion and advice, and give it your own rules for what counts as a winner. A person should still check every report.
Learnings per KPI with the hard numbers, a verdict (winner, loser or inconclusive), a conclusion with possible explanations, and concrete advice that starts with whether to implement the variation. Skip any KPI you didn't measure.
Use an Airtable automation with a "Run a script" action that calls the Claude API, with the API key stored as an Airtable secret. Trigger it with a checkbox, send the record's fields (and images as base64), and write the response back into fields.
Use the commercial Claude API, where data you send is not used to train models. Keep the key in a secret store, avoid extra tools in the chain, and never paste client data into a consumer chat app.
Partly. The data structure and the template are the real work, and those are decisions, not code. The script itself is small, but someone needs to build in the checks, like refusing to run without measured data.