Killed a $500K
vendor.
Built it better.
Replaced Optimizely, a quarter of total IT budget, with an in-house, API-first platform that did what the vendor couldn’t: connect experiments directly to our data lake.
From 1–2 experiments per month to tens per month, post-launch. The compounding flywheel of a self-service platform built right.
A quarter of total IT budget reclaimed. Platform pays for itself on maintenance and infra alone.
Gemini was paying Optimizely roughly $500K a year, about a quarter of total IT budget, for a platform doing the basics of experimentation: feature flags, A/B splits, a UI to spin up tests.
The fatal limitation wasn’t price. It was that the vendor lived outside our data perimeter. Exposure events couldn’t join cleanly to our Snowflake tables. Real impact analysis, the part that turns experiments into compounding learning, was effectively out of reach.
So we were paying premium price for the cheapest half of the value: running tests. The expensive half, understanding them, was on us to build anyway.
I made the case to build it ourselves. The economics were obvious: even modest savings against a $500K contract paid back the team in year one. The product case was the harder argument, and the more important one.
The thesis: an in-house platform, API-first, would do everything Optimizely did at parity uptime, and add the one thing it structurally couldn’t: exposure events flowing directly into our data lake, joined to every other source we already had in Snowflake.
Discipline was the whole strategy. 4 domains, no scope creep. Features, experiments, attributes, audiences. A clean OpenAPI spec for each. Anything outside those four was a no.
- Feature flags, A/B splits, UI
- Vendor-side analytics, walled off from our data
- Exposure events stranded outside Snowflake
- 1–2 experiments per month, ceiling enforced by analysis cost
- 1/4 of total IT budget, every year
- Same flags, splits, audiences, plus 4 clean API domains
- Exposure logging into Kinesis → Snowflake, joined to everything
- Looker dashboards on real exposed-user cohorts
- Tens of experiments per month, self-service
- Pays for itself in year one, compounds after
Replace
Stand up the four API domains and the portal. Hit feature parity with what teams were already using on Optimizely. Match uptime. Cut nothing.
Exceed
Wire exposure logging from the frontends into Kinesis, on into Snowflake. Looker on top. Every experiment now joins back to every other source of truth we own.
Scale
Self-service rollout. Teams spin up experiments without PM or infra bottlenecks. Velocity compounds. The platform earns its keep on capability, not just savings.
Experiment velocity moved from 1–2 a month under the vendor to tens a month on the in-house platform. The unlock wasn’t just speed, it was that every test now produced clean, joinable, cohort-level analysis sitting in Snowflake the moment it concluded.
The economics held. The $500K annual line item became a maintenance and infra cost, a fraction of that and still falling on a per-experiment basis as throughput went up. A quarter of IT budget, returned to the business.
Strategically, this is a clean case study of the build vs. buy decision. The rule: build when both impact and total cost of ownership beat the buy alternative. Both held here: capability the vendor structurally couldn’t deliver, at a TCO well below the contract it replaced.