It looks like virtually each week for the final two years since ChatGPT launched, new giant language fashions (LLMs) from rival labs or from OpenAI itself have been launched. Enterprises are arduous pressed to maintain up with the huge tempo of change, not to mention perceive the right way to adapt to it — which of those new fashions ought to they undertake, if any, to energy their workflows and the customized AI brokers they're constructing to hold them out?
Assist has arrived: AI purposes observability startup Raindrop has launched Experiments, a brand new analytics function that the corporate describes as the primary A/B testing suite designed particularly for enterprise AI brokers — permitting firms to see and examine how updating brokers to new underlying fashions, or altering their directions and power entry, will impression their efficiency with actual finish customers.
The discharge extends Raindrop’s present observability instruments, giving builders and groups a solution to see how their brokers behave and evolve in real-world situations.
With Experiments, groups can observe how modifications — reminiscent of a brand new software, immediate, mannequin replace, or full pipeline refactor — have an effect on AI efficiency throughout thousands and thousands of person interactions. The brand new function is accessible now for customers on Raindrop’s Professional subscription plan ($350 month-to-month) at raindrop.ai.
A Information-Pushed Lens on Agent Growth
Raindrop co-founder and chief know-how officer Ben Hylak famous in a product announcement video (above) that Experiments helps groups see “how literally anything changed,” together with software utilization, person intents, and difficulty charges, and to discover variations by demographic components reminiscent of language. The aim is to make mannequin iteration extra clear and measurable.
The Experiments interface presents outcomes visually, exhibiting when an experiment performs higher or worse than its baseline. Will increase in unfavorable alerts would possibly point out greater job failure or partial code output, whereas enhancements in optimistic alerts might mirror extra full responses or higher person experiences.
By making this knowledge straightforward to interpret, Raindrop encourages AI groups to strategy agent iteration with the identical rigor as fashionable software program deployment—monitoring outcomes, sharing insights, and addressing regressions earlier than they compound.
Background: From AI Observability to Experimentation
Raindrop’s launch of Experiments builds on the corporate’s basis as one of many first AI-native observability platforms, designed to assist enterprises monitor and perceive how their generative AI techniques behave in manufacturing.
As VentureBeat reported earlier this 12 months, the corporate — initially often known as Daybreak AI — emerged to deal with what Hylak, a former Apple human interface designer, known as the “black box problem” of AI efficiency, serving to groups catch failures “as they happen and explain to enterprises what went wrong and why."
At the time, Hylak described how “AI products fail constantly—in ways both hilarious and terrifying,” noting that in contrast to conventional software program, which throws clear exceptions, “AI products fail silently.” Raindrop’s unique platform targeted on detecting these silent failures by analyzing alerts reminiscent of person suggestions, job failures, refusals, and different conversational anomalies throughout thousands and thousands of each day occasions.
The corporate’s co-founders— Hylak, Alexis Gauba, and Zubin Singh Koticha — constructed Raindrop after encountering firsthand the issue of debugging AI techniques in manufacturing.
“We started by building AI products, not infrastructure,” Hylak informed VentureBeat. “But pretty quickly, we saw that to grow anything serious, we needed tooling to understand AI behavior—and that tooling didn’t exist.”
With Experiments, Raindrop extends that very same mission from detecting failures to measuring enhancements. The brand new software transforms observability knowledge into actionable comparisons, letting enterprises take a look at whether or not modifications to their fashions, prompts, or pipelines really make their AI brokers higher—or simply totally different.
Fixing the “Evals Pass, Agents Fail” Downside
Conventional analysis frameworks, whereas helpful for benchmarking, not often seize the unpredictable habits of AI brokers working in dynamic environments.
As Raindrop co-founder Alexis Gauba defined in her LinkedIn announcement, “Traditional evals don’t really answer this question. They’re great unit tests, but you can’t predict your user’s actions and your agent is running for hours, calling hundreds of tools.”
Gauba mentioned the corporate constantly heard a standard frustration from groups: “Evals pass, agents fail.”
Experiments is supposed to shut that hole by exhibiting what really modifications when builders ship updates to their techniques.
The software permits side-by-side comparisons of fashions, instruments, intents, or properties, surfacing measurable variations in habits and efficiency.
Designed for Actual-World AI Habits
Within the announcement video, Raindrop described Experiments as a solution to “compare anything and measure how your agent’s behavior actually changed in production across millions of real interactions.”
The platform helps customers spot points reminiscent of job failure spikes, forgetting, or new instruments that set off surprising errors.
It may also be utilized in reverse — ranging from a recognized drawback, reminiscent of an “agent stuck in a loop,” and tracing again to which mannequin, software, or flag is driving it.
From there, builders can dive into detailed traces to seek out the foundation trigger and ship a repair rapidly.
Every experiment gives a visible breakdown of metrics like software utilization frequency, error charges, dialog length, and response size.
Customers can click on on any comparability to entry the underlying occasion knowledge, giving them a transparent view of how agent habits modified over time. Shared hyperlinks make it straightforward to collaborate with teammates or report findings.
Integration, Scalability, and Accuracy
In response to Hylak, Experiments integrates immediately with “the feature flag platforms companies know and love (like Statsig!)” and is designed to work seamlessly with present telemetry and analytics pipelines.
For firms with out these integrations, it will possibly nonetheless examine efficiency over time—reminiscent of yesterday versus as we speak—with out further setup.
Hylak mentioned groups usually want round 2,000 customers per day to supply statistically significant outcomes.
To make sure the accuracy of comparisons, Experiments screens for pattern dimension adequacy and alerts customers if a take a look at lacks sufficient knowledge to attract legitimate conclusions.
“We obsess over making sure metrics like Task Failure and User Frustration are metrics that you’d wake up an on-call engineer for,” Hylak defined. He added that groups can drill into the precise conversations or occasions that drive these metrics, making certain transparency behind each combination quantity.
Safety and Information Safety
Raindrop operates as a cloud-hosted platform but in addition affords on-premise personally identifiable info (PII) redaction for enterprises that want further management.
Hylak mentioned the corporate is SOC 2 compliant and has launched a PII Guard function that makes use of AI to routinely take away delicate info from saved knowledge. “We take protecting customer data very seriously,” he emphasised.
Pricing and Plans
Experiments is a part of Raindrop’s Professional plan, which prices $350 per 30 days or $0.0007 per interplay. The Professional tier additionally contains deep analysis instruments, matter clustering, customized difficulty monitoring, and semantic search capabilities.
Raindrop’s Starter plan — $65 per 30 days or $0.001 per interplay — affords core analytics together with difficulty detection, person suggestions alerts, Slack alerts, and person monitoring. Each plans include a 14-day free trial.
Bigger organizations can go for an Enterprise plan with customized pricing and superior options like SSO login, customized alerts, integrations, edge-PII redaction, and precedence help.
Steady Enchancment for AI Programs
With Experiments, Raindrop positions itself on the intersection of AI analytics and software program observability. Its deal with “measure truth,” as said within the product video, displays a broader push throughout the business towards accountability and transparency in AI operations.
Somewhat than relying solely on offline benchmarks, Raindrop’s strategy emphasizes actual person knowledge and contextual understanding. The corporate hopes this can permit AI builders to maneuver sooner, establish root causes sooner, and ship better-performing fashions with confidence.
