What we’ve built so far
(November update)

It’s been about two weeks since I started digging into this problem: developers finding out about a leaked AI API key from the bill, often a day or two too late.

In that time, I’ve helped 3 developers get refunds from OpenAI and Anthropic, and started building the dashboard that would have caught their leaks early. This post is a look at both: what I learned from those cases, and what’s built so far.

What 3 refund cases taught me

Each case was different, but a few patterns showed up every time.

1. Everyone found out from the bill.

None of them caught it while it was happening. Provider billing data usually lags by hours, and billing alerts only tell you how much you’ve spent, not what caused it. By the time they noticed, the damage was done, and they still had to dig through logs to figure out where it came from.

2. The first message to support matters more than you’d think.

When you see a surprise charge, the natural reaction is to panic and contact support right away. But that first message often shapes how the case is handled. The cases that went best were the ones where the person revoked the key first, gathered evidence (usage logs, timestamps, which key was used), and then reached out with a clear, calm explanation.

3. Not every spike is a leak.

Sometimes it’s real growth. Sometimes it’s a retry loop or a bug. Sometimes it’s one user abusing a free tier. Each needs a different response, and treating every spike as an emergency just teaches you to ignore alerts. What people needed wasn’t just “spend went up,” but “here’s what it looks like, and here’s what to do.”

4. Most people deal with it alone, quietly.

Very few people post about this publicly. It’s embarrassing, and most just want to fix it and move on. That also means there’s not much guidance out there when it happens to you.

What’s built so far

Everything below came straight out of those cases.

Free Scan

A one-time check of your usage that spots unusual spend. It’s the quickest way to see if something looks off right now, no account needed.

A dashboard that shows who and what, not just how much

Since billing alerts only tell you the total, the dashboard breaks spend down by user, feature, and model. The goal is to answer “what caused this?” in seconds instead of digging through logs.

An SDK wrapper, not a proxy To see spend per user in real time, Brello needs to sit close to your API calls. But I didn’t want a proxy: that would mean your prompts and keys passing through someone else’s server, and your app going down if Brello does. So it’s a drop-in wrapper that runs inside your app:

• Requests go straight to the provider, and your key never leaves your app
• It collects usage metadata only, never prompts or responses
• Data is sent in the background, so if Brello is down, your app keeps running
• The SDK will be open source, so you can check exactly what it sends

Real-time numbers, checked against billing The SDK tracks usage as it happens, then cross-checks it with provider billing data once that arrives (usually a few hours later) to correct the cost. Until then, numbers are marked as pending, so you always know what’s confirmed and what’s an estimate.

Two kinds of alerts Because not every spike is a leak, alerts come in two levels: “usage looks unusual, take a look” and “this looks like a leaked key, rotate it now,” each with what to do next. You can also send a test alert to make sure it reaches your Slack or Discord, without touching a real key.

What’s next

Closed beta, starting around Nov 20

I’m opening Brello to a small group of teams running AI features in production. Details are in the banner at the top of this page.

What I’ll be watching closely during the beta

How fast alerts actually arrive. “In minutes, not the next day” is the core promise, so I want real numbers, not guesses. Whether an alert is enough to act on. When something spikes, can you tell what happened and what to do just from the alert, without opening five dashboards?

After that

Alerts and a checklist when a model you use gets deprecated or repriced, so a provider change doesn’t break your app or your margins More providers, based on what beta users actually run

Tell me what you need If you run AI features in production, I’d love to hear from you: what would make a tool like this worth adding to your stack? And what would you need to see before trusting it in prod?

Reply on x or email hello@brello.app. I read everything.