Hacker Remix

Firebase bill is usually $50, but I was surprised to see a $70k bill in one day

115 points by plainOldText 2 days ago | 158 comments

ChrisMarshallNY 2 days ago

Maybe this has something to do with it?

https://x.com/tamarajtran/status/1867342095033258466

There's a really good reason that I don't link to my apps on this site (or most, for that matter). They are free apps, running on a shoestring budget.

But that whole account looks a little dodgy. The posts make it seem like one of those "I made $30K/mo, from my living room! You can too!" outfits.

danpalmer 2 days ago

People always underestimate how easy it is to sell $70k for $30k.

tasn 2 days ago

As long as you don't literally sell dollar bills, it can actually be quite hard.

danpalmer 1 day ago

And this is why so many in the industry mistake selling $70k for $30k as "product market fit".

andrewmcwatters 2 days ago

I've had the very nice pleasure of always doing business with providers who would simply drop access to the server when bandwidth had been met or exceeded with my service.

But what's odd to me is that I don't often read about other people having the same experience. So... are all of you seriously with hosts that don't shut off access to your servers?

I'd rather that happen than suddenly get a bill I can't afford.

hattmall 2 days ago

Run your cloud contracts (or anything possible to incur liability) through a separate LLC so if there are billing issues you can just move on.

The structure should look like:

"My App LLC" has contract with "My Host LLC" for hosting services. "My Host LLC" provides those services via a cloud provider. If "My Host LLC" racks up a 2 million dollar bill with the cloud provider and goes out of business then I just move to "New Host LLC" and carry on.

It's not as if the patients of a Doctor who fails to pay his office rent would become liable.

This is the entire purpose of LLCs.

standeven 2 days ago

LLC liability protections are not absolute. I wouldn’t be surprised if this would fall under gross negligence or callous indifference or some other legal umbrella that voids the liability protection. I also wouldn’t be surprised if there’s some fine print in the account agreement that creates a personal guarantee.

tessierashpool9 2 days ago

> callous indifference

i like this one. the ultimate law to subjugate everybody once and for all!

jahewson 2 days ago

That’s not going to fly.

Firstly, because cloud contracts generally prohibit renting out the cloud services as-is - you can build a product on top of it but not act as a reseller of their platform.

Secondly, and perhaps most importantly because setting up separate LLCs with the goal of avoiding lawful debts is the demonstrates fraudulent intent and will get your LLCs veil-pierced.

cozzyd 2 days ago

What you mean I can't set up an LLC for every large expense I want to avoid paying?

reverendsteveii 2 days ago

You could, but anything you buy as that LLC would actually belong to the LLC and be subject to seizure and sale when the LLC doesn't pay its bills. That's my thousand-foot understanding of it, at least. If what you said worked people would do it all the time.

disqard 1 day ago

You're doing it wrong.

You need to be a billionaire, and then you need to threaten to sue anyone who dares to ask you to pay.

hdjjhhvvhga 2 days ago

> cloud contracts generally prohibit renting out the cloud services as-is

Doesn't have to be as-is - a simple infra on top of it will do

> setting up separate LLCs with the goal of avoiding lawful debts

If normal bills are paid as usual and only this kind of crap is not paid, Amazon would have a hard time proving in court that the whole point of the daughter LLC was to avoid paying bills.

fargle 2 days ago

ah, yes. the "Shady Roofing Contractor" business model.

"don't worry, all the work has warranted by New Quality Roofing Services IV, LLC"

although, there's very little about that structure that's unique to an LLC. it can be done the same with a corporation.

i'm certain this only works when the amount of debt being ripped off (ahem in dispute) is not a lot more than the cost to sic a really good law firm on you.

feoren 2 days ago

I'm not a lawyer but this sure sounds illegal. Piercing the veil?

mrtksn 2 days ago

In the days before, there used to be a hard spending limit - I know this because of an old official tutorial video on youtube. Last time I checked, there was billing alert functionality and they were recommending creating a function that will halt your service when it hits the limit - the problem is, billing lags behind and you can rack up a hefty sum until anything is triggered.

Amazing service but its a scary one. Every time a social-media-worthy accident happens, they cancel the bill but I wonder how many are burned without recourse.

Maybe if your rack up something modest like 5K accidentally, its better to push it as high as you can and get it declined on your CC and increase your social media prospects :)

jlcummings 2 days ago

It’s weird how normalized over cap billing became acceptable simply because chargeable metrics were not collected/resolved until after the fact. Seems like an obvious gap in the process or a little bit shady.

cozzyd 2 days ago

If you owe Google $70k, you have a problem. If you owe Google $70T, Google has a problem.

iambateman 2 days ago

The idea that an app will experience stratospheric growth so suddenly that costs must be allowed to grow 1,000x hour-over-hour has always been insane to me.

Why does Firebase insist on hanging the sword of Damocles over all of its customers? I’ve read so many stories before and experienced these fears the first time I was setting up Firebase…this has been going on for years

chuckadams 2 days ago

That's every cloud provider. At this point, I think they're actively conspiring to not implement billing caps.

jjnoakes 2 days ago

I use fly.io. I pre-paid for credits and if they run out, things shut down.

No affiliation, just happy to use a provider with actual caps.

danpalmer 2 days ago

I used to think this until I tried architecting out how you'd build a billing cap. I recommend it as a design exercise. It's easy to build a bad billing cap that would slow down services and cause outages, but it's basically impossible to build a good billing cap.

ryao 2 days ago

Oddly, they have no problem shutting things off when the limits of their free plan are exceeded:

https://firebase.google.com/docs/projects/billing/firebase-p...

I am not sure how they can do that, but cannot let people set their own limits on their paid plans.

chen_dev 2 days ago

Limits Reached -> PubSub Notification -> Shutdown Sequence.

Because it's a free plan, the delay between 'limits reached' and actual shutdown only incurs the cost of providing the service during that brief period, not the potential liability of overcharging that might exist on a paid plan.

ffsm8 2 days ago

Is that really a problem though? Just don't bill beyond the cap then and leave the last few requests free, too.

Or write a disclaimer that the billing cap doesn't necessarily cut off at exactly that amount and that there might be an overcharge.

I am pretty sure most people would be okay with either of these options, we didn't need a perfect system, just one that works well enough

pclmulqdq 2 days ago

That cutoff is rarely truly a hard cutoff. The limits are often too low to have a natural test of that, though.

ryao 2 days ago

They could always make the amount over that is given due to their cutoff enforcement being less than perfect free, as it likely already is on the free plan. That would avoid the risk of unbounded bills associated with going on a paid plan.

pclmulqdq 2 days ago

Most of the cloud providers have a less-than-perfect cutoff. It's worse than the cutoff of the free plan, though, because the free plan can be slowed down to have better enforcement, while the commercial plans have performance SLAs to hit.

maeil 2 days ago

> cause outages

That's fine. The major LLM providers work like this. If you're out of credit, or hit your monthly recharge limit, it stops working, bringing down prod with it if your product relies on it. Not heard anyone complain about the concept.

If it's really a problem for you, you can be all enterprisey and contact sales, then they'll be very excited to offer you extremely high limits and post billing.

This way everyone gets what they wants.

danpalmer 1 day ago

To be clear, the outages I'm referring to are not when you hit your billing cap. Try designing a billing system for a cloud provider that implements caps, while still retaining the performance necessary for the services you're providing to make sense, and without introducing huge, common, failure modes.

Havoc 2 days ago

You solve this by opt in, not fancy engineering. There are two classes of customers - those that absolutely can't afford services be cut, and those that absolutely can't afford a 50k bill.

So you deploy an advanced technology known as a radio button to toggle which they want, throw a bunch of ToS & consent agreements about data loss / deletion at the ones opting for hardcaps....and done.

Also reminder that Azure has hard caps for certain account types. This is not a technical problem. They can do this, they just don't want to.

hackingonempty 2 days ago

How is the service being able to answer the question "is there budget available for this action?" different from "is there authorization for this action?"

williamstein 2 days ago

One example - Google Cloud network egress charges aren’t known until up to ~2 days after they happen. Since they can be obscenely expensive (eg $0.23/GB), they can make budget computation difficult.

stefanfisk 2 days ago

What is the cause of this delay?

danpalmer 1 day ago

I don't know for sure, but based on my knowledge as a user, I'd guess it could be something like delivering usage logs from points of presence in the CDN. PoPs can go offline regularly, they're highly dependent on other people's networks, 2 days might be the arbitrary line that has been drawn that gets enough of them in most circumstances, while not being too annoying for customers.

danpalmer 2 days ago

Authorisation is much more cacheable than a value that inherently changes every single time you check it.

Also authorisation revocation is relatively uncommon, which means you can have a fast-path for approval, and then push only the revoked key IDs to just frontend servers.

iterateoften 2 days ago

Look at how insane twilio is.

I set up automatic recharge of $20. A small amount because not much traffic. A bad actor got ahold of our api that didn’t have rate limit yet and started spamming Africa.

Twilio had zero issue charging my credit card every second. Literally I was getting a hundred emails and bank notifications a minute. Brex didn’t stop anything.

Twilio responded that it was my fault. Yeah. I sure 100% probably should have put in that cloudflare rate limit first. But…

How easy would it be for twilio to prevent this on any level? I need rate limits? How about you rate limit credit card charges. Putting $20 recharge limit should mean $20/day or $20/hr not literal unmetered right to charge as much as possible in 20 increments.

Twilio support sent me all this info about protecting myself from African spammers who use the technique to make money from SMS charges. You know what’s more responsible than informing me of this? How about blocking sending sms to country codes known for this from the get-go and optin to send to them.

it was clear the perverse incentives that encourage twilio to massively benefit from being insecure and easily exploitable by spammers.

Ended up costing almost $3k after bill adjustment when our usual spend was $5/mo. not bankruptcy level so after fighting with support just took it as is and learned my lesson. But twilio made *50 years* of revenue in about 10minutes from their own negligence.

nothercastle 2 days ago

It’s probably part of the business model. They rely on the African spammers to improve profits

tasuki 1 day ago

I use digital ocean (a cloud provider) droplets. I know exactly what my bill will be at the end of the month.

yadaeno 2 days ago

Almost all GCP services have quota by default that you need to manually increase.

itake 2 days ago

Imagine if LinkedIn wanted to use Firebase to launch a new product.

I could easily imagine a 1,000x hour over hour growth as the social media grows.

If I was LinkedIn, I’d be very upset if Firebase pulled the cord when everyone was looking at the new launch.

collingreen 2 days ago

Would you be upset if they pulled the cord at the limit you explicitly set beforehad? If not then that's not really the thing the people asking for a billing cap are talking about.

itake 2 days ago

What does "pulled the cord" mean?

I pay for storage. Does this mean they delete my production database? They delete my s3 buckets? They delete my server logs? My message queue?

Then yes! I would be extremely upset. How do you expect the cloud provider to magically know what is safe to "pull the plug" and what is not safe?

like_any_other 2 days ago

Why do you pick unreasonable interpretations, when there are so many trivial, obvious, reasonable options? Even the literal interpretation of "pull the cord", meaning the power cord, would mean the servers shut down, but nothing is deleted. And why do you claim the provider would "magically" have to know these things? These can be communicated through user settings: "if bill exceeds X, do one of the following:

[ ] continue billing me [ ] throttle bandwidth [ ] gracefully shut down servers (data is deleted after 30 days of non-payment)

This is so painfully obvious even for someone that never deals with cloud vendors, that I just don't understand why you would pretend otherwise.

stvltvs 2 days ago

More like make you servers publicly inaccessible rather than delete them. Or throttle bandwidth. Or provide a warning when the cap is approaching. Etc.

jlcummings 2 days ago

Exactly. Why the fixation on one strategy for handling this not so uncommon scenario. It is so common that handling it should be defacto.

This isn’t a pre-paid gas pump use, but that could be one way to present it. We all want to fill as fast as possible. And if your fill spout can handle top rates, you get top fill rates, until you close in on the hard limit. Then it trickles down to the metered drop. Then stops precisely where it needs to.

By accepting/requesting a hard cap, the provider can make clear that in order to be precise, soft caps will go into affect earlier and induce progressive throttling where applicable. If the throttle doesn’t catch the final milliliter or two of gasoline, before the pump shuts off, the provider can and should just let it go. It’s a loss, but comparatively a figurative drop in the bucket.

The other obvious route is predictive where prior usage guide the guardrails. Ordering two eggs is typical for a single meal. Ordering twelve is not. Ordering three or four is unusual for most but if you are a regular diner your habits will be observable.

Any of this predicated on the provider to want to do something. They seem to lack incentives at this point for making it easy. It is stories like op that I avoid well known problematic providers like Firebase who don’t respect and foster long term relationships.