Why do people use Amazon S3 when Backblaze B2 is 1/4 the cost of S3 and also includes a CDN for free. You also get way faster access speeds with Backblaze vs Amazon since they tier their IO speeds.
Unless you're a bootstrapped startup with just a couple people, paying the AWS bill is not something the engineer probably thinks about too much. Setting up a new billing account with another company is just enough friction to just use whatever AWS offers and call it a day.
Also, most employees aren't really incentivized to reduce or minimize infrastructure expenses.
I think a big reason is that people are using the rest of the amazon ecosystem. If your costs aren't primarily storage, you might be willing to pay a premium to use something that integrates nicely with other services you're using. Here's an article[0] that does some other comparisons between providers and mentions things like upload speed and security features.
> Last I checked, Backblaze still stores most data in 1 location, no?
Backblaze now has multiple regions! One in Europe (Netherlands) and one is called "US-West". Quietly the US-West is actually three separate data centers, but your data will only really land in 1 datacenter somewhere in US-West based on a few internal factors.
To be absolutely clear, if you only upload and store and pay for 1 copy of your Backblaze B2 data, it is living in one region. To get a copy in two locations you have to pay twice as much and take some actions. So if this kind of redundancy is important to you for mission critical reasons Backblaze B2 would only be half as expensive as one copy in Amazon S3, not 1/4 as expensive.
In the one copy in one region in Backblaze B2, any file is "sharded" across 20 different servers in 20 different racks in 20 different locations inside that datacenter. This helps insulate against failures like if one rack loses power (like if a power strip goes bad or a circuit breaker blows). But if a meteor hits that 1 datacenter and wipes out all of the equipment in a 1 mile blast radius, you won't be getting that data back unless you have a backup somewhere else.
I've combined cloudflare workers with backblaze to implement etags, signed URLs, etc. Backblaze is part of CF's bandwidth alliance so your bandwidth fee is zero. This makes for a very low monthly cost
Hi, I haven't had time to write up about this, however, I have dumped the majority of the related code here for you and others who are interested in this solution: https://gist.github.com/chocolatkey/a7ef0364e357629e9875521d.... That should help you get started. It includes HMACSHA256 shared secret URL signatures based on IP, expiry, and optional path scope restriction, caching, ETAGs, sentry error reports, access to non-B2 data from a server w/ basic auth, and more... URLs look like this: https://example.com/delivery/UNIQUE_ID/p-001.jpg?token=16fb4... . My B2 bucket is public, however the requested path is also hmac'd with a secret known only to the CF worker to derive the path of the resources in the bucket. It is optimized for my use case of serving EPUB data. I do not guarantee it to be free of flaws, but it's worked well so far.
I am fancying the idea to move our CDN from AWS stack to B2 + CF, thanks to Bandwidth Alliance. There's at least one thing stopping me: for simple schema of hosting static content out of bucket we should deploy Workers just for URL rewriting. Guys from CF recommending that way and not URL rewrites by simple rules[1]. But it puts us in a weak position of raising cost twice: for increased edge trafic AND for increased number of requests.
I looked at Wasabi some time ago, but their pricing is a LOT less simple than their headline says it is.
The major caveats are hidden away in their pricing FAQ: they charge a 1TB minimum if you use less, and there's a 90 days minimum retention period, meaning if you update a file a few times you will pay for the full 90 days of every intermediate version. Additionally, they reserve the right to make you pay for egress if it looks like you transfer more than you have stored.
So all in all, Wasabi might be the right fit for you if you store >1TB of files that are infrequently updated and get less than 1 download/month on average. If you fit that use case, I think their free egress pricing is awesome, but it's definitely not for everyone.
Wasabi does not allow you to use unlimited bandwidth. Your egress is supposed to stay close to your total ingress. So if you are uploading assets that will be access more than a few times in the first month, I think you will be out of spec for wasabi.
Because of vendor lock in. When you move a lot of data between S3 and EC2 it costs nothing (or very cheap). When you move data outside of AWS, there is extra cost, so it might not even be cheaper overall.
I am strongly considering B2 as an option for a dropbox-style system. Something where I run 8-16TB of hot tier on my local LAN, with B2 serving as the slower mass storage tier behind it. It seems that the average B2 access latencies would be ~100-200ms, which is very tolerable for a cache miss on such a massive tier of storage. With this amount of space available you could have pre-fetch rules that do things like pull down entire directories as files within them are accessed.
1. Scale - S3 is big - really really big! You don’t need to care if you store one KB or several petabytes.
2. Tiers: the default on S3 is several way replicated storage with 11 9s of durability with high availability. However you can select from cheaper options with the trade off you are happy with.
3. Cost: S3 has reduced prices several times, you can be reasonably sure your costs will go down over time on per unit basis.
I too use the serverless image handler but it's not perfect. The documentation is really crappy and over the summer they transitioned the whole system from thumbor to sharp and didn't provide great backwards compatibility.
Cloudfare does have a free CDN tier "For individuals with a personal website and anyone who wants to explore Cloudflare." but it's not the same as B2 including a CDN for free, even Azure is apart of the bandwidth alliance.
Right, but it means you can basically (ab?)use Cloudflare to get free egress from B2 storage. Cloudflare won't get too mad until you start hitting terabytes per month; even the free tier doesn't have restrictions.
You can also turn on an extremely aggressive caching policy with a page rule that will keep everything under a given subdomain for a month. This makes the "free CDN" part easy, though again, people who do this run the risk of getting their accounts terminated.
Disclaimer: I work for Backblaze so I'm biased. :-)
> the B2 API is much slower than S3.
This is "generally true" for 1 upload thread. We aren't even sure what Amazon is doing differently, but they can be a little faster in general for 1 thread (some people only see 20% faster, some see as high as 50% faster, might be latency to the datacenter and where you are located).
As long as you use multiple threads, I make the radical claim that B2 can be faster than Amazon S3. The B2 API is slightly better in that we don't go through any load balancers like S3 does, so there is no choke point. What this means is that in B2 40 threads are actually uploading to 40 separate servers in 40 separate "vaults" and none of the threads could possibly know the other threads are uploading and it does not "choke" through a load balancer. This was all designed originally so that 1 million individual laptops could upload backups all at the same time with no issues and no load balancers. And it works great every day.
Practically speaking, for most people in most applications, this means both Amazon S3 and Backblaze B2 are essentially free of any limitations. If you aren't using enough of your bandwidth, spawn a few more threads (on either platform) and soak your upload capacity. But in full disclosure, if your application is only single threaded, yes, B2 tends to be 20% slower for that 1 thread.
Genuinely curious ... do you not assign any value to having a backup outside of Amazon ?
AWS can certainly provide geographical diversity, but on the organizational abstraction layer, all eggs are in one basket, yes ?
Is having organizational redundancy something you assign zero value to, or something whose value conflicts with the egress costs so as to make it a difficult decision ?
I mean we have like 2 millions of line of python code written for lambda, S3, SQS, SNS, Kinesis, Redshift etc using boto3. So if AWS dies, it's not like data backup will save my startup. We're dead.
Not the parent, but they mentioned that they are a startup. AWS "dying" has killed zero startups so far. Time to market has killed many more, same for "not-invented-here" syndrome, and prematurely building for the future.
Maybe? I'm not an influential enough engineer to change something that fundamental. Seniors say it's troubling but they're already married to AWS so it's very expensive to have a plan B. I don't think AWS dying is high on the list of why the startup can die. There are bigger dangers and they can only be solved by writing code that works.
We attempted to be cloud agnostic (using terraform instead of CloudFormation for example) and then later multi-cloud. The amount of complexity and cost around it was just too much.
If AWS goes down, more or less a good portion of the internet goes dark. It's an acceptable risk at this point unless you are truly massive and entirely self contained- if you are using any 3rd party services, IE for auth, payment, whatever- they may be using AWS as well and you are still exposed.
We backup data that's not on S3 outside of AWS (code, operational databases), but most of our S3 data is effectively stuck due to the insane export prices. It's not the end of the world if we were to lose everything in S3 anyway.
To anyone reading this: Don't store lots of small files on S3. It's a terrible idea.
https://www.backblaze.com/b2/cloud-storage.html