Self Proclaimed Internet user and Administrator of Reddthat
- 0 Posts
- 10 Comments
Relevant: https://reddthat.com/comment/8316861 tl;dr. The current centralisation results in a lemmy-verse theoretical maximum for of 1 activity per 0.3 seconds, or 200 activities per minute. As total transfer of packets is just under 0.3 seconds between EU -> AU and back.
Edit: can’t math when sleepy
I use Wasabi storage which is more expensive as they have a minimum space allotment but because my servers are in Aus I had issues with backblaze b2 storage and the latency. (I was dealing with 200-300ms in network latency AU -> US + the time that backblaze takes to store the data).
At that time lemmy/pictrs was not as optimised as it is now so it’s much better now.
Backblaze comes out WAY cheaper per-month if you have servers in us/eu, as close to their regions as possible, but they also charge you for API.
As part of an Object Storage / cdn remember you also might have to pay for egress charges as well. Cloudflare is part of the “Bandwidth Alliance” but that isn’t applicable here as pictrs needs to present the images via its own domain, (such as cdn.reddthat.com). So you’ll still want a CDN infront which will mean you will only pay once for the egress instead of everytime everyone loads it.
- Minio’s free tier is a host-it-yourself. Where as their paid tier are for actual storage hosters, they have a minimum of 100TB/month @ $10/TB.
- B2 has no minimum, and egress + hosting of 20GB cost me… ($0.29)
- Note: They still havn’t billed me because I havn’t passed the $1 mark yet!
- Wasabi (my current choice) is $7/TB (AU, US is $6/TB iirc) with a minimum of 1TB/month with out egress or api charges
Reddthat has… looks up 150GB of object storage now.
I would recommend B2 if you are starting out and are in US/EU. Wasabi in all other regions and have a CDN infront. (and don’t mind burning a little cash for peace of mind)
To get it completely free it would need to be:
- cloudflare > vps
- cloudflare > backblaze
Because Lemmy does not have a way to have all media served from a different url/subdomain (like a CDN). Your setup will be:
- cloudflare > vps
- vps > backblaze
Basically just put cloudflare infront of your vps, and cache everything.
That means if a user uploaded a 10MB file, it would go though your VPS, get uploaded to backblaze. When a user looks at the uploaded image url, on the first request it gets requested from blackblaze, via your vps, and then gets cached on Cloudflare.
Then any extra requests from then on will be via Cloudflare. So you will technically be charged for some api requests and 1x 10MB of egress traffic.
I would recommend Backblaze B2 storage instead of removing post history.
The problem with post removal is that it will affect all other instances’ as you don’t (by default) cache original images from other fediverse instances (iirc)
So for example, my instance Reddthat will reference an image hosted on your instance, which will then 404. (But the thumbnail will be cached (I think))
Backblaze is dirt cheap and the pictrs object migration is now multi threaded making it very easy to migrate now. I was hosting 15GB and it amounted to bills of cents! $5/TB or $0.005 per GB. If you add CloudFlare Infront you then get free egress! Or any CDN that is part of the “Bandwidth alliance”. I now have ~120GB of pictrs storage. After 2-3 months.
Also, there was a recent chat on the lemmy admin matrix channel about setting some variable in pictrs so it does not cache any images. But I don’t remember what it was so maybe check the pictrs documentation.
I did a semi in-depth post on setting it all up here: https://reddthat.com/post/19103
To give you an idea of resources:
- 2 core
- current usage: 20%
- 4g ram
- current usage: 1.5-2gb
Database: 6.8gb Pictures: 13gb
We also use CloudFlare for CDN, as since the majority of connections was web sockets and POST requests it acts as a CDN for all our media.
141 active users, 667 users, 59 Communities, 530 Posts, 1.49K Comments
CPU and RAM are well below problem numbers and will continue to do so for these numbers.
The biggest problem for me is the picture files. Not only do you have to keep all the pictures your users upload, but also all the federated pictures as well! Probably the biggest challenge people that federate.
Since pictrs (since 0.4) now has object storage support, we can migrate to a S3 capable storage finally.
As we have only been online for 2 weeks… the database will grow fast, but object storage grows even faster!
- 2 weeks: 6gb
- 20 weeks: 60gb
- 60 weeks: 180gb
If the database gets that big every instance is going to have problems. The database in its current form has inefficiencies that the devs know about and are eventually working towards fixing.
Lemmy world posted what their server specs are, but they have had problems and scaled out their hardware because they knew they would keep their doors open.
https://lemmy.world/post/75556 - 32Core 128GB ram.
But IMHO it’s complete overkill. But it might not be for their database, considering they have 7.5k active users, which is 50x what I am dealing with.
Since the 11th @ 9am UTC, LW has seen a 2 fold increase of activities. If my insider knowledge is right (and math) it’s 7req/s average up from 3req/s.
Lucky for both of us we are not subbed to every community on LW but I think we are subbed just enough to be affected.