Move to a new, smaller, instance. You can still use lemmy.world as though it was still running at full speed. You can still post to lemmy.world or other federated communities and you experience won’t be so painful.
Lemmy.world is experiencing an influx of Redditors and with us good Redditors come our awful trolls. Growth, along with DDoS attacks have plagued the site since I began using it.
And if the thought of setting up another account annoys you, I’ve made a tool that will migrate your account settings, subscriptions, and blocks: https://github.com/CMahaff/lasim
Now that does require the source instance to be up long enough to download your profile, but after that you can upload to any instance you want and be running like nothing changed in like 2 minutes.
Tbh, I kinda like that we have these growing pains.
Helps folks leave out older expectations of monolithic profit-oriented social platforms. And actually put some money down to help host the specific niche community they really want to exist.
I’ll probably eat these words later but, as of this moment, I stand by it.
If anything this is basically establishing to everyone out there that if you want to kill an instance or encourage people to move to a different one (with different admins who might have different…“styles”), just DDOS it and promote your alternative instance as a refuge.
I’m sticking with .world because the admins there are chill. Don’t feel like rolling the dice on a new instance where some power mods probably set up shop.
It’s a tumblr meme and IKEA reacted to the association by using the shark as a prop in their pro-gay marriage ads in Switzerland. F1nnster has like twenty in a pile in the background as people keep buying them for him.
I wish people would stop using this advice without some caveats. The instance you choose is also about the admins your choosing to have your account under.
I’ll stay on Lemmy.world because I trust the admin there. Any time you jump to a new instance, you better hope it’s run by levelheaded, fair-minded people.
You don’t have to jump to tiny < 100 user instances though, any instance within the top 10 is a good alternative to lemmy.world. If everyone thinks the same as you do, then there would be no point in federation.
Y’know what, you’re right, but in an ideal federated world, it is probably for the best if people branch out further than just the top 10 as well. The instance I’m on probably is not even within the top 50, but it’s fast, performant, and has all my subscriptions. Not sure about the admins, but I also have alts on lemm.ee and sh.itjust.works in case this instance goes bust.
It’s mostly DDOS attacks though, not the influx of normal users.
Really? As site_aggregates table getting 1500 rows updated on every single new comment and post local insert is just the tip of the iceberg of how nobody has scrutinized the PostgreSQL performance. Thank you to lemmy.ca last weekend for looking at AUTO_EXPLAIN on their data.
For all the annoyance, a silver lining is that lemmy.world is testing lemmy at a relatively high scale lemmy doesn’t see anywhere else and so aiding in the development of the software and architectural guidelines for instance management.
There is an instance centered around that actually, though I have no idea the state of it’s stability or funding. I’ve made my account on a more general content furry-themed instance and found it to be pretty well run though, don’t really have any complaints.
I meant, are they defederated from each other - I have taken a look at lemmynsfw.com and don’t recall seeing any content from pornlemmy.com. Though, I don’t remember what my view settings were at the time do maybe I was just looking at local.
I totally agree with your outlook and made a pretty similar post to yours a couple minutes ago. My only addition would be some concern as to why it seems like attacks are causing the downtime. The attacks do encourage improvement, but why do it in the first place. I’m hoping bored enthusiasts. At least it wouldn’t be BS corporate attacks trying to eliminate competition.
Can’t find the post right now but sunaurus said that the main difference is that lemm.ee is the only instance using a horizontal setup, that is, there’s multiple lemmy servers running on multiple servers behind a load balancer, all sharing a database (postgres itself clusters very well). The code isn’t actually made for that so it’s all rather custom and possibly specific to his hoster.
Less technical: sunaurus happens to be a beast of a devop and prolific contributor to the lemmy codebase. As such lemm.ee quite often runs code that’s ahead of the release schedule, addressing stuff that he stumbled across while wearing his sysop hat.
lemdro.id also runs via horizontal scaling behind a load-balancer, soon to expand globally to keep response times down for people everywhere. We’re very resilient :)
I got accounts on lemm.ee, sh.itjust.works and kbin.social. I had one on .world in the beginning, but the performance wasn’t great. Probably too many users.
1993: God, how we would love it if someone could tell us anything was “just that simple”, and then of course when you see a pie chart you go “Oh, a pie chart…”. I mean, it has more religious meaning now than a crucifix to see a pie chart. I mean, because…. why is that so popular? Because it reduces complexity. The complexity is very real but his little soundbites - 1993
@garpunkal@lemm.ee - do you know of the history of site_aggregates PostgreSQL table?
lemmy.ca staff was so frustrated with performance problems a couple weekends ago they cloned a copy of their database Running AUTO_EXPLAIN revealed site_aggregates logic in Lemmy was doing comment = comment + 1 counting against 1500 rows, for every known Lemmy instance in the database, instead of just writing 1 row.
i tried this, is it normal to never receive the verification email? it says verification sent, i tried 4 diff smaller instances, and its been like 10 hours. i checked the spam too
That’s why I joined my local “area of my country” instance. I just need to subscribe to all my communities here and I’ll be good. The app I use (Memmy) let’s you easily switch between profiles on any instance too.
I wanted to stay with World. But they are literally constantly having severe problems (errors, voting, comments). Gave them many chances. I’m sorry World, but I probably would abandon Lemmy if I had to stay, it’s just not an enjoyable experience.
Maybe it’s not even possible to keep a large (targeted) instance working with current limitations and tools? Hope they figure it out. I’m rooming with Stux at Geddit, he’s cool. So many cats…
Having glanced at the code and taken my own instance down a few times cleaning up a surprisingly small number of automated posts, it’s definitely the combination of some design choices in the code and the scale of lemmy.world. Keeping an instance up that has so many posts and communities has been difficult on my instance, and I’m basically the only user. I can imagine with the scale and lemmy.world load and publicity, it’s nearly impossible until some improvements to the data layer are made.
For one example, purging a community with 1k posts and 30k comments (I was messing with a bot) took my instance down for 2 hours with the postgres database pegged at a full core minimum. And then it took down my instance. And then I restarted the database but presumably this was done in a transaction so no progress had been made.
I’m personally impressed with the amount of uptime lemmy.world has managed. And I’m also impressed with lemmy overall, but it’s pretty clear there has been some rapid growth that, as it usually does, exposed some of the limits of the design and requires some improvements for the current scale.
As a user with pretty limited knowledge on servers, I appreciate how easy it is to flip from instance to instance. One goes down and I use a backup account somewhere else. I’m really not keeping close attention to which one I’m on at the moment. It gives me access and lets me comment/post. I’m not trying to build any profile so it doesn’t matter. I could see a major poster or mod want multiple accounts with the same name for consistency sake though.
Problem is that many times it will say “partial outage” but the website doesnt even work so technically it’s a full outage. I assume it’s to keep the uptime % as high as they can. So that 98.XX% uptime isnt very accurate at all.
What is their motivation about lying about uptime? It isn’t a business with advertisers, it is some dudes hobby server and some people who are donating despite what the uptime percentage is
correct, I dont know if it’s automatic to partial outage and manual trigger to full or how that works in their backend. But almost every time I’ve seen a partial (orange) outage, it’s a full blown outage.
I moved to sopuli.xyz because of this. I can still subscribe to all the communities I like so no point in staying in an instance that’s constantly down.
I think it’s more like the previous commentor said. It’s the communities more than the users. Every post, comment, like needs to be sent to every other instance that subscribes to the community. I suspect it’s definitely connected to federation. The reason being, at 20:00 utc yesterday lemmy.world stopped sending my instance anything (previously it was between 2 and 5 messages a second). It only started again at around 00:00 utc. I wonder if they were slowly adding instances back to federation?
In any case the load for that many communities with that many other instances must be huge. The advantages of the fediverse requires that communities AND users are spread between instances. In the current climate, the super instances have most of both and it must be becoming exponentially harder to keep up with hardware requirements for this.
Whose fault is it though? If an instance is capable of 100 concurrent users but everyone flocks to the two or three big instances. What to do? Block instances so they shutdown? Then when the shit really hits the fan there’s nowhere to distribute users to.
In the case of lemmy.world I might suggest they split the instance. Original lemmy.world keeps the communities but has no users. Create a new instance and transfer the users. That way the first instance is dedicated to federating the communities, moving the real time user database hits to a separate database. I’d also suggest preventing the creation of new communities on that instance.
In real terms it’d have been better if the communities were shared between instances more. Making a more even spread of the one to many distribution efforts.
Move to a new, smaller, instance. You can still use lemmy.world as though it was still running at full speed. You can still post to lemmy.world or other federated communities and you experience won’t be so painful.
Lemmy.world is experiencing an influx of Redditors and with us good Redditors come our awful trolls. Growth, along with DDoS attacks have plagued the site since I began using it.
And if the thought of setting up another account annoys you, I’ve made a tool that will migrate your account settings, subscriptions, and blocks: https://github.com/CMahaff/lasim
Now that does require the source instance to be up long enough to download your profile, but after that you can upload to any instance you want and be running like nothing changed in like 2 minutes.
Thanks for the beautiful tool!
Agreed, and starred.
Tbh, I kinda like that we have these growing pains. Helps folks leave out older expectations of monolithic profit-oriented social platforms. And actually put some money down to help host the specific niche community they really want to exist.
I’ll probably eat these words later but, as of this moment, I stand by it.
Yeah, this is the fediverse’s way of telling us to break our old habit of piling on a single site and to spread out.
Except…it’s being DDOS’d, so no, it isn’t.
If anything this is basically establishing to everyone out there that if you want to kill an instance or encourage people to move to a different one (with different admins who might have different…“styles”), just DDOS it and promote your alternative instance as a refuge.
I’m sticking with .world because the admins there are chill. Don’t feel like rolling the dice on a new instance where some power mods probably set up shop.
You do you, but I wouldn’t say that lemm.ee, sh.itjust.works, sopuli.xyz or reddthat.col are power mods
Particularly sh.itjust.works. They’re letting the community vote on instance rules and the Admin is really chill
A gaming instance would take a lot of load.
Isn’t https://lemmy.zip/ gaming focused?
If you can’t tell from the name it’s not going to work well.
Blahaj is queer focused, but there is no way you can get that from the name
It’s a tumblr meme and IKEA reacted to the association by using the shark as a prop in their pro-gay marriage ads in Switzerland. F1nnster has like twenty in a pile in the background as people keep buying them for him.
I didn’t know blahaj was LGBTQ-related. That’s so fucking cool!
I love IKEA.
I wish people would stop using this advice without some caveats. The instance you choose is also about the admins your choosing to have your account under.
I’ll stay on Lemmy.world because I trust the admin there. Any time you jump to a new instance, you better hope it’s run by levelheaded, fair-minded people.
You don’t have to jump to tiny < 100 user instances though, any instance within the top 10 is a good alternative to lemmy.world. If everyone thinks the same as you do, then there would be no point in federation.
Any instance in the top 10 is not a “newer smaller instance.”
Y’know what, you’re right, but in an ideal federated world, it is probably for the best if people branch out further than just the top 10 as well. The instance I’m on probably is not even within the top 50, but it’s fast, performant, and has all my subscriptions. Not sure about the admins, but I also have alts on lemm.ee and sh.itjust.works in case this instance goes bust.
I am subscribed to nine others.
Have you figured out an efficient way to mirror your community subscription list between them?
https://github.com/Ac5000/lemmy_account_sync
I don’t know.
I fouls the perfect instance to match my screen name
Yeah. There is little reason to sign up on the biggest server since you can see and interact with content from any of the servers.
There’s plenty of reason for a new user, but otherwise agreed. Big instances are training wheels… I say from the biggest kbin instance.
Is lemmy.world getting DDoSed? Who would profit from that? (honest question. I hadn’t read this before.)
DDoS are sometimes just people thinking “because I can”, not necessarily motivated by profit.
A smallish scale service like a lemmy server ran by volunteers seems like an easy target, so it wouldn’t be surprising that being the case.
deleted by creator
Really? As site_aggregates table getting 1500 rows updated on every single new comment and post local insert is just the tip of the iceberg of how nobody has scrutinized the PostgreSQL performance. Thank you to lemmy.ca last weekend for looking at AUTO_EXPLAIN on their data.
Is this accurate?
Do you know who runs lemmy.ca, accurate? https://lemmy.ml/post/2502514
Huh? I’m not an instance operator (yet). I was actually asking you a question, does this actually happen? Have you seen this on your own instance?
Was hoping to get some discussion going on this.
Edit: do you have any example logs from your postgres instance demonstrating this?
I’ve exposed raw data from my PostgreSQL tables. And every single new comment and post you could see +1 going on every known server. I haven’t updated to 0.18.3 yet, so it’s still there: https://lemmyadmin.bulletintree.com/query/raw_site_aggregates?output=table
For all the annoyance, a silver lining is that lemmy.world is testing lemmy at a relatively high scale lemmy doesn’t see anywhere else and so aiding in the development of the software and architectural guidelines for instance management.
Yes. These are growing pains. That’s a good thing.
If you want a fast, stable, and we’ll funded instance, try some of the porn ones.
Only if it’s Furry porn.
There is an instance centered around that actually, though I have no idea the state of it’s stability or funding. I’ve made my account on a more general content furry-themed instance and found it to be pretty well run though, don’t really have any complaints.
Cool.
They’re defederated with many of the instances though
They even have video uploading enabled.
( ͡° ͜ʖ ͡°)
Can you recommend any that are good? Lemmynsfw seems to only be celebrities and it sucks.
You have to be signed in to see the actual porn lol
Try pornlemmy.com
Are the porn instances not federated either? Seems like a mistake on their part
The porn instances got defederated by most of the “mainstream” instances, including this one, lemmy.ml.
That strikes me as a bad move, because free porn == user growth
I meant, are they defederated from each other - I have taken a look at lemmynsfw.com and don’t recall seeing any content from pornlemmy.com. Though, I don’t remember what my view settings were at the time do maybe I was just looking at local.
pornlemmy and lemmynsfw are both linked to each other, but there isn’t much content on pornlemmy so far.
Honestly? That’s great. It’s stress-testing Lemmy and, to an extent, ActivityPub.
Growing pains. It’s just gonna improve Lemmy in the long term.
If you don’t like it, use another smaller instance like lemmy.zip or lemm.ee. You know, the entire point of decentralization.
I totally agree with your outlook and made a pretty similar post to yours a couple minutes ago. My only addition would be some concern as to why it seems like attacks are causing the downtime. The attacks do encourage improvement, but why do it in the first place. I’m hoping bored enthusiasts. At least it wouldn’t be BS corporate attacks trying to eliminate competition.
Suffering from success.
Don’t worry though they’ll figure it out. The early days of Reddit were pretty unstable. Definitely sucks for now but it’ll get better
Are you just going to make one of these every single time?
“Ugh” what?
Because it’s annoying.
deleted by creator
I’m indifferent about it.
That’s why I use lemm.ee
I have accounts on a few instances, and lemm.ee is the quickest and most stable of them all. I don’t know what they’re doing, but it’s great.
Can’t find the post right now but sunaurus said that the main difference is that lemm.ee is the only instance using a horizontal setup, that is, there’s multiple lemmy servers running on multiple servers behind a load balancer, all sharing a database (postgres itself clusters very well). The code isn’t actually made for that so it’s all rather custom and possibly specific to his hoster.
Less technical: sunaurus happens to be a beast of a devop and prolific contributor to the lemmy codebase. As such lemm.ee quite often runs code that’s ahead of the release schedule, addressing stuff that he stumbled across while wearing his sysop hat.
lemdro.id also runs via horizontal scaling behind a load-balancer, soon to expand globally to keep response times down for people everywhere. We’re very resilient :)
I got accounts on lemm.ee, sh.itjust.works and kbin.social. I had one on .world in the beginning, but the performance wasn’t great. Probably too many users.
if local.lemmyusers > 15, crash constantly because of PostgreSQL nonsense logic and Rust ORM.
1993: God, how we would love it if someone could tell us anything was “just that simple”, and then of course when you see a pie chart you go “Oh, a pie chart…”. I mean, it has more religious meaning now than a crucifix to see a pie chart. I mean, because…. why is that so popular? Because it reduces complexity. The complexity is very real but his little soundbites - 1993
@garpunkal@lemm.ee - do you know of the history of site_aggregates PostgreSQL table?
huh?
Please explain in detail what “huh” means in this context.
As I said in the comment you replied to: do you know of the history of site_aggregates PostgreSQL table?
Not OP, but I feel like it was Huh? as in what the heck are you talking about and why was it a reply to thier comment
no tell me more?
lemmy.ca staff was so frustrated with performance problems a couple weekends ago they cloned a copy of their database Running AUTO_EXPLAIN revealed site_aggregates logic in Lemmy was doing comment = comment + 1 counting against 1500 rows, for every known Lemmy instance in the database, instead of just writing 1 row.
Dude just move to a small, updated instance with good uptime. I joined aussie.zone and its never down plus feels so much snappier.
i tried this, is it normal to never receive the verification email? it says verification sent, i tried 4 diff smaller instances, and its been like 10 hours. i checked the spam too
🤣🤣 its just you dude. I have switched instances multiple times and never had an issue like you described. Try aussie.zone instance
i am not australian but why not lol
edit: meh nvm, dont wanna wait to be accepted. im american anyway,
Doesn’t matter. Even I am not Australian but I am using aussie.zone
I’m not Australian.
Just in time for lemmy.fmhy.net to come back up
🤫
Oh, you guys are back!
That’s why I joined my local “area of my country” instance. I just need to subscribe to all my communities here and I’ll be good. The app I use (Memmy) let’s you easily switch between profiles on any instance too.
memmy is great!
Until that other one goes down, then memmy really struggles to let you in. At least that’s what happened to me.
I guess I’m betting they don’t go down at once. I guess it can happen tho.
No I mean there’s some weird bug in Memmy. My second account went down and I had to reinstall Memmy to access my first one
Interesting, you should post that on the Memmy community.
I wanted to stay with World. But they are literally constantly having severe problems (errors, voting, comments). Gave them many chances. I’m sorry World, but I probably would abandon Lemmy if I had to stay, it’s just not an enjoyable experience.
Maybe it’s not even possible to keep a large (targeted) instance working with current limitations and tools? Hope they figure it out. I’m rooming with Stux at Geddit, he’s cool. So many cats…
Having glanced at the code and taken my own instance down a few times cleaning up a surprisingly small number of automated posts, it’s definitely the combination of some design choices in the code and the scale of lemmy.world. Keeping an instance up that has so many posts and communities has been difficult on my instance, and I’m basically the only user. I can imagine with the scale and lemmy.world load and publicity, it’s nearly impossible until some improvements to the data layer are made.
For one example, purging a community with 1k posts and 30k comments (I was messing with a bot) took my instance down for 2 hours with the postgres database pegged at a full core minimum. And then it took down my instance. And then I restarted the database but presumably this was done in a transaction so no progress had been made.
I’m personally impressed with the amount of uptime lemmy.world has managed. And I’m also impressed with lemmy overall, but it’s pretty clear there has been some rapid growth that, as it usually does, exposed some of the limits of the design and requires some improvements for the current scale.
As a user with pretty limited knowledge on servers, I appreciate how easy it is to flip from instance to instance. One goes down and I use a backup account somewhere else. I’m really not keeping close attention to which one I’m on at the moment. It gives me access and lets me comment/post. I’m not trying to build any profile so it doesn’t matter. I could see a major poster or mod want multiple accounts with the same name for consistency sake though.
You can check lemmy.world’s status page to see when it’s down.
Problem is that many times it will say “partial outage” but the website doesnt even work so technically it’s a full outage. I assume it’s to keep the uptime % as high as they can. So that 98.XX% uptime isnt very accurate at all.
What is their motivation about lying about uptime? It isn’t a business with advertisers, it is some dudes hobby server and some people who are donating despite what the uptime percentage is
correct, I dont know if it’s automatic to partial outage and manual trigger to full or how that works in their backend. But almost every time I’ve seen a partial (orange) outage, it’s a full blown outage.
Oh so that’s why there seems to be less content in my feed than normal?
I moved to sopuli.xyz because of this. I can still subscribe to all the communities I like so no point in staying in an instance that’s constantly down.
Problem is, so many communities are on .world now, so it hurts even from another instance
I hope people on the Fediverse will finally learn not to choose the biggest instance all the time
I think it’s more like the previous commentor said. It’s the communities more than the users. Every post, comment, like needs to be sent to every other instance that subscribes to the community. I suspect it’s definitely connected to federation. The reason being, at 20:00 utc yesterday lemmy.world stopped sending my instance anything (previously it was between 2 and 5 messages a second). It only started again at around 00:00 utc. I wonder if they were slowly adding instances back to federation?
In any case the load for that many communities with that many other instances must be huge. The advantages of the fediverse requires that communities AND users are spread between instances. In the current climate, the super instances have most of both and it must be becoming exponentially harder to keep up with hardware requirements for this.
That’s a very valid point. Sometimes I question if very small instances (1-10 users) are not more detrimental than anything to the general performance
Whose fault is it though? If an instance is capable of 100 concurrent users but everyone flocks to the two or three big instances. What to do? Block instances so they shutdown? Then when the shit really hits the fan there’s nowhere to distribute users to.
In the case of lemmy.world I might suggest they split the instance. Original lemmy.world keeps the communities but has no users. Create a new instance and transfer the users. That way the first instance is dedicated to federating the communities, moving the real time user database hits to a separate database. I’d also suggest preventing the creation of new communities on that instance.
In real terms it’d have been better if the communities were shared between instances more. Making a more even spread of the one to many distribution efforts.
sounds like a cool idea. hit Ruud up once they’re less busy.
Ya still down for me. Even says my account doesn’t even exist.