UPDATE: The latest RC version of Lemmy-ui (0.18.2-rc.2) contains fixes for the issue, but if you believe you were vulnerable, you should still rotate your JWT secret after upgrading! Read below for instructions. Removing custom emoji is no longer necessary after upgrading.
Original post follows:
This post is intended as a central place that admins can reference regarding the XSS incident from this morning.
What happened?
A couple of the bigger Lemmy instances had several user accounts compromised through stolen authentication cookies. Some of these cookies belonged to admins, these admin cookies were used to deface instances. Only users that opened pages with malicious content during the incident were vulnerable. The malicious content was possible due to a bug with rendering custom emojis.
Stolen cookies gave attackers access to all private messages and e-mail addresses of affected users.
Am I vulnerable?
If your instance has ANY custom emojis, you are vulnerable. Note that it appears only local custom emojis are affected, so federated content with custom emojis from other instances should be safe.
I had custom emojis on my instance, what should I do?
This should be enough to mitigate now:
- Remove custom emoji
DELETE FROM custom_emoji_keyword;
DELETE FROM custom_emoji;
- Rotate your JWT secret (invalidates all current login sessions)
-- back up your secret first, just in case
SELECT * FROM secret;
-- generate a new secret
UPDATE secret SET jwt_secret = gen_random_uuid();
- Restart Lemmy server
If you need help with any of this, you can reach out to me on Matrix (@sunaurus:matrix.org) or on Discord (@sunaurus)
Legal
If your instance was affected, you may have some legal obligations. Please check this comment for more info: https://lemmy.world/comment/1064402
More context:
Anything about running your own Lemmy instance. Including how to install it, maintain and customise it.
Be sure to check out the docs: https://join-lemmy.org/docs/en/administration/administration.html
If you have any problems, describe them here and we will try to help you fixing them.
- 0 users online
- 3 users / day
- 4 users / week
- 4 users / month
- 4 users / 6 months
- 1 subscriber
- 63 Posts
- 273 Comments
- Modlog
- mods:
Thanks for posting. There really should be a button which allows the admins to log everyone out for crisis situations like there I think
Changing the JWT secret does this. So instead of a button, its a line of code, making it less likely to be done by mistake.
True as that is, not every admin has DB access and those with DB access might be AFK.
Surely this exploit proves that itโs best to minimise the number of administrative actions available in the UI?
Destructive actions, sure. This would not be one such actions and it would be self-defeating.
In crisis situations like this, the last thing you want is to rely on some portion of the application UI working.
If the UI doesnโt work, thereโs always the DB directly. But itโs good to have that option.
JWT secret keys are not in the DB (speaking typically, maybe for Lemmy they are, but that would be very surprising), thatโs typically an environment variable or configuration file sort of thing.
In any case, this isnโt the part thatโs broken, it doesnโt need fixed.
Itโs in the DB. Check the OP
I am also not suggesting a fix. I am suggesting an improvement
Oof, okay well thatโs not how I wouldโve done it. The JWT secret in the database itself could be a vulnerability (e.g., someone that gains read only access to the database could then use that as a wedge to create any JWT they wanted). Iโm not sure if thatโs actually worth bringing up or not (itโs a bit of an odd case).
It used to be in a config file but they moved it to the db so it could be autogenerated on first startup I believe. I guess itโs better than newb instances running around with a jwt secret of โchangemeโ but stillโฆ
Hmโฆ They couldโve edited the config or just
exit(1)if the credential is the default, but very fair.afaik to generate those tokens, you configure a secret in an enviroment variable. You cannot generate tokens from looking at valid tokens within the database. Thus storing active tokens in the database is fine since you can always purge all active tokens as this post has also suggested.
This is the secret, not the tokens.
oh i see, they want to delete the secret instead of the active tokens. Yeah now i get what you mean. Seems kinda odd.
I donโt know enough about Lemmyโs JWT design, but some JWT designs donโt store the JWT in a database at all, so the only correct response is to regenerate the secret and kill all the sessions by them failing the validity checks.
๐
โฆthatโs what i just said?https://lemmy.dbzer0.com/comment/793036No, you said you canโt generate valid tokens within the database. I just told you this is the secret, not the tokens (that is present in the database).
This bug also needs to be fixed so that a user logging out / changing password actually correctly invalidates their JWT token: https://github.com/LemmyNet/lemmy/issues/3364
I suspect that last night even if an admin knew their account was compromised, they probably couldnโt do anything about it without DB access.
Okay it is insane that that issue is 2 weeks old and was not prioritized and because of that the hacker was gifted 2 weeks to prepare an attack.
I love Lemmy and am grateful for the work of the devs. But I see huge issues with the LemmyNet governance.
Previously it was captchas that were removed because one of the two main devs had a strong personal belief that captchas are โuselessโ and wanted to impose that belief upon everyone, which then led to an enormous wave of bots with the 0.18.0 as captchas were removed. Iโm glad he was then convinced that he was wrong to remove them and then took steps to revert that decision, but it took too long and many instances suffered performance problems by being forced to stay with the 0.17.6 version because it had captchas.
Right now the same thing happens again with issues about security being left unanswered for two weeks. I believe right now all feature developments should be paused and a security audit of the whole code base should be the #1 priority.
This is just very bad. Proper governance and prioritization would have avoided exposing minors to lemon party porn and other disgusting content.
I can only imagine how helpless the admin whose account was compromised felt if she didnโt have access to the database to invalidate those tokens, and it could have been prevented if that issue was properly prioritized.
Changing password does invalidate tokens, but the rest of that is accurate.
It seems to me that the scope of this could have been mitigated with a simple privilege separation policy for admin server accounts but I see a lot of (what looks like) server admins using that account as their daily driver.
Also, lemmy-ui should post a security advisory to their github.
Welp, that is some simple lapse of separation-of-duties principles that shouldnโt have been.
I donโt pretend to know the demographics of lemmy server admins, but my gut says itโs predominantly hobbyists and devops types, rather than grizzled system admins.
People in DevOps should know better than reusing their accounts. It doesnโt take a seasoned system admin to know about basic OPSEC and separation amongst profiles.
Agreed. Iโm pointing it out in hopes that becomes one of the takeaways of this incident.
(Unrelated, why are you marked at a bot account?)
Wait, why am I marked as a bot account?
This is the first time Iโve heard about this. Whom should I ask for help?
Itโs a checkbox in the setting for your account. You should be able to change it.
Hi, could you check now?
I still see it.
Yes and no. Admin accounts often remain logged in as a practical matter. They canโt see incoming reports, registration applications, etc. if the account isnโt logged in. And there is no โmiddle tierโ/sitemod or customizable permissions allowing for anything between community mod and instance admin that would mitigate the need to use admin accounts day to day.
I still wouldnโt use the admin account as my daily driver. Leave it open in another browser/private tab specifically to perform admin actions (as noted) but not for browsing/posting/community modding. I understand how thatโs a pain but given how early days we are with this platform and the high probability of more issues surfacing, itโs a necessary pain.
Iโd really like to see the devs add some tools to mitigate future risk and further protect admin accounts. The least of which being that admin actions require stronger validation than a browser side cookie, and frequent re-validation to perform admin actions.
iโve thought about switching to a different account for non-admin tasks but I was a normal user here first and I want to keep using this account for normal posting too. I think using a different account is kind of pointless from a security standpoint, because performing mod and admin actions requires looking at user-generated content anyway and there is no way around that (except for maybe moving to a very clipboard-heavy workflow which would make everything take much longer).
There is work underway to fix these XSS bugs in a more comprehensive way (better CSP, and HttpOnly cookiesโฆ) which will hopefully land soon.
(i am not a lemmy developer but iโm reading the github and matrix chatโฆ)
I mean. You could still be a normal user and have a separate admin account.
I get your point. But itโs a huge risk.
Using a separate account doesnโt substantially mitigate the risk. It might reduce the chances of getting randomly exploited, but itโs easy to post things admins need to see to do their job so any attacker wanting to target admins would still be able to even if we used separate accounts.
Systemically fixing the XSS problems is necessary either way :)
edit: actually I guess most common admin activities could be separated from the rarely used ones, at leastโฆ or the infrequent actions could simply require re-authenticating. That wouldnโt be a bad idea.
To be clear Iโm not talking about separate mod accounts. Iโm talking just admin.
that wouldnโt have necessarily stopped this attack I donโt think, but yeah, probably a good idea on multiple levels.
If the separate admin window was open, and a tagged reply or PM was sent to the admin account I think that would render the emote in the notification and trigger the exploit
Fair point. I know that I donโt have the answers. I do think that admin actions need to be more stringently scrutinized. Maybe something like a โsudoโ model where your a normal acct 99% of the time and admin actions require a temporary elevation.
Thanks for the info.
My personal instance has no custom emojiโs, yet the SQL to remove onload= from content removed 16 effected comments. Does that mean federated instances are effected?
This does not mean your instance was affected. Youโre just cleaning up comments which may or may not have successfully worked on other instances by abusing the emoji bug there.
Roger okay. Thanks for clearing that up.
Iโm curious why the UI update to 0.18.2-rc.1 isnโt mentioned anywhere in this thread. I understood that updating the UI was a fix for this exploit.
My post predates the new versions, but I am editing the post now!
Awesome, thank you!
edit: you may want to specify that itโs
lemmy-uithat has the update, notlemmy.Weโll likely do a lemmy release tomorrow with this fix included.
Arenโt โprivate messagesโ not really private on lemmy anyway?
Under normal circumstances, they are only visible to:
It is still considered a breach of user data if such messages are leaked.
โprivateโ messages are not private anywhere.
unless theyโre E2EE
How did the hackers get the cookies in the first place? Compromised devices on the clients?
Iโm not affiliated with lemmy.world or any other instance but I do software and can explain some of the jargon above. XSS is the abbreviation of one of the most common attacks we see on websites, cross-site scripting. This attack works by having some vulnerable code which arbitrarily executes some javascript on a users browser.
In this case, the attacker seems to have found a vulnerability where a specially crafted character is executed when users read posts or comments containing it. In this case, it was especially bad because of how passwords are stored. When you log in to pretty much any website, passwords are stored in the form of cookies; small pieces of data that are passed back and forth to the web server and the client automatically. Usually, these cookies are set to not be readable by javascript, but in this case, it appears that that flag was not set. This allowed the XSS exploit to be sent back to a computer which was set up to grab these cookies.
One thing to note is that although the cookies were stolen, our passwords wouldnโt have immediately been compromised. Our login cookies store jwt tokens; a cryptographically signed message proving that we provided a password previously. Thatโs not to say that further escalation isnโt possible, but its hard to say for certain how far the hack went before being noticed.
Thank you for the detailed explanation :)
Stealing sessions/cookies is the new trend, it bypasses 2FA and hackers do not even have to know passwords.
Client does not have to be infected, it can happen inside the browser, there is no known protection at this time.
You can only mitigate, log out after you finish the session, like banking and clean or do not save essential cookies.
There are various vectors though: https://blog.netwrix.com/2022/11/29/bypassing-mfa-with-pass-the-cookie-attack
Why is lemmy creating html nodes concatenating strings? This is going to happen again and again
I should have expected this judging by the state of the WS bugs
This is not the general way of doing things in Lemmy. It was just a mistake that unfortunately did not get caught in code review. These things happen from time to time (even in projects which are FAR bigger and better funded than Lemmy)
Itโs probably worth having someone audit the codebase to ensure that thereโs no other places that build HTML using string concatenation. https://github.com/LemmyNet/lemmy-ui/pull/1897 looks like a temporary patch rather than an actual fix. I can try to take a look if I get some free time.
As an example, Googleโs own website domain was once bought by someone other than Google after they discovered it on a list of domains for sale. Out of curiosity, they tried to buy it and were (to their surprise) successful. The person owned it for about a minute before Google caught on and reversed the transaction. https://money.cnn.com/2016/01/29/technology/google-domain-purchase/index.html