ruud

There was another attack going on (as you might have noticed). We're working on a fix. In the meantime, we've blocked the listing of comments, so we at least aren't down, but it did break comments.

Hope to have a fix in the next hour. Stay tuned!

Update OK we've implemented a fix, again many thanks to @[email protected] for his assistance. This will prevent the outages we've seen last couple of days. Let's see what they will come up with next...

A few days ago I saw some cool JoinLemmy stickers created by @[email protected] . I asked her if she could also create lemmy.world stickers, and she did!

You can see and order them here, also check the other cool stickers in her shop.

Thanks for creating them!

Lemmy.world has been down between 02:00 UTC and 05:45 UTC. This was caused by the database spiking to 100% cpu (all 32 cores/64 threads!) due to inefficient queries been fired to the db very often.

I’ve collected the logs and we’ll be checking how to prevent this. (And what caused this)

Update The upgrade was done, DB migrations took around 5 minutes. We'll keep an eye out for (new) issues but for now it seems to be OK.

Original message We will upgrade lemmy.world to 0.18.3 today at 20:00 UTC+2 (Check what this isn in your timezone). Expect the site to be down for a few minutes. ""Edit"" I was warned it could be more than a few minutes. The database update might even take 30 minutes or longer.

Release notes for 0.18.3 can be found here: https://github.com/LemmyNet/lemmy/blob/main/RELEASES.md

(This is unrelated to the downtimes we experienced lately, those are caused by attacks that we're still looking into mitigating. Sorry for those)

Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.

Most of these 'attacks' are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.

For the other attacks, we are using them to investigate and implement measures like rate limiting etc.

As requested by some users: 'old' style now accessible via https://old.lemmy.world

Code can be found here: https://github.com/rystaf/mlmym , created by Ryan (Is he here?) (Yes he appears to be! @[email protected] ! Thanks for this awesome front-end!)

Thanks to @[email protected] for another release with awesome enhancements, see release notes here: https://lemmy.world/post/1558795

I blogged about what happened in June, and the financial overview.

It's always the small things you overlook...

The docker-compose.yml I copied from somewhere when setting up lemmy.world apparently was missing the external network for the pictrs container.. So pictrs was working, as long as it got the images via Lemmy. Getting the images via URL didn't work...

Looks like it's working now. Looks a whole lot better with all the images :-)

Edit For existing posts: Edit the post, then Save. (No need to change anything). This also fetches the image.

(Duplicate post :-) see https://lemmy.world/post/1375042)

We've installed Voyager and it's reachable at https://m.lemmy.world, you can browse Lemmy, and login there (also if your account isn't on lemmy.world)

PS Thanks go out to @stux@[email protected] , he came up with the idea (see https://m.geddit.social).

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

• There is an vulnerability which was exploited
• Several people had their JWT cookies leaked, including at least one admin
• Attackers started changing site settings and posting fake announcements etc

Our mitigations:

• We removed the vulnerability
• Deleted all comments and private messages that contained the exploit
• Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.