Lemmy.world Announcements

Hey everyone, so as I'm sure everyone is aware Lemmy.World has been experiencing several outages throughout the last few days.

We have been investigating the root cause of these outages but believe that they are related to our current hosting provider (Hetzner) blocking access from ClouldFlare as (we think) they believe that our CDN is a DDoS'er, and is causing these disconnects to our backend server, problematic for sure.

We've opened support tickets with our current provider and are awaiting a response. We have no issue with being as transparent as possible with downtime. Anyone that is curious, can feel free to check out https://status.lemmy.world and https://dash.lemmy.world for up to the minute outage information. We are also looking into other fediverse friendly methods of posting status and outage updates

In the meantime, we are evaluating alternative hosting options and solutions to provide a high level of reliability to you, our users. Really, we want to say thanks to everyone for soldiering through all our technical growing pains.

Cheers

• LW Infra Team

Earlier, after review, we blocked and removed several communities that were providing assistance to access copyrighted/pirated material, which is currently not allowed per Rule #1 of our Code of Conduct. The communities that were removed due to this decision were:

• [email protected]
• [email protected]
• [email protected]

We took this action to protect lemmy.world, lemmy.world's users, and lemmy.world staff as the material posted in those communities could be problematic for us, because of potential legal issues around copyrighted material and services that provide access to or assistance in obtaining it.

This decision is about liability and does not mean we are otherwise hostile to any of these communities or their users. As the Lemmyverse grows and instances get big, precautions may happen. We will keep monitoring the situation closely, and if in the future we deem it safe, we would gladly reallow these communities.

The discussions that have happened in various threads on Lemmy make it very clear that removing the communites before we announced our intent to remove them is not the level of transparency the community expects, and that as stewards of this community we need to be extremely transparent before we do this again in the future as well as make sure that we get feedback around what the planned changes are, because lemmy.world is yours as much as it is ours.

Hello there!

It has been a while since our last update, but it's about time to address the elephant in the room: downtimes. Lemmy.World has been having multiple downtimes a day for quite a while now. And we want to take the time to address some of the concerns and misconceptions that have been spread in chatrooms, memes and various comments in Lemmy communities.

So let's go over some of these misconceptions together.

"Lemmy.World is too big and that is bad for the fediverse".

While one thing is true, we are the biggest Lemmy instance, we are far from the biggest in the Fediverse. If you want actual numbers you can have a look here: https://fedidb.org/network

The entire Lemmy fediverse is still in its infancy and even though we don't like to compare ourselves to Reddit it gives you something comparable. The entire amount of Lemmy users on all instances combined is currently 444,876 which is still nothing compared to a medium sized subreddit. There are some points that can be made that it is better to spread the load of users and communities across other instances, but let us make it clear that this is not a technical problem.

And even in a decentralised system, there will always be bigger and smaller blocks within; such would be the nature of any platform looking to be shaped by its members. 

"Lemmy.World should close down registrations"

Lemmy.World is being linked in a number of Reddit subreddits and in Lemmy apps. Imagine if new users land here and they have no way to sign up. We have to assume that most new users have no information on how the Fediverse works and making them read a full page of what's what would scare a lot of those people off. They probably wouldn't even take the time to read why registrations would be closed, move on and not join the Fediverse at all. What we want to do, however, is inform the users before they sign up, without closing registrations. The option is already built into Lemmy but only available on Lemmy.ml - so a ticket was created with the development team to make these available to other instance Admins. Here is the post on Lemmy Github.

Which brings us to the third point:

"Lemmy.World can not handle the load, that's why the server is down all the time"

This is simply not true. There are no financial issues to upgrade the hardware, should that be required; but that is not the solution to this problem.

The problem is that for a couple of hours every day we are under a DDOS attack. It's a never-ending game of whack-a-mole where we close one attack vector and they'll start using another one. Without going too much into detail and expose too much, there are some very 'expensive' sql queries in Lemmy - actions or features that take up seconds instead of milliseconds to execute. And by by executing them by the thousand a minute you can overload the database server.

So who is attacking us? One thing that is clear is that those responsible of these attacks know the ins and outs of Lemmy. They know which database requests are the most taxing and they are always quick to find another as soon as we close one off. That's one of the only things we know for sure about our attackers. Being the biggest instance and having defederated with a couple of instances has made us a target.  

"Why do they need another sysop who works for free"

Everyone involved with LW works as a volunteer. The money that is donated goes to operational costs only - so hardware and infrastructure. And while we understand that working as a volunteer is not for everyone, nobody is forcing anyone to do anything. As a volunteer you decide how much of your free time you are willing to spend on this project, a service that is also being provided for free.

We will leave this thread pinned locally for a while and we will try to reply to genuine questions or concerns as soon as we can.

Edit: Upgrade finished

We will upgrade lemmy.world to v0.18.4 tomorrow at 20:00 UTC+2 (Check what this isn in your timezone. Expect the site to be down for a few minutes.

Release notes for 0.18.4 can be found here: https://github.com/LemmyNet/lemmy/blob/main/RELEASES.md

We added an extra UI for our users. Photon by Xylight is an interface with a sleek, modern look and has all the bells and whistles you would need. Moderator tools included! You can access the UI from https://photon.lemmy.world.

Visit Photon's project page on github or get in touch with the developer in the [email protected] community. If you want to use photon on other instances you can do so from https://phtn.app.

Update: @Xylight has set up a donation link so if you like photon as much as I do, show him some love! https://www.buymeacoffee.com/xylight

We've launced a public discord server at https://discord.gg/lemmyworld

The reason for why we choose discord is because it was easier to moderate and manage than other options. Besides we also had a discord-bot guru by the name of Rooki who created a neat bot that allowed us to connect/verify discord users to their Lemmy World account.

So if you are a moderator and you are looking for some extra hands to moderate your community, or if you need to contact anyone from the LW team, this is the most efficient way to do it!

Come have a look!

Lemmy.World is looking for 4 new Systems operators to help with our growing community.

Volunteers will assist our existing systems team with monitoring and maintenance.

We’re ideally looking for chill folks that want to give back to their community and work on our back-end infrastructure. Must have 4+ years of professional experience working in systems administration. We are not looking for junior admins at this time. Please keep in mind that, while this is a volunteer gig, we would ask you to be able to help at least 5-10 hours a week. We also understand this is a hobby and that family and work comes first.

Applicants must be okay with providing their CV and/or LinkedIn profile AND sitting for a video interview. This is due to the sensitivity of the infrastructure you will have access to.

We are an international team that works from both North America EST time (-4) and Europe CEST (+2) so we would ask that candidates be flexible with their availability.

If you are in AEST (+10) or JST (+9) please let us know, as we are looking for at least one Sysadmin to help out during our overnight.

You may be asked to participate in an on-call pool. Please keep in mind that this is a round-robin style pool, so it's alright if you're busy as it will just move along the chain.

If you're interested and want to apply, click here.

We recently received a message from a concerned Rammy user regarding their instance not having an active admin team.

We have made attempts to contact the Rammy admins, which other instance admins have tried as well, to determine their current status. Due to their admins being absent and their unmoderated content growing in numbers, we will defederate from Rammy. If and when this situation changes, we will be happy to reevaluate our approach. It should be noted that any instances that have abandoned admin teams will be defederated.

There was another attack going on (as you might have noticed). We're working on a fix. In the meantime, we've blocked the listing of comments, so we at least aren't down, but it did break comments.

Hope to have a fix in the next hour. Stay tuned!

Update OK we've implemented a fix, again many thanks to @[email protected] for his assistance. This will prevent the outages we've seen last couple of days. Let's see what they will come up with next...

A few days ago I saw some cool JoinLemmy stickers created by @[email protected] . I asked her if she could also create lemmy.world stickers, and she did!

You can see and order them here, also check the other cool stickers in her shop.

Thanks for creating them!

Update:
The comments from this post will not be removed as to preserve the discussion around the announcement. Any continued discussions outside of this thread that violate server rules will be removed. We feel that everyone that has an opinion, and wanted to vent, has been heard.

————-

Original post:
Yesterday, we received information about the planned federation by Hexbear. The announcement thread can be found here: https://www.hexbear.net/post/280770. After reviewing the thread and the comments, it became evident that allowing Hexbear to federate would violate our rules.

Our code of conduct and server rules can be found here.

The announcement included several concerning statements, as highlighted below:

• “Please try to keep the dirtbag lib-dunking to hexbear itself. Do not follow the Chapo Rules of Posting, instead try to engage utilizing informed rhetoric with sources to dismantle western propaganda. Posting the western atrocity propaganda and pig poop balls is hilarious but will pretty quickly get you banned and if enough of us do it defederated.”
• “The West's role in the world, through organizations such as NATO, the IMF, and the World Bank - among many others - are deeply harmful to the billions of people living both inside and outside of their imperial core.”
• “These organizations constitute the modern imperial order, with the United States at its heart - we are not fooled by the term "rules-based international order." It is in the Left's interest for these organizations to be demolished. When and how this will occur, and what precisely comes after, is the cause of great debate and discussion on this site, but it is necessary for a better world.”

The rhetoric and goal of Hexbar are clear based on their announcement: to "dismantle western propaganda" and "demolish organizations such as NATO” shows that Hexbar has no intention of "respecting the rules of the community instance in which they are posting/commenting.” It’s to push their beliefs and ideology.

In addition, several comments from a Hexbear admin, demonstrate that instance rules will not be respected.

Here are some examples:

“I can assure you there will be no lemmygrad brigades, that energy would be better funneled into the current war against liberalism on the wider fediverse.”

“All loyal, honest, active and upright Communists must unite to oppose the liberal tendencies shown by certain people among us, and set them on the right path. This is one of the tasks on our ideological front.”

• https://lemmy.world/comment/121850
• https://lemmy.world/comment/1487168
• https://lemmy.world/comment/1476084
• https://lemmy.world/comment/171595
• https://www.hexbear.net/comment/3648500

Overall community comments:

• https://www.hexbear.net/comment/3526128
• https://www.hexbear.net/comment/3526086
• https://www.hexbear.net/comment/3652828

To clarify, for those who have inquired about why Hexbear versus Lemmygrad, it should be noted that we are currently exploring the possibility of defederating from Lemmygrad as well based on similar comments Hexbear has made.

• https://lemmygrad.ml/post/158656
• https://lemmygrad.ml/comment/882559
• https://lemmygrad.ml/comment/540170
• https://lemmygrad.ml/comment/446529

Defederation should only be considered as a last resort. However, based on their comments and behavior, no positive outcomes can be expected.

We made the decision to preemptively defederate from Hexbear for these reasons. While we understand that not everyone may agree with our decision, we believe it is important to prioritize the best interests of our community.

Lemmy.world has been down between 02:00 UTC and 05:45 UTC. This was caused by the database spiking to 100% cpu (all 32 cores/64 threads!) due to inefficient queries been fired to the db very often.

I’ve collected the logs and we’ll be checking how to prevent this. (And what caused this)

Update The upgrade was done, DB migrations took around 5 minutes. We'll keep an eye out for (new) issues but for now it seems to be OK.

Original message We will upgrade lemmy.world to 0.18.3 today at 20:00 UTC+2 (Check what this isn in your timezone). Expect the site to be down for a few minutes. ""Edit"" I was warned it could be more than a few minutes. The database update might even take 30 minutes or longer.

Release notes for 0.18.3 can be found here: https://github.com/LemmyNet/lemmy/blob/main/RELEASES.md

(This is unrelated to the downtimes we experienced lately, those are caused by attacks that we're still looking into mitigating. Sorry for those)

We just added Alexandrite to the server, it's an alternative desktop UI for Lemmy created by Sheodox who worked tirelessly to make the necessary changes to we could host it ourselves here. So go to https://a.lemmy.world and have a look!

He continues to update it constantly, you can follow the development on his github page or in his community. If you like what you see and want to support him, why not buy him a coffee? :)

For those who don't have Lemmy World as their home instance and want to use Alexandrite, either ask your instance admins to add it or go to https://alexandrite.app!

Edit: I should probably have mentioned that Alexandrite is meant for desktop!

Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.

Most of these 'attacks' are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.

For the other attacks, we are using them to investigate and implement measures like rate limiting etc.

As requested by some users: 'old' style now accessible via https://old.lemmy.world

Code can be found here: https://github.com/rystaf/mlmym , created by Ryan (Is he here?) (Yes he appears to be! @[email protected] ! Thanks for this awesome front-end!)

We have received numerous reports from users about the closure of the c/android community. While we fully support the original community owners' decision to move to another instance, it will eventually be necessary to open up the community on Lemmy.world. The beauty of the fediverse is that multiple communities on the same subject can exist in different instances. However, if you can no longer moderate a community on Lemmy for any reason, it is important to pass it on to individuals who are willing and able to do so.

To ensure the best interests of our instance members, it is necessary to establish boundaries. Holding onto a community name cannot be a permanent arrangement. It's important to consider our users' ongoing interest in the community if they wish it to continue. While we acknowledge the objective of consolidating communities, current community members ultimately decide whether they wish to join the new community at lemdro.id.

To ensure a smooth transition, we will keep the community locked for another week, providing ample time to inform the active user base about the move to the new instance at https://lemmy.world/c/[email protected].

Thanks to @[email protected] for another release with awesome enhancements, see release notes here: https://lemmy.world/post/1558795

I blogged about what happened in June, and the financial overview.

It's always the small things you overlook...

The docker-compose.yml I copied from somewhere when setting up lemmy.world apparently was missing the external network for the pictrs container.. So pictrs was working, as long as it got the images via Lemmy. Getting the images via URL didn't work...

Looks like it's working now. Looks a whole lot better with all the images :-)

Edit For existing posts: Edit the post, then Save. (No need to change anything). This also fetches the image.

(Duplicate post :-) see https://lemmy.world/post/1375042)

The lemmy.world instance was just updated to version 0.18.2. The login issues that were being reported (for example, here) are now resolved.

For release notes: https://lemmy.world/post/1339018

Edit for those who still have issues logging in:

• When using a browser: clear cookies and cache
• When using an app: remove your lemmy.world account and add it again.

We've installed Voyager and it's reachable at https://m.lemmy.world, you can browse Lemmy, and login there (also if your account isn't on lemmy.world)

PS Thanks go out to @stux@[email protected] , he came up with the idea (see https://m.geddit.social).

While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

• There is an vulnerability which was exploited
• Several people had their JWT cookies leaked, including at least one admin
• Attackers started changing site settings and posting fake announcements etc

Our mitigations:

• We removed the vulnerability
• Deleted all comments and private messages that contained the exploit
• Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been 'stolen' and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

There has been significant discussion in recent weeks regarding Meta/Threads. We would like to express our disappointment with the negative and threatening tone of some of these discussions. We kindly ask everyone to engage in civil discourse and remember that not everyone will share the same opinions, which is perfectly acceptable.

When considering whether or not to defederate from Threads, we're looking for a decision based on facts that prioritize your safety. We strive to remain neutral to make an informed choice.

First, there seem to be some misconceptions about how the Fediverse operates based on several posts. We’ve compiled some resource links to help explain the details and address any misunderstandings.

Fed Tips , Fediverse , ActivityPub

Initial Thoughts:

It seems unlikely that Meta will federate with Lemmy. When/if Meta adopts ActivityPub, it will likely affect Mastodon only rather than Lemmy, given Meta's focus on being a Twitter alternative at the moment.

Please note that we have a few months before Threads will even federate with Mastodon, so we have some time to make the right decision.

Factors to Consider:
Factors to consider if Meta federates with Lemmy:

Privacy - While it’s true that Meta's privacy settings for the app are excessive, it’s important to note that these settings only apply to users of the official Threads app and do not impact Lemmy users. It’s worth mentioning that Lemmy does not collect any personal data, and Meta has no means of accessing such data from this platform. In addition, when it comes to scraping data from your post/comments, Meta doesn’t need ActivityPub to do that. Anyone can read your profile and public posts as it is today.

Moderation - If a server hosts a substantial amount of harmful content without performing efficient and comprehensive moderation, it will create an excessive workload for our moderators. Currently, Meta is utilizing its existing Instagram moderation tools. Considering there were 95 million posts on the first day, this becomes worrisome, as it could potentially overwhelm us and serve as a sufficient reason for defederation.

Ads - It’s possible if Meta presents them as posts.

Promoting Posts - It’s possible with millions of users upvoting a post for it to trend.

Embrace, extend, and extinguish (EEE) - We don't think they can. If anyone can explain how they technically would, please let us know. Even if Meta forks Lemmy and gets rid of the original software, Lemmy will survive.

Instance Blocking - Unlike Mastodon, Lemmy does not provide a feature for individual users to block an instance (yet). This creates a dilemma where we must either defederate, disappointing those who desire interaction with Threads, or choose not to defederate, which will let down those who prefer no interaction with Threads.

Blocking Outgoing Federation - There is currently no tool available to block outgoing federation from lemmy.world to other instances. We can only block incoming federation. This means that if we choose to defederate with our current capabilities, Threads will still receive copies of lemmy.world posts. However, only users on Threads will be able to interact with them, while we would not be able to see their interactions. This situation is similar to the one with Beehaw at the moment. Consequently, it leads to significant fragmentation of content, which has real and serious implications.

Conclusion:
From the points discussed above, the possible lack of moderation alone justifies considering defederation from Threads. However, it remains to be seen how Meta will handle moderation on such a large scale. Additionally, the inability of individuals to block an instance means we have to do what is best for the community.

If you have any added points or remarks on the above, please send them to @[email protected].