lwadmin

Hey everyone, so as I'm sure everyone is aware Lemmy.World has been experiencing several outages throughout the last few days.

We have been investigating the root cause of these outages but believe that they are related to our current hosting provider (Hetzner) blocking access from ClouldFlare as (we think) they believe that our CDN is a DDoS'er, and is causing these disconnects to our backend server, problematic for sure.

We've opened support tickets with our current provider and are awaiting a response. We have no issue with being as transparent as possible with downtime. Anyone that is curious, can feel free to check out https://status.lemmy.world and https://dash.lemmy.world for up to the minute outage information. We are also looking into other fediverse friendly methods of posting status and outage updates

In the meantime, we are evaluating alternative hosting options and solutions to provide a high level of reliability to you, our users. Really, we want to say thanks to everyone for soldiering through all our technical growing pains.

Cheers

• LW Infra Team

Earlier, after review, we blocked and removed several communities that were providing assistance to access copyrighted/pirated material, which is currently not allowed per Rule #1 of our Code of Conduct. The communities that were removed due to this decision were:

• [email protected]
• [email protected]
• [email protected]

We took this action to protect lemmy.world, lemmy.world's users, and lemmy.world staff as the material posted in those communities could be problematic for us, because of potential legal issues around copyrighted material and services that provide access to or assistance in obtaining it.

This decision is about liability and does not mean we are otherwise hostile to any of these communities or their users. As the Lemmyverse grows and instances get big, precautions may happen. We will keep monitoring the situation closely, and if in the future we deem it safe, we would gladly reallow these communities.

The discussions that have happened in various threads on Lemmy make it very clear that removing the communites before we announced our intent to remove them is not the level of transparency the community expects, and that as stewards of this community we need to be extremely transparent before we do this again in the future as well as make sure that we get feedback around what the planned changes are, because lemmy.world is yours as much as it is ours.

Hello there!

It has been a while since our last update, but it's about time to address the elephant in the room: downtimes. Lemmy.World has been having multiple downtimes a day for quite a while now. And we want to take the time to address some of the concerns and misconceptions that have been spread in chatrooms, memes and various comments in Lemmy communities.

So let's go over some of these misconceptions together.

"Lemmy.World is too big and that is bad for the fediverse".

While one thing is true, we are the biggest Lemmy instance, we are far from the biggest in the Fediverse. If you want actual numbers you can have a look here: https://fedidb.org/network

The entire Lemmy fediverse is still in its infancy and even though we don't like to compare ourselves to Reddit it gives you something comparable. The entire amount of Lemmy users on all instances combined is currently 444,876 which is still nothing compared to a medium sized subreddit. There are some points that can be made that it is better to spread the load of users and communities across other instances, but let us make it clear that this is not a technical problem.

And even in a decentralised system, there will always be bigger and smaller blocks within; such would be the nature of any platform looking to be shaped by its members. 

"Lemmy.World should close down registrations"

Lemmy.World is being linked in a number of Reddit subreddits and in Lemmy apps. Imagine if new users land here and they have no way to sign up. We have to assume that most new users have no information on how the Fediverse works and making them read a full page of what's what would scare a lot of those people off. They probably wouldn't even take the time to read why registrations would be closed, move on and not join the Fediverse at all. What we want to do, however, is inform the users before they sign up, without closing registrations. The option is already built into Lemmy but only available on Lemmy.ml - so a ticket was created with the development team to make these available to other instance Admins. Here is the post on Lemmy Github.

Which brings us to the third point:

"Lemmy.World can not handle the load, that's why the server is down all the time"

This is simply not true. There are no financial issues to upgrade the hardware, should that be required; but that is not the solution to this problem.

The problem is that for a couple of hours every day we are under a DDOS attack. It's a never-ending game of whack-a-mole where we close one attack vector and they'll start using another one. Without going too much into detail and expose too much, there are some very 'expensive' sql queries in Lemmy - actions or features that take up seconds instead of milliseconds to execute. And by by executing them by the thousand a minute you can overload the database server.

So who is attacking us? One thing that is clear is that those responsible of these attacks know the ins and outs of Lemmy. They know which database requests are the most taxing and they are always quick to find another as soon as we close one off. That's one of the only things we know for sure about our attackers. Being the biggest instance and having defederated with a couple of instances has made us a target.  

"Why do they need another sysop who works for free"

Everyone involved with LW works as a volunteer. The money that is donated goes to operational costs only - so hardware and infrastructure. And while we understand that working as a volunteer is not for everyone, nobody is forcing anyone to do anything. As a volunteer you decide how much of your free time you are willing to spend on this project, a service that is also being provided for free.

We will leave this thread pinned locally for a while and we will try to reply to genuine questions or concerns as soon as we can.

Edit: Upgrade finished

We will upgrade lemmy.world to v0.18.4 tomorrow at 20:00 UTC+2 (Check what this isn in your timezone. Expect the site to be down for a few minutes.

Release notes for 0.18.4 can be found here: https://github.com/LemmyNet/lemmy/blob/main/RELEASES.md

Lemmy.World is looking for 4 new Systems operators to help with our growing community.

Volunteers will assist our existing systems team with monitoring and maintenance.

We’re ideally looking for chill folks that want to give back to their community and work on our back-end infrastructure. Must have 4+ years of professional experience working in systems administration. We are not looking for junior admins at this time. Please keep in mind that, while this is a volunteer gig, we would ask you to be able to help at least 5-10 hours a week. We also understand this is a hobby and that family and work comes first.

Applicants must be okay with providing their CV and/or LinkedIn profile AND sitting for a video interview. This is due to the sensitivity of the infrastructure you will have access to.

We are an international team that works from both North America EST time (-4) and Europe CEST (+2) so we would ask that candidates be flexible with their availability.

If you are in AEST (+10) or JST (+9) please let us know, as we are looking for at least one Sysadmin to help out during our overnight.

You may be asked to participate in an on-call pool. Please keep in mind that this is a round-robin style pool, so it's alright if you're busy as it will just move along the chain.

If you're interested and want to apply, click here.

We recently received a message from a concerned Rammy user regarding their instance not having an active admin team.

We have made attempts to contact the Rammy admins, which other instance admins have tried as well, to determine their current status. Due to their admins being absent and their unmoderated content growing in numbers, we will defederate from Rammy. If and when this situation changes, we will be happy to reevaluate our approach. It should be noted that any instances that have abandoned admin teams will be defederated.

Update:
The comments from this post will not be removed as to preserve the discussion around the announcement. Any continued discussions outside of this thread that violate server rules will be removed. We feel that everyone that has an opinion, and wanted to vent, has been heard.

————-

Original post:
Yesterday, we received information about the planned federation by Hexbear. The announcement thread can be found here: https://www.hexbear.net/post/280770. After reviewing the thread and the comments, it became evident that allowing Hexbear to federate would violate our rules.

Our code of conduct and server rules can be found here.

The announcement included several concerning statements, as highlighted below:

• “Please try to keep the dirtbag lib-dunking to hexbear itself. Do not follow the Chapo Rules of Posting, instead try to engage utilizing informed rhetoric with sources to dismantle western propaganda. Posting the western atrocity propaganda and pig poop balls is hilarious but will pretty quickly get you banned and if enough of us do it defederated.”
• “The West's role in the world, through organizations such as NATO, the IMF, and the World Bank - among many others - are deeply harmful to the billions of people living both inside and outside of their imperial core.”
• “These organizations constitute the modern imperial order, with the United States at its heart - we are not fooled by the term "rules-based international order." It is in the Left's interest for these organizations to be demolished. When and how this will occur, and what precisely comes after, is the cause of great debate and discussion on this site, but it is necessary for a better world.”

The rhetoric and goal of Hexbar are clear based on their announcement: to "dismantle western propaganda" and "demolish organizations such as NATO” shows that Hexbar has no intention of "respecting the rules of the community instance in which they are posting/commenting.” It’s to push their beliefs and ideology.

In addition, several comments from a Hexbear admin, demonstrate that instance rules will not be respected.

Here are some examples:

“I can assure you there will be no lemmygrad brigades, that energy would be better funneled into the current war against liberalism on the wider fediverse.”

“All loyal, honest, active and upright Communists must unite to oppose the liberal tendencies shown by certain people among us, and set them on the right path. This is one of the tasks on our ideological front.”

• https://lemmy.world/comment/121850
• https://lemmy.world/comment/1487168
• https://lemmy.world/comment/1476084
• https://lemmy.world/comment/171595
• https://www.hexbear.net/comment/3648500

Overall community comments:

• https://www.hexbear.net/comment/3526128
• https://www.hexbear.net/comment/3526086
• https://www.hexbear.net/comment/3652828

To clarify, for those who have inquired about why Hexbear versus Lemmygrad, it should be noted that we are currently exploring the possibility of defederating from Lemmygrad as well based on similar comments Hexbear has made.

• https://lemmygrad.ml/post/158656
• https://lemmygrad.ml/comment/882559
• https://lemmygrad.ml/comment/540170
• https://lemmygrad.ml/comment/446529

Defederation should only be considered as a last resort. However, based on their comments and behavior, no positive outcomes can be expected.

We made the decision to preemptively defederate from Hexbear for these reasons. While we understand that not everyone may agree with our decision, we believe it is important to prioritize the best interests of our community.

We have received numerous reports from users about the closure of the c/android community. While we fully support the original community owners’ decision to move to another instance, it will eventually be necessary to open up the community on Lemmy.world. The beauty of the fediverse is that multiple communities on the same subject can exist in different instances. However, if you can no longer moderate a community on Lemmy for any reason, it is important to pass it on to individuals who are willing and able to do so.

To ensure the best interests of our instance members, it is necessary to establish boundaries. Holding onto a community name cannot be a permanent arrangement. It’s important to consider our users’ ongoing interest in the community if they wish it to continue. While we acknowledge the objective of consolidating communities, current community members ultimately decide whether they wish to join the new community at lemdro.id.

To ensure a smooth transition, we will keep the community locked for another week, providing ample time to inform the active user base about the move to the new instance at [email protected].

We have received numerous reports from users about the closure of the c/android community. While we fully support the original community owners' decision to move to another instance, it will eventually be necessary to open up the community on Lemmy.world. The beauty of the fediverse is that multiple communities on the same subject can exist in different instances. However, if you can no longer moderate a community on Lemmy for any reason, it is important to pass it on to individuals who are willing and able to do so.

To ensure the best interests of our instance members, it is necessary to establish boundaries. Holding onto a community name cannot be a permanent arrangement. It's important to consider our users' ongoing interest in the community if they wish it to continue. While we acknowledge the objective of consolidating communities, current community members ultimately decide whether they wish to join the new community at lemdro.id.

To ensure a smooth transition, we will keep the community locked for another week, providing ample time to inform the active user base about the move to the new instance at https://lemmy.world/c/[email protected].

There has been significant discussion in recent weeks regarding Meta/Threads. We would like to express our disappointment with the negative and threatening tone of some of these discussions. We kindly ask everyone to engage in civil discourse and remember that not everyone will share the same opinions, which is perfectly acceptable.

When considering whether or not to defederate from Threads, we're looking for a decision based on facts that prioritize your safety. We strive to remain neutral to make an informed choice.

First, there seem to be some misconceptions about how the Fediverse operates based on several posts. We’ve compiled some resource links to help explain the details and address any misunderstandings.

Fed Tips , Fediverse , ActivityPub

Initial Thoughts:

It seems unlikely that Meta will federate with Lemmy. When/if Meta adopts ActivityPub, it will likely affect Mastodon only rather than Lemmy, given Meta's focus on being a Twitter alternative at the moment.

Please note that we have a few months before Threads will even federate with Mastodon, so we have some time to make the right decision.

Factors to Consider:
Factors to consider if Meta federates with Lemmy:

Privacy - While it’s true that Meta's privacy settings for the app are excessive, it’s important to note that these settings only apply to users of the official Threads app and do not impact Lemmy users. It’s worth mentioning that Lemmy does not collect any personal data, and Meta has no means of accessing such data from this platform. In addition, when it comes to scraping data from your post/comments, Meta doesn’t need ActivityPub to do that. Anyone can read your profile and public posts as it is today.

Moderation - If a server hosts a substantial amount of harmful content without performing efficient and comprehensive moderation, it will create an excessive workload for our moderators. Currently, Meta is utilizing its existing Instagram moderation tools. Considering there were 95 million posts on the first day, this becomes worrisome, as it could potentially overwhelm us and serve as a sufficient reason for defederation.

Ads - It’s possible if Meta presents them as posts.

Promoting Posts - It’s possible with millions of users upvoting a post for it to trend.

Embrace, extend, and extinguish (EEE) - We don't think they can. If anyone can explain how they technically would, please let us know. Even if Meta forks Lemmy and gets rid of the original software, Lemmy will survive.

Instance Blocking - Unlike Mastodon, Lemmy does not provide a feature for individual users to block an instance (yet). This creates a dilemma where we must either defederate, disappointing those who desire interaction with Threads, or choose not to defederate, which will let down those who prefer no interaction with Threads.

Blocking Outgoing Federation - There is currently no tool available to block outgoing federation from lemmy.world to other instances. We can only block incoming federation. This means that if we choose to defederate with our current capabilities, Threads will still receive copies of lemmy.world posts. However, only users on Threads will be able to interact with them, while we would not be able to see their interactions. This situation is similar to the one with Beehaw at the moment. Consequently, it leads to significant fragmentation of content, which has real and serious implications.

Conclusion:
From the points discussed above, the possible lack of moderation alone justifies considering defederation from Threads. However, it remains to be seen how Meta will handle moderation on such a large scale. Additionally, the inability of individuals to block an instance means we have to do what is best for the community.

If you have any added points or remarks on the above, please send them to @[email protected].

We're closing this thread. Everything that could be said has been said. Thank you

Original Post:
Today, we want to inform everyone that we have decided to defederate from https://exploding-heads.com/. We understand that defederating should always be a last resort, and individuals can certainly block communities. However, blocking alone does not prevent potential harm to vulnerable communities.

After carefully reviewing the instance, reported posts, and multiple comments from the community, we have concluded that exploding-heads is not adhering to the Lemmy or Citizen Code of Conduct. Therefore, we cannot, in good faith, continue to federate with an instance that consistently promotes hate, racism, and bullying.

Examples:
https://lemmy.world/post/577526 - Community Moderator Harassment
https://exploding-heads.com/post/92194 - Systemadmin Post
https://exploding-heads.com/post/90780 - Systemadmin Post
https://exploding-heads.com/post/91488 - Systemadmin Post
https://exploding-heads.com/post/93725 - Community Moderator Post

Again, deciding to defederate from an instance is not taken lightly. In the future, we will continue to review instances on a case-by-case bases.

As for our community, please refrain from posting or commenting with hateful words as well. Arguing back and calling people names is not the solution. The best course of action is to report the posts or comments violating our server rules.

Lemmy Code of Conduct
https://join-lemmy.org/docs/code_of_conduct.html
Citizen Code of Conduct https://github.com/stumpsyn/policies/blob/master/citizen_code_of_conduct.md

“We are committed to providing a friendly, safe, and welcoming environment for all, regardless of level of experience, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, nationality, or other similar characteristic.”