Lemmy maintainer

  • 36 Posts
  • 147 Comments
Joined 5 years ago
cake
Cake day: January 17th, 2020

help-circle








  • nutomic@lemmy.mlOPMtoAnnouncements@lemmy.mlLemmy AMA March 2025
    link
    fedilink
    English
    arrow-up
    9
    ·
    27 days ago

    The stack is great, I wouldnt want to change anything. Postgres is very mature and performant, with a high focus on correctness. It can sometimes be difficult to optimize queries, but there are wizards like @dullbananas@lemmy.ca who know how to do that. Anyway there is no better alternative that I know of. Rust is also great, just like Postgres it is very performant and has a focus on correctness. Unlike most programming languages it is almost impossible to get any runtime crashes, which is very valuable for a webservice.

    The high performance means that less hardware is required to host a given number of users, compared to something like NodeJS or PHP. For example when kbin.social was popular, I remember it had to run on multiple beefy servers. Meanwhile lemmy.ml is still running on a single dedicated server, with much more active users. Or Mastodon having to handle incoming federation activities in background tasks which makes the code more complicated, while Lemmy can process them directly in the HTTP handler.

    Nevertheless, scaling for more users always has its surprises. I remember very early in development, Lemmy wasnt able to handle more than a dozen requests per second. Turns out we only used a single database connection instead of a connection pool, so each db query was running after that last one was finished, which of course is very slow. It seems obvious in retrospect, but you never notice this problem until there are a dozen or so users active at the same time.

    With the Reddit migration two years ago a lot of performance problems came up, as active users on Lemmy suddenly grew around 70 times. You can see some of that in the 0.18.x release announcements. One part of the solution was to add missing database indexes. Another was to remove websocket support, which was keeping a connection open for each user. That works fine with 100 users, but completely breaks down with 1000 or more.

    After all there is nothing I would do different really. It would have been good to know about these scaling problems earlier, but thats impossible. In fact for my project Ibis (federated wiki) Im using the exact same architecture as Lemmy.














  • Mainly SEO spam with text copied from other sites and lots of ads/referral links to make the owner a profit. But after thinking about it more, those would be rather easy to filter based on ad code in the HTML.

    A much bigger challenge will be the ranking of search results. When searching for a term and there are 100 pages in the index that contain it, which of these pages should be shown first? Google developed the Pagerank when they started out, so that might be a good starting point to research further.


  • This sounds like a very interesting idea. I agree that Yacy doesnt work, when I checked it out years ago it was a completely bloated mess. Not sure how viable how your idea is, because Im not familiar with webrings, and not sure how the federation will work. Anyway the main challenge for this project will be to actually give useful search results, both early on when there are very few crawlers, and also later once spammers try to abuse it.



  • Funny, Mastodon just posted a similar thing about creating a foundation. But the problem is, the existence of a foundation does nothing to prevent billionaires from controlling social media. For billionaires its very easy to donate a few hundred thousand USD to the foundation and gain influence that way. I expect that Bluesky will be fine for the first years (maybe like early Twitter), but sooner or later the foundation will take decisions that the users dont like, and there is nothing they can do about it.

    In my view, the only way to avoid influence from billionaires is to avoid any large centralized structures. In the Fediverse there are dozens of platforms and thousands of instances. Even if a billionaire were to take control over a couple of projects or large instances, people would create forks in a matter of days. Some admins would block these corrupted instances, and their users would barely notice that anything changed.

    So Bluesky is just trying to repeat something that has already failed. The Fediverse is the future, but it will take a long time for most people to understand that.