Sometime early this morning, our database started having issues, most likely due to a disk corruption issue.

Unfortunately the postgres database was unrecoverable, and we had to restore from a nightly backup, so a few hours of posts and comments will be missing.

Apologies again for the downtime, and for the disruption.

  • v_krishna@lemmy.ml
    link
    fedilink
    English
    arrow-up
    17
    ·
    9 months ago

    How does lemmy federation work in this case? Conceivably after being restored from backup the lemmy.ml instance could see those few hours of lost history as federated to other lemmy instances and resync it back as the host instance. Obv I’m vastly oversimplifiying things but what happens today?

    • nutomic@lemmy.mlM
      link
      fedilink
      English
      arrow-up
      1
      ·
      9 months ago

      That doesnt work because we generally cant trust remote servers. Plus we dont even know where to fetch from, so wed have to run a complete crawl of all known instances which isnt practical.

  • spaphy@lemmy.ml
    link
    fedilink
    English
    arrow-up
    12
    arrow-down
    2
    ·
    9 months ago

    I AM TALKING IN ALL CAPS BECAUSE I AM MAD

    HOW DARE THIS FREE SERVICE I USE GO DOWN HOW EVER WILL I POST MEMES FROM MY OPAL THRONE

    Thanks for coming to my Ted talk.

    also thank you for hosting us and working on the issue!

  • Omega_Haxors@lemmy.ml
    link
    fedilink
    English
    arrow-up
    8
    ·
    edit-2
    9 months ago

    When I saw LemmyMl was down I went to sleep and had a dream where I was eating extremely large fries that that were labeled on the fryholder “Sadistically Large” (chicken fries as well as potato fries) and drinking probiotic shakes so dangerous they literally had to be bundled with a free antibiotic shake by law which of course I took one sip of in front of the clerk to calm them down before guzzling the other shake down with impunity. Woke up before I found find out what the burger was. Maybe those anti-woke types are on to something.

  • Ramin Honary@lemmy.ml
    link
    fedilink
    English
    arrow-up
    5
    ·
    edit-2
    9 months ago

    I saw the “502 Gateway” HTTP error message, but I never even closed my browser tab. I refreshed again today and Lemmy.ml was back online! Thanks for your hard work!