sh.itjust.works Main Community @sh.itjust.works TheDude @sh.itjust.works 9 mo. ago

Update on SJW upgrade to v19.3

Hello sh.itjust.works community,

Many of you have been eager to get an update about when the sh.itjust.works instance will get it's upgrade to the latest version of lemmy. Here's a update along with a tentative timeline.

In December 2023 I purchased a new server for this community. It took me awhile but I eventually made the time to get it racked at the local datacenter. For the sysadmins lingering and those interested here are the specs:

Dual Xeon 2.9Ghz CPUs (32 cores total)
256GB ram
4 x 1TB SSD in raid 10 (with room to add 6 more disks)
10gbit networking

While I'm ready to proceed with the upgrade, I've decided to first migrate this instance over to the new hardware. Here are two reasons.

Those of you who have been around long enough may remember that I've been running this instance on "borrowed" unused resources that were available at the time. There are no more resources available for this instance to grow.
There are reports that the latest version of lemmy may use more resources. Given we are among the bigger instances, should I end up in a situation where I need to increase resources to keep things fast I'll be restricted.

Here's the tentative timeline:

Task	Date	Expected Downtime
Migration to new server	Tuesday February 27 2024 @ 8:00PM ET	90 Minutes
Upgrade to V19.3	Thursday February 29 2024 @ 8:00PM ET	Up to 120 Minutes

If anything major goes wrong on the 27th I will revert back the changes and bring the instance back up on the current server.
If anything major goes wrong on the 29th I will revert back using an earlier snapshot. If that fails, I will restore from a backup.

During these two planned events those who want to provide moral support or who want to get periodic updates are more than welcome to join us on our matrix channel

=========================================================
Update February 29 2024
We've successfully completed the upgrade to v1.9.3. I'm happy to announce that we did it in an astonishing 27 minutes, a whole 93 minutes under what was expected. The extra leg work that was done over the last few weeks combined with the better hardware definitely played a part. Looking over the processes, it looks like the service responsible for images is still doing some work so it's possible that you will come across some broken images. I'll be keeping on eye on that over the next bit and make adjustments if needed. Thank you all for the support and to all of you who kept me company on our matrix channel. Have a good evening.

=========================================================
Update February 27 2024
We've successfully completed the migration. I'm happy to announce that this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected. Please give this instance a chance to catch up what it missed but we should be good within the next 30 or so minutes. Thank you

123

Fediverse @lemmy.world Blaze @reddthat.com 9 mo. ago

Update on SJW upgrade to v19.3

123 comments

So you bought a server for us? I really need to contribute financially
Every time I see SJW, I think "Social Justice Warrior" not "Shit Just Works."

Omnissiah's blessing on the upgrade @TheDude!
Some reason it never crossed my mind that someone could run their own instance on their own hardware and not like a cloud provider. That's super cool!
Good luck with the upgrade!
Thanks for running this place, and thanks for the info.
What do you expect me to do for that 90 and 120 minutes??
Good luck. Yours brought me to Lemmy and then I made my own soon after.
Thank you for all your time and effort.
The Dude, as per usual, abides.

Thank you again for all you do.
Thank you for everything you are doing. Take a few days if you have to fit this migration and update. I've hosted many servers... I always feel I need to do so much for everyone. Curb that thought, if you can. We genuinely appreciate what you're doing here.
Lol SJW
Not on your instance but good luck!
I appreciate everything you do for this instance. Also, thanks for posting the specs of the new build. It’s pretty neat to see what’s going on behind the curtain.
Good luck - I previously moved Feddit.dk to a new server and recently upgraded to 0.19.3 without any problems during either upgrade - hope you have the same experience!
Good luck ! We did it for jlai.lu, it was a long week-end :)
Wow, so new and snappy, smells like fresh cut mahogany in here
Thank you. We love you very much!
We can't thank you enough and we all love you. You're a rock star.
Thank you for everything you do!
Thanks Dude
Dude, you're the Scotty of the fediverse, you rock.
Question. Why would you go with 1tb ssds instead of larger hdds? Isn't the space and price more important than the speed for this use?

You could get double the space (2tb hdd) for the same price as a 1 tb ssd.

Just wondering.
this instance is now running on its new hardware dedicated solely to this community! We experienced just under 40 minutes of downtime which is a whole 50 minutes less than expected

Woohoo, that's awesome! Thank you so much!
Not on your instance, but wishing you luck on the migration and updates!
Thanks for the update.

Looking forward to the new update.
Good luck from one small instance that just moved VPS providers and upgraded to 0.19.3. I ho no issues except for email died and I had to figure out the new hosts email gateway that was lacking lacking in some of it's documentation
Looks like the site was down this morning. Was the migration done earlier ? or just some downtime ? Good luck for the migration, It's an opportunity to use my alt account on fedia, so not that bad
Woohoo!

Now, can anyone point out what -if anything- changed for us end users?
Just want to say Good Luck and we're all counting on you!
Nice job on the migration! Excited for the update!
Please let us throw monies at you
Félicitation à toute l'équipe, merci pour votre travail
I appreciate everything being done on our behalf. Thanks Dude!
Dang, I was looking forward to being productive today, but you were just too quick.
That's a relatively low spec server, I thought it would be worse. What does the co-location cost?
How can I help, my friend?
Good luck, Dude! I'm sooo looking forward to seeing what I previously upvoted.
So how much do you pay for the server?
GG
Good luck with the update in (if I'm right in the timezone calculation) a couple of hours...
Thank you!!!
I've encountered a problem, not sure if it's related to the update or not. On both Eternity and browser my Subscribed feed shows posts from All. From Eternity only upvoting posts fails with a 401 code, but I seem to be able to do it on the browser. Any one else encountered some like that?
Lets go! Congrats on the upgrade of the hardware!
Great work Dude!

Looks like pict-rs uploads by my account from today are working, still waiting on pictures from a few days ago and beyond. (Well prioritized, in that case.)

Thank you!
How does one go about the blocking whole instances in 19.3?
I know Scaled isn't selectable in Sync so you have to set it as default in Lemmy settings to use it. Does anyone know what other new features have no/limited compatibility with Sync? Or any other app for that matter?

Edit: I already had the sort for subscribed and everything overridden so I had to figure out how to clear that (since scaled isn't an option when I try to change the override) but now it's good.

The trick is to hit the correct cancel btw 🙄
Pictures don't seem to show up... but I'm guessing this is a caching issue, so it should be resolved by itself.
Wow congrats ! I'm glad it went well :D
Can we please unpin this now? It’s the top of all of my feeds still.

You've viewed 123 comments.