This is the second time today I've had to cut over to the backup WAN (which is much slower). Fiber provider is royally screwing the pooch today, and I currently have no ETA. The only silver lining is it's a widespread issue, so hopefully they're inclined to address the problem in a timely manner.

To relieve congestion on the backup connection, I've temporarily disabled external access to pictrs. All requests (GET or POST) to /pictrs will return a 404 response until the main connection is back online.

Sorry for the inconvenience. Please direct all rage at the shit fiber provider I foolishly "upgraded" to.

Status as of 5:17PM:

Appalachian Power is the absolute worst about giving notifications for scheduled outages. The. Fucking. Worst.

I got an email at at 8:05 PM saying there would be a planned, scheduled outage at 8:00 PM and then 45 seconds later was plunged into darkness. Didn't bother to switch on the generator :sigh: and just decided to sit on the patio and get drunk in the waning daylight.

This is not the first time they've done that, and it probably won't be the last. Guess my next purchase is an automatic transfer switch.

Did I say "fuck Appalachian Power"? Because Fuck. Appalachian. Power.

Sorry about that. lol

Let me start off by saying, in my official admin capacity, that I really don't know if I want to be here anymore (see the problem outlined below). That has implications beyond me just deleting my account and going outside since I would not allow the instance to run unattended. I'm also unsure if my backup admin would have any interest in taking over full time.

If it does come to that, there will be a subsequent announcement and either a transition plan or a sunset plan.

That is still up in the air, but for now....

The Problem

I just wanted to take a moment here to address Lemmy's extremism problem / lynch-mob mentality and how DubVee is responding to that.

Last month, a new site rule was added expressly forbidding any form of extremism with regards to violence. Every rule is there for a reason, and this was added as a direct response to the increasingly violent rhetoric I've been seeing from users.

In full:

I don't care which end of the spectrum you're on. Any post/

This Weekend

There will be some periodic downtime this weekend as I do some physical cleanup / rearrangement to my equipment. I'll try to keep these to a minimum, but at some point, everything is going to need to go offline for a bit in order to re-cable the UPS and power connections.

Plan is Saturday afternoon between 1 and 4 pm. Hopefully any interruptions will be brief.

Next Week

I'm switching to a new fiber provider next week (FTTH vs current FTTN), so there will be a brief disruption as I cut over to that. That downtime should only be a matter of seconds since both connections will be active, but just in case it turns into a whole thing, I'm mentioning it now. The new fiber is supposed to be installed Tuesday afternoon, and I plan to do the cutover around 7PM local time.

Next Few Months

I'll be expanding the storage server in the coming weeks, but that should not have any immediate impact on DubVee's normal operation.

With these, I hope to provide a better

I've been working on the latest release of Tesseract for the last few weeks, and I'm happy to announce the first beta release of the upcoming 1.4.0 series.

I daily drive my dev version throughout the development process, so all the bugs I've encountered/introduced should be fixed. The beta test on the main instance is more of a formality and final shakedown. That said, please report any bugs you encounter either through Github or by describing them in a comment here.

1.4.0 is a significant update as it drops all backwards compatibility with the 0.18.x API and introduces several new features. It's also had some additional polish since 1.3.0 and added things that have been requested for quite some time (pasting images into post/comments, etc).

As always: thanks in advance for being my beta testers. Appreciate it!

Change log for 1.4.0 if you want to see what's new: https://github.com/asimons04/tesseract/blob/1.4.0/ChangeLog.md

If you've noticed content from Lemmy World lagging by a few hours the past few days, it's not just you.

Long story short, it's a problem with how Lemmy sends activities, and it's heavily impacted by latency between sending and receiving server and creates a fixed upper limit on the number of activities per second that can be sent. Lemmy World is hosted in Finland, and DubVee on the US east coast. There's only so much I can do to work around the inherent latency of a trans-Atlantic link.

We're not alone in this. Some instances, particularly in Australia, New Zealand, and US west coast, are impacted more dramatically.

While there's always some federation delay/backlog, it only escalates to noticeable levels every so often. Not sure why, but I've gone over our infrastructure top to bottom several times and cannot find any reason, on this end, for these events. The graph in the post shows the number of activities Dubvee is lagging behind Lemmy World over the last 30 days.

Nor

Finally got around to upgrading to 0.19.3.

Upgrade did not go smoothly due to multiple DB migration step failures and less than helpful logging from Lemmy during the DB migrations. (surprised Pikachu).

After digging through to figure out what it was complaining about, the upgrade was finally able to be completed.

You will likely need to log in again (at least I did).

I'm in the process of migrating the pict-rs database to Postgres. Pict-rs has to be offline during this process, so images here will appear to be broken for a bit. You also won't be able to upload any images for the duration of this migration.

Estimated time is about 20-30 minutes.

Currently, we host 4 UIs:

• Tesseract (default UI at https://dubvee.org)
• Mlmym (old Reddit looking UI at https://old.dubvee.org)
• Alexandrite (desktop UI at https://alex.dubvee.org)
• Lemmy-UI (crappy "default" UI at https://lemmy.dubvee.org).

Due to all the bot crawler traffic that slips through, I'm having a hard time distinguishing actual usage from bot traffic. That said, it does seem like Mlmym and Alexandrite are rarely used by actual people (there's a lot of bot traffic to filter out that spoofs its user agent, so I could easily be wrong here).

Update: Mlmym does all the Lemmy API calls server-side, so all of the actual usage traffic was actually originating from my server (which I was filtering out).

I'm looking to decommission Mlmym (the old Reddit style) and Alexandrite.

I just updated both of those to the latest, so apologies for the brief disruption if you were using them, but ultimately I'd like to trim down the selection to just the ones being actively

I'm not quite done with the 1.3.0 release of Tesseract, but the last few bits are going to take longer than I expected. So I think I'm going to hold off and add those in a 1.3.x release a bit later.

I've gutted, re-implemented, and just flat-out re-wrote large portions of the application over the last month or two. I've also been daily driving the dev version, and I guess a couple other people were too. That said, it is stable enough for daily use and ready to get some more eyes on it before an official release.

So, please bear with me if you hit any bugs (I've fixed all the ones I've found so far). If you would be willing and kind enough, either submit a Github issue if you find a bug or just throw a reply to this post with as many details as you can provide.

Hope to run a few betas here before officially releasing 1.3.0.

Apologies for the downtime.

We had some severe weather come through on Tuesday that caused extensive damage to power infrastructure throughout the area. I thought my location had been spared the worst of it, but a tree decided to fall on the transmission line servicing this area Wednesday morning (according to the power company, anyway). By that point, there were about 55,000 other outages ahead of my area in line awaiting repairs (no hard feelings there).

Since power outages are rare here, at least ones lasting longer than an hour or two, I only keep about 2 gallons of fuel on hand for the generator. At typical 3/4 load, that usually last about 4-5 hours. Again, long outages are extremely rare here, so that's usually more than enough runtime. Sadly, this outage lasted much longer (even though I was able to stretch the generator runtime by slightly hacking my UPS)

Federated content should now be coming back in, but it'll take a while to catch up.

Lessons learned:

• Mother n

Weather / power related. May go offline for a bit until power is restored. Currently on UPS.

Update 1: Yep, expect downtime. UPS ran dry and moved over to aux generator. ETA from power company is tomorrow at 11 PM EST.

Update 2: Still no change in ETA. Have had to shed some load from the generator because my UPS's refuse to charge from it and every time the fridge kicks on, a random piece of my equipment would reboot :sigh:.

Hopefully my reduced power budget will stabilize that.

On the bright side, the primary network connection has remained up and haven't had to switch over to the (much slower) backup.

Update #3: (17:06) Generator running on fumes now and will probably be going offline soon to ensure a safe shutdown and save some fuel to cool the fridge tomorrow if power is still out. Apologizes in advance. I usually have very reliable power here and typically never need the generator for more than a few hours.

Update #4 (04/04/2024 14:48): Power restored 12 hours a

Lemmy World

There was an issue today with content from Lemmy World not federating to us. After pulling my hair out and testing the DubVee stack top to bottom, I got in touch with the awesome admins over at LW to try to figure out the problem. After they gave their federation service workers a good, stern talking to, we're now receiving content again.

Unfortunately, due to the amount of time (about 4 hours) the federation messages were getting stuck, there is a huge backlog that needs to catch up: about 90,000 messages. As of this writing, there are still 55,024 queued ActivityPub messages in flight (that includes backlog and current messages).

So just a heads up if you're not seeing posts/votes/comments from there or if they don't show up for a while. They will eventually arrive, though they'll be in the feed based on when they were published rather than when they arrive. On the bright side, they are receiving posts/votes/comments from us; you just might not get a reply ri

If you've noticed a large number of bulk post removals performed by me in the modlog recently and are wondering if I've gone full authoritarian: no, I haven't gone mad with power. Those were posts that were already removed by other community mods or posts that were self-deleted by their creators. Some were legitimate spam or otherwise violated server rules, but the bulk of them were just cleanup.

"If they're already removed/self-deleted, why remove them again?", you might ask.

Well, Lemmy treats Pictrs (the media subsystem, basically) like a black hole - images go in but it never removes them. When posts are modded or self-deleted, any media attached to them lives forever in pictrs with no clean way to remove them later*. That wastes a huge amount of disk space on my hosting stack for media that will never see the light of day again.

I'm not okay with that for so many reasons. Yeah, object storage is cheap, but why be wasteful?

The Lemmy + Pictrs integration...well, let'

Apologies for the brief ~20 minute outage. Had a loss of power and one of the two UPS's failed without warning. Had to manually move equipment over to the remaining UPS until power is restored. The cold boot takes frigging forever and was responsible for the bulk of those 20 minutes, but everything should now be back up and running.

We also lost the primary internet connection and are running on the failover, so things may be a bit sluggish.

Utility company said a tree was at fault, so will likely have both electric and network service restored in an hour or two. Unless UPS-A decides to die on me too.

If you've noticed that you're landing on tesseract.dubvee.org by default now, it's because I've done some restructuring to Dubvee's infrastructure to take advantage of the new features I've written into Tesseract. It also means you will need to login to your account again. Sorry about that, but it's a small price to pay for the benefits we'll see.

If you have Dubvee installed as a PWA, you'll probably also need to "re-install" it from the tesseract. domain. It'll redirect gracefully, but unless you resinstall, it will have a title bar and not look app-like. Again, my apologies.

Why?

I've added image proxying and caching to the Tesseract server process which allows it to act as a pseudo-CDN. This takes a significant burden off of my Lemmy + Pict-rs server while also making images and other media load faster for users. It also reduces the load on other Lemmy instances by not repeatedly fetching the same media. Win/win/win!

Additional Perks

I've left this Tesseract

They've got a massive spam problem, and their mod actions do not federate out. So, while they may be cleaning up spam on their end, none of that federates to Lemmy and we get stuck cleaning up the mess in its entirety.

Have had good experiences with users from kbin.social, so it's not them that's the problem. Looking at the magazines on the Kbin side, they are cleaning up spam, so the mods/admins are on the ball. But without those cleanup actions federating out, they're still contributing to fediverse-wide spam that everyone else has to deal with.

Also, Kbin doesn't seem to have the concept of a registration application, only a captcha, which seems to be a very low barrier to entry for the spam bots that keep popping up there.

https://codeberg.org/Kbin/kbin-core/issues/570

Once that issue with Kbin core is addressed and mod actions from Kbin start federating, we will resume federation with kbin.social.

Update: We've re-federated but have removed all communities

Figured it was time for another status update. Will try to keep it brief.

System/Infrastructure Status

We are within allowances for all metered resources (bandwidth, disk, etc) and will continue to be so for the foreseeable future. Translated: it still isn't costing me any extra money to host DubVee.

Image Uploads Have Been Re-Enabled (With Limits)

Image uploads have been re-enabled but with a limit of 150kb. This is mostly to support setting your profile avatars, but I don't have enough granular control to limit it to just that purpose. As such, you can add images to posts/comments as long as the file you're uploading is less than 150KB.

Consolidating Lemmy Frontends

For quite some time, I've offered just about every alternative Lemmy front-end available. Logs have shown some to never be used, including by me. To simplify administration, I've removed Voyager and Photon from the offerings.

The remaining options are:

• [Alexandrite](https://alex.dubvee.org

Now that this instance has been running for a little over two months, I thought it was a good time to zoom out a bit and do a little reflection.

Two Month Reflections

First, I cannot believe it's only been two months; it seems way longer than that (in both a good way and an "I'm so tired, you guys" way). While I do run other public-facing services, they're much more hands-off as far as day-to-day work is concerned. Lemmy, on the other hand, is very much like a living organism, almost a pet, that requires near constant supervision, interaction, and attention. The codebase is being rapidly developed, updated, and patched, new vulnerabilities and rough edges being discovered and mitigated, and bad actors constantly looking to exploit, DoS, spam, troll, or just shit all over the place.

It's not all bad, though. In the same two months, many new and exciting projects have sprung up to give Lemmy a fresh face with new UIs. DubVee is happy to offer several of those as official

Did you know there's more than one way to Lemmy? Since Lemmy usage has taken off, so have the projects to develop alternate frontends. When one seems promising, we will usually adopt it as an option.

Below are the official and 3rd party Lemmy user interfaces supported by DubVee:

Lemmy UI

The default Lemmy experience through the official frontend.

• URL: https://dubvee.org
• Github: https://github.com/LemmyNet/lemmy-ui

Photon

A sleek, mobile and desktop friendly interface that can be installed as a PWA (progressive web app). This is currently my "editor's choice" for Lemmy mobile access.

• URL: https://m.dubvee.org
• Github: https://github.com/Xyphyn/photon

Voyager

A progressive web app (PWA) designed for mobile. Has both an Android and iOS (default) skin.

• URL: https://voyager.dubvee.org
• Github: https://github.com/aeharding/voyager/blob/main/Dockerfile

Mlmym

An old-Reddit style UI for Lemmy. Definitely nostalgic and scratches the "R