What's up? - Sunday weekly

No issues at the moment but need to update a few containers when I get the chance. I also need to set up contacts sync in radicale for the address book and integrate it with Thunderbird and davdroid.
In the near term I've been working on a plan to make sure my keepass db is accessible to my SO and family in the event of my demise. I recently lost a dear friend and had to gain access to his stuff for his family, luckily he didn't have the linux partition encrypted so I got a recovery shell then remounted the disk and changed the password and could then also mount the windows partition once I logged in.
It made me think as all my stuff is encrypted and there is no way someone would guess it nor crack it so I'm writing documentation and leaving it with family members.
The documentation explains how to use keepass and who to contact for support. Im leaving the db with family members and the password with a select few people that dont have the db. My SO will have access to all the info too.
I'll update the db periodically and give them a newer version but keep the same password
I encourage you all to consider this too.

I'll just start! Personally, I'm tinkering with my local network to create a subnet for my homelab.
I want to set up Lemmy and Audiobookshelf next, but I want to tweak the infrastructure a bit before hosting more stuff.
Before the firewall thing, I set up authentik and am integrating it in more services. Migration was mostly straightforward so far in Bookstack and Paperless. Also the proxy authentication is pretty cool, finally being able to ditch basic auth in Prometheus was cool.
- Majority of openrc/hardened/selinux binhost setup is done, need to figure out the small things.
  Lemmy was also giving a bit of a headache, fiddled with limits some more.
  I'm fairly certain there's been an attempt to play with some opnsense config, but there was only time to install the updates. Or maybe this was last week 🤔

I'm currently reconsidering using a couple mikrotik for some layer 3 hardware offloading.
Not really homelab, but close.
I have a project that gets integrated with another network for an event. I'm thinking of using 2x crs504 (cause I'm using mlag for servers, think vrrp or whatever for "public" (it's all internal) ip) and seeing if I can get l3hw working as a router.
While I could sit on a subnet of the "host" network, having a gateway that traffic goes through allows me to test and prove everything for my system in my homelab, with just the final integration being a do-in-a-time-crunch problem.
I'm already using the crs504s for networking (I bought them ages ago, thinking 25gbps was going to be as easy as 10gbps. It's all running at 10gbps), and this saves having to use something as a router, cuts down on rack space, all sorts of benefits. I think.
Anyone have any experience with mikrotik l3hw offloading?
My actual homeland is just a NAS and some networking. It's a small flat, it's just me. Not complicated, no need to give me more headaches!

Updated my NAS recently and Immich's database stopped working due to some PostgreSQL update that needs something changed manually, so I need to get my head around that.
Also trying to get a tablet to run as a 2nd satellite for HomeAssistant voice commands and no matter what I do, only the 1st one responds to wakeword... but I tend to give up after everyone's gone to bed as I'm literally in a room on my own talking to myself...
- PostgreSQL Updates AFAIK require manual Backup / Restore of the Database. But better look that up. I think the last one I did was:
  Stop the Application Containers (here the Immich ones, so only PostgreSQL runs)
  Backup the Database
  Stop the PostgreSQL Container
  Change to the new PostgreSQL Version
  Start the PostgreSQL Container
  Restore the Database
  Start the Application Containers
  As I said, better look it up first, this is just how I remember the process (but not the backup / restore commands).
  
  Ok, thanks. Yeah, I don't use PostgreSQL much, so I have to get my head into it.
  That said, it's only for Immich, so I could just wipe it and start again which might be quicker... it only took 3 days to scan the photos, it might take me longer to update the database 😉

I just set up some geofencing on pfsense, found alot of traffic that I didn't know was happening. That freaked me out.
- Using PfBlockerNG?
  It's a great piece of software, but I had a hell of a time blocking some countries for torrents.
  A single IP in China was repeatedly downloading an Ubunto ISO, I think due to the various methods of peers finding each other, so in the end I had to create an additional alias to block outgoing traffic even though only I was only allowing specific other countries in.
  
  Yup! PFBlockerNG. I thought the GeoIP thing from Maxmind was paid, since the setup asked for a license key. Nope, free. Just had to register.
- Interesting, I think I should do the same for the services that are only used to people real close.

I am finally in a position to have hardware running at home without it bothering anyone, so I cobbled together the hardware peaces I thrifted for over the years.
I played around with Proxmox and lxc containers, which are awesome, but not really useful for my usecase. I currently needed the essentials to get started and to finally have some kind of backups.
So TrueNAS scale it is. I got the ACLs down quickly, so the built in apps are no problem. But some things are not suited to be run as a built in app, I found. To avoid these headaches, I created an ubuntu server vm and a network bridge to allow for host access, and spun up those containers there.
I went for too little storage on the vm in the begiining (10G) so of course it filled up to the brim in a day. So I had to learn how to extend an lvm. Which worked only after I made some space available. It was so full, even mkdir failed.

2012 macbook pro in the cupboard as server. Put nixos on it and it flies now, only wish the Magsafe connection was a bit more stable.
Its got gigabit so should be good.

I kinda-sorta finalized my migration to a smaller setup with my mail+web server. I've been running a small MSP business for several years and as customers flee right and left mostly to microsoft (due to 365 setup pricing) it's been in a decline for quite a while. So, I finally pulled the plug and shut down the business side of things and downscaled that to a single VPS with a handful of domains, email service and a few simple wodrpress sites.
Also I kinda-sorta moved all of my photo archive of 20+ years to immich and set up a backup scheme for it, which is now (only) 2-1-1. I also need more storage for that thing, but it needs to wait for few days until paycheck and after that migration I can finish importing all the photos I have laying around. That also requires some reconfiguration of my disk arrays, copying couple of terabytes from system to another and back again, but that's relatively easy thing to do, but it takes "a while" to accomplish.
After that there's a long list of things to do, but mostly I'll spend my free time and money to improve the current setup as quickly as possible in the immediate future.

I set up Affine and Kanboard to help with various projects. I got fed up of Notion, Trello, and/or a git repository full of documents.
- I used to host kanboard for a while, maybe I should set it up again for my homelab
  
  I do like Trello, but Kanboard is a pretty good replacement. Lacking some polish, but it's a trade-off I'm willing to make.

I moved my Home Assistant from Proxmox VM to a older Lenovo Laptop we had stored as we thought the charger wasn't working. We are preparing to move so it was my job to check that laptop as well as 2 others. 2 I am not going to use and e-scrap those later this week after yanking the drives out (I don't trust anyone with my old drives). It turned out, the charger works just fine! I just installed it early in the morning (Midnight) and so far, it seems just as responsive if maybe more than what I had on the Proxmox host so that's a win on my end. Plus, I was able to give it the full 8gb of RAM it has instead of the 4gb I gave it in Promxox and somehow it's showing lighter use than what I had in the VM. 2.8gb vs. 4-5gb it reported from the Home Assistant Hardware details when in the VM.

I like this thread :-)
I have just checked off a long standing item in my backlog: implementing OIDC on at least two apps. I've used a remote keycloak instance for authention for my household and so far so good. Now I'll try to understand the configurations a little better before take on other items on my backlog.

Needed more ports for core switch so decided to mess with Cisco. Not because I like or respect that company or need their features but rather to get real world experience with it. This has only reinforced my hatred of them, but it's currently functional
Also needed to get 10G up, so went with microtik. This is also a curious little system, but at least it wasn't $10,000 MSRP (Cisco came from eBay). Still need to configure vlans here: thinking I'll just keep RouterOS but make it not route.
- Thought of a problem: I only pulled 300MB/s in initial test of 10G to my NAS (direct from PC) . Theoretical is 1.2GB/s right? NAS is 5 disk RAID5 spinning rust which interfaces to NAS expansion through USB3. so, this has to be the issue right?
  Fix: cut out the USB for slight gain, add SSD cache for medium gain, or switch to all SSD to hit theoretical... Right?

Currently still fixing alpine Linux lxc running docker that decided to stop being able to network after a PVR update.
I've managed to migrate my services to debian-based docker Lxc, but it bothers me that I can't figure this out.
Best I have so far is that flushing the iptables in alpine lxc works temporarily.

I have been trying to get my own mastodon instance up and running behind traefik this weekend
I maneged to get it up and running yesterday, but I get some permission errors when trying to upload like a profile image etc.
I found a few issues that sort of mentioned permission errors, but they suggested running chown in the container, which is not possible, and running chown on the linked volume, which looks like an empty folder, and it didn't solve the issue.
Hoping to fix it when I get back home later.
It's nice to have something to fiddle with as the world is falling apart around you.
- I've had similar problems with some other Fedi service once and it was indeed a permission problem. Good luck!

I replaced my old 1gbs router with a 2.5gbs one from Protectli! It has coreboot which was the ultimate deciding factor for me, even though it came at a premium.
Installing OpenWRT wasn't difficult, but configuring it correctly to use all the ports took some retries.
I also have been trying to install a pixelfed instance of my own on Podman, but no luck so far; I keep failing at the image building part. I wish they would release images of their own.

I spent half a day trying to get acme-dns + Cert Warden up and running and failed miserably. And I think I will give up on it. That does not happen usually, but during my debugging sessions I have seen that the acme-dns project is not maintained regularly since quite a while. The current maintainer just has not enough time, but tries to prepare the project for a move to a new GitHub organization, so more people can help with the project. Until then, Issues and PRs accumulate, so I am not sure anymore if I should stick to acme-dns or just do it differently.
Why did I pick this scenario? Because of Let's Encrypt certificates and my DNS provider does not allow fine-grained API Keys for DNS management. This means, that currently the processes that request certificates in my Network need the API Key for the dns-challenge for Let's Encrypt.
Ways around that are by either using Let's Encrypt alternate (I think it is called DNS alias mode) method where you can request Certificates for your main domain, but put the TXT records for the DNS challenge on another Domain. One way is to just use a 2nd Domain for that if you have one.
I tried to do it with a Subdomain of my Main Domain that I delegate to acme-dns. The whole acme-dns, Domain delegation stuff etc. works fine, but I am not able to get this hooked up to Cert Warden properly and end up with error messages that make no sense to me and since I do not find any further information in the logs, as I said, I just gave up yesterday evening... for now ;-)
Another thing I am struggling sometimes is my Pi-Hole + Unbound setup where Unbound for no reason just returns a NXDOMAIN for some queries and I can not figure out why, under which circumstances and when that happens. It just seems to be random and a restart / cache clearing etc. does not fix it.

So, slightly tangential, but I have a failed home automation project this past week.
I have been using an unofficial integration for my mini-splits for a few years. The guy who wrote it likes to disappear for 6 months at a time and it seems like it may be abandoned. It finally stopped working after a home assistant update.
I had bought some ESP based replacement dongles about a year ago and decided to finally use them. Well, not all of the features worked, so I set about writing my own firmware.
That ended up working even less well. I wasted a lot of time and effort trying to get my firmware to work before giving up and just moving to the fork of the original Home Assistant integration for the official dongles.
I hate being beholden to third party stuff like this because I have robust automation setup for my mini-splits and updates can completely break them and be a massive pain to fix.
I'm not sad I tried and failed so much as I'm just sad it didn't work. I may try again sometime in the future.

I replaced my tabbyml code assistant this week with ollama+continue.dev. But I'm having issues with speed. I think this is because I switched from code qwen 2.5B (ish) to Deepeek Coder 9B (ish) and I think I'm pushing the limits of my GPU. Maybe I'll spend today sorting out which models I want to use and which computers I want to use them on so I dont run into this issue (I've got ollama on 2 computers with 3 GPUs shared between them, for a total of 24GB VRAM)

I set up Jellyfin, it was easy but I'm only allowing local network access