What's up, selfhosters? - The Sunday thread
Almost forgot before going to bed but I feel bi-weekly is a good rhythm for this.
Let us know what you set up lately, what kind of problems you currently think about or are running into, what new device you added to your homelab or what interesting service or article you found.
I'm currently looking to connect an NVMe SSD to a Pi 4 I have in a differences location to finally have proper 3-2-1 backups. I'm trying to find a NVMe to USB adapter that will work though.
I plan on setting up the *arr suite and getting rid of Netflix, Crunchyroll, Amazon Prime and Disney+
I just got cactus comments working on my writefreely blog. Cactus comments needs matrix, so I got matrix with element set up. It was an incredibly frustrating journey of learning, but it turns out the final bit wasn't too complicated. I'm running on TruNas, which I hear you shouldn't do, but it's too late haha.
I also got the whatsapp bridge set up, but it doesn't look like matrix supports disappearing messages, so I left some rooms it created b/c one of the larger groups easily took up a gig of storage before disappearing messages was turned on.
Anyway, writefreely has federated cactus comments that you can currently only sign in with a matrix account lol. I can try and change that later. I host the photos with picsur.
I think that I need to collect my notes and memories and put it into a blog post, but I don't really want to turn my blog (if I even keep up with it) into a blog about how to self host a blog.
In hindsight, I don't think the comments were worth it, but whatever.
ITT: lots of busted pihole v6 updates
Finally got started with Grafana, Prometheus and Meshtastic.
I've had two failed harddrives in the last month. Not sure if bad batch or what. Thankfully the order these were on only were the two drives so may not see more. They are under warranty but it's still a pain!
Otherwise I'm enjoying Mealie lately for my recipes. Kinda nice having them all in one place but accessible by anyone in the house.
I found Mealie to be a bit bloated and not fitting my needs too well. I moved to KitchenOwl. Small project but I love it.
I love mealie as well. My wife and I keep all our recipes in mealie.
I feel bi-weekly is a good rhythm for this.
What does biweekly mean to you? Twice a week, or once every two weeks? If it's the latter, I prefer to use fortnightly, since it's not ambiguous.
Yeah, nobody other than Brits use fortnightly anymore.
I mean every other week. I wasn't aware of the other interpretation, but I think in combination with "The Sunday thread" it's unambiguous?
I have never heard fortnightly, but then I'm not a native speaker. Is that commonly used?
Trying to get my hands dirty with LLM, Ollama and Web Scrapping.
I don't understand most of it , but hey, that's the fun. No complaints.
Immich. Wanted to exclusively use the external libraries features in read only.
Set it up once in its own Proxmox LXC under Docker. Set it up all properly started scanning my entire library. And when I woke up again it had crashed and I couldn't recover it.
Started over the following morning and only gave it access to 2024 instead of everything. And it filled up to 30gb/40gb I gave it with thumbnails and files and such. Guess it crashed the other day because it took up too much room.
Guess I'll start over again, and ensure all the config files and thumbnails are stored on my NAS so they can take up the space they need to without overloading the main (small SSD) on my server.
Pihole 6 broke my DNS (dnsmasq), and since I had a fw rule in opnsense to only use pihole's DNS, and deny public DNS access, it was an early rise for me :)
Unbound broke on both of mine day one of v6 and I’ve still not gone and fixed it. Sigh.
And that's why you have either a backup for your DNS or know whats auto-updated ;)
As you mention opnsense:
What do you mean with fw rules to only use pihole dns?
This sounds partly like a DHCP config and partly like a deny (hardcoded) DNS requests and to please use what DHCP supplied (looking at you google/amazon)
I did have backups, it was an easy fix. I had a pihole -up on a crontab for years, probably not the best idea :)
FW rule accept :53 from pihole only, deny :53 from all. I had some devices with hardcored DNS settings (8.8.8.8).
Damn... DNS issue early in the morning... What a nightmare 😂! Hope you got enough caffeine.
Realised my jellyfin lxc had a maxed out bootdisk yesterday, haven't been using it for a while. Luckily I have decent backups setup so I was able to restore a backup from late January when it wasn't filled yet. A quick library rescan and everything was up and running again.
Had a hard drive fail my main zfs array. First time I have experienced a disk failure so it was a bit worrying. Thankfully I had added an additional drive to expand the array so I was able to quickly rebuild to that drive. Currently shopping for a replacement. From now on I think I will keep a cold spare just in case this happens again. I just wish hard drives would stop increasing in price.
You save some money by buying recertified drives from Serverpartdeals.
After having upgraded my Pi-Hole to v6, for some reason yesterday it started to not recognize any of the blocklists. So, I resetted it and now it works.
I'm in the process of doing an initial restic sync of my primary storage to B2 as offsite backup and while I'm at it finally got around having a look at resticprofiles to simplify my restic backups on all my systems. Highly recommend it as it reduced my mental overhead of doing regular backups quite a bit!
My pihole exploded yesterday, all my fault. A couple of years ago, I created a script called via cron to update pihole's services every other week. This was great, until now when it updated to v6 at 4am. To make matters worse, I neglected to automate raspian updates, meaning it was very out of date, and was no longer compatible with pihole-FTL (thinking back, I thought I automated it too, but I guess not).
I took an image after creating a pihole "teleporter" backup, and began formatting. In my lack of caffeine and focus, I missed that my teleporter file was corrupt after I had successfully wiped the SD card. Thankfully I had that image as I was able to mount it and retrieve my blocklists via sqlite, otherwise I would have had to start from scratch.
One good thing that came out of it (for my taste, anyway) was that I swapped the OS on the pi to fedora. No more debian around here!
Tomorrow, I plan on setting up some backup automation for my pi, as it's the only machine missing backups at this point.
Why so hostile sounding against debian?
What does fedora better?
I don't mean to sound hostile, that's probably my past demons coming out. Like I said in my last comment, it's really apt that I hate. It would constantly break or put me into dependency hell and I haven't had to deal with that (yet) with Fedora.
I haven't put my finger on it, but Fedora, for whatever reason, also just feels faster.
@AmbiguousProps @tofuwabohu why would you privilege fedora over Debian? Asking because I am trying to do the reverse.
It's mostly personal preference, but I have grown to hate apt in general. I used it for over a decade and constantly got in dependency hell. I've yet to have anything like that happen on Fedora, especially Silverblue and CoreOS.
Finally got my lemmy instance fully updated.
Been improving my backup scripts in advance of adding backup to a server.
Updated servers and other services.
I upgraded immich without breaking everything. That's always reason to celebrate.
I’m running the Immich Flatpak specifically for this reason. It’s always one version out of date but always self updates without issues :)
flatpak
You mean as a client or a server?
How exactly does stuff get broken? Never rly had a problem bumping up the version in docker. The only issue has been the playstore version taking longer to push updates sometimes for the mobile apps.
A few versions ago I upgraded it and some default port configs changed rendering it unusable. Since my upgrades are a docker command, I had to go hunt down the error message. It didn't take long, but it def broke the setup.
I feel you 😂
Since it's winter and I mostly don't want to leave my house, I busted out an unused Raspberry Pi 4b a couple weeks ago. Started with CasaOS and AdGuard. Have now added a few other services including Navidrome to serve up a lot of local-area music for myself and friends. Got a Cloudflare tunnel set up, then some authentication through CF as well. And finally secured a static IP from my ISP. This is the farthest along I've ever gotten with any of this and it's been going great. Nearly every hurdle I've encountered I've been able to work through.
Two things causing me grief today though:
Try the OCCWeb app in nextcloud apps.
Thanks! It just threw an error at me when I launched it, but I'll see what I can do. Based of the warnings in the admin panel, there isn't anything critical for me to address, I just hate that orange.
Finally moved all my lxc onto a lower-power Xeon D host, consumes 1/3 the electricity of my previous Dell R430, same essential performance.
I'm setting up Seafile and trying to swap everything from docker to podman. The longer term goal is that once everything is on podman, I'll get a new NVME drive and install MicroOS so I can retire my old SATA SSD (I've had it for 10 years or so, across 3 PCs).
I'm also considering setting up Forgejo and getting a worker to build my Rust projects.
I'm a new selfhoster and reached the limit on what my DS923+ can handle after setting up an Immich instance (on top of qbitorrent, radarr/sonarr, plex). So I picked up a mini PC this week and migrated the Immich stack over (pointing to an NFS mount for the NAS!) and now it's running super smooth 🙌 Now I'm hype to move over more services and eventually start separating out media services from mission-critical stuff like photos when I have another machine handy.
I wanted to set up local domain resolution for my devices in order to stop having to visit sites with the local 192.168.1.x IP, so I started following some guides to run dnsmasq on the mini PC (Ubuntu Server) and add entries to /etc/hosts. It was pretty easy to get working OK, but for whatever reason the DNS doesn't seem to be working on a fresh boot. My local workstation can't ping the custom DNS entries for my devices until I sudo systemctl restart dnsmasq on the mini PC, after which everything works fine, which leads me to believe it's some weird boot order problem? I'm trying not to screw with it too much before bed, but hopefully I can figure out what's going on this week.
If you want to have domains assigned to local IP addresses, you can also use Pihole as a local DNS! It's a very nice tool for adblocking on network level anyways, can only recommend it.
Highly suggest putting Caddy on a machine, forwarding port 443 and 80 to caddy, and then letting it do your reverse-proxy stuff. Register a domain name, give it your IP address, and then tell caddy that 'immich.yourdomain.bleh' goes to port 78789 and plex goes to 'media.yourdomain.bleh' port 89898 -- Caddy handles all of the TLS stuff, handshaking, you name it - so you can have secure sites with proper certs.
Then make sure those things are isolated from your home network through vlans if your router supports it.
You can get fancier with it using a tailscale and getting some datacenter IP to forward into your network
Thanks for the advice, I didn't know a reverse proxy was what I was setting up though I've seen that term all over. I think Caddy is likely in my future but I already have basic access to my home network through a Wireguard tunnel for now so I was hoping dnsmasq could solve for my case without getting too fancy or exposing any ports. I think I should probably try to learn about reverse proxies more generally to figure out the next steps forward.
You can use https://schedule.lemmings.world/ to automate the posts. Or, given the community we're in, you can selfhost it!
This week I've been doing some work on my GOG Downloader to finally back up all my GOG stuff when I buy new disks, that's pretty much it for my selfhost/homeserver stuff this week.
I didn't know that, cool! Though I should probably talk to the mods before setting up such a thing.
I'm the one who files the most bug reports on github under a different name. Our instance runs on Lemmy Schedule, so thanks!
I've been working on some bash scripts to help manage my media files. I've been slowly working on learning more bash and I'm pretty pleased with my progress. After I finish this bash book I'm reading (can't remember the title atm), I think I'm gonna jump into awk.
Bash is a really great shell, but consider trying out a functional shell scripting language like Elvish (which is also a shell). Syntatically it's pretty similar and not hard to pickup, but it's stupid powerful. A cool example is updating different servers via ssh in parallel using a servers.json file;
json
[
{"name": "server.com", "user": "root", "identity": "~/.ssh/private_key0", "cmd": "apt update; apt upgrade -y"},
{"name": "serverb.com", "user": "root", "identity": "~/.ssh/private_key1", "cmd": "pacman -Syu"},
{"name": "serverc.com", "user": "root", "identity": "~/.ssh/private_key2", "cmd": "apk update; apk upgrade"}
]
and a little elvish magic;
elvish
var hosts = (from-json < servers.json)
peach {|h|
ssh $h[user]@$h[name] -i $h[identity] $h[cmd] > ssh-$h[name].log
} $hosts
Just run the script and boom, done. You can even swap out peach which is parallel each for each if you want to do each command procedurally--but I really love using peach, especially with file operations over many different files. Linux is fast, but peach is fuckin' crazy fast. Especially for deleting files (fd -e conf -t file | peach {|x| rm $x }, or one thing that I do is extract internal subs (so they play on my chromecast) in my Jellyfin server, using elvish makes it really fast;
elvish
fd -e mkv | peach {|x| ffmpeg -i $x -map 0:s:0 $x.srt }
Find all *.mkv files, pass the filenames through ffmpeg (using peach) and extract the first subtitle as filename.mkv.srt. Takes only about a few seconds to do thousands and thousands of video files. I highly recommend it for home-labbers.
Pretty dumb example, but peach is like 6x faster;
elvish
❯ time { range 0 1000 | each {|x| touch $x.txt }}
5.2591751s
❯ time { range 0 1000 | peach {|x| touch $x.txt }}
776.2411ms
Pushed Wireguard back onto my network. I've been a Tailscale user for a couple of years, but never really saw the need for it for me as I'm the only user of the service. :)
I will freely admit though, there's nothing wrong with the service and honestly is great if you are behind a CGNAT router or don't want to use Cloudflare for your tunneling.
Just swapped VPS hosts from ssdnodes to MassiveGRID. Got a pretty sweet deal, so I'm pretty excited.
Got my services transferred over this week and it's been fun as hell. It's interesting because I was discussing Portainer with my buddy and he has Portainer on his local PC to connect to his remote instances and with hindsight it sounds obvious of course, but it's such a nice little setup. Just finished setting up my Jellyfin reverse proxy so I'm gonna watch a movie and chill.
I used Portainer for a while and still like it for checking out networking stuff, but try out Dockge! It's more open sourcey and basic, but makes updating easier.
Dockge
Dockage was nice, and it was much simpler, however, I had to leverage more docker commands via my VPS with it, because there weren't a lot of options, specifically network settings.
Many issues this week:
But at least i got my Velero backups working on an private S3
A third, and hopefully final attempt at getting an iredmail setup going. SPF, DKIM & DMARC all checking out fine. It's actually working this time. Need to get the ISP to change our PTR record though, last bit of the puzzle.
Also picked up a used negate device, so we now have pfsense fronting everything. That's allowed me to move the original router to a better location and put it in AP mode.
Emby media server moved off a Synology and into a proxmox container. Finally, we can stream high def with the hardware acceleration we weren't getting before.
I finally got link warden up and running, but I'm chasing down some failures on a few websites.
Also realized that me biting the bullet for unlimited bandwidth (screw you Comcast!) means I can run archive team warrior, so that's been going.
IMO linkwarden was a real PITA. I've been trying linkding and it's been really great so far. I've had no issues like I had with linkwarden.
I spent half a dozen hours this weekend trying to get Proxmox running on a 2nd hand laptop, but I can't get it to run without sounding like a jet engine. The machine did fine when I ran Mint and used it as a laptop - but even after blacklisting the dGPU and forcing all the CPU cores to powersaving, I'm still making heat like crazy.
Plan B is to put Mint back on it and install podman and see if fan noise is a problem then. But I'd rather have podman running in an unprivileged LXC.
Did you check Mint recently? If it's been a while, it could also be dust buildup at the fan.
Its only been a few weeks, but I should give it a good blowout regardless.
Hmmm you might be able to first install Debian 12 and make sure the fan control works properly, then just install the proxmox application inside of that
https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
Got Prometheus and Grafana setup with https on my Talos Linux cluster. Tried to use cert-manager with a DNS01 Challenge with Let's Encrypt but was using a local TLD and found out it won't issue it. So I had to switch to a local issuer. Was using metallb to gain a routable ip, I used the nginx-ingress controller for Prometheus and Grafana. Next time I can tinker I'll place the rest of my services behind it.
I hadn't heard of Talos Linux, sounds cool! We are using haproxy as ingress controller with stepca for local certificates at work.
Personally I'm mostly involved with my homelab migration so there's not too much on the selfhosting page except os updates. I set up meshmini earlier to access my thin clients via vPro/AMT but I need to configure the clients before being able to actually using meshmini. Once I'm done with that I'll finally be able to set up Lemmy and Pine pods.
My selfhosted stuff currently works fine without me doing much which feels good and lets me focus on hardware stuff currently.
Experimented with selfhosting a Woodpecker CI as a complement to my Forgejo.
Works quite nicely, I just need to set up a native ARM64 agent as the overhead of cross compilation on x86_64 is quite big.
Why not just use forgejo's actions and runner?
Woodpecker is more mature and I can control access better since I am not the only one using my Forgejo. But I think at some point the built in ones might reach feature parity.
Experimented with selfhosting a Woodpecker CI as a complement to my Forgejo.
If you need CI, check out OneDev. It's a git solution that comes with an integrated CI solution.
I like iOS shortcuts. This week, I created an iOS shortcut to scan my Plex library. Now this may seem weird since there is an option to scan a library from the official Plex iOS app and there are also options to scan the library automatically or periodically. For various reasons (excuses), I didn't like that the official app only lets you scan one library at a time and I have automatic/periodic scans turned off to avoid network drive access, so I created the shortcut to scan from my phone any time I felt like I wanted to trigger it.
https://{ip_address}:{port}/library/sections/all/refresh?X-Plex-Token={plex_token}