State of the shork!
State of the shork!
So it's been a few days, where are we now?
I also thought given the technical inclination of a lot of our users that you all might be somewhat interested in the what, how and why of our decisions here, so I've included a bit of the more techy side of things in my update.
Bandwidth
So one of the big issues we had was the heavy bandwidth caused by a massive amount of downloaded content (not in terms of storage space, but multiple people downloading the same content).
In terms of bandwidth, we were seeing the top 10 single images resulting in around 600GB+ of downloads in a 24 hour period.
This has been resolved by setting up a frontline caching server at pictrs.blahaj.zone, which is sitting on a small, unlimited 400Mbps connection, running a tiny Caddy cache that is reverse proxying to the actual lemmy server and locally caching the images in a file store on its 10TB drive. The nginx in front of lemmy is 301 redirecting internet facing static image requests to the new caching server.
This one step alone is saving over $1,500/month.
Alternate hosting
The second step is to get away from RDS and our current fixed instance hosting to a stand-alone and self-healing infrastructure. This has been what I've been doing over the last few days, setting up the new servers and configuring the new cluster.
We could be doing this cheaper with a lower cost hosting provider and a less resiliant configuration, but I'm pretty risk averse and I'm comfortable that this will be a safe configuration.
I woudn't normally recommend this setup to anyone hosting a small or single user instance, as it's a bit overkill for us at this stage, but in this case, I have decided to spin up a full production grade kubernetes cluster with a stacked etcd inside a dedicated HA control plane.
We have rented two bigger dedicated servers (64GB, 8 CPU, 2TB RAID 1, 1 GBPS bandwidth) to run our 2 databases (main/standby), redis, etc on. Then a the control plane is running on 3 smaller instances (2GB, 2 CPU each).
All up this new infrastructure will cost around $9.20/day ($275/m).
Current infrastructure
The current AWS infrastructure is still running at full spec and (minus the excess bandwidth charges) is still costing around $50/day ($1500/m).
Migration
Apart from setting up kubernetes, nothing has been migrated yet. This will be next.
The first step will be to get the databases off the AWS infrastucture first, which will be the biggest bang for buck as the RDS is costing around $34/day ($1,000/m)
The second step will be the next biggest machine which is our Hajkey instance at Blåhaj zone, currently costing around $8/day ($240/m).
Then the pictrs installation, and lemmy itself.
And finally everything else will come off and we'll shut the AWS account down.
I tried to update my profile picture for the first time since the migration and now I don't have a profile picture at all, anyone else noticed issues? Image upload attempted from the webui settings page at lemmy.blahaj.zone.
Yeah same here, assuming its just a migration hiccup
@NoStressyJessie@lemmy.blahaj.zone Just log files filling up a partition. It should be good to go again now
appreciate the dedication and transparency! I'm a developer myself but im still learning the basics when it finds to clusters and scaling
a) holy 💩 i had no idea this was so expensive b) please include the ko-fi link for us to help support in future updates.
(link found in other comments)
It's not supposed to be, that's the issue. :)
well yeah, but even once the costs are reduced 10x or whatever there will still be costs and it's still be good to support its continued existence.
@supakaity@lemmy.blahaj.zone I'm pleased that this post has been up for 30 minutes and no one has jumped in with "Acktually..." and proceeded to tell you how you're doing it wrong. jk /jk
Thankfully I have no clue about this stuff, but appreciate the detailed update nonetheless. #Hajkey #blahajzone
I think it's because what they are saying is pretty solid. It's the exact solution I'd recommend
@jo @supakaity@lemmy.blahaj.zone That's because the ones who are complaining are wisely doing so in their own spaces, as they ought to. They're also revealing just how much they know about things, in the usual armchair quarterback / bleacher flyhalf fashion.
now mixed destruction reading journey plough smoke tomorrow spade addition quite bed feather green seed dust common attack fertile rub care journey true design soft motion no key take behaviour tongue red roof road effect across purpose pencil money wash if poor space taste pin sail stick brain where secretary
@supakaity@lemmy.blahaj.zone thanks for the detailed update!
through bad account war expert fact send bag together print clear rough harmony porter bridge very paste safe tongue nerve net still music seat mother together wound reaction map cough rate farm nose clock screw elastic work note sound wave writing destruction exchange reason wide feather parcel fiction selection seem
lift answer verse powder healthy shame poison mine bag little fiction cotton polish but fiction I reason voice make black like leg destruction pipe bent box middle sneeze wood impulse shut soup pin meeting care that poison lip wax river milk carriage hearing experience swim open desire town strange open
rub print let system wood tax do hat board news toe only gun normal unit gun cord amount day wing noise whistle roll glove learning event loss only small lip dependent open blood probable view crack idea glove animal rice country push short ring blade cook round river may thin
/qa/ won
box disgust old balance throat year insurance thing point heart pull agreement time fall fixed statement chance stick soup umbrella profit throat stocking coal surprise island dependent desire dead argument cause island sister direction damage strong arch digestion wall religion this discovery wide steel attempt touch foot range bad way
/qa/ won
question shoe while blow change account hammer net event number worm nerve how representative statement back stamp measure cake interest automatic simple in friend far comparison complex man distribution important past twist little porter industry war while comparison card shelf care against field window tall sock unit pen bulb pain
approval burn decision leaf when safe pipe milk leather free part chief table plane broken steam new sticky week tree small rain monkey than conscious bag bell hat field size form humour doubt now chance crush pin orange straight salt say advertisement bad substance trouble girl back limit wood after
/qa/ won
plant sound meeting enough bright carriage wide eye angle important wood polish sticky rice last blood chance join decision warm clock solid smile earth flower paint waste stem against needle yesterday damage metal material come year punishment division smash development east addition attack early flower here teaching degree pig weather
/qa/ won
free man meat art danger middle chance old delicate knife question part law who wall punishment hammer material hole brother tail sudden memory possible open company dear farm bridge wheel a winter letter danger science front thin wool step flag stick berry sudden camera little needle town tooth present range
/qa/ won
kettle very bottle belief goat motion sun approval run reading woman wound but sweet smash value operation increase test at army shake comb worm healthy crime waste as instrument humour farm protest full bird monkey off paper parallel wet polish foot healthy note bright relation past seat warm leaf umbrella
Once pict-rs updates to allow directly serving images from object storage- wouldn't it be beneficial to migrate it to an object storage that allows unlimited egress like Cloudflare R2?
Thank you for all your communication about how the server is being run. I always feel in good hands here on Blahaj :)
house question every verse experience high view bell loss sticky account kiss stem connection glove plant leg rain ornament tree grain here money snow together other probable net run past care silver nose board mine soap coal face broken knowledge start toe key step knee steam animal learning night sex
Absolutely wild to me that moving off AWS + setting up the caching server will bring overall costs down by around a factor of ten. So glad y’all are capable of the advanced technical junk, and super thankful that you’re willing and able to host the various blahaj.zone instances!
operation profit open company disease against note brass soap leg machine bucket tight enough organization dead respect connection full nose substance knot cheese straight pipe adjustment short push morning fish any soft copper no parcel butter size word stop slip colour through orange arm moon breath payment destruction ant match
father gold throat paint same lip order so turn growth take high growth sudden voice rain exchange fight butter hole probable expert mind thread button rest scissors hate growth drawer sugar name basket regret seem feather now cow under carriage expansion nut pipe current sad pen respect smash bread farm
island money rod comb heart knife form summer hand hate work conscious blood debt amount jump purpose only jewel turn chance hope sweet far before be till bird match prose fold ice late he second error will minute smoke system ready jump push than kettle grip before bent high come
yesterday do tray seem moon window solid mountain sharp green slope weather rail join prison come fight belief look bone amusement to solid army man deep heart sex act expert white slow noise family substance bread waiting pipe distribution tall store future position experience attraction sharp finger plane off elastic
wool blow poor attraction office or hook view brake coal fruit rule pencil wave touch picture copper approval than awake nose force slope wax edge manager idea pain baby ready shelf word political clock society slow root sticky potato number true under other account little deep for stem test key
much lip amount band river crime pencil down or poor operation space north mother attention agreement expert stomach cheese lift station birth cup brick door cork map wound committee female rain bee paste hour smile work invention fight company wet harbour band impulse same shelf even curtain payment love skirt
make snake place crush umbrella poor free start memory be trade fly porter shoe gold cart hanging spoon dependent that whistle sister scissors school wash powder engine jump young thin wheel rat start cup any parcel rat debt fiction watch have rest poison need in smile tree than unit support
scale decision full pain cough healthy measure doubt push snake instrument request competition root almost sheep yellow by letter price fiction cotton carriage exchange doubt military street request collar map meat question paste development false ray war learning distance angle low jelly nail still ticket grain shock roof girl healthy
How can we donate to keep the infrastructure up and running?
There's a KoFi! https://ko-fi.com/blahajzone