Remote View

2y ago

/kbin server update - or how the server didn't blow up

Currently, on the main instance, people have created 40191 accounts (+214 marked as deleted). I don't know how many are active because I don't monitor it, but once again, I greet all of you here :) In recent days, the traffic on the website has been overwhelming. It's definitely too much for the basic docker-compose setup, primarily designed for development use. I was aware of the possible consequences of the situation happening on Reddit, but I assumed that most people would migrate to one of the Lemmy instances, which already has an established position. I hoped that a few stray enthusiasts would find their way to kbin ;)

The first step was to upscale the VPS to a higher version (66.91EUR). It quickly turned out that it wasn't enough. I had to enable CF protection just to keep the website responsive, but the response times were still very slow. At this stage, the instance was practically unusable. The next step was a full migration to a dedicated server (100EUR, the current hardware). It can be done relatively quickly, so it resulted in a 5-minute technical break. Despite the much higher parameters, it didn't get any better. It became clear that the problem didn't lie there. I'm really frustrated when it comes to server administration. That was the moment when I started looking for help. Or rather, it found me.

A couple days ago I wrote about how kbin qualified for the Fast Forward program. To be honest, I did it out of pure curiosity and completely forgot because a lot was happening during that time. During the biggest fire incident, Hannah ( @haubles ) reached out with a proposal to help. I outlined the situation (in short: the server is dying, I don't even know what I need, help! ;). She quickly connected us with Vlad ( @vvuksan ) and Renaud ( @renchap ). I was probably too tired because I don't know if the whole operation lasted 60 minutes or 6 hours, but after a series of precise questions and getting an understanding of the situation, the guys themselves adjusted the entire job. I love working with experts, and it's not often that you come across individuals so well-versed in the fediverse. Thanks to Hannah's kindness, we will be staying there a bit longer. Currently, fastly.com handles the caching layer and processes images. Hence those cool moving thumbnails ;)

Things were going well at that point. I could disable Cloudflare protection. Probably thanks to that, many of you are here today, and we got to know each other a bit better :) However, even then, when I tried to enable federation, the server would stop working.

Around the same time, Piotr ( @piotrsikora ), whom I already knew from the Polish fediverse, contacted me. He is the administrator of the Polish Mastodon instance pol.social, operates within the ftdl.pl foundation, and specializes in administering applications with a very similar tech stack. I made the decision to grant him server access. It only took him a few moments, and he came back to me with a few tips that allowed us to enable federation. In the following days, there was more of it, and we managed to reach the current level. I think it's not too bad.

Nevertheless, managing the instance has taken up about 60% or more of my time so far, which prevents me from fully focusing on current tasks. That's why I would like to collaborate with Piotr and hand over full care of the server to him. Piotr will also take care of the security side. Now I have to take this much more seriously. We still need to work out the terms of cooperation, but I want you to know the direction I intend to pursue.

We also need to migrate to a new environment because one server will sooner or later become insufficient. This time, I want to be prepared for it. This may be associated with transient issues with the website in the coming days.

The next two updates will still be about project funding (I still can't believe what happened) and moderation. The following ones will be more technical, with descriptions of changes and what contributors are doing on Codeberg. I would like to be here more often, but not as an admin, just as myself.

Thank you all for this.

P.S. In private messages, I also received numerous offers of help that I didn't even have a chance to read and respond to. You are the best!

175 comments

hi all, you can buy ernest a cup of coffee aka give him some cash for his incredibly hard work! https://www.buymeacoffee.com/kbin
- Yassss I'm supporter 666🤘
  
  🐐
- That 100 Euro server also needs some coins inserted :)
- Done! Thanks @ernest! This site is so cool!
- Ernest should set up a patreon. I'd pay a monthly sub to him for this great website.
  It's hard ot believe it's only a month or two old.
  
  100%
- Unfortunately someone just tried to use my bank account the other day, so I'm waiting for a new card to come in, but I got this bookmarked for when I do. I think it's worth the cost of a cup of coffee every now and then.
- Done!
- Done! Keep up the good work!
- Yes, done!
- What payment options are there? This seems to be only credit card.
- If I was Ernest I’d be scared stupid
  
  Know wut i mean, Vern?
- Easiest $15 I've spent this month. Let's keep Kbin going!

I hoped that a few stray enthusiasts would find their way to kbin
They did. "A few" as a portion of the whole, where the whole is much larger than you ever expected it would be.
One thing I have learned in my own technical career is that you can't do everything. Even if there was enough time in a day for you to do everything, you can't know everything, at least not sufficiently enough to be most effective. You have to depend on other experts, and delegate to them what you are not qualified to do. This requires a pretty high level of trust, and makes it so that you need to develop people management skills.
Lots of technical people are short on people management skills. I don't know where you sit on that spectrum, but you may need to consider bringing on an "overseer," kind of like a project manager, to keep tabs on all of the technical resources involved - yourself included. This will help ensure that concerns are prioritized appropriately, and that communication and messaging about those priorities are consistent and clear.
I've been in that kind of position, and I take a bit of pride in my use of words. I'm happy to give any advice you like.
- I sense an opportunity here: managed hosting of Fediverse services. I don't mean a managed host where you can install/run the services. I mean a top-to-bottom setup, management, backup, upgrades, monitoring, etc. so the only thing you need to do is administer the community. I'd love to set up several Fediverse services for my local community, and I know there is an audience that would also love that, but I also know I do not want to invest the time it takes to manage the technical side of that.
  Having easier to set up instances would help in relieving the pressure from the more popular current instances. We're starting to see those hosting options come on line for some of the services, but not fast enough for my liking.
  
  Love it. I'm sure there are some spam-evaluation services up or starting soon that use AI, but that will also soon be important as we become more of a target.
  
  Check out Grow Your Own Services. It doesn't have everything (e.g., kbin, Lemmy, Calckey) but it does offer what I think you're looking for with a number of the other more popular services.
  
  @bourbonmakesitbetter We're a small independent "mom & pop" managed hosting provider focused on Mastodon and we're currently testing out kbin, send us an email at keith[@]thunderhost[.]com with the size of your community and we might be able to assist
  
  Something like this already exists for Mastodon (check out masto.host, and there are also a few others). But a service supporting all fediverse softwares would be cool.
  
  @bourbonmakesitbetter elest.io provide a service like this kbin should be added in a few days
  @haubles @vvuksan @renchap @piotrsikora @ernest @Nougat

@ernest @vvuksan @renchap @piotrsikora Thank you, Ernest 💞 we're so glad we can help and support you, and the project.

Hell yes, ernest. Keep up the momentum, stay humble of your shortcomings, and don't burn yourself out.
We all love this community you developed, and hopefully we can attract the kinds of people who are as open, humble, courageous, and intelligent as you seem to have gravitating towards you in your personal and professional life.

It's cool to see renchap helping out here in addition to his work with mastodon.social! The amount of cross-pollination and coordination within the fediverse is so cool to see.
- @MeowdyPardner @haubles fascinating reading comments like this and having to figure out the context for “here” (as I read it via Ivory with my mastodon.social account)
  One of the things that will have to evolve in our discourse online in the fediverse is the distributed nature of “here”

- Sometimes the world is divided into before and after. For Earnest this must be one of those moments

Thank you for the update @ernest ! Do you have a webpage for donations ?
- https://www.buymeacoffee.com/kbin
  
  This link needs to be on the sidebar or something
  
  Thanks!
- https://www.buymeacoffee.com/kbin
  
  Done!

Hey @Ernest and @piotrsikora,
I haven't looked too closely at how kbin is architected yet, but would it benefit from horizontal scaling? I do full-time development of tooling to administrate very large k8s clusters for a company that you've probably interacted with today without knowing it. Not sure if k8s is the right orchestration system for you, but I'd be more than happy to provide some input on a potential migration to k8s (if kbin is a good fit there). I know there's a community on Matrix as well — I'll try to reach out there too, although it may be a bit.
- If the post is anything to go by it's using the included "mostly for dev work only, mostly" docker-compose files. It would absolutely be able to be scaled out since at it's core it's just a webapp with workers. The app is already configured to use Redis for session storage so should be able to go super wide.
  Only limitation is how performant you could make your postgres cluster.
  
  Bullseye
- Hi @Badabinski
  K8S is one of option, but we decided to use some mix of bare-metal and docker swarm.
  Almost everything is prepare to grow horizontally. Only (like always) problem is in database, and also we want to have flexible software that run on big cluster and small node without changes in code.
  Give us few days, and after that we will show something ;)
  @haubles @vvuksan @renchap @ernest
- I was thinking the same thing. Shouldn't this be one of the cases where k8s shines with a horizontal autoscaler? Wouldn't want to manage your own k8s though, so I imagine managed k8s is the best option. If it's the cheapest option is another question.
  @Babinski do you know if there are other horizontal autoscaling options besides k8s?
  
  As @BiggestBulb said, most cloud providers have container platforms that support horizontal scaling, although generally not as elegantly as k8s (imo, others may disagree). Also totally agree about managed providers. EKS, AKS, and GKE weren't suitable for what we use k8s for (very large shared clusters) until recently, so we've been administrating our own custom k8s distro. The managed stuff has gotten a lot better, and I'd definitely recommend that for running kbin. Running k8s yourself is hard, etcd is an evil bastard. I've had plenty of chances to see what works and what doesn't in my role, however. There are some development/deployment patterns that are robust, and there are many that are not.
  
  I'm not familiar with the architecture of the app (nor where it's hosted), but if it happens to be on AWS then you should be able to spin up an ECS cluster (especially since it's already containerized) and load balance it that way with an ALB configured during setup. Imo that would be the fastest way to do it (again, assuming this is on AWS)

Thank you ernest... and fastly... and welcome aboard Piotr!
OSPs make me so happy :)

Thanks for the update, and keep up the amazing work!

I read the whole thing with a smile on my face. So wholesome!

Thank you for creating this space for us :)

Thanks for the update. Sounds like you're having an "exciting" time. You've done a great job so far and I hope that the additional help you are receiving means that you might be able to take some time off in the near future, for your own sake.
Edit: buy Ernest a coffee for those able to contribute

I'm so confused. We only have 3500 accounts on Kbin? So we are miles and miles less than Lemmy?
edit No, I just need coffee
- Nope, we are at 35k+
  
  I need coffee
- Edited ;)
  
  That might also need an indicator to avoid abuse.
  
  Hah, thanks Ernest!

Kudos, @ernest. What you've put together in such a short time is amazing. I'm glad you're getting the help you need.

ITT: When your little hobby completely blows up in your face and takes over your life.
Thanks @ernest - we all appreciate it!

Keep notes. This will make a great documentary some day!
I am not well versed with the way all this works so this might be a stupid question: Is there one central server in one location for this site? Could outside, remote server space be used to share the work?
- @digitallyfree
  Yes, and we are going to change it in a few days:)
  @haubles @vvuksan @renchap @ernest @huskola
  
  Could I put one in my house with a 1Gig fiber connection and let someone else run it remotely? Just curious.

@ernest Fantastic news and big thanks to all that are helping! @ernest I feel you will keep seeing several thousand users join in daily as the reddit shitshow continues to play out. There are now 10 apps being developed that will further support the transition especially as the July 1st shutdown takes place. That’s probably when a much bigger wave of new users will come in. Is your infrastructure scalable to allow for this onslaught? Thank you and everyone else again for your hard work!

Nice work! I can't imagine what would happen if a hobby project of mine is suddenly used by 35k users, it would be insanely stressful, keep it up, we are loving it here.

Welcome aboard, Piotr!

Thank you @ernest! Please consider setting up a Patreon, or similar, account so I can donate monthly. I've bought you a few coffees but I'd like it to be automatic.

Thanks for the update 👍 💚 ... I like the approach you've taken.

@ernest this was an extraordinary situation with several crisis points on a platform in early development (as far as mass usage is concerned) and you were able to keep a cool head, keep things going and also keep people informed of the situations as they were developing. I am happy to read that you have help now (and it looks like really great expert help) and that you can take a break/retirement from admin and enjoy it with the rest of us. Bravo!

@ernest
Can you break down your costs in a spreadsheet?
Many scaling issues can be solved by throwing money at it - and I think seeing the cost in black and white might help!

That must've been quite the ride.
Eagerly awaiting the updates on moderation and project funding. I think it'll be good to be transparent about costs and monetisation. We all know what happened with Reddit in the end.

@ernest your achievements with this piece of software is going to go down in internet history, i hope you know that. you have already written your name into the history books. you deserve congratulations and very sincere thanks.

- Infrastructure costs money. Communities need to remove bad actors. These are just facts of life.
  Communities require continual investment. This is as true online, as it is for in-person communities, so it's important for people to donate when and if they can, if they find value in a community. The key thing is, a community needing to be invested in is very different from needing, or wanting, to turn a profit. In fact, community and profit-seeking are kind of mutually exclusive.
  The profit motive inevitably strangles the community.
- There are no free lunches. You want a nice platform? You've gotta pay for it somehow. You want a platform not filled with spam? You gotta keep it clean somehow. Funding and moderation. I really hope that kbin can survive off of donations, and I have already financially contributed fairly heavily in pursuit of that. Ernest is doing the right thing by communicating about these issues early, and often. Part of why Reddit got so fucked up was that there was no community engagement when it came to raising money. That's the nature of all for-profit platforms IMO.
  This is where federated, FOSS platforms can do better.
- Same song, new instruments, that's all. Just brace for the ads and the monetization. Be ready to be a customer rather than a user, or be ready to move on.
  
  I receive support from the people here, and I also want to clearly communicate and report expenses. I feel a sense of responsibility. Many magazines have been abandoned, and that also needs to be addressed.
  
  Do you think Ernest should run and develop this whole site out of his personal income?
  
  Or spin up your own instance. Pay the $6/month it'll cost you, and never worry about ads and admins exploiting you for profit. You always have an out here.
  Plus, it's not actually that expensive. Centralized services have a problem because not only do they need to cover hosting costs, but also development costs, moderating costs, and then they need to answer to shareholders and provide a return on investment. Here, everything's being done by volunteers. Development, hosting, etc. None of it is being done for profit, by businesses.
  Look at it this way. If there are, say, 30k active users on the server, and hosting costs are 100 Eur/month, then it just takes 20 people donating 5 Eur/month to keep things going. And if every server has similar costs and similar benefactors, then this space is golden.
  Costs, of course, will be higher over time, but there are ways to share them. Media sharing is already a thing, and if more media shares crop up, costs can be kept pretty low and people with resources and a love of community can keep things going indefinitely.
  You just have to trust a small fraction of users to actually support not being the product.
  
  Yeah, don't even get me started about all the ads on Wikipedia. /s Servers cost money. Money has to come from somewhere, that doesn't mean ad dollars, but likely donations/grants. Moderation is a given for any public space because people could start posting CP, and I don't think anyone wants that.

Look how far we've come in just a week :)

That’s amazing! Thank you for what you’re doing 3

Thanks for the update, Ernest. Maybe a tracker can be posted on the sidebar to make sure we're supplying you with enough 'coffee' to keep the lights on? A lot of us are loving this platform enough to want to invest in your work and I hope everyone will crowdsource funds to keep you afloat.

You handled this situation beautifully. Very often people don't know when to share responsibility and ask for help and their project dies in critical situations like that.
Well done, have fun with your now very alive and thriving project!

This is some dedicated effort here to keep this going, but is anyone not worried that there is basically one guy with all the keys to this thing? Dont want to sound too doomerish but if Ernest disappears tomorrow, wouldn't that be the end of kbin?
- This is just one instance. The code is open and can be forked. That's why if I happen to get hit by a bus tomorrow, Piotr will have the keys to the server room ;)
  
  Thanks for the reply and thats good to know!

I just want to say that I.. we appreciate all of you and all of the work that you are doing! We are happy to be here!

Thanks for the update!
My only caveat is that I wish I could see posts and comments from lemmy instances users when they post/comment on a kbin magazine.
- Can't you?

Huge shoutout to you and all of those who have gotten kbin to this point. Super crazy to have seen it go from struggling to a place I can happily and easily browse

that is such a great update, never knew the stuff that goes behind the scenes. You got this!

I had a feeling that the silence over the last few days was a sign that a huge amount of work was happening behind the scenes. I also think you’re doing an excellent job of communicating with us. Thank you for all your effort that’s allowed this community to grow.

Thanks for your transparency and hard work! Loving the site and looking forward to it getting even better!

Thanks for the hard work ernest!! I'm glad you were able to find some help and take some workload off you.

@ernest Regarding servers... did you have a look at Hetzner's server auctions. They tend to have 8c/16t servers for 40-50 bucks.
Also, I've seen kbin uses PHP at it's core. Do you consider switching to a golang stack, which is known to be more resource-friendly than PHP.
- That isn't the issue.
  A complete rewrite of the application might add capacity, but its vertical, you stack increase load in one instance. No matter how much performance you extract eventually you run out of capacity.
  As scales increase you need to add horizontal capacity. This is the idea of adding 2, 10, 100 servers. That means breaking out services into stateless parts which can run concurrently (or managed state behaviour).
  This is where something like Kubernetes comes into play, since its designed to manage docker images over hubdreds of servers. Instead of using every last bit of capacity from one server you spread it.
  Similarly postgres like most SQL platforms doesn't particularly scale beyond 1 instance.
  Facebook invented Apache Cassandra for this reason, it was the first NoSQL database and is designed to deploy in multiples of its replicaset number (3 is the default).
  Having data spread over 3, 30, 300 is less efficient, but you know have 3,30, 300 servers responding.
  The other advantage is horizontal scaling is fault tolerant by design.
  There is an argument for compiled languages like Go, C# and Java, but honestly the next big win is making as much as possible scale horizontally.
- Methinks that a rewrite from PHP to Go would be a pretty massive undertaking. PHP's performance characteristics have gotten a lot better as the language and various runtimes have improved, although it's not anything like Go. I think the best route would be for someone to implement another federated link aggregation system in Go, so then we'd have a diverse selection to choose from — Lemmy in Rust, kbin in PHP, this hypothetical new platform in Go, along with everything else out there. A heterogeneous system is good for the continued health of the threadiverse IMO.

Looking forward to the follow-up posts with technical details, if you do find the time to write them up ofc! As a new kbin user, my thanks for all the hard work and for welcoming us here 3

I'm glad you are here. Thank you for your work.

Glad you got help with server administration! Hopefully account migration is a feature that can be implemented, I would be happy to move my account to a less crowded instance, once it/they come to fruition.

Thank you!

thanks for proving the community this - really enjoying this place 3

Thank you everyone!

Thank you for the update

Thank you for everything you and the contributors are doing!

Thank you for the update, @ernest!

Great update!
Keep up the good work.

Exciting times indeed! Thanks for the updates, it's always interesting and fun to read these. Seems like you've adjusted pretty well considering the massive influx of users in less than 2 weeks, hopefully it gets a bit easier and less stressful from this point forward.

Amazing work, and very much appreciated.

I'd been doing a little shopping around over the past few weeks as I've been getting ready to properly leave my 11 year old (cringe) Reddit account behind. Lemmy does seem promising as well, but I do have some concerns about the developers, and while of course no one is perfect, as it's still the early days, I'd much prefer to throw my support behind people to whom I can do so guilt-free. So far at least, this has felt right up my alley, and while I am trying to use this moment to cut down a bit on my internet time in general, I'm definitely happy to be here!
- The biggest reason I'm here, is Ernest. Getting in when a project is just getting off the ground, let's users have some say about the direction of things and features. And Ernest is very responsive to the community and asking for feedback. That's exciting, and makes the whole thing feel more of a community.

Your work is appreciated, @ernest !

175 comments