My struggle from a UNIX background in the modern "cloud" world

Permanently Deleted

We can't go back to the old way of doing things but we can't keep maintaining these fundamentally flawed systems either.

That's a great way of putting it, thanks. I'm actually only 30 years old (lol). Sometimes I feel there's so few people who've ever used or written software at this level in the part of the industry I find myself in. It seems more common to throw money at Amazon, Microsoft, and more staff.

I've replaced big Java systems with small Go programs and rescued stalled projects trying to adopt Kubernetes. My fave was a failed attempt to adopt k8s for fault tolerance when all that was going on was an inability to code around TCP resets (concurrent programming helped here). That team wasn't "unskilled"; they were just normal people being crushed by complexity. I could help because they just weren't familiar with the kind of problem solving I was, nor what tooling is available without installing extra stuff and dependencies.

Thanks for your understanding :)

I see that the problem arises from the "visionary, but lower experienced newer developers (compared to the past generation) " trying to fix a world where "don't touch it if it works crowd who has seen all old timers" built, by putting each layer over the older one. It has all the capabilities, but there is no "single vision", no "well defined api".

Old established paradigms are being broken. Some conventions are forgotten, new tooling and perspectives are being built.

Sure this means there is an unfortunate clash is happening.

I can't say if this is a better, or wiser world or not, however I can only say this is the way now. You can adapt, try to embrace and push forward things or you can try to stay away and become one of the legendary Cobol developer crowd. We know they are there in the wild, but we can't find them.

A lot of simple parts build up in predictable ways to accomplish big things. The complexity is spread out and minimized.

This has always felt untrue to me. The command line has always been simple parts. However we cannot argue that this applies to all Unix-like systems: The monolithic Linux kernel, Kerberos, httpd, SAMBA, X windowing, heck even OpenSSL. There's many examples of tooling built on top of Unix systems that don't follow that philosophy.

The traditional Unix way of doing things is definitely very outdated though.

Depends on what you mean. "Everything is a file"? Sure, that metaphor can be put to rest. "Low coupling, high cohesion"? That's even more valid now for cloud architectures. You cannot scale a monolith efficiently these days.

In the end, Kubernetes is trying to impose a semi-distributed model of computation on a very NOT distributed operating system to the detriment of system complexity, maintainability, and security.

Kubernetes is more complex than a single Unix system. It is less complex than manually configuring multiple systems to give the same benefits of Kubernetes in terms of automatic reconciliation, failure recovery, and declarative configuration. This is because those three are first class citizens in Kubernetes, whereas they're just afterthoughts in traditional systems. This also makes Kubernetes much more maintainable and secure. Every workload is containerized, every workload has predeclared conditions under which it should run. If it drifts out of those parameters Kubernetes automatically corrects that (when it comes to reconciliation) and/or blocks the undesirable behaviour (security). And Kubernetes keeps an audit trail for its actions, something that again in Unix land is an optional feature.

If you work with the Kubernetes model then you spend 10% more time setting things up and 90% less time maintaining things.

9P is much simpler and more elegant than HTTP

It also has negligible adoption compared to HTTP. And unless it provides an order of magnitude advantage over HTTP, then it's going to be unlikely that developers will use it. Consider git vs mercurial. Is the latter better than git? Almost certainly. Is it 10x better? No, and that's why it finds it hard to gain traction against git.

A filesystem does not exclusively mean an on-disk representation of a tree of files with a single physical point of origin. A filesystem can be just as “highly available” and distributed as any other way of representing resources of a system if not more so because of its abstractness.

Even an online filesystem does not guarantee high availability. If I want highly available data I still need to have replication, leader election, load balancing, failure detection, traffic routing, and geographic distribution. You don't do those in the filesystem layer, you do them in the application layer.

Also, you’re “disappointed” in me? Lmao

Nice ad hominem. I guess it's rules for thee, but not for me.

And how do you manage containers? With bespoke tools and infrastructure removed from the file abstraction. Which is another way Kubernetes is removed from the Unix way of doing things. Unless I’m mistaken, it’s been a long time since I touched Kubernetes.

So what's the problem? Didn't you just say that the Unix way of doing things is outdated? Let the CSI plugin handle the filesystem side if things, and let Kubernetes focus on the workload scheduling and reconciliation.

It’s not a preconception. They engaged with your way of doing things and didn’t like it.

Dismissal based on flawed anecdote is preconception.

By what standard? The standard of you and your employer? In general, you seem to be under the impression that the conventional hegemonic corporate “cloud” way of doing things is the only correct way and that everyone else is unskilled and not flexible.

No. I'm not married to the "cloud" way of doing things. But if someone comes to me and says "Hey boblin, we want to implement something on system foo, can you help us?" and I am not used to doing things the foo way I will say "I'm not familiar with it but let's talk about your requirements, and why you chose foo" instead of "foo is for bureaucrats, I don't want to use it". I'd rather hire an open-mined junior than a gray-bearded Unix wizard that dismisses anything unfamilar. And I will also be the first person to reject use cases for Kubernetes when they do not make sense.

just that you should be more open-minded and not judge everyone else seeking a different path to the conventional model of cloud/distributed computing as naive, unskilled people making “bad-faith arguments”.

There are scenarios where cloud compute just does not make sense, like HPC. If the author had led with something like that, then they would have made a better argument. But instead they went for

cloud-native tooling feels like it’s meant for bureaucrats in well-paid jobs,

,

In the 90s my school taught us files and folders when we were 8 years old

, and

When you finally specify all those flags, neatly namespaced with . to make it feel all so very organised, you feel like you’ve achieved something. Sunk-cost fallacy kicks in: look at all those flags that I’ve tuned just so - it must be robust and performant!

It's hard to not take that as bad faith.

Probably need to keep in mind incidental versus essential complexity here.

Go on...

Because this is how much of what we use already is implemented. Significant effort goes in to portability, interoperability and balancing compromises. When I’m doing software development e.g. writing HTTP APIs (of which I apparently know nothing about ;) ) - I feel like I’ve got a responsibility to carefully balance what I expose as some user-configurable thing versus something managed internally by the application. Sometimes, thankfully, the application doesn’t even have to think about it al all - like what TCP flags to set when I dial some service.

In the case of vmalert, the binary makes no assumptions as to default behaviour because it was not meant to be run standalone. It comes as part of a container with specific environment variables, which in turn is packaged as a Helm chart which has sane configurations. Taking the vmalert binary by itself is like taking a kerberos server binary without its libraries and config files in /etc files and complaining that it's not working.

You bring up containers which is a great example of some cool features provided by the Linux kernel to solve interesting problems. If you’re interested, have a look at FreeBSD’s Jails, Plan 9 and LXC. Compare the interface to all these systems, both at the library level and userspace, and compare the applications developed using those systems. How easy is it to get going? How much do I need to keep in my head when using these features? Docker, Kubernetes, and the rest all have made different tradeoffs and compromises.

I am very well versed in jails, chroot, openvz, LXC, etc. OCI containers are in a different class - don't think of them as an OS-like environment, think of them as a self-contained, packaged service. Docker is then one example of a runtime runtime on which those services run, and Kubernetes is an orchestrator that managed containers in runtimes. And yes, there are some tradeoffs and compromises, but those are well within the bounds of the Pareto principle - remove the 10% long tail of features on the host, reduce user-facing complexity by 90%.

Another one I think about is SQLite. Some seriously clever smarts. Huge numbers of people don’t know anything about for-loops, C, or B-Trees but can read & write SQL. That’s technology at its best.

Are you arguing that Kubernetes doesn't do that for you? Because with Kubernetes I can say "run the service in this container with these settings and so many replicas", attach some conditions like "stop sending traffic to any one container that takes longer than N seconds to respond" and "restart the container if a certain command returns an error", and just let it run. I can do a rolling upgrade of the nodes and Kubernetes will reschedule the containers on any other available node, it can load balance traffic, I can update the spec of a deployment and Kubernetes will do a zero-downtime upgrade for me. Try implementing the same on a Unix system. You'd need a way to push configs (Ansible, Puppet, etc?). You need load balancing and leader election (Keepalived?). You need error detection. You need DNS. You need to run the services. You need to ensure there's no library conflict. There's a LOT of complexity that a Kubernetes user does not need to worry about any more. Tell me that's not serious smarts and technology at its best.

What I’m struggling with are thoughts of significant vested commercial interest in exposing this kind of detail, fuelling multi-billion dollar service industries. Feelings of being an outsider despite understanding how it all fits together.

You seem to be conflating Kubernetes and cloud services. Being a cloud native technology does not mean it has to run on a managed cloud service. It just means that it has certain expectations as to how workloads run on it, and if those expectations are met then it makes certain promises about how it will behave.

Have you ever written this kind of software before?

I have contributed to several similar open source projects, yes. What about it?

It sounds like you are comfortable with the status quo of this part of the software industry, and I’m truly jealous!

I am comfortable with my knowledge of this part of the software industry. There is no status quo - there's currently an equilibrium, yes, but it is a tenuous one. I know the tools I use today will likely not be the same tools I will be using a decade from now. But I also know that the concepts and architectures I learn from managing these tools will still be applicable then, and I can stay agile enough to adapt and become comfortable in a new ecosystem. I would urge you to consider the same approach for yourself.