Looking over k3s and it really pushes the IoT/embedded aspect.
Any comments on what makes it different to a server-oriented k8s solution? Why wouldnt I run it on servers?
- by default packaged as a single binary, so is extremely easy to install, even uses SQLite for storage (though that can be swapped out for others as well)
- includes the functionality that you'd expect, like local storage, load balancing, ingress, while at the same time getting rid of some of the unnecessary plugins that you'd get in other distros
- as a consequence of the above, has a small runtime footprint, so running K8s clusters on VPSes with 2 to 4 GB of RAM is no longer a pipe dream, also has way less overhead on the actual nodes that you want to manage (think along the lines of a few hundred MB)
- also includes a variety of tools with it for managing your cluster more easily, so you don't need to install those separately
k3s, by default, uses sqlite for storage instead of etcd. This is one of the ways you can get a performance improvement.
If you are going to run a cluster you're likely going to want to have your database be an HA cluster. While k3s can support other databases, like PostgreSQL, it doesn't have handling for clusters in the config.
This is a limitation I see for k3s in clusters, right now.
I would love to see cluster handling in k3s. etcd can be a scaling bottleneck and replacing it with PostgreSQL could help it scale much higher.
Disclaimer, I work on Rancher Desktop which uses k3s
At my job, we use k3s not for IoT or embedded, but for deployment to back-office "servers" in field offices for deployments of services that don't need HA in that environment.
I'm running it on an Oracle Cloud Free Tier server, it's nice because the 1Gb of ram they include is less than the requirement for k8s but plenty for k3s