Cross-container/vm communication security on Proxmox
Cross-container/vm communication security on Proxmox
Hey, I've got a bunch of services all running in their own containers/vms on Proxmox. All of these have their own ips that are accessible from my network.
I also have a container with a reverse proxy, which acts as a gateway for access to these services (it's IP is the only one allowed to go through the firewall of each service).
These services have http servers, no encryption. Could someone on my network listen to comms between a service and my reverse proxy?
Would have to play around with VLANs if that's the case...
Thanks
If your goal is to improve security you would have to look into e2e encryption. This means network traffic needs to be encrypted both between client and proxy as well as between proxy and service. Your volumes should be also encrypted. You didn’t elaborate on your proxmox/network setup. I will assume that you have multiple proxmox hosts and external router perhaps with switch between them. Traffic this way flows between multiple devices. With security mindset you’re assuming network can’t be trusted. You need to apply layered approach and use sparation of physical devices, VLANs, ACLs, separate network interfaces for management and services for respective networks. Firewall rules on router, proxmox and VM.
Some solutions
- separate network for VM/CT. Instead of using network routable IP going to your router you can create new bridge on separate CIDR without specifying gateway. Add bridge to every VM that needs connectivity. Use new bridge IPs to communicate between VMs. Further you can configure proxmox to communicate between nodes in ring network P2P instead using switch/router. This requires at least 2 dedicated NICs on Proxmox host. This separates network but doesn’t encrypt.
Encryption:
- You could run another proxy on same VM as service just to encrypt traffic if service doesn’t support that. Then have your proxy connect to that proxy instead of service directly. This way unencrypted traffic doesn’t leave VM. Step up would be to use certificate validation. Step up from there would be to use internal certificate authority and issue certificates from there as well as validate using CA cert.
- Another alternative is to use overlay network between proxy and VM. There are bunch of different options. Hashicorp consul network could be interesting project. There are more advanced projects combining zero trust concepts like nebula.
- if you start building advanced overlay networks you may as well look at kubernetes as it streamlines deployment of both services and underlying infrastructure. You could deploy calico with wire guard network. Setup gets more complicated for a simple home lab.
All boils down to the question why you do self hosting? If it’s to learn new tech then go for it all the way. Experiment and fail often so you learn what works and what doesn’t. If you want to focus on reliability and simplicity don’t overcomplicate things. You will spend too much time troubleshooting and have your services unavailable. Many people run everything on single node just running docker with networks between services to separate internal services from proxy traffic. Simplicity trumps everything if you can’t configure complex networks securely.
I have just one proxmox hosts which runs everything.
I wasn't sure if there was any traffic leaked out of the bridge (as it would be to everyone with wifi), though the more I think about it, the less sense it'd make for that to be the case.
I self-host for a myriad of reasons, including a dev server, so ideally I need uptime. Might look into some more advanced stuff when I've got time though.
Appreciate the response btw!
Unless someone has physical access to the ports/switch that the traffic flows through, they would not be able to see anything besides broadcast/multicast traffic if they were just snooping with Wireshark. The internal switch of proxmox and any hardware switch you have will forward unicast traffic to the ports those Mac's reside on, so without port mirrors setup, no one but you should be able to see that traffic.
Good to hear, thanks!
Encryption would be a good idea between the service and the reverse proxy. I’d also look into Proxmox firewall, which is pretty robust. Here’s the link to the documentation: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pve_firewall. Definitely worth perusing.
I've got firewall set up, services are only accessible through the reverse proxy, was more concerned about something like logging into keycloak and having the password leaked with MitM or another attack of the sort.
Gotcha. That makes sense. My own thoughts are that if you mitigate all of the attack surfaces you can, it ends up coming down to the robustness of the particular app. I’ve never played with keycloak, so can’t speak intelligently about that, but I’ve got authentik setup in a similar configuration. I limit access so the only way in is via either the reverse proxy or the PVE console and basically keep an ear out for security and software updates.
As I type this, it occurs to me that perhaps there’s a fail2ban integration that could be added to limit credential guessing at the keycloak webui.
Is the reverse proxy also on Proxmox/the same hardware?
I am interested in this topic as well