1y ago

docker conflicts with host network, causing no internet connectivity

Hello! Thanks to everyone who helped me yesterday, but I could not solve the issue. I however identified the problem. I opened a new thread to better focus on the real problem.

I just set up a debian server, installed docker from the official repo and then immich using docker compose. When I run docker compose up -d (after a few seconds) my server loses internet connectivity: it can access local addresses (I can ssh into it) but cannot access the internet: ping linux.org fails.

if I put down immich (docker compose down) it starts working again. the issue is probably that docker is creating a network bridge that conflicts with the host.

After searching the web, I tried to create /etc/docker/daemon.json like this:

 json

    
{
  "bip": "172.18.0.1/24",
  "default-address-pools": [
    {"base": "172.19.0.0/16", "size": 24}
  ]
}

after the changes systemctl restart docker, then wait 10 seconds for everything to restart correctly.
I tried different configurations of addresses here (every stackoverflow answer gave different values, so I tried all of them), but none of them worked. I don't know how to get which values to put here (if this is actually the solution)

a strange behavior I observed is that running ip route flush 0/0 temporary solves the problem, until the restart of docker, and Immich works normally (at least the "normal" behavior, I don't know if this affects some functionalities)

Any tips? This is my really first experience in self hosting and I have to admit it, I thought it would have been easier :P

In case you'll need it, here's the output of ip addr show:

 undefined

    
0.0.0.0 dev veth4c84e92 scope link
0.0.0.0 dev veth1f88dcc scope link
0.0.0.0 dev vethda721de scope link
0.0.0.0 dev vethd123481 scope link
0.0.0.0 dev veth23a05f6 scope link
default dev veth4c84e92 scope link # this line and the line below disappear after the flush
default dev veth1f88dcc scope link # (this one)
default via 192.168.1.1 dev enp1s0
169.254.0.0/16 dev veth23a05f6 proto kernel scope link src 169.254.6.247
169.254.0.0/16 dev vethd123481 proto kernel scope link src 169.254.226.60
169.254.0.0/16 dev vethda721de proto kernel scope link src 169.254.248.163
169.254.0.0/16 dev veth1f88dcc proto kernel scope link src 169.254.136.146
169.254.0.0/16 dev veth4c84e92 proto kernel scope link src 169.254.29.133
169.254.0.0/16 dev enp1s0 scope link metric 1000
172.16.0.0/20 dev br-237d14e56e71 proto kernel scope link src 172.16.0.1
172.18.0.0/24 dev docker0 proto kernel scope link src 172.18.0.1 linkdown
192.168.1.0/24 dev enp1s0 proto kernel scope link src 192.168.1.4
192.168.1.1 dev enp1s0 scope link

EDIT: I gave up. I removed debian and installed fedora, and now it all works like a charm

14 comments

Humm... this seems rather strange. Maybe show us you docker-compose to have a look on how you configured immich's network?
169.254.0.0/16 are APIPA addresses . So this a network misconfiguration.
After searching the web, I tried to create /etc/docker/daemon.json
This is not how you configure a docker network. This is only used if your local networks overlaps with docker's defaut network.
The easiest way i can think on how to make your docker-compose work is to reinstall docker and use the host network. DO NOT forget to delete the /etc/docker/daemon.json file in case you want a fresh start. This seems a badly network configuration on you docker stack !
This will give you a good starting point! After that try to configure a bridge network for your docker compose.
Normally after you get the gist on how docker works, it's rather easy!
- Maybe show us you docker-compose to have a look on how you configured immich’s network?
  I didn't change anything, just followed the instructions on the wiki:
  undefined
  
  version: "3.8" # # WARNING: Make sure to use the docker-compose.yml of the current release: # # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml # # The compose file on main may not be compatible with the latest release. # name: immich services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} command: [ "start.sh", "immich" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env ports: - 2283:3001 depends_on: - redis - database restart: always immich-microservices: container_name: immich_microservices image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} # extends: # file: hwaccel.yml # service: hwaccel command: [ "start.sh", "microservices" ] volumes: - ${UPLOAD_LOCATION}:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro env_file: - .env depends_on: - redis - database restart: always immich-machine-learning: container_name: immich_machine_learning image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release} volumes: - model-cache:/cache env_file: - .env restart: always redis: container_name: immich_redis image: redis:6.2-alpine@sha256:c5a607fb6e1bb15d32bbcf14db22787d19e428d59e31a5da67511b49bb0f1ccc restart: always database: container_name: immich_postgres image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee env_file: - .env environment: POSTGRES_PASSWORD: ${DB_PASSWORD} POSTGRES_USER: ${DB_USERNAME} POSTGRES_DB: ${DB_DATABASE_NAME} volumes: - pgdata:/var/lib/postgresql/data restart: always volumes: pgdata: model-cache:
  The easiest way i can think on how to make your docker-compose work is to reinstall docker and use the host network
  I'll try as soon as I can and post here the results, thanks for the time and help!
  This seems a badly network configuration on you docker stack !
  oops, I have lots to learn I guess XD
  
  So I got it up and running in 10 minutes just by copy/pasting the docker-compose.yaml and .env files. So their configuration files are working flawlessly.
  Either you have a router misconfiguration or a docker network misconfiguration. Either way If I were you I would first start without duckdns.org domain name and without to much complex network configuration. Start slow and build up to more complex configurations.
  Leave your router defaults network configuration, without any open ports.
  See if your spare laptop server has internet access when everything is defaulted (if not that's the first thing to solve)
  Check if your networks configuration is in the inet 172.17.0.1/16 brd 172.17.255.255 range (dockers default bridge network)
  Default routes on your laptop
  DHCP or manual
  
  The important part is to make your laptop have internet access without changing to much, the default DHCP works great !
  Fresh docker installation and don't forget to delete your json file (/etc/docker/daemon.json)
  Try again with the docker-compose.yaml and .env from immich's github
  If your network configuration is wrong from the beginning, you are in for bad times specially if you are going to use duckdns ! Try to make it work on your local network first and than you can go crazy.
  Also if you do not know what you are doing, please don't make your containers accessible to the web ! Rather use a wireguard server to access all your containers from everywhere in the world with a secure tunnel !
  If you're a beginner, there is alot to grasp before having a good working laptop server :)
  
  Immich isn't the easiest docker stack ! I will up it on my own server and give you some feedback.
  Because their isn't any network configuration in the compose it uses the default docker network. Thus... maybe... it overlaps with your own network, that's what your first post was about :)
  I will look into it and if nobody comes up with an anwser before me, will give you some feedback on how it went and try to find out what's wrong !
  
  First find out what's your debian network configuration if it's a fresh install and everything is installed by default, you get your ip and network from DHCP
  undefined
  
  > ip a
  If your ethernet or wifi networks is in the inet 172.17.0.1/16 brd 172.17.255.255 range it overlaps with the default docker bridge network.

What firewall are you using? Docker doesn't like non-iptables firewalls and it has been more than once that I changed my nftables config and really the whole networking stack to figure that out. I have a ubuntu server vm which had some iptables save-restore unit activated which was messing with my rules, that was fun to debug.
- The way Docker interacts with the firewall as if it owned the whole system and in particular that this ancient issue is still open without any official comment is one of the reasons I consider the Docker project to be one of the worst widely used open source projects out there.
- I haven't set up a firewall yet, I wanted first to make sure that everything works before installing one (I was thinking of ufw)
  I have a clean Debian install, I don't know if there are some rules OOTB that can interfere...
  
  Better check, you definitely already have a firewall running since docker needs it for NAT. A fresh debian has, as far as I know nftables and iptables-nft installed.

Try perhaps the solution in https://unix.stackexchange.com/questions/648084/docker-interface-tears-down-wifi-internet
- Debian does not use NetworkManager I think, because there is no /etc/NetworkManager/ directory :(

14 comments