I have recently been learning about podman, a tool for running containers that has a command syntax that matches Docker, but that does not require a Docker daemon and which does not require root privileges.
I ran into some unexpected problems publishing ports with Podman, which had to do with my default DROP policy on the iptables FORWARD chain. Below I will demonstrate some of the differences between Docker and Podman in terms of iptables changes, and provide a workaround for Podman.
To test the differences, I used Amazon AWS EC2 t2.nano instances based on the CentOS 7 (x86_64) – with Updates HVM AMI. In my test I am going to run a mock HTTP server using netcat, so I opened port 8080 to the world in the AWS security group for these EC2 instances.
Docker
After launching the EC2 instance, I ran through the following steps to configure the host:
sudo yum updatesudo reboot- Follow steps to install Docker on CentOS:
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.reposudo yum install docker-ce
sudo yum install iptables-service
Change the FORWARD chain policy to DROP in /etc/sysconfig/iptables:
:FORWARD DROP [0:0]
Restart iptables:
sudo systemctl restart iptables
I tried running a simple container:
sudo docker run --rm -it --public 8080:80 alpine sh
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
Oh right, the Docker daemon needed to be running:
sudo systemctl start docker
I ran the simple container again, this time successfully:
sudo docker run --rm -it --public 8080:80 alpine sh
From inside the container, I started netcat listening on port 80:
# nc -l -p 80
I was then able to connect to the container from the outside world, e.g. my desktop, like so:
curl 3.85.222.191:8080
I saw the request header appear via netcat within the container:
GET / HTTP/1.1
Host: 3.85.222.191:8080
User-Agent: curl/7.56.1
Accept: */*
From there I entered:
Hello, world
[^d]
And confirmed that the response text was received by curl. The entire communication was successful.
Docker sets up some routes so that traffic to a container IP address is handled by the docker0 interface:
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default ip-172-31-80-1. 0.0.0.0 UG 0 0 0 eth0
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.31.80.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
Docker makes a lot of changes to iptables rules, most of which are filter rules (there are a couple tweaks to NAT rules, which have ignored for now). Primarily I want to focus on the filter rules in the FORWARD chain, since that chain directly affects packets coming in on the eth0 interface that are destined for the docker0 interface.
$ sudo iptables -v -L FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
39 2093 DOCKER-USER all -- any any anywhere anywhere
39 2093 DOCKER-ISOLATION-STAGE-1 all -- any any anywhere anywhere
7 621 ACCEPT all -- any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED
13 660 DOCKER all -- any docker0 anywhere anywhere
19 812 ACCEPT all -- docker0 !docker0 anywhere anywhere
0 0 ACCEPT all -- docker0 docker0 anywhere anywhere
0 0 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
Let’s look at what each rule is doing:
- All forwarded packets are sent to the DOCKER-USER chain. In this case, the DOCKER-USER chain RETURNs all packets, and so are further processed by the other FORWARD chain rules.
- All forwarded packets are sent to the DOCKER-ISOLATION-STAGE-1 chain. This one is a bit more complex, so I’m going to wave my hands and say that all the packets I sent survived this rule and are RETURNed for further processing.
- Any packets forwarded from any interface to the
docker0 interface are accepted if the connection state is RELATED or ESTABLISHED.
- Any packets forwarded from any interface to the
docker0 interface are directed to the DOCKER chain. More on this in a minute, but our connection should be ACCEPTed here.
- Any packets from the
docker0 interface to any other interface are accepted
- Any packets from the
docker0 interface to the docker0 interface are accepted
- All other packets are REJECTed
The DOCKER chain is only populated while the container is running, so if you run iptables while the container is stopped you may not see any rules here:
$ sudo iptables -v -L DOCKER
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
1 52 ACCEPT tcp -- !docker0 docker0 anywhere ip-172-17-0-2.ec2.internal tcp dpt:http
This rule accepts TCP packets from interfaces other than docker0 destined for docker0 on destination port 80.
Podman
After launching the EC2 instance, I ran through the following steps to configure the host:
sudo yum updatesudo rebootsudo yum install podmansudo yum install iptables-services
Change the FORWARD chain policy to DROP in /etc/sysconfig/iptables:
:FORWARD DROP [0:0]
Restart iptables:
sudo systemctl restart iptables
I ran a simple container:
sudo podman run --rm -it --publish 8080:80 alpine sh
From inside the container, I started netcat listening on port 80:
# nc -l -p 80
I attempted to connect to the container from the outside world, e.g. my desktop:
curl 18.206.214.125:8080
However, the connection failed due to a timeout.
I attempted to connect to the container from its host:
curl 10.88.0.1:8080
This worked, and I see the request header appear in the container as expected:
GET / HTTP/1.1
User-Agent: curl/7.29.0
Host: 10.88.0.1:8080
Accept: */*
Why wasn’t the connection completing from my desktop? I suspected that something wasn’t routing between the interfaces correctly.
In the routing table, note that the CNI (Container Networking Interface) used by Podman uses a different default range of RFC 1918 IP addresses (10.88.0.0/16 instead of Docker’s 172.17.0.0/16):
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default ip-172-31-80-1. 0.0.0.0 UG 0 0 0 eth0
10.88.0.0 0.0.0.0 255.255.0.0 U 0 0 0 cni0
link-local 0.0.0.0 255.255.0.0 U 1002 0 0 eth0
172.31.80.0 0.0.0.0 255.255.240.0 U 0 0 0 eth0
I looked at the iptables FORWARD chain, where is was clear that all forwarded packets first jump to the CNI-FORWARD chain:
$ sudo iptables -v -L FORWARD
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
4 212 CNI-FORWARD all -- any any anywhere anywhere /* CNI firewall plugin rules */
4 212 REJECT all -- any any anywhere anywhere reject-with icmp-host-prohibited
Next I looked at the iptables CNI-FORWARD chain:
$ sudo iptables -v -n -L CNI-FORWARD
Chain CNI-FORWARD (1 references)
pkts bytes target prot opt in out source destination
4 212 CNI-ADMIN all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI firewall plugin rules */
0 0 ACCEPT all -- * * 0.0.0.0/0 10.88.0.11 ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- * * 10.88.0.11 0.0.0.0/0</pre>
<ol>
<li>The CNI-FORWARD chain first jumps to the CNI-ADMIN chain, which in this case was empty.</li>
<li>Next it ACCEPTs any packets from RELATED or ESTABLISHED connections destined for the container IP address.</li>
<li>Next it ACCEPTS any packets from the container to any destination</li>
</ol>
If none of those rules match, it returns to the FORWARD chain where it hits the REJECT rule. What's missing is a rule that allows NEW connection packets destined for the container. There are a variety of ways to do this. At first I tried adding an overly broad rule directly to the FORWARD chain:
<pre><code>sudo iptables -I FORWARD -p tcp ! -i cni0 -o cni0 -j ACCEPT
That rule ACCEPTs any TCP packets from an interface other than cni0 bound for cni0. And it worked: after added that rule I was able to reach the container from my desktop via port 8080. However, that rule allows much more than was Docker allows. I removed that rule and added a more specific rule to the CNI-FORWARD chain:
sudo iptables -I CNI-FORWARD -p tcp ! -i cni0 -o cni0 -d 10.88.0.11 --dport 80 -j ACCEPT
That also worked, and much more closely resembled what Docker added to iptables.
Why doesn’t Podman add the necessary iptables rule automatically?
Good question. I’m using the podman package provided via yum from the CentOS extras repository, which provides podman version 0.11.1.1. It may be that future releases will address this. It may also be the case that Podman does not anticipate a default DROP policy on the FORWARD chain (although default DROP is best practice).
In any case, it is useful to know a little bit more about how container tools use iptables. It is also important to note that, at least at this point, Docker is easier to use than Podman.
