Recently I built docker images for omnibus builds. Omnibus packages must be built on the target distro so I needed images for centos, debian and ubuntu. Usually I build such docker images on my laptop which is running Ubuntu; I try to make the builds as repeatable as possible using the excellent packer tool and when I publish images I build them again (using a build server) on cloud instances. In this case, I was using GCE CentOS 7 instance to perform the build and I was surprised when a build that was smooth on my laptop failed miserably. The build was for Ubuntu precise and the build step that failed was the the creation of the omnibus user. This was really weird since creating a user is a trivial task that was unexpected to fail.
After I did a little digging, I found that running the following command in an Ubuntu container running on CentOS 7 host fails with return code 12:
useradd -m -d /home/test test
On an Ubuntu host using the same image (ubuntu:precise) or a CentOS image (centos:centos6) on either CentOS or Ubuntu host works fine. I checked man useradd and found that return code 12 is “can’t create home directory”, which immediately pointed towards SELinux as the culprit. Ubuntu does not use SELinux and I suspected it was blocking the Ubuntu userland tools. I ran setenforce 0 to switch SELinux to permissive mode and tried again, but got return code 12 once more… weird.
It was time to bring in the heavy guns. I ran the container again, this time in privileged mode and used strace to see what useradd was doing:
....
access("/home/test", F_OK) = -1 ENOENT (No such file or directory)
open("/proc/filesystems", O_RDONLY) = 8
fstat(8, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f64448c4000
read(8, "nodev\tsysfs\nnodev\trootfs\nnodev\tb"..., 1024) = 310
gettid() = 55
open("/proc/self/task/55/attr/current", O_RDONLY) = 9
read(9, "system_u:system_r:svirt_lxc_net_"..., 4095) = 47
close(9) = 0
close(8) = 0
munmap(0x7f64448c4000, 4096) = 0
open("/etc/selinux/config", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/selinux/targeted/contexts/files/file_contexts.subs_dist", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/selinux/targeted/contexts/files/file_contexts.subs", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/selinux/targeted/contexts/files/file_contexts", O_RDONLY) = -1 ENOENT (No such file or directory)
....
close(3) = 0
exit_group(12) = ?
It seems that useradd is trying to read selinux config files to get the configuration of the context to set for new users. Of course, Ubuntu images don’t have any such configuration files so it fails. I then looked at the source code of the shadow package:
--- useradd.c
1763 static void create_home (void)
1764 {
1765 if (access (user_home, F_OK) != 0) {
1766 #ifdef WITH_SELINUX
1767 if (set_selinux_file_context (user_home) != 0) {
1768 fail_exit (E_HOMEDIR);
1769 }
1770 #endif
--- libmisc/copydir.c
126 int set_selinux_file_context (const char *dst_name)
127 {
128 /*@null@*/security_context_t scontext = NULL;
129
130 if (!selinux_checked) {
131 selinux_enabled = is_selinux_enabled () > 0;
132 selinux_checked = true;
133 }
134
135 if (selinux_enabled) {
136 /* Get the default security context for this file */
137 if (matchpathcon (dst_name, 0, &scontext) < 0) {
138 if (security_getenforce () != 0) {
139 return 1;
140 }
141 }
For some reason, it seems some Ubuntu userland tools are compiled with SELinux support; They detect the presence the of SELinux and try to set the default context for the new user. So although SELinux was not blocking useradd (SELinux was switched to permissive mode) useradd still failed because it tried to use a feature that is not supported in Ubuntu.
This is clearly a problem in the Ubuntu image and not docker itself but it does highlight the degree of reliance distribution packages have on implicit and sometime undocumented configurations and APIs. While docker enthusiasts claim you can “run any app anywhere” this is unfortunately not true in many cases. Many userland tools are coupled to kernel features, kernel modules, distro specific kernel configurations, etc. Moreover, this example shows that generic linux/gnu tools may behave differently in the presence of different kernels despite claims of transparent compatibility. Over the years we have built a complex web of interdependence between kernelspace, userspace, compile-time configurations and runtime configurations; it will takes years to untangle this mess and most distributions seem to be interested in other tasks. sigh.