docker seems to start but can no longer connect to it

  • I'm running LE 8.x with Kodi 17.6 and have installed docker from LE Add-on Services. I run some custom Server applications inside that docker and it had been running quite well for several months.

    Since last month or two docker has stopped running properly. Even after a reboot my script that connected to docker is no longer able to do so:

    docker -H tcp://0.0.0.0:2900 run -v <some_parameters> node myAppServer.js

    This command throws an error :

    docker: Cannot connect to the Docker daemon at tcp://0.0.0.0:2900. Is the docker daemon running?.

    I have confirmed that docker is running via 'systemctl status docker' command

    systemctl status docker returns:


    ---------------- start of status -----------------

    service.system.docker.service - Docker Application Container Engine

    Loaded: loaded (/storage/.kodi/addons/service.system.docker/system.d/service.system.docker.service; enabled; vendor preset: disa

    bled)

    Active: active (running) since Sun 2019-07-07 05:25:10 EDT; 7min ago

    Docs: Docker Documentation | Docker Documentation

    Process: 3543 ExecStartPre=/storage/.kodi/addons/service.system.docker/bin/docker-config (code=exited, status=0/SUCCESS)

    Main PID: 3546 (dockerd)

    CGroup: /system.slice/service.system.docker.service

    3546 /storage/.kodi/addons/service.system.docker/bin/dockerd --exec-opt native.cgroupdriver=systemd --log-driver=j

    ournald --group=root --graph=/storage/.kodi/userdata/addon_data/service.system.docker/docker --storage-driver=overlay2 --storage-op

    t overlay2.override_kernel_check=1

    3553 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start

    -timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --runtime-args

    --systemd-cgroup=true

    Jul 07 05:25:08 libreelec dockerd[3546]: time="2019-07-07T05:25:08.023889000-04:00" level=error msg="Failed to load container mount 1889

    d5fc421e3d2de52191371d2a500a05a59a12cadf4462b9b40d73e05eb77c: mount does not exist"

    Jul 07 05:25:08 libreelec dockerd[3546]: time="2019-07-07T05:25:08.695033000-04:00" level=info msg="Firewalld running: false"

    Jul 07 05:25:08 libreelec dockerd[3546]: time="2019-07-07T05:25:08.797745000-04:00" level=warning msg="Could not load necessary modules

    for IPSEC rules: Running modprobe xfrm_user failed with message: `modprobe: WARNING: Module xfrm_user not found in directory /lib/m

    odules/3.14.29`, error: exit status 1"

    Jul 07 05:25:09 libreelec dockerd[3546]: time="2019-07-07T05:25:09.184144000-04:00" level=info msg="Default bridge (docker0) is assigned

    with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"

    Jul 07 05:25:10 libreelec dockerd[3546]: time="2019-07-07T05:25:10.610251000-04:00" level=info msg="Loading containers: done."

    Jul 07 05:25:10 libreelec dockerd[3546]: time="2019-07-07T05:25:10.824833000-04:00" level=warning msg="Not using native diff for overlay

    2: opaque flag erroneously copied up, consider update to kernel 4.8 or later to fix"

    Jul 07 05:25:10 libreelec dockerd[3546]: time="2019-07-07T05:25:10.849314000-04:00" level=info msg="Daemon has completed initialization"

    Jul 07 05:25:10 libreelec dockerd[3546]: time="2019-07-07T05:25:10.849477000-04:00" level=info msg="Docker daemon" commit=1.13.1 graphdr

    iver=overlay2 version=1.13.1

    Jul 07 05:25:10 libreelec systemd[1]: Started Docker Application Container Engine.

    Jul 07 05:25:10 libreelec dockerd[3546]: time="2019-07-07T05:25:10.916821000-04:00" level=info msg="API listen on /var/run/docker.sock"

    ---------------- end of status. -----------------

    I have uninstalled and reinstalled docker and it still doesn't work.

    I did notice one odd thing though - I left the system running for several days and then tried to start my custom application in docker and it WORKED! so it is almost as if upon reboot it fails for some reason and takes a while (could be several minutes, hours or days - no idea) and will then allow me to connect. This happened once, I have since rebooted the box and am yet to connect successfully.

    Please advice on how to resolve this. also is there any change that happened to docker that would have caused this to stop working ?
    My first install of docker add-on had "auto-update" enabled, so it may have updated docker automatically causing some incompatibility that broke this. I did not keep track of docker version that was first installed vs one that I have now so I don't know for sure if an upgrade caused this broken functionality.

    Thanks in advance.

  • journalctl -r gets me the following logs for docker:

    --------- start of logs ------------

    Jul 07 05:36:30 libreElec nmbd[2851]: This response was from IP 192.168.222.41, reporting an IP address of 192.168.222.41.

    Jul 07 05:36:30 libreElec nmbd[2851]: query_name_response: Multiple (2) responses received for a query on subnet 192.168.222.160 for name WORKGROUP<1d>.

    Jul 07 05:36:30 libreElec nmbd[2851]: [2019/07/07 05:36:30.533454, 0] ../source3/nmbd/nmbd_namequery.c:109(query_name_response)

    Jul 07 05:31:30 libreElec nmbd[2851]: This response was from IP 192.168.222.41, reporting an IP address of 192.168.222.41.

    Jul 07 05:31:30 libreElec nmbd[2851]: query_name_response: Multiple (2) responses received for a query on subnet 192.168.222.160 for name WORKGROUP<1d>.

    Jul 07 05:31:30 libreElec nmbd[2851]: [2019/07/07 05:31:30.903969, 0] ../source3/nmbd/nmbd_namequery.c:109(query_name_response)

    Jul 07 05:28:32 libreElec connmand[2760]: ntp: adjust (slew): +0.014074 sec

    Jul 07 05:26:30 libreElec nmbd[2851]: This response was from IP 192.168.222.41, reporting an IP address of 192.168.222.41.

    Jul 07 05:26:30 libreElec nmbd[2851]: query_name_response: Multiple (2) responses received for a query on subnet 192.168.222.160 for name WORKGROUP<1d>.

    Jul 07 05:26:30 libreElec nmbd[2851]: [2019/07/07 05:26:30.968947, 0] ../source3/nmbd/nmbd_namequery.c:109(query_name_response)

    Jul 07 05:26:25 libreElec systemd[1]: Started Cleanup of Temporary Directories.

    Jul 07 05:26:25 libreElec systemd[1]: Starting Cleanup of Temporary Directories...

    Jul 07 05:25:10 libreElec dockerd[3546]: time="2019-07-07T05:25:10.916821000-04:00" level=info msg="API listen on /var/run/docker.sock"

    Jul 07 05:25:10 libreElec systemd[1]: Started Docker Application Container Engine.

    Jul 07 05:25:10 libreElec dockerd[3546]: time="2019-07-07T05:25:10.849477000-04:00" level=info msg="Docker daemon" commit=1.13.1 graphdriver=overlay2 version=1.13.1

    Jul 07 05:25:10 libreElec dockerd[3546]: time="2019-07-07T05:25:10.849314000-04:00" level=info msg="Daemon has completed initialization"

    Jul 07 05:25:10 libreElec dockerd[3546]: time="2019-07-07T05:25:10.824833000-04:00" level=warning msg="Not using native diff for overlay2: opaque flag erroneously copied up, consider update to kernel 4.8 or later to fix"

    Jul 07 05:25:10 libreElec dockerd[3546]: time="2019-07-07T05:25:10.610251000-04:00" level=info msg="Loading containers: done."

    Jul 07 05:25:09 libreElec dockerd[3546]: time="2019-07-07T05:25:09.184144000-04:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"

    Jul 07 05:25:08 libreElec dockerd[3546]: time="2019-07-07T05:25:08.797745000-04:00" level=warning msg="Could not load necessary modules for IPSEC rules: Running modprobe xfrm_user failed with message: `modprobe: WARNING: Module xfrm_user not found in directory /lib/modules/3.14.29`, error: exit status 1"

    Jul 07 05:25:08 libreElec dockerd[3546]: time="2019-07-07T05:25:08.695033000-04:00" level=info msg="Firewalld running: false"

    Jul 07 05:25:08 libreElec dockerd[3546]: time="2019-07-07T05:25:08.023889000-04:00" level=error msg="Failed to load container mount 1889d5fc421e3d2de52191371d2a500a05a59a12cadf4462b9b40d73e05eb77c: mount does not exist"

    Jul 07 05:25:07 libreElec dockerd[3546]: time="2019-07-07T05:25:07.960196000-04:00" level=info msg="Loading containers: start."

    Jul 07 05:25:07 libreElec dockerd[3546]: time="2019-07-07T05:25:07.959342000-04:00" level=warning msg="mountpoint for pids not found"

    Jul 07 05:25:07 libreElec dockerd[3546]: time="2019-07-07T05:25:07.959063000-04:00" level=warning msg="Your kernel does not support cgroup blkio weightt_device"

    Jul 07 05:25:07 libreElec dockerd[3546]: time="2019-07-07T05:25:07.959009000-04:00" level=warning msg="Your kernel does not support cgroup blkio weightt"

    Jul 07 05:25:07 libreElec dockerd[3546]: time="2019-07-07T05:25:07.958960000-04:00" level=warning msg="Your kernel does not support cgroup rt runtime"

    Jul 07 05:25:07 libreElec dockerd[3546]: time="2019-07-07T05:25:07.958914000-04:00" level=warning msg="Your kernel does not support cgroup rt period"

    Jul 07 05:25:07 libreElec dockerd[3546]: time="2019-07-07T05:25:07.958864000-04:00" level=warning msg="Your kernel does not support cgroup cfs quotas"

    Jul 07 05:25:07 libreElec dockerd[3546]: time="2019-07-07T05:25:07.958761000-04:00" level=warning msg="Your kernel does not support cgroup cfs period"

    Jul 07 05:25:07 libreElec dockerd[3546]: time="2019-07-07T05:25:07.958047000-04:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"

    Jul 07 05:25:07 libreElec dockerd[3546]: time="2019-07-07T05:25:07.563168000-04:00" level=warning msg="Using pre-4.0.0 kernel for overlay2, mount failures may require kernel update"

    Jul 07 05:25:06 libreElec dockerd[3546]: time="2019-07-07T05:25:06.527650000-04:00" level=info msg="libcontainerd: new containerd process, pid: 3553"

    Jul 07 05:25:06 libreElec systemd[1]: Starting Docker Application Container Engine...

    Jul 07 05:25:06 libreElec systemd[1]: Stopped Docker Application Container Engine.

    Jul 07 05:25:06 libreElec kodi.sh[3171]: EOFError: socket closed

    Jul 07 05:25:06 libreElec kodi.sh[3171]: raise EOFError('socket closed')

    Jul 07 05:25:06 libreElec kodi.sh[3171]: File "/storage/.kodi/addons/service.system.docker/lib/dockermon.py", line 94, in watch

    Jul 07 05:25:06 libreElec kodi.sh[3171]: dockermon.watch(print_notification)

    Jul 07 05:25:06 libreElec kodi.sh[3171]: File "/storage/.kodi/addons/service.system.docker/default.py", line 266, in run

    Jul 07 05:25:06 libreElec kodi.sh[3171]: File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner

    Jul 07 05:25:06 libreElec kodi.sh[3171]: Traceback (most recent call last):

    Jul 07 05:25:06 libreElec kodi.sh[3171]: Exception in thread Thread-1:

    Jul 07 05:25:06 libreElec dockerd[2790]: time="2019-07-07T05:25:06.321981000-04:00" level=error msg="libcontainerd: failed to receive event from containerd: rpc error: code = 13 desc = transport is closing"

    Jul 07 05:25:06 libreElec systemd[1]: Stopping Docker Application Container Engine...

    Jul 07 05:25:06 libreElec dockerd[2790]: time="2019-07-07T05:25:06.312042000-04:00" level=info msg="Processing signal 'terminated'"

    Jul 07 05:25:06 libreElec dockerd[2790]: time="2019-07-07T05:25:06.311994-04:00" level=info msg="stopping containerd after receiving terminated"

    ------------ end of log --------------