[Docker] Docker Swarm 모드로 stack deploy 시 /bin/sh 커맨드 실행 오류 해결 방법

■ 문제 상황

기존에 Docker Compose로 실행하던 프로젝드로 Docker Swarm 모드로 바꿔서 실행해야해서 작업을 진행하고 있는데, oathkeeper 컨테이너가 계속 오류가 발생해 Exited 되고 있었다.

hyper@node1:/opt/gopath/src/github.com/honeybee/sandbox-multi/test/bdd$ docker container ls -a
CONTAINER ID   IMAGE                                        COMMAND                  CREATED             STATUS                           PORTS                                                                      NAMES
b3ad856e52a2   oryd/oathkeeper:v0.38.4-alpine               "oathkeeper /bin/sh …"   About an hour ago   Exited (255) About an hour ago                                                                              cms_oathkeeper-proxy.1.vukqzhez9nexn1ohfk3ubcy8p
9c0f6e68ac4a   oryd/oathkeeper:v0.38.4-alpine               "oathkeeper /bin/sh …"   About an hour ago   Exited (255) About an hour ago                                                                              cms_oathkeeper-proxy.1.p84ltjg68y14yguko13u2x1hu
acee0a064f32   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            About an hour ago   Up About an hour                 7051/tcp                                                                   sidetree_peer1-org3.1.f9jjzqvr6a51vbbs13xkg1ljy
d900f7a431e4   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            About an hour ago   Up About an hour                 7051/tcp                                                                   sidetree_peer0-org3.1.rv4pinebeh46hwu9swwa8pw4k
5a2059740014   oryd/oathkeeper:v0.38.4-alpine               "oathkeeper /bin/sh …"   About an hour ago   Exited (255) About an hour ago                                                                              cms_oathkeeper-proxy.1.o31rt3e61z9nurxijjkw69u43
9b0e1c5410c7   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            About an hour ago   Up About an hour                 7051/tcp                                                                   sidetree_peer1-org2.1.qexzenlupp5wnar7b53xj4skl
0103295d445d   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            About an hour ago   Up About an hour                 7051/tcp                                                                   sidetree_peer0-org2.1.8x2xrghp9z5vdgy84usxyh13p
cea477bf6290   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            About an hour ago   Up About an hour                 7051/tcp                                                                   sidetree_peer1-org1.1.sri841b5b3nog16xlo29q7hhs
72b263c7e4f0   oryd/oathkeeper:v0.38.4-alpine               "oathkeeper /bin/sh …"   About an hour ago   Exited (255) About an hour ago                                                                              cms_oathkeeper-proxy.1.8bkw5uyzi5hh5mfj6a4x4fvwz
a6ae6e1a51f8   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            About an hour ago   Up About an hour                 7051/tcp                                                                   sidetree_peer0-org1.1.1dxw209ygv25fpmr2m7heqq97
1aa7a2b80297   hyperledger/fabric-orderer:amd64-2.2.1       "orderer"                About an hour ago   Up About an hour                 7050/tcp                                                                   sidetree_orderer.1.hc4d7bbxzspicv3fnlr3j3i1b
fa6f3dc176ba   strapi/strapi:3.0.6-alpine                   "docker-entrypoint.s…"   About an hour ago   Up About an hour                 1337/tcp                                                                   cms_strapi.1.454pbg2cvjklt60031w00m58m
8bc6584a1df1   couchdb:3.1                                  "tini -- /docker-ent…"   About an hour ago   Up About an hour                 4369/tcp, 5984/tcp, 9100/tcp                                               dbs_shared-couchdb.1.6ulwzru0pibr2vdij7ktoj8t4
e219e29b20d1   traefik:v2.4                                 "/entrypoint.sh --pr…"   About an hour ago   Up About an hour                 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   dbs_proxy.1.l2dj6sr874l4mhg1ccew09af7
044e7f8c601a   mysql:8.0.20                                 "docker-entrypoint.s…"   About an hour ago   Up About an hour                 3306/tcp, 33060/tcp                                                        dbs_mysql.1.sv2japlqudiva9u81mqpu5v4k
05dccf7643e2   hyperledger/fabric-tools:amd64-2.0.0-alpha   "//bin/bash -c 'FABR…"   About an hour ago   Exited (0) About an hour ago                                                                                gracious_mahavira
e792cc2146ec   hyperledger/fabric-tools:amd64-2.0.0-alpha   "//bin/bash -c 'FABR…"   About an hour ago   Exited (0) About an hour ago                                                                                great_sutherland

 

cms_oathkeeper-proxy.* 의 이름을 가진 컨테이너가 오류가 발생해 종료되는 컨테이너다.

cms 라는 이름의 stack(스택)을 배포하기 위한 서비스 정의 .yml 파일은 다음과 같다.

#
# SecureKey Technologies Inc. All Rights Reserved.
#
# SPDX-License-Identifier: Apache-2.0
#

version: '3.3'

services:

  oathkeeper-proxy:
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
      placement:
        constraints: [node.hostname == node1]
      labels:
        - traefik.enable=true
        - traefik.docker.network=sandbox_swarm
        - traefik.constraint-label=traefik-public
        - traefik.http.routers.oathkeeper-proxy.rule=Host(`oathkeeper-proxy.trustbloc.local`)
        - traefik.http.routers.oathkeeper-proxy.tls=true
        - traefik.http.routers.oathkeeper-proxy.tls.domains[0].main=*.trustbloc.local
        - traefik.http.routers.oathkeeper-proxy.entrypoints=https
        - traefik.http.services.oathkeeper-proxy.loadbalancer.server.port=4455
    image: oryd/oathkeeper:v0.38.4-alpine
    ports:
      - "4455:4455"
    # will add self-signed certificate to the “trusted list” because oathkeeper doesn't trust self-signed certificate
    # https://github.com/ory/oathkeeper/issues/181
    # remove it when using real certificate
    entrypoint: ""
    command: /bin/sh -c "cp /etc/tls/trustbloc-dev-ca.crt /usr/local/share/ca-certificates/;update-ca-certificates;oathkeeper serve proxy --config /oathkeeper/config.yaml"
    user: root
    environment:
      - LOG_LEVEL=debug
      - PORT=4455
      - ISSUER_URL=https://oathkeeper-proxy.trustbloc.local
      - VIRTUAL_HOST=oathkeeper-proxy.trustbloc.local
      - VIRTUAL_PORT=4455
    restart: on-failure
    volumes:
      - ../oathkeeper:/oathkeeper
      - ../keys/tls:/etc/tls
    networks:
      - sandbox_swarm

  strapi:
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
      placement:
        constraints: [node.hostname == node1]
    hostname: strapi
    image: strapi/strapi:3.0.6-alpine
    entrypoint: ""
    command:  /bin/sh -c "/usr/local/bin/docker-entrypoint.sh strapi;/tmp/strapi_configure.sh& strapi develop"
    environment:
      - DATABASE_CLIENT=mysql
      - DATABASE_HOST=mysql
      - DATABASE_PORT=3306
      - DATABASE_NAME=strapi
      - DATABASE_USERNAME=user
      - DATABASE_PASSWORD=secret
      - VIRTUAL_HOST=cms.trustbloc.local
    ports:
      - 1337:1337
    volumes:
      - ../scripts/strapi_configure.sh:/tmp/strapi_configure.sh
      - ../oathkeeper:/oathkeeper
    networks:
      - sandbox_swarm

networks:
  sandbox_swarm:
    external:
      name: sandbox_swarm

 

해당 컨테이너의 로그를 출력하니 다음과 같은 오류가 출력되었다.

Error: unknown command "/bin/sh" for "oathkeeper"
Run 'oathkeeper --help' for usage.
unknown command "/bin/sh" for "oathkeeper"

 

oathkeeper-proxy 서비스는 /bin/sh -c "cp /etc/tls/trustbloc-dev-ca.crt /usr/local/share/ca-certificates/;update-ca-certificates;oathkeeper serve proxy --config /oathkeeper/config.yaml" command(명령어)를 실행하도록 되어 있는데, /bin/sh 앞에 oathkeeper 명령어가 추가로 붙어서 명령이 제대로 실행되지 않고 있었다.

hyper@node1:$ docker container inspect cms_oathkeeper-proxy.1.vukqzhez9nexn1ohfk3ubcy8p
...(생략)
        "Config": {
            ...(생략)
            "Cmd": [
                "/bin/sh",
                "-c",
                "cp /etc/tls/trustbloc-dev-ca.crt /usr/local/share/ca-certificates/;update-ca-certificates;oathkeeper serve proxy --config /oathkeeper/config.yaml"
            ],
            "Image": "oryd/oathkeeper:v0.38.4-alpine@sha256:7236e11e2d452bcb5c59c705e8fbf0d8545a93f5c3e7f1e271483751c0010976",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "oathkeeper"
            ],
            ...(생략)
            

 

여기서 보면 oathkeeper-proxy 서비스의 entrypoint를 ""로 오버라이드(override) 했는데, 해당 내용이 제대로 적용되지 않아 기존의 Dockerfile에 정의된 entrypoint인 oathkeeper가 설정되었고, 그로 인해 정의한 command 앞에 oathkeeper 명령어가 붙은 것이었다.

 

■ 해결 방법

● /bin/sh를 entrypoint로 설정

entrypoint를 ""에서 "/bin/sh"로 변경하고, command에서 /bin/sh를 제거한다.

...
  oathkeeper-proxy:
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
      placement:
        constraints: [node.hostname == node1]
      labels:
        - traefik.enable=true
        - traefik.docker.network=sandbox_swarm
        - traefik.constraint-label=traefik-public
        - traefik.http.routers.oathkeeper-proxy.rule=Host(`oathkeeper-proxy.trustbloc.local`)
        - traefik.http.routers.oathkeeper-proxy.tls=true
        - traefik.http.routers.oathkeeper-proxy.tls.domains[0].main=*.trustbloc.local
        - traefik.http.routers.oathkeeper-proxy.entrypoints=https
        - traefik.http.services.oathkeeper-proxy.loadbalancer.server.port=4455
    image: oryd/oathkeeper:v0.38.4-alpine
    ports:
      - "4455:4455"
    # will add self-signed certificate to the “trusted list” because oathkeeper doesn't trust self-signed certificate
    # https://github.com/ory/oathkeeper/issues/181
    # remove it when using real certificate
    entrypoint: "/bin/sh"  # 이 부분 수정
    command: -c "cp /etc/tls/trustbloc-dev-ca.crt /usr/local/share/ca-certificates/;update-ca-certificates;oathkeeper serve proxy --config /oathkeeper/config.yaml"  # 이 부분 수정
    user: root
    environment:
      - LOG_LEVEL=debug
      - PORT=4455
      - ISSUER_URL=https://oathkeeper-proxy.trustbloc.local
      - VIRTUAL_HOST=oathkeeper-proxy.trustbloc.local
      - VIRTUAL_PORT=4455
    restart: on-failure
    volumes:
      - ../oathkeeper:/oathkeeper
      - ../keys/tls:/etc/tls
    networks:
      - sandbox_swarm
...

 

 

이렇게 설정하고 실행하면 해당 컨테이너가 문제없이 정상적으로 실행되는 것을 확인할 수 있다!

hyper@node1:$ docker container ls -a
CONTAINER ID   IMAGE                                        COMMAND                  CREATED          STATUS                       PORTS                                                                      NAMES
31055d10a8b7   ghcr.io/trustbloc/edv:0.1.6                  "edv-rest start"         7 minutes ago    Up 7 minutes                                                                                            edv_edv.1.e0lmgsz1irduad7reewldx4vr
f5d65cdcdafd   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            9 minutes ago    Up 9 minutes                 7051/tcp                                                                   sidetree_peer1-org1.1.h7qfk4cc9cmgs8vhtitvuea1m
c1063063b0a1   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            9 minutes ago    Up 9 minutes                 7051/tcp                                                                   sidetree_peer0-org1.1.4w4s1m4woa4s6zzpbwu70k5av
91e1c6bd1373   hyperledger/fabric-orderer:amd64-2.2.1       "orderer"                9 minutes ago    Up 9 minutes                 7050/tcp                                                                   sidetree_orderer.1.oc7rb2608z581h3ymkv0dcd1x
a0e5955ecb29   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            9 minutes ago    Up 9 minutes                 7051/tcp                                                                   sidetree_peer1-org3.1.ttv1wtlaqoy0rv26fetmdnstm
3bc9a5cbf240   oryd/oathkeeper:v0.38.4-alpine               "/bin/sh -c 'cp /etc…"   9 minutes ago    Up 9 minutes                                                                                            cms_oathkeeper-proxy.1.mimdk3ezbotlsmdki5wvxjbcq
cfa558710759   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            9 minutes ago    Up 9 minutes                 7051/tcp                                                                   sidetree_peer0-org3.1.p8450njjbe9fsw4ov3h9lfga8
d451ffa3273b   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            9 minutes ago    Up 9 minutes                 7051/tcp                                                                   sidetree_peer1-org2.1.s8s4ore3lwezlmro7lcbdzbdv
a36b26e487b5   strapi/strapi:3.0.6-alpine                   "docker-entrypoint.s…"   9 minutes ago    Up 9 minutes                 1337/tcp                                                                   cms_strapi.1.jmy2ju8v7d7j4imap5c0fbjm0
0b75eac04cb8   ghcr.io/trustbloc/sidetree-fabric:0.6.0      "fabric-peer"            9 minutes ago    Up 9 minutes                 7051/tcp                                                                   sidetree_peer0-org2.1.30peqfr9a79xctue9sn8m1rwd
7fd4f5f87b98   couchdb:3.1                                  "tini -- /docker-ent…"   9 minutes ago    Up 9 minutes                 4369/tcp, 5984/tcp, 9100/tcp                                               dbs_shared-couchdb.1.nex56jwtpjy29ubu0sk6tbs0b
f929a9c3297d   traefik:v2.4                                 "/entrypoint.sh --pr…"   9 minutes ago    Up 9 minutes                 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   dbs_proxy.1.l9eppmsvev1jd591lb9p5c57q
3b70348066c5   mysql:8.0.20                                 "docker-entrypoint.s…"   10 minutes ago   Up 10 minutes                3306/tcp, 33060/tcp                                                        dbs_mysql.1.lvorxm4m5bbc2qunh7r4r8tht
a1ae4944e840   hyperledger/fabric-tools:amd64-2.0.0-alpha   "//bin/bash -c 'FABR…"   10 minutes ago   Exited (0) 10 minutes ago                                                                               quirky_sanderson
5d0c07677c4e   hyperledger/fabric-tools:amd64-2.0.0-alpha   "//bin/bash -c 'FABR…"   10 minutes ago   Exited (0) 10 minutes ago                                                                               nostalgic_keller

 

entrypoint를 "/bin/sh"로 설정했을 때 오버라이드가 되어 잘 동작하는 것을 보고 ""(공백) 으로 설정하는 것을 인식하지 못하는 것 같아 []로 설정하고 실행해보았는데 이것도 똑같이 인식하지 못했다.

 

Docker Compose에서는 정상적으로 실행이 됐었는데 Docker Swarm 모드에서는 제대로 인식이 안되는 것 같다...