Docker Container Unresponsive After 1-2 Days with ShinyProxy Deployment

zhaohongxin0 · October 6, 2024, 5:34am

Hello ShinyProxy Community,

I am experiencing a recurring issue with ShinyProxy where Docker containers used to serve Shiny apps become unresponsive after running continuously for one or two days. Below are some details about my setup and the errors I am seeing:

ShinyProxy Configuration:

Using Docker Swarm as the backend
Containers are set to use 4 CPU cores and 6GB memory each
ShinyProxy configuration includes several apps deployed with container-network: sp-example-net
ShinyProxy configuration allows for Docker pre-initializing, but container re-use is disabled

Here is an excerpt from my configuration:

Problem Details:

After one or two days, I can access the ShinyProxy menu, but clicking on an app link results in the app failing to start, sometimes displaying a blank page. The ShinyProxy navigation bar and user information are displayed, but the app interface itself remains blank.
Restarting the Docker Swarm service running ShinyProxy temporarily resolves the issue, but it reoccurs every one or two days.
At any time, I can successfully log in using OpenID authentication. However, after logging in, I am unable to launch any Shiny app. The app page remains blank, but the ShinyProxy navigation bar and user information are still visible. Restarting the Docker Swarm service running ShinyProxy is required to restore functionality.

Relevant Logs (ShinyProxy):

The following log entries show the issue occurring repeatedly before the containers become unresponsive:

I noticed that the logs repeatedly mention that the Docker Swarm service is “not ready yet” and that the container is “unresponsive.” This seems to indicate a timeout or scaling issue. The containers are already locally available, so they should not need to be pulled again, yet ShinyProxy keeps retrying.

My Observations:

The problem seems periodic, usually after the container has been running for a couple of days.
Even though my server has 16 cores and 64GB of RAM, and the Docker instances are limited to 4 cores and 6GB each, I face these issues even when there are only a few users.
It also seems that when one Shiny container becomes unresponsive or gets stuck, it impacts new containers as well, with ShinyProxy struggling to start new ones.
I have another server with the same apps and identical configuration except for the authentication type (set to none instead of openid ). This server does not experience the same issues. The server without the issue never encounters a blank screen problem. Additionally, the apps on the problematic server are larger and have more features, which may result in longer loading times.

Questions:

Could this problem be related to a Shiny app running a long or infinite process, leading to Docker becoming unresponsive or causing a resource bottleneck?
Given the configurations I provided, are there any best practices I might be missing to ensure better container management and stability?
Could the use of openid authentication be contributing to this issue, especially since the server without authentication (none ) does not face the same problem?

Any help or suggestions would be greatly appreciated.

Thank you in advance for your support!

Configuration Summary:

Docker Backend: Docker Swarm
Memory Limit: 6GB per container
CPU Limit: 4 cores per container
Server Specs: 16 cores, 64GB RAM
Issue Frequency: After 1-2 days of container uptime
Authentication Type: OpenID (on problematic server), None (on stable server)

confuguration:

proxy:
  stop-proxies-on-shutdown: true
  recover-running-proxies: false  
  container-wait-time: 200000
  hide-navbar: false
  usage-stats-url: micrometer
  title: My APP
  heartbeat-rate: 10000
  heartbeat-timeout: 180000
  port: 8080
  authentication: openid
  admin-groups: scientists
  openid:
    auth-url: http://auth0.b-ht.com:8135/oauth2/authorize
    token-url: http://auth0.b-ht.com:8135/oauth2/token
    jwks-url: http://auth0.b-ht.com:8135/oauth2/jwks
    userinfo-url: http://auth0.b-ht.com:8135/user
    logout-url: http://auth0.b-ht.com:8135/delToken
    client-id: messaging-client2
    client-secret: 4536
    username-attribute: sub
    roles-claim: authorities
    SHINYPROXY_USERGROUPS: scientists
    scopes: [profile]
  
  container-backend: docker-swarm
  docker:
    internal-networking: true
  specs:
  - id: mstatavip
    display-name: app1
    description: app1
    container-cmd: ["R", "-e", "shiny::runApp('/home/rstudio/mstatavip')"]
    container-image: zhaohongxin0/mstatavip2
    access-groups: [admins, Vipgroup]
    container-network: sp-example-net
    minimum-seats-available: 3
    allow-container-re-use: false
    container-memory-limit: 6g
    container-cpu-limit: 4
    
  - id: tutor
    display-name: app2
    description: app2
    container-cmd: ["R", "-e", "shiny::runApp('/home/rstudio/tutor')"]
    container-image: zhaohongxin0/mstatavip2
    access-groups: [admins, Vipgroup]
    container-network: sp-example-net
    container-memory-limit: 6g
    container-cpu-limit: 4    
  - id: journal
    display-name: app3
    description: app3
    container-cmd: ["R", "-e", "shiny::runApp('/home/rstudio/journal')"]
    container-image: zhaohongxin0/mstatavip2
    access-groups: [admins, Vipgroup]
    container-network: sp-example-net
    container-memory-limit: 6g
    container-cpu-limit: 4
  - id: predictapp
    display-name: app4
    description: app4
    container-cmd: ["R", "-e", "shiny::runApp('/home/rstudio/predictapp')"]
    container-image: zhaohongxin0/mstatavip2
    access-groups: [admins, Vipgroup]
    container-network: sp-example-net
    container-memory-limit: 6g
    container-cpu-limit: 4    
  - id: chatstata
    display-name: app5
    description: app5
    container-cmd: ["R", "-e", "RTutor::run_app()"]
    container-image: zhaohongxin0/mstatavip2
    access-groups: [admins, Vipgroup]
    container-network: sp-example-net
    minimum-seats-available: 1
    allow-container-re-use: false
    container-memory-limit: 6g
    container-cpu-limit: 4

logging:
  file:
    shinyproxy.log

server:
  forward-headers-strategy: native  
  servlet.session.timeout: -1
management:
  metrics:
    export:
      prometheus:
        enabled: true

shinyproxy log right before the issue occur:

{"log":"2024-10-06T04:03:03.817Z  INFO 1 --- [haringScaler-13] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=aa413422-b449-4e9d-b76a-414d49af2776] Started DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:03:03.817159379Z"}
{"log":"2024-10-06T04:03:27.260Z  INFO 1 --- [GlobalEventLoop] e.o.c.s.hearbeat.ActiveProxiesService    : [user=4fbdf587-817e-4e7c-a2ce-36f347515db4 proxyId=03cf5fc5-7bfe-40f6-95a4-5d3999933146 specId=freetrial] Releasing inactive proxy [silence: 64300ms]\n","stream":"stdout","time":"2024-10-06T04:03:27.261105001Z"}
{"log":"2024-10-06T04:03:27.261Z  INFO 1 --- [ProxyService-15] e.o.c.b.d.p.ProxySharingDispatcher       : [user=4fbdf587-817e-4e7c-a2ce-36f347515db4 proxyId=03cf5fc5-7bfe-40f6-95a4-5d3999933146 specId=freetrial delegateProxyId=95d6fd0c-2f7e-44ce-86de-97a512817e2a seatId=af1cf523-cabd-4475-bd4f-7d162a1be9e2] Seat released\n","stream":"stdout","time":"2024-10-06T04:03:27.261214881Z"}
{"log":"2024-10-06T04:03:27.261Z  INFO 1 --- [ProxyService-15] e.o.containerproxy.service.ProxyService  : [user=4fbdf587-817e-4e7c-a2ce-36f347515db4 proxyId=03cf5fc5-7bfe-40f6-95a4-5d3999933146 specId=freetrial] Proxy released\n","stream":"stdout","time":"2024-10-06T04:03:27.261264152Z"}
{"log":"2024-10-06T04:03:27.261Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=95d6fd0c-2f7e-44ce-86de-97a512817e2a] DelegateProxy cannot be re-used, marking for removal\n","stream":"stdout","time":"2024-10-06T04:03:27.2612702Z"}
{"log":"2024-10-06T04:03:27.261Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=95d6fd0c-2f7e-44ce-86de-97a512817e2a seatId=af1cf523-cabd-4475-bd4f-7d162a1be9e2] Removed seat\n","stream":"stdout","time":"2024-10-06T04:03:27.26134874Z"}
{"log":"2024-10-06T04:03:29.689Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=95d6fd0c-2f7e-44ce-86de-97a512817e2a] Stopping DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:03:29.689272555Z"}
{"log":"2024-10-06T04:05:47.260Z  INFO 1 --- [GlobalEventLoop] e.o.c.s.hearbeat.ActiveProxiesService    : [user=1bf5a1aa-ce15-4075-bc10-69963d50a873 proxyId=08a30413-e119-4434-8a3b-c21629520f0d specId=freetrial] Releasing inactive proxy [silence: 75161ms]\n","stream":"stdout","time":"2024-10-06T04:05:47.260985775Z"}
{"log":"2024-10-06T04:05:47.261Z  INFO 1 --- [ProxyService-15] e.o.c.b.d.p.ProxySharingDispatcher       : [user=1bf5a1aa-ce15-4075-bc10-69963d50a873 proxyId=08a30413-e119-4434-8a3b-c21629520f0d specId=freetrial delegateProxyId=5b2a9281-a3fa-465a-947e-5e23cca0c4b0 seatId=fe4976fb-aa20-412f-95ac-d2698dd4f2ed] Seat released\n","stream":"stdout","time":"2024-10-06T04:05:47.261128341Z"}
{"log":"2024-10-06T04:05:47.261Z  INFO 1 --- [ProxyService-15] e.o.containerproxy.service.ProxyService  : [user=1bf5a1aa-ce15-4075-bc10-69963d50a873 proxyId=08a30413-e119-4434-8a3b-c21629520f0d specId=freetrial] Proxy released\n","stream":"stdout","time":"2024-10-06T04:05:47.261163642Z"}
{"log":"2024-10-06T04:05:47.261Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=5b2a9281-a3fa-465a-947e-5e23cca0c4b0] DelegateProxy cannot be re-used, marking for removal\n","stream":"stdout","time":"2024-10-06T04:05:47.261181778Z"}
{"log":"2024-10-06T04:05:47.261Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=5b2a9281-a3fa-465a-947e-5e23cca0c4b0 seatId=fe4976fb-aa20-412f-95ac-d2698dd4f2ed] Removed seat\n","stream":"stdout","time":"2024-10-06T04:05:47.261237379Z"}
{"log":"2024-10-06T04:05:49.690Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=5b2a9281-a3fa-465a-947e-5e23cca0c4b0] Stopping DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:05:49.690445679Z"}
{"log":"2024-10-06T04:07:51.869Z  INFO 1 --- [  XNIO-1 task-4] e.o.containerproxy.service.UserService   : Anonymous user logged out [user: 87157836-4621-4970-a998-e0bf536db7c8]\n","stream":"stdout","time":"2024-10-06T04:07:51.869632868Z"}
{"log":"2024-10-06T04:09:59.905Z  INFO 1 --- [  XNIO-1 task-4] e.o.containerproxy.service.UserService   : User logged in [user: f33e64a8-bd7e-4d9c-813a-09a7f3368e61]\n","stream":"stdout","time":"2024-10-06T04:09:59.905949408Z"}
{"log":"2024-10-06T04:10:00.669Z  INFO 1 --- [ProxyService-15] e.o.containerproxy.service.ProxyService  : [user=f33e64a8-bd7e-4d9c-813a-09a7f3368e61 proxyId=8d394eba-db00-4148-b9ab-6f0dbadf6177 specId=freetrial] Starting proxy\n","stream":"stdout","time":"2024-10-06T04:10:00.669930667Z"}
{"log":"2024-10-06T04:10:00.669Z  INFO 1 --- [ProxyService-15] e.o.c.b.d.p.ProxySharingDispatcher       : [user=f33e64a8-bd7e-4d9c-813a-09a7f3368e61 proxyId=8d394eba-db00-4148-b9ab-6f0dbadf6177 specId=freetrial delegateProxyId=aa413422-b449-4e9d-b76a-414d49af2776 seatId=cc96158c-a66a-42b6-991a-74f9ca003f13] Seat claimed\n","stream":"stdout","time":"2024-10-06T04:10:00.669963106Z"}
{"log":"2024-10-06T04:10:00.669Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial] Scale up required, trying to create 1 DelegateProxies\n","stream":"stdout","time":"2024-10-06T04:10:00.670014624Z"}
{"log":"2024-10-06T04:10:00.669Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=62e27db3-1115-4c9c-a792-b09071fe6275] Creating DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:10:00.670025854Z"}
{"log":"2024-10-06T04:10:00.669Z  INFO 1 --- [ProxyService-15] e.o.containerproxy.service.ProxyService  : [user=f33e64a8-bd7e-4d9c-813a-09a7f3368e61 proxyId=8d394eba-db00-4148-b9ab-6f0dbadf6177 specId=freetrial] Proxy activated\n","stream":"stdout","time":"2024-10-06T04:10:00.670030214Z"}
{"log":"2024-10-06T04:10:00.670Z  INFO 1 --- [haringScaler-13] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=62e27db3-1115-4c9c-a792-b09071fe6275] Preparing DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:10:00.670157325Z"}
{"log":"2024-10-06T04:10:00.670Z  INFO 1 --- [haringScaler-13] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=62e27db3-1115-4c9c-a792-b09071fe6275] Starting DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:10:00.670502457Z"}
{"log":"2024-10-06T04:10:05.796Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (11/40)\n","stream":"stdout","time":"2024-10-06T04:10:05.796672574Z"}
{"log":"2024-10-06T04:10:07.808Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (12/40)\n","stream":"stdout","time":"2024-10-06T04:10:07.808585954Z"}
{"log":"2024-10-06T04:10:09.821Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (13/40)\n","stream":"stdout","time":"2024-10-06T04:10:09.821196166Z"}
{"log":"2024-10-06T04:10:11.832Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (14/40)\n","stream":"stdout","time":"2024-10-06T04:10:11.83261815Z"}
{"log":"2024-10-06T04:10:13.844Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (15/40)\n","stream":"stdout","time":"2024-10-06T04:10:13.844532187Z"}
{"log":"2024-10-06T04:10:15.855Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (16/40)\n","stream":"stdout","time":"2024-10-06T04:10:15.85544685Z"}
{"log":"2024-10-06T04:10:17.867Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (17/40)\n","stream":"stdout","time":"2024-10-06T04:10:17.867414942Z"}
{"log":"2024-10-06T04:10:19.877Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (18/40)\n","stream":"stdout","time":"2024-10-06T04:10:19.878065361Z"}
{"log":"2024-10-06T04:10:21.888Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (19/40)\n","stream":"stdout","time":"2024-10-06T04:10:21.888175727Z"}
{"log":"2024-10-06T04:10:23.897Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (20/40)\n","stream":"stdout","time":"2024-10-06T04:10:23.898005669Z"}
{"log":"2024-10-06T04:10:25.908Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (21/40)\n","stream":"stdout","time":"2024-10-06T04:10:25.908391266Z"}
{"log":"2024-10-06T04:10:27.919Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (22/40)\n","stream":"stdout","time":"2024-10-06T04:10:27.919119086Z"}
{"log":"2024-10-06T04:10:29.929Z  INFO 1 --- [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (23/40)\n","stream":"stdout","time":"2024-10-06T04:10:29.929358497Z"}
{"log":"2024-10-06T04:10:36.946Z  WARN 1 --- [haringScaler-13] e.o.shinyproxy.ShinyProxyTestStrategy    : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Container unresponsive, trying again (11/25): http://2dfab14cc920:3838\n","stream":"stdout","time":"2024-10-06T04:10:36.946598614Z"}
{"log":"2024-10-06T04:10:38.947Z  WARN 1 --- [haringScaler-13] e.o.shinyproxy.ShinyProxyTestStrategy    : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Container unresponsive, trying again (12/25): http://2dfab14cc920:3838\n","stream":"stdout","time":"2024-10-06T04:10:38.947169372Z"}
{"log":"2024-10-06T04:10:45.022Z  INFO 1 --- [haringScaler-13] e.o.shinyproxy.ShinyProxyTestStrategy    : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Container responsive\n","stream":"stdout","time":"2024-10-06T04:10:45.022391159Z"}
{"log":"2024-10-06T04:10:45.022Z  INFO 1 --- [haringScaler-13] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=62e27db3-1115-4c9c-a792-b09071fe6275 seatId=3339b410-eea8-4b61-80e3-c7c0fff13287] Created Seat\n","stream":"stdout","time":"2024-10-06T04:10:45.022425658Z"}
{"log":"2024-10-06T04:10:45.022Z  INFO 1 --- [haringScaler-13] e.o.c.b.d.p.ProxySharingScaler           : [specId=freetrial delegateProxyId=62e27db3-1115-4c9c-a792-b09071fe6275] Started DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:10:45.022430983Z"}
{"log":"2024-10-06T04:10:51.503Z  INFO 1 --- [ProxyService-15] e.o.containerproxy.service.ProxyService  : [user=de799849-6f00-4317-b5a2-4eb47b180701 proxyId=98bec556-3197-48be-a560-80dd7182375c specId=chatstata] Starting proxy\n","stream":"stdout","time":"2024-10-06T04:10:51.503994625Z"}
{"log":"2024-10-06T04:10:51.503Z  INFO 1 --- [ProxyService-15] e.o.c.b.d.p.ProxySharingDispatcher       : [user=de799849-6f00-4317-b5a2-4eb47b180701 proxyId=98bec556-3197-48be-a560-80dd7182375c specId=chatstata delegateProxyId=19799eed-74db-4911-a296-5218d33b517a seatId=f133e9a7-a70b-499e-8c9b-c56ce3069406] Seat claimed\n","stream":"stdout","time":"2024-10-06T04:10:51.50403379Z"}
{"log":"2024-10-06T04:10:51.504Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata] Scale up required, trying to create 1 DelegateProxies\n","stream":"stdout","time":"2024-10-06T04:10:51.504154506Z"}
{"log":"2024-10-06T04:10:51.504Z  INFO 1 --- [ProxyService-15] e.o.containerproxy.service.ProxyService  : [user=de799849-6f00-4317-b5a2-4eb47b180701 proxyId=98bec556-3197-48be-a560-80dd7182375c specId=chatstata] Proxy activated\n","stream":"stdout","time":"2024-10-06T04:10:51.504171952Z"}
{"log":"2024-10-06T04:10:51.504Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata delegateProxyId=3d95455c-03c8-472d-902d-16481d27a6a8] Creating DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:10:51.504175786Z"}
{"log":"2024-10-06T04:10:51.504Z  INFO 1 --- [haringScaler-16] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata delegateProxyId=3d95455c-03c8-472d-902d-16481d27a6a8] Preparing DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:10:51.50426105Z"}
{"log":"2024-10-06T04:10:51.504Z  INFO 1 --- [haringScaler-16] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata delegateProxyId=3d95455c-03c8-472d-902d-16481d27a6a8] Starting DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:10:51.504477365Z"}
{"log":"2024-10-06T04:10:56.625Z  INFO 1 --- [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (11/40)\n","stream":"stdout","time":"2024-10-06T04:10:56.62592566Z"}
{"log":"2024-10-06T04:10:58.637Z  INFO 1 --- [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (12/40)\n","stream":"stdout","time":"2024-10-06T04:10:58.637479991Z"}
{"log":"2024-10-06T04:11:00.648Z  INFO 1 --- [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (13/40)\n","stream":"stdout","time":"2024-10-06T04:11:00.648545415Z"}
{"log":"2024-10-06T04:11:02.659Z  INFO 1 --- [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (14/40)\n","stream":"stdout","time":"2024-10-06T04:11:02.659547066Z"}
{"log":"2024-10-06T04:11:04.670Z  INFO 1 --- [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (15/40)\n","stream":"stdout","time":"2024-10-06T04:11:04.670217371Z"}
{"log":"2024-10-06T04:11:05.180Z  INFO 1 --- [  XNIO-1 task-6] e.o.containerproxy.service.UserService   : User logged in [user: b28f0354-3cc5-4d5c-838f-10e15f5fd8aa]\n","stream":"stdout","time":"2024-10-06T04:11:05.18025247Z"}
{"log":"2024-10-06T04:11:05.347Z  INFO 1 --- [  XNIO-1 task-4] e.o.containerproxy.service.UserService   : User logged in [user: 6225df14-9fee-4065-aa8a-89fd2030ebc9]\n","stream":"stdout","time":"2024-10-06T04:11:05.347739713Z"}
{"log":"2024-10-06T04:11:05.384Z  INFO 1 --- [ProxyService-15] e.o.containerproxy.service.ProxyService  : [user=b28f0354-3cc5-4d5c-838f-10e15f5fd8aa proxyId=6e3c8f4d-226b-4ea4-979b-c62eff111412 specId=chatstata] Starting proxy\n","stream":"stdout","time":"2024-10-06T04:11:05.387521134Z"}
{"log":"2024-10-06T04:11:05.384Z  INFO 1 --- [ProxyService-15] e.o.c.b.d.p.ProxySharingDispatcher       : [user=b28f0354-3cc5-4d5c-838f-10e15f5fd8aa proxyId=6e3c8f4d-226b-4ea4-979b-c62eff111412 specId=chatstata] Seat not immediately available\n","stream":"stdout","time":"2024-10-06T04:11:05.387543741Z"}
{"log":"2024-10-06T04:11:05.384Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata] Scale up required, trying to create 1 DelegateProxies\n","stream":"stdout","time":"2024-10-06T04:11:05.387565313Z"}
{"log":"2024-10-06T04:11:05.384Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata delegateProxyId=7fe105d2-57eb-4139-920f-ea2d02ce4f2f] Creating DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:11:05.387569181Z"}
{"log":"2024-10-06T04:11:05.384Z  INFO 1 --- [haringScaler-15] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata delegateProxyId=7fe105d2-57eb-4139-920f-ea2d02ce4f2f] Preparing DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:11:05.387572729Z"}
{"log":"2024-10-06T04:11:05.384Z  INFO 1 --- [haringScaler-15] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata delegateProxyId=7fe105d2-57eb-4139-920f-ea2d02ce4f2f] Starting DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:11:05.387576044Z"}
{"log":"2024-10-06T04:11:05.511Z  INFO 1 --- [ProxyService-14] e.o.containerproxy.service.ProxyService  : [user=6225df14-9fee-4065-aa8a-89fd2030ebc9 proxyId=bc147863-4493-4d0f-8818-3596cdff3b49 specId=chatstata] Starting proxy\n","stream":"stdout","time":"2024-10-06T04:11:05.511733155Z"}
{"log":"2024-10-06T04:11:05.511Z  INFO 1 --- [ProxyService-14] e.o.c.b.d.p.ProxySharingDispatcher       : [user=6225df14-9fee-4065-aa8a-89fd2030ebc9 proxyId=bc147863-4493-4d0f-8818-3596cdff3b49 specId=chatstata] Seat not immediately available\n","stream":"stdout","time":"2024-10-06T04:11:05.511764657Z"}
{"log":"2024-10-06T04:11:05.511Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata] Scale up required, trying to create 1 DelegateProxies\n","stream":"stdout","time":"2024-10-06T04:11:05.515448228Z"}
{"log":"2024-10-06T04:11:05.511Z  INFO 1 --- [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata delegateProxyId=dde145cf-53ce-4c6d-9a1e-c3bcb07dcb50] Creating DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:11:05.515468885Z"}
{"log":"2024-10-06T04:11:05.512Z  INFO 1 --- [haringScaler-14] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata delegateProxyId=dde145cf-53ce-4c6d-9a1e-c3bcb07dcb50] Preparing DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:11:05.515471826Z"}
{"log":"2024-10-06T04:11:05.512Z  INFO 1 --- [haringScaler-14] e.o.c.b.d.p.ProxySharingScaler           : [specId=chatstata delegateProxyId=dde145cf-53ce-4c6d-9a1e-c3bcb07dcb50] Starting DelegateProxy\n","stream":"stdout","time":"2024-10-06T04:11:05.515474718Z"}
{"log":"2024-10-06T04:11:06.679Z  INFO 1 --- [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (16/40)\n","stream":"stdout","time":"2024-10-06T04:11:06.679512587Z"}
{"log":"2024-10-06T04:11:08.687Z  INFO 1 --- [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend  : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (17/40)\n","stream":"stdout","time":"2024-10-06T04:11:08.688065216Z"}

tdekoninck · October 21, 2024, 2:49pm

Hi, thank you for your detailed questions!

Could this problem be related to a Shiny app running a long or infinite process, leading to Docker becoming unresponsive or causing a resource bottleneck?

ShinyProxy is able to run apps for multiple days or weeks, this should not be a problem.

Given the configurations I provided, are there any best practices I might be missing to ensure better container management and stability?

You mention that you are using Docker swarm, but it seems you are using only a single node. In that case I strongly advice to use the regular docker backend and disable swarm on the docker daemon. In general the docker backend/engine is more stable than docker swarm.

Could the use of openid authentication be contributing to this issue, especially since the server without authentication (none ) does not face the same problem?

No, it should not cause the issue. Usually when a blank page is shown, ShinyProxy cannot communicate with the container. Either because the container disappeared or because of some network issue. When this happens, you could check the admin panel, and find the IP of the broken app and check whether the container associated to this IP still exists.

zhaohongxin0 · October 30, 2024, 5:05pm

Thank you for your detailed reply!

I have reverted to the Docker backend rather than Docker Swarm, as recommended. However, the issue continues to appear every day or two. When it happens, I can log in, and the navbar is visible, but after clicking the “Admin” button to enter the admin page, everything below the navbar remains blank, including the section where app IP addresses should display.

Additionally, I have another server running the same Shiny app setup, and this server doesn’t encounter this issue. Key differences on the stable server include setting the login method to “none,” limiting maximum session data to 20, restricting file uploads to 1MB, and having fewer app functionalities. These missing functionalities in the stable server may involve extensive R calculations, which makes me wonder: could heavy R computations cause the entire Docker setup to crash?

However, I would have thought a single Docker container crash wouldn’t affect other containers or cause all user containers to disappear. Only the ShinyProxy container and pre-initializing containers remain visible in the Docker backend once the issue happens, as shown in the logs I previously shared.

After each failure, I can restore the service by running the following commands to restart the ShinyProxy container:

sudo docker rm -f $(sudo docker ps -aq)
sudo docker run -d -v /var/run/docker.sock:/var/run/docker.sock:ro --group-add $(getent group docker | cut -d: -f3) --net sp-example-net -p 8080:8080 zhaohongxin0/mstatavip_shinyproxy

Any insights or recommendations you might have would be very helpful. Thank you again!

btw, the shinyproxy log info before the isssue occurred:

"log":"\n","stream":"stdout","time":"2024-10-30T16:11:18.528224025Z"}
{"log":"2024-10-30T16:11:22.181Z ERROR 1 --- [   XNIO-1 I/O-1] io.undertow.request                      : UT005071: Undertow request failed HttpServerExchange{ CONNECT dnspod.qcloud.com:443}\n","stream":"stdout","time":"2024-10-30T16:11:22.181612259Z"}
{"log":"\n","stream":"stdout","time":"2024-10-30T16:11:22.181641083Z"}
{"log":"java.lang.IllegalArgumentException: UT000068: Servlet path match failed\n","stream":"stdout","time":"2024-10-30T16:11:22.181644299Z"}
{"log":"\u0009at io.undertow.servlet.handlers.ServletPathMatchesData.getServletHandlerByPath(ServletPathMatchesData.java:83) ~[undertow-servlet-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181647294Z"}
{"log":"\u0009at io.undertow.servlet.handlers.ServletPathMatches.getServletHandlerByPath(ServletPathMatches.java:133) ~[undertow-servlet-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.18165043Z"}
{"log":"\u0009at io.undertow.servlet.handlers.ServletInitialHandler.handleRequest(ServletInitialHandler.java:148) ~[undertow-servlet-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181653185Z"}
{"log":"\u0009at io.undertow.server.handlers.HttpContinueReadHandler.handleRequest(HttpContinueReadHandler.java:69) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181655911Z"}
{"log":"\u0009at org.springframework.boot.web.embedded.undertow.DeploymentManagerHttpHandlerFactory$DeploymentManagerHandler.handleRequest(DeploymentManagerHttpHandlerFactory.java:74) ~[spring-boot-3.2.6.jar!/:3.2.6]\n","stream":"stdout","time":"2024-10-30T16:11:22.181658636Z"}
{"log":"\u0009at io.undertow.server.handlers.ProxyPeerAddressHandler.handleRequest(ProxyPeerAddressHandler.java:129) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181661361Z"}
{"log":"\u0009at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181675357Z"}
{"log":"\u0009at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:265) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181678152Z"}
{"log":"\u0009at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181680887Z"}
{"log":"\u0009at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181711184Z"}
{"log":"\u0009at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.18171419Z"}
{"log":"\u0009at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181717777Z"}
{"log":"\u0009at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) ~[xnio-api-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181720452Z"}
{"log":"\u0009at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291) ~[xnio-api-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181723066Z"}
{"log":"\u0009at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) ~[xnio-api-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181725671Z"}
{"log":"\u0009at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) ~[xnio-api-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181728256Z"}
{"log":"\u0009at org.xnio.nio.QueuedNioTcpServer2.acceptTask(QueuedNioTcpServer2.java:178) ~[xnio-nio-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181731061Z"}
{"log":"\u0009at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612) ~[xnio-nio-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181733656Z"}
{"log":"\u0009at org.xnio.nio.WorkerThread.run(WorkerThread.java:479) ~[xnio-nio-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.181736251Z"}
{"log":"\n","stream":"stdout","time":"2024-10-30T16:11:22.181738836Z"}
{"log":"2024-10-30T16:11:22.537Z ERROR 1 --- [   XNIO-1 I/O-2] io.undertow.request                      : UT005071: Undertow request failed HttpServerExchange{ CONNECT example.com:443}\n","stream":"stdout","time":"2024-10-30T16:11:22.537964857Z"}
{"log":"\n","stream":"stdout","time":"2024-10-30T16:11:22.53798762Z"}
{"log":"java.lang.IllegalArgumentException: UT000068: Servlet path match failed\n","stream":"stdout","time":"2024-10-30T16:11:22.537991657Z"}
{"log":"\u0009at io.undertow.servlet.handlers.ServletPathMatchesData.getServletHandlerByPath(ServletPathMatchesData.java:83) ~[undertow-servlet-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.537995154Z"}
{"log":"\u0009at io.undertow.servlet.handlers.ServletPathMatches.getServletHandlerByPath(ServletPathMatches.java:133) ~[undertow-servlet-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.537999352Z"}
{"log":"\u0009at io.undertow.servlet.handlers.ServletInitialHandler.handleRequest(ServletInitialHandler.java:148) ~[undertow-servlet-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538003179Z"}
{"log":"\u0009at io.undertow.server.handlers.HttpContinueReadHandler.handleRequest(HttpContinueReadHandler.java:69) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538006816Z"}
{"log":"\u0009at org.springframework.boot.web.embedded.undertow.DeploymentManagerHttpHandlerFactory$DeploymentManagerHandler.handleRequest(DeploymentManagerHttpHandlerFactory.java:74) ~[spring-boot-3.2.6.jar!/:3.2.6]\n","stream":"stdout","time":"2024-10-30T16:11:22.538021513Z"}
{"log":"\u0009at io.undertow.server.handlers.ProxyPeerAddressHandler.handleRequest(ProxyPeerAddressHandler.java:129) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538026212Z"}
{"log":"\u0009at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538029509Z"}
{"log":"\u0009at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:265) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538032885Z"}
{"log":"\u0009at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538036232Z"}
{"log":"\u0009at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538039648Z"}
{"log":"\u0009at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538043185Z"}
{"log":"\u0009at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57) ~[undertow-core-2.3.13.Final.jar!/:2.3.13.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538047543Z"}
{"log":"\u0009at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) ~[xnio-api-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538050929Z"}
{"log":"\u0009at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291) ~[xnio-api-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538054225Z"}
{"log":"\u0009at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) ~[xnio-api-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538057461Z"}
{"log":"\u0009at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) ~[xnio-api-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538060708Z"}
{"log":"\u0009at org.xnio.nio.QueuedNioTcpServer2.acceptTask(QueuedNioTcpServer2.java:178) ~[xnio-nio-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538064284Z"}
{"log":"\u0009at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612) ~[xnio-nio-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.5380675Z"}
{"log":"\u0009at org.xnio.nio.WorkerThread.run(WorkerThread.java:479) ~[xnio-nio-3.8.8.Final.jar!/:3.8.8.Final]\n","stream":"stdout","time":"2024-10-30T16:11:22.538070807Z"}
{"log":"\n","stream":"stdout","time":"2024-10-30T16:11:22.538074153Z"}

tdekoninck · November 4, 2024, 9:20am

A simple crash of the process inside a container should not make the other contains crash. However, if the server is running out of memory, it might be that the kernel starts killing some processes and therefore other containers may crash (oomkill).
You can prevent this by setting container-memory-request to the same value as container-memory-limit, this way the system reserves the memory for the container and you are sure that you don’t over-commit your memory on the server.

Regarding the error message: it seems your ShinyProxy server tries to connect to dnspod.qcloud.com, which seems to be a DNS server of your cloud provider. I’m not sure why ShinyProxy would connect to this. It’s expected that it connects to your OpenID provider. It seem its’ related to UT000068: Servlet path match failed\n , which could be caused by using a wrong domain/path to access ShinyProxy.

Are you using any loadbalancer, proxy, WAF or something like a CDN in front of ShinyProxy?