Hello ShinyProxy Community,
I am experiencing a recurring issue with ShinyProxy where Docker containers used to serve Shiny apps become unresponsive after running continuously for one or two days. Below are some details about my setup and the errors I am seeing:
ShinyProxy Configuration:
- Using Docker Swarm as the backend
- Containers are set to use 4 CPU cores and 6GB memory each
- ShinyProxy configuration includes several apps deployed with
container-network: sp-example-net - ShinyProxy configuration allows for Docker pre-initializing, but container re-use is disabled
Here is an excerpt from my configuration:
Problem Details:
- After one or two days, I can access the ShinyProxy menu, but clicking on an app link results in the app failing to start, sometimes displaying a blank page. The ShinyProxy navigation bar and user information are displayed, but the app interface itself remains blank.
- Restarting the Docker Swarm service running ShinyProxy temporarily resolves the issue, but it reoccurs every one or two days.
- At any time, I can successfully log in using OpenID authentication. However, after logging in, I am unable to launch any Shiny app. The app page remains blank, but the ShinyProxy navigation bar and user information are still visible. Restarting the Docker Swarm service running ShinyProxy is required to restore functionality.
Relevant Logs (ShinyProxy):
The following log entries show the issue occurring repeatedly before the containers become unresponsive:
I noticed that the logs repeatedly mention that the Docker Swarm service is “not ready yet” and that the container is “unresponsive.” This seems to indicate a timeout or scaling issue. The containers are already locally available, so they should not need to be pulled again, yet ShinyProxy keeps retrying.
My Observations:
- The problem seems periodic, usually after the container has been running for a couple of days.
- Even though my server has 16 cores and 64GB of RAM, and the Docker instances are limited to 4 cores and 6GB each, I face these issues even when there are only a few users.
- It also seems that when one Shiny container becomes unresponsive or gets stuck, it impacts new containers as well, with ShinyProxy struggling to start new ones.
- I have another server with the same apps and identical configuration except for the authentication type (set to
noneinstead ofopenid). This server does not experience the same issues. The server without the issue never encounters a blank screen problem. Additionally, the apps on the problematic server are larger and have more features, which may result in longer loading times.
Questions:
- Could this problem be related to a Shiny app running a long or infinite process, leading to Docker becoming unresponsive or causing a resource bottleneck?
- Given the configurations I provided, are there any best practices I might be missing to ensure better container management and stability?
- Could the use of
openidauthentication be contributing to this issue, especially since the server without authentication (none) does not face the same problem?
Any help or suggestions would be greatly appreciated.
Thank you in advance for your support!
Configuration Summary:
- Docker Backend: Docker Swarm
- Memory Limit: 6GB per container
- CPU Limit: 4 cores per container
- Server Specs: 16 cores, 64GB RAM
- Issue Frequency: After 1-2 days of container uptime
- Authentication Type: OpenID (on problematic server), None (on stable server)
confuguration:
proxy:
stop-proxies-on-shutdown: true
recover-running-proxies: false
container-wait-time: 200000
hide-navbar: false
usage-stats-url: micrometer
title: My APP
heartbeat-rate: 10000
heartbeat-timeout: 180000
port: 8080
authentication: openid
admin-groups: scientists
openid:
auth-url: http://auth0.b-ht.com:8135/oauth2/authorize
token-url: http://auth0.b-ht.com:8135/oauth2/token
jwks-url: http://auth0.b-ht.com:8135/oauth2/jwks
userinfo-url: http://auth0.b-ht.com:8135/user
logout-url: http://auth0.b-ht.com:8135/delToken
client-id: messaging-client2
client-secret: 4536
username-attribute: sub
roles-claim: authorities
SHINYPROXY_USERGROUPS: scientists
scopes: [profile]
container-backend: docker-swarm
docker:
internal-networking: true
specs:
-
id: mstatavip
display-name: app1
description: app1
container-cmd: [“R”, “-e”, “shiny::runApp(‘/home/rstudio/mstatavip’)”]
container-image: zhaohongxin0/mstatavip2
access-groups: [admins, Vipgroup]
container-network: sp-example-net
minimum-seats-available: 3
allow-container-re-use: false
container-memory-limit: 6g
container-cpu-limit: 4 -
id: tutor
display-name: app2
description: app2
container-cmd: [“R”, “-e”, “shiny::runApp(‘/home/rstudio/tutor’)”]
container-image: zhaohongxin0/mstatavip2
access-groups: [admins, Vipgroup]
container-network: sp-example-net
container-memory-limit: 6g
container-cpu-limit: 4 -
id: journal
display-name: app3
description: app3
container-cmd: [“R”, “-e”, “shiny::runApp(‘/home/rstudio/journal’)”]
container-image: zhaohongxin0/mstatavip2
access-groups: [admins, Vipgroup]
container-network: sp-example-net
container-memory-limit: 6g
container-cpu-limit: 4 -
id: predictapp
display-name: app4
description: app4
container-cmd: [“R”, “-e”, “shiny::runApp(‘/home/rstudio/predictapp’)”]
container-image: zhaohongxin0/mstatavip2
access-groups: [admins, Vipgroup]
container-network: sp-example-net
container-memory-limit: 6g
container-cpu-limit: 4 -
id: chatstata
display-name: app5
description: app5
container-cmd: [“R”, “-e”, “RTutor::run_app()”]
container-image: zhaohongxin0/mstatavip2
access-groups: [admins, Vipgroup]
container-network: sp-example-net
minimum-seats-available: 1
allow-container-re-use: false
container-memory-limit: 6g
container-cpu-limit: 4
logging:
file:
shinyproxy.log
server:
forward-headers-strategy: native
servlet.session.timeout: -1
management:
metrics:
export:
prometheus:
enabled: true
shinyproxy log right before the issue occur:
{“log”:“2024-10-06T04:03:03.817Z INFO 1 — [haringScaler-13] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=aa413422-b449-4e9d-b76a-414d49af2776] Started DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:03:03.817159379Z”}
{“log”:“2024-10-06T04:03:27.260Z INFO 1 — [GlobalEventLoop] e.o.c.s.hearbeat.ActiveProxiesService : [user=4fbdf587-817e-4e7c-a2ce-36f347515db4 proxyId=03cf5fc5-7bfe-40f6-95a4-5d3999933146 specId=freetrial] Releasing inactive proxy [silence: 64300ms]\n”,“stream”:“stdout”,“time”:“2024-10-06T04:03:27.261105001Z”}
{“log”:“2024-10-06T04:03:27.261Z INFO 1 — [ProxyService-15] e.o.c.b.d.p.ProxySharingDispatcher : [user=4fbdf587-817e-4e7c-a2ce-36f347515db4 proxyId=03cf5fc5-7bfe-40f6-95a4-5d3999933146 specId=freetrial delegateProxyId=95d6fd0c-2f7e-44ce-86de-97a512817e2a seatId=af1cf523-cabd-4475-bd4f-7d162a1be9e2] Seat released\n”,“stream”:“stdout”,“time”:“2024-10-06T04:03:27.261214881Z”}
{“log”:“2024-10-06T04:03:27.261Z INFO 1 — [ProxyService-15] e.o.containerproxy.service.ProxyService : [user=4fbdf587-817e-4e7c-a2ce-36f347515db4 proxyId=03cf5fc5-7bfe-40f6-95a4-5d3999933146 specId=freetrial] Proxy released\n”,“stream”:“stdout”,“time”:“2024-10-06T04:03:27.261264152Z”}
{“log”:“2024-10-06T04:03:27.261Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=95d6fd0c-2f7e-44ce-86de-97a512817e2a] DelegateProxy cannot be re-used, marking for removal\n”,“stream”:“stdout”,“time”:“2024-10-06T04:03:27.2612702Z”}
{“log”:“2024-10-06T04:03:27.261Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=95d6fd0c-2f7e-44ce-86de-97a512817e2a seatId=af1cf523-cabd-4475-bd4f-7d162a1be9e2] Removed seat\n”,“stream”:“stdout”,“time”:“2024-10-06T04:03:27.26134874Z”}
{“log”:“2024-10-06T04:03:29.689Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=95d6fd0c-2f7e-44ce-86de-97a512817e2a] Stopping DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:03:29.689272555Z”}
{“log”:“2024-10-06T04:05:47.260Z INFO 1 — [GlobalEventLoop] e.o.c.s.hearbeat.ActiveProxiesService : [user=1bf5a1aa-ce15-4075-bc10-69963d50a873 proxyId=08a30413-e119-4434-8a3b-c21629520f0d specId=freetrial] Releasing inactive proxy [silence: 75161ms]\n”,“stream”:“stdout”,“time”:“2024-10-06T04:05:47.260985775Z”}
{“log”:“2024-10-06T04:05:47.261Z INFO 1 — [ProxyService-15] e.o.c.b.d.p.ProxySharingDispatcher : [user=1bf5a1aa-ce15-4075-bc10-69963d50a873 proxyId=08a30413-e119-4434-8a3b-c21629520f0d specId=freetrial delegateProxyId=5b2a9281-a3fa-465a-947e-5e23cca0c4b0 seatId=fe4976fb-aa20-412f-95ac-d2698dd4f2ed] Seat released\n”,“stream”:“stdout”,“time”:“2024-10-06T04:05:47.261128341Z”}
{“log”:“2024-10-06T04:05:47.261Z INFO 1 — [ProxyService-15] e.o.containerproxy.service.ProxyService : [user=1bf5a1aa-ce15-4075-bc10-69963d50a873 proxyId=08a30413-e119-4434-8a3b-c21629520f0d specId=freetrial] Proxy released\n”,“stream”:“stdout”,“time”:“2024-10-06T04:05:47.261163642Z”}
{“log”:“2024-10-06T04:05:47.261Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=5b2a9281-a3fa-465a-947e-5e23cca0c4b0] DelegateProxy cannot be re-used, marking for removal\n”,“stream”:“stdout”,“time”:“2024-10-06T04:05:47.261181778Z”}
{“log”:“2024-10-06T04:05:47.261Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=5b2a9281-a3fa-465a-947e-5e23cca0c4b0 seatId=fe4976fb-aa20-412f-95ac-d2698dd4f2ed] Removed seat\n”,“stream”:“stdout”,“time”:“2024-10-06T04:05:47.261237379Z”}
{“log”:“2024-10-06T04:05:49.690Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=5b2a9281-a3fa-465a-947e-5e23cca0c4b0] Stopping DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:05:49.690445679Z”}
{“log”:“2024-10-06T04:07:51.869Z INFO 1 — [ XNIO-1 task-4] e.o.containerproxy.service.UserService : Anonymous user logged out [user: 87157836-4621-4970-a998-e0bf536db7c8]\n”,“stream”:“stdout”,“time”:“2024-10-06T04:07:51.869632868Z”}
{“log”:“2024-10-06T04:09:59.905Z INFO 1 — [ XNIO-1 task-4] e.o.containerproxy.service.UserService : User logged in [user: f33e64a8-bd7e-4d9c-813a-09a7f3368e61]\n”,“stream”:“stdout”,“time”:“2024-10-06T04:09:59.905949408Z”}
{“log”:“2024-10-06T04:10:00.669Z INFO 1 — [ProxyService-15] e.o.containerproxy.service.ProxyService : [user=f33e64a8-bd7e-4d9c-813a-09a7f3368e61 proxyId=8d394eba-db00-4148-b9ab-6f0dbadf6177 specId=freetrial] Starting proxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:00.669930667Z”}
{“log”:“2024-10-06T04:10:00.669Z INFO 1 — [ProxyService-15] e.o.c.b.d.p.ProxySharingDispatcher : [user=f33e64a8-bd7e-4d9c-813a-09a7f3368e61 proxyId=8d394eba-db00-4148-b9ab-6f0dbadf6177 specId=freetrial delegateProxyId=aa413422-b449-4e9d-b76a-414d49af2776 seatId=cc96158c-a66a-42b6-991a-74f9ca003f13] Seat claimed\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:00.669963106Z”}
{“log”:“2024-10-06T04:10:00.669Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial] Scale up required, trying to create 1 DelegateProxies\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:00.670014624Z”}
{“log”:“2024-10-06T04:10:00.669Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=62e27db3-1115-4c9c-a792-b09071fe6275] Creating DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:00.670025854Z”}
{“log”:“2024-10-06T04:10:00.669Z INFO 1 — [ProxyService-15] e.o.containerproxy.service.ProxyService : [user=f33e64a8-bd7e-4d9c-813a-09a7f3368e61 proxyId=8d394eba-db00-4148-b9ab-6f0dbadf6177 specId=freetrial] Proxy activated\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:00.670030214Z”}
{“log”:“2024-10-06T04:10:00.670Z INFO 1 — [haringScaler-13] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=62e27db3-1115-4c9c-a792-b09071fe6275] Preparing DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:00.670157325Z”}
{“log”:“2024-10-06T04:10:00.670Z INFO 1 — [haringScaler-13] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=62e27db3-1115-4c9c-a792-b09071fe6275] Starting DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:00.670502457Z”}
{“log”:“2024-10-06T04:10:05.796Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (11/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:05.796672574Z”}
{“log”:“2024-10-06T04:10:07.808Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (12/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:07.808585954Z”}
{“log”:“2024-10-06T04:10:09.821Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (13/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:09.821196166Z”}
{“log”:“2024-10-06T04:10:11.832Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (14/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:11.83261815Z”}
{“log”:“2024-10-06T04:10:13.844Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (15/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:13.844532187Z”}
{“log”:“2024-10-06T04:10:15.855Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (16/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:15.85544685Z”}
{“log”:“2024-10-06T04:10:17.867Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (17/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:17.867414942Z”}
{“log”:“2024-10-06T04:10:19.877Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (18/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:19.878065361Z”}
{“log”:“2024-10-06T04:10:21.888Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (19/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:21.888175727Z”}
{“log”:“2024-10-06T04:10:23.897Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (20/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:23.898005669Z”}
{“log”:“2024-10-06T04:10:25.908Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (21/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:25.908391266Z”}
{“log”:“2024-10-06T04:10:27.919Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (22/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:27.919119086Z”}
{“log”:“2024-10-06T04:10:29.929Z INFO 1 — [haringScaler-13] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Docker Swarm Service not ready yet, trying again (23/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:29.929358497Z”}
{“log”:“2024-10-06T04:10:36.946Z WARN 1 — [haringScaler-13] e.o.shinyproxy.ShinyProxyTestStrategy : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Container unresponsive, trying again (11/25): http://2dfab14cc920:3838\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:36.946598614Z”}
{“log”:“2024-10-06T04:10:38.947Z WARN 1 — [haringScaler-13] e.o.shinyproxy.ShinyProxyTestStrategy : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Container unresponsive, trying again (12/25): http://2dfab14cc920:3838\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:38.947169372Z”}
{“log”:“2024-10-06T04:10:45.022Z INFO 1 — [haringScaler-13] e.o.shinyproxy.ShinyProxyTestStrategy : [user=null proxyId=62e27db3-1115-4c9c-a792-b09071fe6275 specId=freetrial] Container responsive\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:45.022391159Z”}
{“log”:“2024-10-06T04:10:45.022Z INFO 1 — [haringScaler-13] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=62e27db3-1115-4c9c-a792-b09071fe6275 seatId=3339b410-eea8-4b61-80e3-c7c0fff13287] Created Seat\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:45.022425658Z”}
{“log”:“2024-10-06T04:10:45.022Z INFO 1 — [haringScaler-13] e.o.c.b.d.p.ProxySharingScaler : [specId=freetrial delegateProxyId=62e27db3-1115-4c9c-a792-b09071fe6275] Started DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:45.022430983Z”}
{“log”:“2024-10-06T04:10:51.503Z INFO 1 — [ProxyService-15] e.o.containerproxy.service.ProxyService : [user=de799849-6f00-4317-b5a2-4eb47b180701 proxyId=98bec556-3197-48be-a560-80dd7182375c specId=chatstata] Starting proxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:51.503994625Z”}
{“log”:“2024-10-06T04:10:51.503Z INFO 1 — [ProxyService-15] e.o.c.b.d.p.ProxySharingDispatcher : [user=de799849-6f00-4317-b5a2-4eb47b180701 proxyId=98bec556-3197-48be-a560-80dd7182375c specId=chatstata delegateProxyId=19799eed-74db-4911-a296-5218d33b517a seatId=f133e9a7-a70b-499e-8c9b-c56ce3069406] Seat claimed\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:51.50403379Z”}
{“log”:“2024-10-06T04:10:51.504Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata] Scale up required, trying to create 1 DelegateProxies\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:51.504154506Z”}
{“log”:“2024-10-06T04:10:51.504Z INFO 1 — [ProxyService-15] e.o.containerproxy.service.ProxyService : [user=de799849-6f00-4317-b5a2-4eb47b180701 proxyId=98bec556-3197-48be-a560-80dd7182375c specId=chatstata] Proxy activated\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:51.504171952Z”}
{“log”:“2024-10-06T04:10:51.504Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata delegateProxyId=3d95455c-03c8-472d-902d-16481d27a6a8] Creating DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:51.504175786Z”}
{“log”:“2024-10-06T04:10:51.504Z INFO 1 — [haringScaler-16] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata delegateProxyId=3d95455c-03c8-472d-902d-16481d27a6a8] Preparing DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:51.50426105Z”}
{“log”:“2024-10-06T04:10:51.504Z INFO 1 — [haringScaler-16] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata delegateProxyId=3d95455c-03c8-472d-902d-16481d27a6a8] Starting DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:51.504477365Z”}
{“log”:“2024-10-06T04:10:56.625Z INFO 1 — [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (11/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:56.62592566Z”}
{“log”:“2024-10-06T04:10:58.637Z INFO 1 — [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (12/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:10:58.637479991Z”}
{“log”:“2024-10-06T04:11:00.648Z INFO 1 — [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (13/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:00.648545415Z”}
{“log”:“2024-10-06T04:11:02.659Z INFO 1 — [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (14/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:02.659547066Z”}
{“log”:“2024-10-06T04:11:04.670Z INFO 1 — [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (15/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:04.670217371Z”}
{“log”:“2024-10-06T04:11:05.180Z INFO 1 — [ XNIO-1 task-6] e.o.containerproxy.service.UserService : User logged in [user: b28f0354-3cc5-4d5c-838f-10e15f5fd8aa]\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.18025247Z”}
{“log”:“2024-10-06T04:11:05.347Z INFO 1 — [ XNIO-1 task-4] e.o.containerproxy.service.UserService : User logged in [user: 6225df14-9fee-4065-aa8a-89fd2030ebc9]\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.347739713Z”}
{“log”:“2024-10-06T04:11:05.384Z INFO 1 — [ProxyService-15] e.o.containerproxy.service.ProxyService : [user=b28f0354-3cc5-4d5c-838f-10e15f5fd8aa proxyId=6e3c8f4d-226b-4ea4-979b-c62eff111412 specId=chatstata] Starting proxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.387521134Z”}
{“log”:“2024-10-06T04:11:05.384Z INFO 1 — [ProxyService-15] e.o.c.b.d.p.ProxySharingDispatcher : [user=b28f0354-3cc5-4d5c-838f-10e15f5fd8aa proxyId=6e3c8f4d-226b-4ea4-979b-c62eff111412 specId=chatstata] Seat not immediately available\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.387543741Z”}
{“log”:“2024-10-06T04:11:05.384Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata] Scale up required, trying to create 1 DelegateProxies\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.387565313Z”}
{“log”:“2024-10-06T04:11:05.384Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata delegateProxyId=7fe105d2-57eb-4139-920f-ea2d02ce4f2f] Creating DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.387569181Z”}
{“log”:“2024-10-06T04:11:05.384Z INFO 1 — [haringScaler-15] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata delegateProxyId=7fe105d2-57eb-4139-920f-ea2d02ce4f2f] Preparing DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.387572729Z”}
{“log”:“2024-10-06T04:11:05.384Z INFO 1 — [haringScaler-15] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata delegateProxyId=7fe105d2-57eb-4139-920f-ea2d02ce4f2f] Starting DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.387576044Z”}
{“log”:“2024-10-06T04:11:05.511Z INFO 1 — [ProxyService-14] e.o.containerproxy.service.ProxyService : [user=6225df14-9fee-4065-aa8a-89fd2030ebc9 proxyId=bc147863-4493-4d0f-8818-3596cdff3b49 specId=chatstata] Starting proxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.511733155Z”}
{“log”:“2024-10-06T04:11:05.511Z INFO 1 — [ProxyService-14] e.o.c.b.d.p.ProxySharingDispatcher : [user=6225df14-9fee-4065-aa8a-89fd2030ebc9 proxyId=bc147863-4493-4d0f-8818-3596cdff3b49 specId=chatstata] Seat not immediately available\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.511764657Z”}
{“log”:“2024-10-06T04:11:05.511Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata] Scale up required, trying to create 1 DelegateProxies\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.515448228Z”}
{“log”:“2024-10-06T04:11:05.511Z INFO 1 — [GlobalEventLoop] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata delegateProxyId=dde145cf-53ce-4c6d-9a1e-c3bcb07dcb50] Creating DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.515468885Z”}
{“log”:“2024-10-06T04:11:05.512Z INFO 1 — [haringScaler-14] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata delegateProxyId=dde145cf-53ce-4c6d-9a1e-c3bcb07dcb50] Preparing DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.515471826Z”}
{“log”:“2024-10-06T04:11:05.512Z INFO 1 — [haringScaler-14] e.o.c.b.d.p.ProxySharingScaler : [specId=chatstata delegateProxyId=dde145cf-53ce-4c6d-9a1e-c3bcb07dcb50] Starting DelegateProxy\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:05.515474718Z”}
{“log”:“2024-10-06T04:11:06.679Z INFO 1 — [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (16/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:06.679512587Z”}
{“log”:“2024-10-06T04:11:08.687Z INFO 1 — [haringScaler-16] e.o.c.backend.docker.DockerSwarmBackend : [user=null proxyId=3d95455c-03c8-472d-902d-16481d27a6a8 specId=chatstata] Docker Swarm Service not ready yet, trying again (17/40)\n”,“stream”:“stdout”,“time”:“2024-10-06T04:11:08.688065216Z”}