Server unresponsive: uncontrolled replicated containers

Hi,

Recently our shinyproxy server got unresponsive with “Failed to start app. The server does not have enough capacity to start application. Please try again later”. When reviewing the services and we noticed that our services are getting replicated automatically in fraction of seconds and new containers are getting started immediately. When I checked the EC2 instance, we found 74 containers and services running.

After redeploying the application, everything works fine initially. But a couple of days later, same problem appeared again and the number of replicated services and containers starts increasing rapidly again.

We are running shinyproxy on a single VM with docker swarm backend without Operator. We have set the maximum total number of instances to 15 and are using 2 pre-iniitalized containers. So it’s already strange that the number of instances could exceed the maximum of 15 set. See below our application.yml

Has anyone had a similar experience and knows the cause and solution?

Best,

Ivo


proxy:
title: Omics Playground
logo-url: ``https://www.openanalytics.eu/shinyproxy/logo.png
container-backend: docker-swarm
docker:
internal-networking: true
landing-page: SingleApp
port: 8080
authentication: none
hide-navbar: true
max-total-instances: 15
favicon-path: /opt/shinyproxy/favicon.ico
container-log-path: /omicsplayground/container-logs
specs:
- id: omicsplayground
container-image: bigomics/omicsplayground:devel
container-network: app-network
display-name: Omics Playground v4
resource-name: OPG-#{proxy.id}
minimum-seats-available: 2
seats-per-container: 1
allow-container-re-use: false
container-proxy-url: /
container-volumes: [...]
logging:
file:
name: shinyproxy.log

Hi

This is indeed not the desired behaviour. The big number of containers could happen if the containers are crashing or new containers are failing to start.
Could you keep track of the ShinyProxy logs, and if this happens again, share the logs here?