Shinyproxy with a docker swarm


#1

Hello friends!

First of all, thanks a lot for the great job you are doing with shinyproxy. I have used r and shiny for a while but I haven’t had the need to scale (plus the options to do so where limited).

Now that I found shinyproxy I decided to give it a try. I might have been a little ambitious for my first try and stumbled upon something.

tl;dr
how do we use an auto-scaling docker swarm in aws from shinyproxy?

Setup:

  • EC2 machine working as an ubuntu 16.04 dev environment. Used to create docker images and do first tests with shinyproxy.
  • Auto scaling docker swam launched with docker cloud on AWS.

Goal:

  • Run shinyproxy on the dev machine or in a swarm manager and be able to support an arbitrary number of concurrent users.

What has worked so far:

For the last step, notice that an important part is
export DOCKER_HOST=tcp://127.0.0.1:32768

in this way we use the remote docker swarm instead of the local daemon.

Problem
I run shinyproxy on the dev machine that is connected to the docker swarm, I open it in the browser and when I click on an app.

It works!
But… doing a docker ps shows the shiny app is running on the dev machine and not on the swarm.

I try changing the configuration file application.yml

#url: http://localhost:2375
url: http://127.0.0.1:32768

Then I try again

Error
Status code: 500
Message: Failed to start container: Request error: POST http://127.0.0.1:32768/services/create: 400, body: {"message":"rpc error: code = InvalidArgument desc = port '20000' is already in use by service '01_hello_20000' (mmaamskif3xrj5rrcjc81xxut) as an ingress port"}
Stack Trace:
eu.openanalytics.ShinyProxyException: Failed to start container: Request error: POST http://127.0.0.1:32768/services/create: 400, body: {"message":"rpc error: code = InvalidArgument desc = port '20000' is already in use by service '01_hello_20000' (mmaamskif3xrj5rrcjc81xxut) as an ingress port"}
at eu.openanalytics.services.DockerService.startProxy(DockerService.java:392)
at eu.openanalytics.services.DockerService.getMapping(DockerService.java:195)

Ok, maybe the docker swarm is already doing something on port 20000. Let’s change that in the application.yml

port-range-start: 55000

and try again

Error
Status code: 500
Message: Failed to start container: Failed to inspect swarm service tasks

Stack Trace:
eu.openanalytics.ShinyProxyException: Failed to start container: Failed to inspect swarm service tasks
at eu.openanalytics.services.DockerService.startProxy(DockerService.java:392)
at eu.openanalytics.services.DockerService.getMapping(DockerService.java:195)
at eu.openanalytics.controllers.AppController.startApp(AppController.java:63)

But if I ssh to the swarm manager and check docker, I can see the app was launched there

$docker ps
CONTAINER ID	6494ca8aaafd
IMAGE		leofranco/image:latest
COMMAND		"R -e shiny::runApp(…"
CREATED        47 seconds ago
STATUS         Up 46 seconds
PORTS		3838/tcp
NAMES		image_55000.1.52bcs4pzje31f2dyb4sj8qh0p

Also, closing shiyproxy doesn’t kill the shiny app running in the swarm.

I have looked for more information on the docker swarm setup for shinyproxy but I haven’t found much. Does anybody have a set of steps that have worked (especially using docker cloud and aws)?

It definitely doesn’t help that I am new to shinyproxy, docker and docker swarm. Maybe I have missed something obvious?

Thanks!!
Leo

p.s. thanks again to the team for an awesome product!


#2

Hi @leo.franco,

Can you give it a try with the latest release, 1.1.0? The previous version did indeed have an issue where unsuccessful startups were not correctly cleaned up. This explains your first error message, about port 20000 being in use.

Note that in version 1.1.0, you have to configure the backend as explained here:
https://www.shinyproxy.io/configuration/#container-back-ends