Hello friends!
First of all, thanks a lot for the great job you are doing with shinyproxy. I have used r and shiny for a while but I haven’t had the need to scale (plus the options to do so where limited).
Now that I found shinyproxy I decided to give it a try. I might have been a little ambitious for my first try and stumbled upon something.
tl;dr
how do we use an auto-scaling docker swarm in aws from shinyproxy?
Setup:
- EC2 machine working as an ubuntu 16.04 dev environment. Used to create docker images and do first tests with shinyproxy.
- Auto scaling docker swam launched with docker cloud on AWS.
Goal:
- Run shinyproxy on the dev machine or in a swarm manager and be able to support an arbitrary number of concurrent users.
What has worked so far:
- Run shiny app docker image on dev machine
- Run shinyproxy on dev machine (multiple apps and multiple users… pretty cool!)
- Run docker image of shiny app on the swarm following instructions from https://docs.docker.com/docker-cloud/cloud-swarm/connect-to-swarm/#connect-to-a-swarm-with-a-docker-cloud-generated-run-command
For the last step, notice that an important part is
export DOCKER_HOST=tcp://127.0.0.1:32768
in this way we use the remote docker swarm instead of the local daemon.
Problem
I run shinyproxy on the dev machine that is connected to the docker swarm, I open it in the browser and when I click on an app.
It works!
But… doing a docker ps shows the shiny app is running on the dev machine and not on the swarm.
I try changing the configuration file application.yml
#url: http://localhost:2375
url: http://127.0.0.1:32768
Then I try again
Error
Status code: 500
Message: Failed to start container: Request error: POST http://127.0.0.1:32768/services/create: 400, body: {"message":"rpc error: code = InvalidArgument desc = port '20000' is already in use by service '01_hello_20000' (mmaamskif3xrj5rrcjc81xxut) as an ingress port"}
Stack Trace:
eu.openanalytics.ShinyProxyException: Failed to start container: Request error: POST http://127.0.0.1:32768/services/create: 400, body: {"message":"rpc error: code = InvalidArgument desc = port '20000' is already in use by service '01_hello_20000' (mmaamskif3xrj5rrcjc81xxut) as an ingress port"}
at eu.openanalytics.services.DockerService.startProxy(DockerService.java:392)
at eu.openanalytics.services.DockerService.getMapping(DockerService.java:195)
Ok, maybe the docker swarm is already doing something on port 20000. Let’s change that in the application.yml
port-range-start: 55000
and try again
Error
Status code: 500
Message: Failed to start container: Failed to inspect swarm service tasks
Stack Trace:
eu.openanalytics.ShinyProxyException: Failed to start container: Failed to inspect swarm service tasks
at eu.openanalytics.services.DockerService.startProxy(DockerService.java:392)
at eu.openanalytics.services.DockerService.getMapping(DockerService.java:195)
at eu.openanalytics.controllers.AppController.startApp(AppController.java:63)
But if I ssh to the swarm manager and check docker, I can see the app was launched there
$docker ps
CONTAINER ID 6494ca8aaafd
IMAGE leofranco/image:latest
COMMAND "R -e shiny::runApp(…"
CREATED 47 seconds ago
STATUS Up 46 seconds
PORTS 3838/tcp
NAMES image_55000.1.52bcs4pzje31f2dyb4sj8qh0p
Also, closing shiyproxy doesn’t kill the shiny app running in the swarm.
I have looked for more information on the docker swarm setup for shinyproxy but I haven’t found much. Does anybody have a set of steps that have worked (especially using docker cloud and aws)?
It definitely doesn’t help that I am new to shinyproxy, docker and docker swarm. Maybe I have missed something obvious?
Thanks!!
Leo
p.s. thanks again to the team for an awesome product!