"Container did not respond in time" ... but I can connect to the spawned App


#1

I’m trying to get the examples working and I’m hitting an odd snag with the containerized Docker example. I’m using the example from 02-containerized-docker-engine

I can authenticate using LDAP, I see my two apps listed. When I click on one I can see it spawn another docker image. And I can even connect to the spawned image at http://localhost:20000 and the app works great… until the ShinyProxy times out and kills it returning the error:

Status code: 500

Message: Container did not respond in time`

Stack Trace:
eu.openanalytics.containerproxy.ContainerProxyException: Container did not respond in time
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.startProxy(AbstractContainerBackend.java:115)
at eu.openanalytics.containerproxy.service.ProxyService.startProxy(ProxyService.java:213)

Full stack trace at the bottom of this post.

I read through the troubleshooting (well written and helpful!) and I was suspicious that my app docker image was not spawning. But I was able to connect to the spawned app image, which leaves me totally flummoxed. How can I manually connect to the spawned app image yet ShinyProxy does not think the app is up? I’m at a bit of a loss as to how to debug. Any help is appreciated.

My application.yml file is as follows:

proxy:
  title: Open Analytics Shiny Proxy
  logo-url: http://www.openanalytics.eu/sites/www.openanalytics.eu/themes/oa/logo.png
  landing-page: /
  heartbeat-rate: 10000
  heartbeat-timeout: 60000
  container-wait-time: 30000
  port: 8181
  authentication: ldap
  admin-groups: Users
  
  ldap:
    url: ldap://dc01.myorg.com:389/dc=myorg,dc=com
    manager-dn: CN=Long\, James,OU=Users,OU=New York,OU=Offices,DC=myorg,DC=com
    manager-password: xxxxxxxxxxx
    user-search-base:
    user-search-filter: (sAMAccountName={0})
    group-search-base: OU=Users,OU=New York,OU=Offices
    group-search-filter: (member={0})
      
  # Docker configuration
  docker:
    cert-path: /home/none
    port-range-start: 20000
    container-protocol: http
    
  specs:
  - id: 01_hello
    display-name: Hello Application
    description: Application which demonstrates the basics of a Shiny app
    container-cmd: ["R", "-e", "shinyproxy::run_01_hello()"]
    container-image: openanalytics/shinyproxy-demo
    access-groups: 
  - id: 06_tabsets
    container-cmd: ["R", "-e", "shinyproxy::run_06_tabsets()"]
    container-image: openanalytics/shinyproxy-demo
    access-groups: 

logging:
  file:
    shinyproxy.log

The full text of my stack trace is:

Error
Status code: 500

Message: Container did not respond in time

Stack Trace: 
eu.openanalytics.containerproxy.ContainerProxyException: Container did not respond in time
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.startProxy(AbstractContainerBackend.java:115)
at eu.openanalytics.containerproxy.service.ProxyService.startProxy(ProxyService.java:213)
at eu.openanalytics.shinyproxy.controllers.AppController.getOrStart(AppController.java:89)
at eu.openanalytics.shinyproxy.controllers.AppController.startApp(AppController.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:870)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:776)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:881)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:64)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:94)
at eu.openanalytics.containerproxy.ContainerProxyApplication$ProtectedPathHandler.handleRequest(ContainerProxyApplication.java:111)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:336)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

The logs on the ShinyProxy server show the following (which tell me that it’s looking in the right place http://localhost:20000):


2019-02-04 00:23:39.999  INFO 1 --- [  XNIO-2 task-6] c.s.docker.client.DefaultDockerClient    : Starting container with Id: e5a55f990af8e24cd363880c932c73a09946fc73bcc3c95a40cd22f32a9d9239
2019-02-04 00:23:42.487  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (2/15): http://localhost:20000
2019-02-04 00:23:44.491  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (3/15): http://localhost:20000
2019-02-04 00:23:46.494  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (4/15): http://localhost:20000
2019-02-04 00:23:48.498  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (5/15): http://localhost:20000
2019-02-04 00:23:50.499  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (6/15): http://localhost:20000
2019-02-04 00:23:52.501  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (7/15): http://localhost:20000
2019-02-04 00:23:54.503  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (8/15): http://localhost:20000
2019-02-04 00:23:56.504  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (9/15): http://localhost:20000
2019-02-04 00:23:58.474  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (10/15): http://localhost:20000
2019-02-04 00:24:00.476  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (11/15): http://localhost:20000
2019-02-04 00:24:02.479  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (12/15): http://localhost:20000
2019-02-04 00:24:04.484  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (13/15): http://localhost:20000
2019-02-04 00:24:06.487  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (14/15): http://localhost:20000
2019-02-04 00:24:08.489  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (15/15): http://localhost:20000
2019-02-04 00:24:11.040 ERROR 1 --- [  XNIO-2 task-6] io.undertow.request                      : UT005023: Exception handling request to /app/01_hello

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is eu.openanalytics.containerproxy.ContainerProxyException: Container did not respond in time
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:986) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:881) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
...

#2

It feels as if my ShinyProxy image can’t “see” my spawned app image. So I went back through the configuration but it looks ok. I set up a bridge which I can see when I docker network ls:

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
a015852c5631        bridge              bridge              local
77873acb6686        host                host                local
a9bad54ec5a4        none                null                local
66ec60b631da        sp-example-net      bridge              local

and I’m using that bridge when I start my ShinyServer:

sudo docker run -d -v /var/run/docker.sock:/var/run/docker.sock --net sp-example-net -p 8181:8181 shinyproxy-example

#3

For debugging I also changed the yml config file to use networking as shown in the example on github.

  docker:
    internal-networking: true

which changes slightly what’s going on in the log:

2019-02-04 00:48:29.046  INFO 1 --- [  XNIO-2 task-7] c.s.docker.client.DefaultDockerClient    : Starting container with Id: 83c490b804e3b4d7bbb3464a1dec7dca041ed6ad632376142f818a0e18666790
2019-02-04 00:48:32.201  WARN 1 --- [  XNIO-2 task-7] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (2/30): http://83c490b804e3:3838

However I am unable to connect to http://83c490b804e3:3838 from my browser. Nor can I connect to localhost:3838. Which makes sense. Otherwise the behavior is the same: still does not connect.


#4

Since you are using a container to run shinyproxy, you have to let the containers know where the network they should be running from.

So container-network: sp-example-net is a necessity.

See https://github.com/openanalytics/shinyproxy-config-examples/blob/master/02-containerized-docker-engine/application.yml


#5

Moreover, if you are using OSX, my experience is I have to run another process in order to access the docker daemon correctly. See https://github.com/openanalytics/shinyproxy/issues/20


#6

Yep, that’s it. Thank you so much! In my debugging I had managed to copy chunks from another example in there.

For future travelers, you need the container-network in the specs and also internal-networking in the docker settings:

  docker:
    internal-networking: true