Occasional "Failed to start container" exceptions

#1

We are using shinyproxy to host ~20 shiny apps. We usually have at most 5 concurrent users (total - only a couple apps are used at a time).

A few times per week, a user runs into a “Failed to start container” exception. From what I’ve searched, the underlying exception (DockerTimeoutExceptIon) and the fact that it only occurs intermittently is different from other scenarios. Here is the full stack trace:
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is eu.openanalytics.containerproxy.ContainerProxyException: Failed to start container
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:986) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:881) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:855) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) ~[spring-security-web-5.0.3.RELEASE.jar!/:5.0.3.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:64) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:94) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
at eu.openanalytics.containerproxy.util.ProxyMappingManager$ProxyPathHandler.handleRequest(ProxyMappingManager.java:159) ~[containerproxy-0.8.1.jar!/:0.8.1]
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) [undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) [undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [undertow-servlet-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:336) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-1.4.22.Final.jar!/:1.4.22.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: eu.openanalytics.containerproxy.ContainerProxyException: Failed to start container
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.startProxy(AbstractContainerBackend.java:110) ~[containerproxy-0.8.1.jar!/:0.8.1]
at eu.openanalytics.containerproxy.service.ProxyService.startProxy(ProxyService.java:213) ~[containerproxy-0.8.1.jar!/:0.8.1]
at eu.openanalytics.shinyproxy.controllers.AppController.getOrStart(AppController.java:100) ~[classes!/:2.2.1]
at eu.openanalytics.shinyproxy.controllers.AppController.startApp(AppController.java:68) ~[classes!/:2.2.1]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:870) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:776) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978) ~[spring-webmvc-5.0.4.RELEASE.jar!/:5.0.4.RELEASE]
… 79 common frames omitted
Caused by: com.spotify.docker.client.exceptions.DockerTimeoutException: Timeout: POST http://127.0.0.1:2375/containers/bda71f5df807a6a2d25911ac427fd3f8c98f0a26a701ef11be71aa3d7c130dbe/start
at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2705) ~[docker-client-8.11.2.jar!/:8.11.2]
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2663) ~[docker-client-8.11.2.jar!/:8.11.2]
at com.spotify.docker.client.DefaultDockerClient.containerAction(DefaultDockerClient.java:700) ~[docker-client-8.11.2.jar!/:8.11.2]
at com.spotify.docker.client.DefaultDockerClient.containerAction(DefaultDockerClient.java:685) ~[docker-client-8.11.2.jar!/:8.11.2]
at com.spotify.docker.client.DefaultDockerClient.startContainer(DefaultDockerClient.java:680) ~[docker-client-8.11.2.jar!/:8.11.2]
at eu.openanalytics.containerproxy.backend.docker.DockerEngineBackend.startContainer(DockerEngineBackend.java:83) ~[containerproxy-0.8.1.jar!/:0.8.1]
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.doStartProxy(AbstractContainerBackend.java:126) ~[containerproxy-0.8.1.jar!/:0.8.1]
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.startProxy(AbstractContainerBackend.java:107) ~[containerproxy-0.8.1.jar!/:0.8.1]
… 95 common frames omitted
Caused by: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[na:1.8.0_131]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[na:1.8.0_131]
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[na:1.8.0_131]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[na:1.8.0_131]
at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2661) ~[docker-client-8.11.2.jar!/:8.11.2]
… 101 common frames omitted
Caused by: javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:481) ~[jersey-apache-connector-2.22.2.jar!/:na]
at org.glassfish.jersey.apache.connector.ApacheConnector$1.run(ApacheConnector.java:491) ~[jersey-apache-connector-2.22.2.jar!/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
at jersey.repackaged.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299) ~[jersey-guava-2.26-b03.jar!/:na]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) ~[na:1.8.0_131]
at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:50) ~[jersey-guava-2.26-b03.jar!/:na]
at jersey.repackaged.com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37) ~[jersey-guava-2.26-b03.jar!/:na]
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:487) ~[jersey-apache-connector-2.22.2.jar!/:na]
at org.glassfish.jersey.client.ClientRuntime.lambda$null$3(ClientRuntime.java:187) ~[jersey-client-2.26.jar!/:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[jersey-common-2.26.jar!/:na]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[jersey-common-2.26.jar!/:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[jersey-common-2.26.jar!/:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[jersey-common-2.26.jar!/:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[jersey-common-2.26.jar!/:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312) ~[jersey-common-2.26.jar!/:na]
at org.glassfish.jersey.client.ClientRuntime.lambda$createRunnableForAsyncProcessing$4(ClientRuntime.java:163) ~[jersey-client-2.26.jar!/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
… 3 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_131]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_131]
at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[na:1.8.0_131]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_131]
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[httpcore-4.4.9.jar!/:4.4.9]
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[httpcore-4.4.9.jar!/:4.4.9]
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[httpcore-4.4.9.jar!/:4.4.9]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[httpclient-4.5.5.jar!/:4.5.5]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.5.jar!/:4.5.5]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.9.jar!/:4.4.9]
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.9.jar!/:4.4.9]
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[httpclient-4.5.5.jar!/:4.5.5]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.9.jar!/:4.4.9]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.9.jar!/:4.4.9]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.5.jar!/:4.5.5]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.5.jar!/:4.5.5]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.5.jar!/:4.5.5]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) ~[httpclient-4.5.5.jar!/:4.5.5]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.5.jar!/:4.5.5]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.5.jar!/:4.5.5]
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:435) ~[jersey-apache-connector-2.22.2.jar!/:na]
… 21 common frames omitted

A few other details:
Docker 18.06.1-ce
ShinyProxy 2.2.1
CentOS 7

Any leads on what could be going wrong would be appreciated!

#2

Did You try to increase the timeout limit in shinyproxy application.yml https://www.shinyproxy.io/configuration/ ? Sometimes shiny applications take a longer time to load all necesarry libraries etc.

#3

Which timeouts in particular are you referring to? I recently increased heartbeat-timeout and container-wait-time to 20 minutes but still saw this error occur before that timeout expired (I restarted shinyproxy for the config change to take effect - is that enough?).

#4

To be more precise, the error is being thrown 30 seconds after “Starting container with Id: …” is logged. I don’t see anything in my config file that is set to 30 seconds (or ever has been), so it’s possible that I’m missing whatever option that is and it’s using a default. I haven’t been able to find what that would be, though.

#5

I realize my replies may not have notified you. I had some follow up questions.

#6

I think the problem is somewhere else. Try to run the container manually in an interactive mode, i. e. run the command
docker run --rm -it -p 9999:3838 -d [your_image]
After this command is executed your app should be available on port 9999 in a browser.
Check the output in the console. Maybe it’s a matter of the image, not shinyproxy configuration.

#7

I’ve tried this ~10 times (since we’ve seen it as an intermittent issue) but haven’t seen anything except the expected output.

#8

Try to run many containers (the apps you mentioned at the beginning) at the same time, you can run containers manually. Maybe the problem is related to the number of containers running at the same time. There may be a problem with resources. In this case this link could be helpfull https://docs.docker.com/v17.09/engine/admin/resource_constraints/#configure-the-realtime-scheduler