With some load on our ShinyProxy instance, we see apps failing to start with ShinyProxy 3.1.1 (we don’t see that with ShonyProxy 3.0.2)
eu.openanalytics.containerproxy.ProxyFailedToStartException: Container with index 0 failed to start
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.startProxy(AbstractContainerBackend.java:132) ~[containerproxy-1.1.1.jar!/:1.1.1]
at eu.openanalytics.containerproxy.backend.dispatcher.DefaultProxyDispatcher.startProxy(DefaultProxyDispatcher.java:49) ~[containerproxy-1.1.1.jar!/:1.1.1]
at eu.openanalytics.containerproxy.service.ProxyService.startOrResumeProxy(ProxyService.java:471) ~[containerproxy-1.1.1.jar!/:1.1.1]
at eu.openanalytics.containerproxy.service.ProxyService.lambda$startProxy$5(ProxyService.java:293) ~[containerproxy-1.1.1.jar!/:1.1.1]
at eu.openanalytics.containerproxy.service.ProxyService.lambda$action$12(ProxyService.java:638) ~[containerproxy-1.1.1.jar!/:1.1.1]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
Caused by: eu.openanalytics.containerproxy.ContainerFailedToStartException: Docker container failed to start
at eu.openanalytics.containerproxy.backend.docker.DockerEngineBackend.startContainer(DockerEngineBackend.java:195) ~[containerproxy-1.1.1.jar!/:1.1.1]
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.startProxy(AbstractContainerBackend.java:126) ~[containerproxy-1.1.1.jar!/:1.1.1]
... 9 common frames omitted
Caused by: org.mandas.docker.client.exceptions.DockerTimeoutException: Timeout: GET http://localhost:2375/images/<DOCKER_IMAGE>/json
at org.mandas.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2578) ~[docker-client-7.0.8-OA-3.jar!/:na]
at org.mandas.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2438) ~[docker-client-7.0.8-OA-3.jar!/:na]
at org.mandas.docker.client.DefaultDockerClient.inspectImage(DefaultDockerClient.java:1228) ~[docker-client-7.0.8-OA-3.jar!/:na]
at eu.openanalytics.containerproxy.backend.docker.DockerEngineBackend.isImagePresent(DockerEngineBackend.java:316) ~[containerproxy-1.1.1.jar!/:1.1.1]
at eu.openanalytics.containerproxy.backend.docker.DockerEngineBackend.startContainer(DockerEngineBackend.java:102) ~[containerproxy-1.1.1.jar!/:1.1.1]
... 10 common frames omitted
Caused by: jakarta.ws.rs.ProcessingException: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:533) ~[jersey-apache-connector-3.1.6.jar!/:na]
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300) ~[jersey-client-3.1.6.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:687) ~[jersey-client-3.1.6.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:709) ~[jersey-client-3.1.6.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:703) ~[jersey-client-3.1.6.jar!/:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-3.1.6.jar!/:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-3.1.6.jar!/:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-3.1.6.jar!/:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:391) ~[jersey-common-3.1.6.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:703) ~[jersey-client-3.1.6.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[jersey-client-3.1.6.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:422) ~[jersey-client-3.1.6.jar!/:na]
at org.mandas.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2436) ~[docker-client-7.0.8-OA-3.jar!/:na]
... 13 common frames omitted
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.14.jar!/:4.5.14]
at org.glassfish.jersey.apache.connector.ApacheConnector.apply(ApacheConnector.java:485) ~[jersey-apache-connector-3.1.6.jar!/:na]
... 25 common frames omitted
Reverting back to 3.0.2 fixes the problem, so we are quite sure the problem is actually related to the new ShinyProxy version.
While you increased the readTimeoutMillis() in containerproxy/src/main/java/eu/openanalytics/containerproxy/backend/docker/AbstractDockerBackend.java at master · openanalytics/containerproxy · GitHub, might it be that increasing the connectTimeoutMillis() as well could help here?
It seems, the default timeout is rather short: docker-client/src/main/java/org/mandas/docker/client/builder/BaseDockerClientBuilder.java at master · dmandalidis/docker-client · GitHub
Thanks & best regards