Hello,
I’m seeking assistance with an error that’s preventing certain pods on our server from closing properly. We’ve been dealing with this issue for a while in our Kubernetes environment where ShinyProxy is running. The problem revolves around certain pods that fail to close automatically and linger until we manually intervene to close them. After an examination of the logs, we’ve found errors originating from ShinyProxy as the root cause. The problem is that ShinyProxy doesn’t try to close them later, leaving these pods in an open state for hours. As an example, I’ve included the full log snippet below, which shows a timeout error. In this example the Read timed out error is the root cause, then after that the containerProxy fails to stop the container and leave it open.
eu.openanalytics.containerproxy.ContainerProxyException: Failed to stop container
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.stopProxy(AbstractContainerBackend.java:130) ~[containerproxy-1.0.2.jar!/:1.0.2]
at eu.openanalytics.containerproxy.service.ProxyService.lambda$stopProxy$6(ProxyService.java:318) ~[containerproxy-1.0.2.jar!/:1.0.2]
at eu.openanalytics.containerproxy.service.ProxyService.lambda$action$11(ProxyService.java:599) ~[containerproxy-1.0.2.jar!/:1.0.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_342]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_342]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_342]
at java.lang.Thread.run(Thread.java:750) ~[na:1.8.0_342]
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [delete] for kind: [Pod] with name: [sp-pod-69b290d8-cea7-4ad2-ab03-29dbb8054b6f] in namespace: [shiny] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159) ~[kubernetes-client-api-6.3.1.jar!/:na]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.deleteAll(BaseOperation.java:508) ~[kubernetes-client-6.3.1.jar!/:na]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.delete(BaseOperation.java:467) ~[kubernetes-client-6.3.1.jar!/:na]
at eu.openanalytics.containerproxy.backend.kubernetes.KubernetesBackend.doStopProxy(KubernetesBackend.java:601) ~[containerproxy-1.0.2.jar!/:1.0.2]
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.stopProxy(AbstractContainerBackend.java:128) ~[containerproxy-1.0.2.jar!/:1.0.2]
... 7 common frames omitted
Caused by: java.io.IOException: Read timed out
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:533) ~[kubernetes-client-6.3.1.jar!/:na]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:570) ~[kubernetes-client-6.3.1.jar!/:na]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:554) ~[kubernetes-client-6.3.1.jar!/:na]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleDelete(OperationSupport.java:327) ~[kubernetes-client-6.3.1.jar!/:na]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.deleteAll(BaseOperation.java:502) ~[kubernetes-client-6.3.1.jar!/:na]
... 10 common frames omitted
Caused by: javax.net.ssl.SSLException: Read timed out
at sun.security.ssl.Alert.createSSLException(Alert.java:127) ~[na:1.8.0_342]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:324) ~[na:1.8.0_342]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:267) ~[na:1.8.0_342]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:262) ~[na:1.8.0_342]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:138) ~[na:1.8.0_342]
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1397) ~[na:1.8.0_342]
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1305) ~[na:1.8.0_342]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:440) ~[na:1.8.0_342]
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:379) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) ~[okhttp-4.9.3.jar!/:na]
at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517) ~[okhttp-4.9.3.jar!/:na]
... 3 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_342]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[na:1.8.0_342]
at java.net.SocketInputStream.read(SocketInputStream.java:171) ~[na:1.8.0_342]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[na:1.8.0_342]
at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:464) ~[na:1.8.0_342]
at sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165) ~[na:1.8.0_342]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:109) ~[na:1.8.0_342]
... 23 common frames omitted
2023-09-27 08:15:39.373 INFO 1 --- [pool-2-thread-636] e.o.containerproxy.service.ProxyService : [user=669d1505a7754e12cea6efc493edcdf91740578db61ab8d0a62aa73394d319ec proxyId=0d27149e-74af-4bfb-9169-a5422eb3c5a1 specId=Skynet_Application] Proxy released