Readiness Pod Patch

ShinyProxy
#1

Hello,
I have shinyproxy [2.6.0] running in openshift (aka kubernetes). The issue I am having is when an application launches, and the image is not cached on the worker node, the image pull time is long enough that the pod throws a timeout error. The image is coming from an image repo.

Error
Status code: 500

Message: Container failed to start

Caused by: eu.openanalytics.containerproxy.ContainerProxyException: Container did not become ready in time
at eu.openanalytics.containerproxy.backend.kubernetes.KubernetesBackend.startContainer(KubernetesBackend.java:293)
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.doStartProxy(AbstractContainerBackend.java:163)
at eu.openanalytics.containerproxy.backend.AbstractContainerBackend.startProxy(AbstractContainerBackend.java:129)
at eu.openanalytics.containerproxy.service.ProxyService.startProxy(ProxyService.java:279)

I thought increasing the container wait time would help this issue (container-wait-time: 200000), however, this time does not seem to be used before the container is running. Looking at the source code, it seems that this timeout is related to kubernetes internal ‘readiness’. Attached is the image of the relevant shinyproxy kubernetes backend code and the path to that code:

shinyproxy kubernetes backend java

This makes me believe there needs to be a pod patch for readiness/readiness probe/liveliness. Can anyone comment if this is correct? If it is, is there a sample of adding these as pod patches?

shinyproxy_kubernetes
shinyproxy_kubernetes931×277 14.5 KB

#2

Worth noting ive tried the pod patch for a readiness probe and startup probe like this

- op: add
  path: /spec/containers/0/readinessProbe
  value:
   - readinessProbe:
       httpGet:
         path: /actuator/health/readiness
         port: 9090
       periodSeconds: 10
       initialDelaySeconds: 120

and like this

   - op: add
     path: /spec/containers/0/readinessProbe
     value: |
       readinessProbe:
         httpGet:
           path: /actuator/health/readiness
           port: 9090
         periodSeconds: 10
         initialDelaySeconds: 120

Both have thrown errors

#3

Hi

The Container did not become ready in time message will not be solved by adding readiness or liveness probes. Those are used for indicating the status of a Pod to Kubernetes, but only after the Pod has started (and thus after the image has been pulled).

By default ShinyProxy waits up to 60 seconds for the image to be pulled and the pod to start up. You can increase this timeout using the proxy.kubernetes.pod-wait-time option: https://shinyproxy.io/documentation/configuration/#kubernetes

1 Like
#4

Yes! that is exactly what i was looking for! thanks for the response.

1 Like