ShinyProxy Container alive while process executing?

sukanya30 · February 21, 2019, 12:05am

Hi,
I am running ShinyProxy with Kubernetes and have heartbeat-timeout set to 15 mins.

My R Shiny App, accesses a ML model hosted in third-party platform and is supposed to wait for the model to run and display the returned results from platform. (This works fine in my local R Studio Shiny app).

Does heart-beat timeout mean that the container serving a user will be killed after “timeout seconds” even if the Shiny app is waiting for the ML model to execute and has not returned any result yet from the third-party API call?
i.e.

Kindly please help me solving this. Thanks!

shrektan · February 18, 2019, 11:38am

No, it means after you closed the tab and tried to reconnect to the app, as long as it’s within 15min , you will connect to the same container.

sukanya30 · February 21, 2019, 12:06am

Thank you so much for your reply.

Just to confirm - if Shiny app is waiting for a third party API call to finish and return results and is showing a loading symbol meanwhile, then however long does the API call take to execute, Shiny Proxy app will always wait for its completion, correct?

shrektan · February 18, 2019, 4:16pm

It’s true in my opinion. Through there are possibilities that the browser will disconnect the session if no response for too long. You can actually build a simple app to test. Better approach is running the API in a new process… as stated in https://shiny.rstudio.com/articles/async.html

sukanya30 · February 21, 2019, 12:07am

Thank you for your response!

With the same Shiny App (that calls a ML model via an API call), When I run my model the first time after logging into the app, it works fine.

After a few minutes of inactivity, say 10 minutes, I rerun the API call and it never runs to completion, I keep on seeing loading symbol.

Only way I can make it run is log out, log in again and re-run and it executes fine.

Can you please tell me the solution to this problem? I have heartbeat-timeout set to 15 minutes.

This is an urgent issue for us, and would be of huge help if you can provide a solution. Thanks!

shrektan · February 18, 2019, 11:43pm

Blocking the main process for too long may result in disconnected session, I think the browser has some kind of settings.

As I said, the simple and definitely working solution is that you can run the ml in a separate process. Consider using the callr package. Just call a background process to run and let the main process regularly check the status of the ml process.

sukanya30 · February 21, 2019, 12:08am

Hi,

We debugged more. Looks like there is a bug in ShinyProxy jar’s interaction with Kubernetes deployment.

SCENARIO - We have Shiny app deployed with ShinyProxy on Kubernetes. Our app calls a light-weight ML model hosted in third party platform via API call. The execution time of this API call in local RStudio Shiny App is 30 second, which is the same with ShinyProxy deployed app when we access the model via API call for the first time.
I wait for 5-6 minutes without interacting with the ShinyProxy app, leaving it idle. Nothing executing.
After that, I re-run the lightweight ML model API call and the container hangs for about 19-25 minutes and suddenly gives result successfully after.

DEBUGGING AND RESULT:
Login in App and run model first time - executed in 30 sec - Our DevOps team did a CURL request on ShinyProxy jar and found that the jar was able to contact the container which contains our app’s docker image.

Wait for 6 minutes and re-run the model - total execution time 20 minutes - DevOps team did a CURL request again on ShinyProxy jar and found that the containers had hanged and CURL was not returning any response for 19 minutes and gave a response in the last minute and we could simultaneously see results populated on the Shiny app.

This does not look like any async processing or session or timeout issue.
I also tried changing the browser settings as you had mentioned above. But saw the same issue.
Would appreciate your insights on this.

shrektan · February 19, 2019, 4:58pm

I’ve never used Kubernetes. But according to my limited experiences, I suggest:

Ensure it has nothing to do with your image itself (You may use docker run --rm -p 3838:3838 $(your image) R -e "shiny::runApp(...)" and see if the app in 127.0.0.1:3838 has the same issue.
If TRUE, then it has nothing to do with shinyproxy.
If it works fine, then it may relate to shinyproxy although I doubt it. It’s because shinyproxy’s hosting model is quite simple and elegant, thus difficult to go wrong… IMHO.

Anyway, It’s just my personal guesses… I’d like to hear from the shinyproxy team…

hangs for about 19-25 minutes and suddenly gives result successfully after

BTW, not sure if it helps. I have a similar experience when I happen to close the tab while a async process is running… I will get the result (e.g., an output email) whenever I reconnect to the app again…

sukanya30 · February 21, 2019, 1:19am

So my team and I debugged with extensive logging.
We observed that after 6+minutes of inactivity on the RShiny website deployed with ShinyProxy, if we initiate an API call, the website screen keeps on showing loading symbol for 20-25 minutes but the call never reaches the ML model.

Only after 20-25mins the ML model is invoked and gives back the result in its usual timeframe.

Shiny logs ----
Calling Algorithm. Request sent at - 2019-02-20 23:24:38

Algorithm API call logs ----
2019-02-20 23:40:52.403353 - Inside Algorithm Function, initiated the model
2019-02-20 23:40:52.742326 - Call has finished, returning results

I can confirm that there is no problem with the image as the same image inside a normal docker container deployed without ShinyProxy does not show this behavior.

print(session$isClosed()) debugging statement gave “False” so I am assuming Shiny “session” object has not lost any connection.

Do you have any insights to check if something is dying/timing out during inacvity that might be preventing the Shiny app to truly initiate API call until only very late?
If yes, can you share any code snippets for the same?

sukanya30 · February 21, 2019, 12:01am

@fmichielssen – Hi! I saw some of your useful tips on other people’s issues. Would you have any insight on the issue I am facing? It is regarding inactivity of Shiny app deployed with ShinyProxy that breaks something and API calls from my app do not reach third party platform until after 20-25 minutes.

Thanks!

fmichielssen · February 26, 2019, 7:25am

Hi @sukanya30,

If I understand correctly, the following happens?

User opens the Shiny app
User clicks a button to trigger an API call to your backend model API
It takes 16 minutes for the request to ‘arrive’ in the backend

Some thoughts:

It doesn’t look like the front channel (between user’s browser and Shinyproxy) is involved in the issue. This is where the heartbeat takes place. If the heartbeat times out, the container will be killed. As long as there is a living websocket channel between the browser and Shinyproxy, the heartbeat should not time out.

The problem seems to be in the communication channel between the Shiny app (docker container in kubernetes) and the backend API (third party platform). Do you know exactly how the docker container can reach the backend API? Which networks, firewalls, etc are involved in this route?

sukanya30 · February 27, 2019, 3:05am

Hi @fmichielssen,
Thank you for getting back to me.

Slight correction:

User opens the Shiny app
User clicks a button to trigger an API call to backend model API. Works fine and gives result back in 1 minute.
User is inactive on Shiny app for 6+ minutes. (Important - this causes the issue below)
User clicks a button to trigger an API call to backend model API. It suddenly takes 16 minutes for the request to ‘arrive’ in the backend instead of a few seconds as in step 2.

As to your point on problem being in communication channel of Shiny app (docker container in kubernetes) and the backend API -
Kubernetes uses port 80 and 443, both of which are open. The load-balancer of Kubernetes is behind a firewall, however, if this was a firewall issue, I would not have been able to hit the API successfully first time.

The same shiny app in a simple docker container in my local (without ShinyProxy) can hit the API multiple times no matter however long the user is inactive on the app and gives results back within a minute.

fmichielssen · February 27, 2019, 3:58pm

Hi @sukanya30,

Are you running shinyproxy inside kubernetes as well (internal-networking: true), or outside the cluster?
An additional test that might be helpful, is to CURL the target pod, bypassing the shinyproxy route. You’ll find the target URL in the Endpoint column of the admin page. If internal-networking: true then the URL will be a cluster-internal URL, and you have to CURL from within another pod.

sukanya30 · March 1, 2019, 1:10am

Hi @fmichielssen. Yes we are running shinyproxy inside kubernetes with “internal-networking: true”. We had done the CURL test you mentioned and following are our observations:

– When we CURL before the inactivity period of 6+ minutes - we get back result instantly.
– After 6+minutes of inactivity, if we trigger the API call, and the application is showing loading symbol for 15 minutes (assuming it is hung), and we do the same CURL procedure, it hangs too. We do not get back any CURL results for 15 minutes. After 15 minutes, when the Shiny app successfully shows ML model results, the CURL command gives result too.

Please note that we tried deploying shinyproxy and our shiny app in a standalone docker container and saw the exact same issues.

However, if we deploy just our shiny app in a docker container, we NEVER see this problem occur. The only reason this method does not serve our purpose is because we need multiple people to access the shiny app and shinyproxy is the feasible free solution for us.

sukanya30 · March 1, 2019, 7:15pm

Hi @fmichielssen,

My team and I can get on a call and demo the issue to you, if that will be helpful.

Please let me know.