SHINYPROXY_USERGROUPS is empty

I am running Shiny Proxy (openanalytics/shinyproxy-operator:2.3.1)
SSO works, but I have more than 200 groups so msgraph needs to return the groups.
I must be missing something, is there a debug setting that I could set to see if msgraph is responding.

I have followed the instructions from here.

However it is still empty
This is my config.

  openid:
    auth-url: https://login.microsoftonline.com/12345678-1234-1234-abcdefghijlk/oauth2/v2.0/authorize
    token-url: https://login.microsoftonline.com/12345678-1234-1234-abcdefghijlk/oauth2/v2.0/token
    jwks-url: https://login.microsoftonline.com/common/discovery/v2.0/keys
    client-id: abcdefgh-1234-1234-1234-abcdefjhighds
    client-secret: lJM8Q~ka,sdfbnq2389roidacjkq..
    username-attribute: name
    scopes: ["openid", "offline_access", "email", "profile"]
    roles-claim: groups
  ms-graph:
    token-url: https://login.microsoftonline.com/12345678-1234-1234-abcdefghijlk/oauth2/v2.0/token
    client-id: abcdefgh-1234-1234-1234-abcdefjhighds
    client-secret: lJM8Q~ka,sdfbnq2389roidacjkq..
    tenant-id: 12345678-1234-1234-abcdefghijlk

These are the API permissions in Entra ID

Microsoft Graph (7)
email Delegated View users' email address No  Granted for BD
Group.Read.All Application Read all groups Yes Not granted for BD
offline_access Delegated Maintain access to data you have given it access to No Granted for  D
openid Delegated Sign users in No Granted for BD
profile Delegated View users' basic profile No Granted for BD
User.Read Delegated Sign in and read user profile No Granted for BD
User.Read.All Application Read all users' full profiles Yes Not granted for BD