[Question] kuberay autoscaler v2 rayservice compatibility

[CuriousDolphin]

Hello!
Is kuberay autoscaler compatible with RayService (in RayClusterConfig) or only in plain RayCluster resource?

i'have already configured kube cluster autoscaler (and is working with other pods), and i want to scale up gpu machine when a new job is launched, and scale down when is finished, i use JobSubmissionApi from a python client.
i've tried to launch a job with the submission api, but remain pending and autoscaler is not triggered.

my template is like this (i use helm with argocd); i need to create a standalone RayCluster instead of RayService?

apiVersion: ray.io/v1
kind: RayService
metadata:
  name: ray-model-server
spec:
  serveConfigV2: |
    logging_config:
      encoding: TEXT
      log_level: DEBUG
      logs_dir: null
      enable_access_log: true

    applications:
    - name: model-server
      route_prefix: /
      import_path: model_server:app
      deployments:
        - name: model-server
          num_replicas: {{ .Values.server_replica }}
          max_ongoing_requests: {{ .Values.server_max_ongoing_requests }}
          max_queued_requests: {{ .Values.server_max_queued_requests }}
          ray_actor_options:
            num_gpus: {{ .Values.server_gpus }}
            num_cpus: {{ .Values.server_cpu }}
            accelerator_type: {{ .Values.server_accelerator_type }}

  rayClusterConfig:
    enableInTreeAutoscaling: true
    autoscalerOptions:
      version: v2
      upscalingMode: Default
      idleTimeoutSeconds: 60
      imagePullPolicy: IfNotPresent
      # Optionally specify the Autoscaler container's securityContext.
      securityContext: {}
      envFrom: []
      resources:
        limits:
          cpu: "500m"
          memory: "512Mi"
        requests:
          cpu: "500m"
          memory: "512Mi"
          nvidia.com/gpu: 1
    rayVersion: "{{ .Values.rayVersion }}" 
    # Ray head pod template.
    headGroupSpec:
      rayStartParams:
        dashboard-host: "0.0.0.0"
        metrics-export-port: "8080"
      #pod template
      template:
        spec:
          serviceAccountName: {{ .Values.serviceAccountName }}
          nodeSelector:
            eks.amazonaws.com/nodegroup: {{ .Values.head_nodegroup }}
          containers:
          - name: ray-head
            image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
            resources:
              limits:
                cpu: {{ .Values.head_cpu }}
                memory: {{ .Values.head_memory }}
              requests:
                cpu: {{ .Values.head_cpu }}
                memory: {{ .Values.head_memory }}
            ports:
            - containerPort: 6379
              name: gcs-server
              protocol: TCP
            - containerPort: 8265 # Ray dashboard
              name: dashboard
              protocol: TCP
            - containerPort: 10001
              name: client
              protocol: TCP
            - containerPort: 8000
              name: serve
              protocol: TCP
            - containerPort: 8080
              name: metrics
              protocol: TCP
            - containerPort: 44217
              name: as-metrics # autoscaler
              protocol: TCP
            - containerPort: 44227
              name: dash-metrics # dashboard
              protocol: TCP
            env:
            - name: RAY_GRAFANA_IFRAME_HOST
              value: {{ .Values.grafana_iframe_host }}
            - name: RAY_GRAFANA_HOST
              value: {{ .Values.grafana_host }}
            - name: RAY_PROMETHEUS_HOST
              value: {{ .Values.prometheus_host }}
            - name: RAY_RUNTIME_ENV_HOOK
              value: ray._private.runtime_env.uv_runtime_env_hook.hook
    workerGroupSpecs:
    - replicas: {{ .Values.initialWorkers }}
      minReplicas: {{ .Values.minWorkers }}
      maxReplicas: {{ .Values.maxWorkers }}
      numOfHosts: 1
      groupName: {{ .Values.nodegroup }}
      rayStartParams: {}
      #pod template
      template:
        spec:
          serviceAccountName: {{ .Values.serviceAccountName }}
          nodeSelector:
            eks.amazonaws.com/nodegroup: {{ .Values.nodegroup }}
          tolerations:
          - key: "nvidia.com/gpu"
            operator: "Exists"
            effect: "NoSchedule"
          volumes:
          - name: model-cache
            persistentVolumeClaim:
              claimName: ray-efs-pvc
          containers:
          - name: ray-worker 
            image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
            lifecycle:
              preStop:
                exec:
                  command: [ "/bin/sh", "-c", "ray stop" ]
            resources:
              limits:
                cpu: {{ .Values.workerCpu }}
                memory: {{ .Values.workerMemory }}
              requests:
                cpu: {{ .Values.workerCpu }}
                memory: {{ .Values.workerMemory }}
                nvidia.com/gpu: 1
            volumeMounts:
            env:
            - name: ENVIRONMENT
              value: {{ .Values.environment }}

[fscnick]

Hi, After brief testing based on the information you provide, the RayService autoscaler could work if it stays on the same driver process. Calling JobSubmissionApi might create another driver process resulting in the newly created pod failed on readiness probe.

If it is not the case you encountered, would you mind to provide more detail about this issue like how to reproduce step by step or the more error message?

[CuriousDolphin]

actually might be my problem, but i don't sure because in my case i didnt see a new pod, even if there are resources available in the two service machines and the job is executed in those two machines.
is there any way to force the driver to use the service one?

The succeded job is executed in service worker (the same of the model-server) because there is 0.2 gpu available.
the other one is pending for resources.

this is the output of ray status in head pod:

all ray pods:

what other information would be helpful to you? i have full access to both the cluster and the codebase.

[fscnick]

is there any way to force the driver to use the service one?

I didn't know that. However, it is not the recommended way to use the RayService and triggering some work via JobSubmissionApi. Is any specific reason that it should stick on the RayService? Could it use RayJob instead?

Is the behavior you expected the RayService autoscaler will create another pod with GPU so that it would have enough resources to run the job?

What is the meaning of Con... in the last picture? There is a gray block in the worker pod. Does it mean something?

[CuriousDolphin]

I didn't know that. However, it is not the recommended way to use the RayService and triggering some work via JobSubmissionApi. Is any specific reason that it should stick on the RayService? Could it use RayJob instead?

I setupped RayService because is the faster way to rise up a ray cluster with a ray model server on kube, in the context of kubernetes you mean create a "RayJob Resource"? in this case i want to use the submission api for running remote job that maybe not are declared in any template; eg: a random script in a remote working_dir.

Is the behavior you expected the RayService autoscaler will create another pod with GPU so that it would have enough resources to run the job?

exactly!!

What is the meaning of Con... in the last picture? There is a gray block in the worker pod. Does it mean something?
Containers!

the gray pod is "wait-gcs-ready", i think is the probe that launch workers when the head is ready.

So, I'm actually thinking that the problem is with how I launch the jobs and relative the drivers, but I still haven't figured out what is the correct way to do what I would like to do.

[fscnick]

I setupped RayService because is the faster way to rise up a ray cluster with a ray model server on kube, in the context of kubernetes you mean create a "RayJob Resource"? in this case i want to use the submission api for running remote job that maybe not are declared in any template; eg: a random script in a remote working_dir.

I guess this method might already come across your mind. Use RayCluster in conjunction with autoscaler and bring up ray model server manually on it. It could create pods as need by submitting job through the submission api and terminate after job done.

Sorry, I'm not sure if there is a way to use the existed driver or not.