HAMi 是一个用于管理 Kubernetes 集群中异构 AI 计算设备的开源平台。其前身为 k8s-vGPU-scheduler，可在多个容器和工作负载之间实现设备共享。
Volcano 是基于 Kubernetes 的容器批量计算平台，主要用于高性能计算场景。
本文结合两者的能力进行实践，参考资料：https://project-hami.io/zh/tutorials/labs/volcano-vgpu-gang-queue
前提条件
Kubernetes 已部署，版本 >= 1.16，本文使用 RKE2 v1.35.6+rke2r1
GPU 节点已接入集群
GPU 节点已安装内核驱动和 NVIDIA Container Toolkit
GPU 节点默认 Runtime 已设置为 Nvidia
集群没有安装 NVIDIA 或 HAMi 的 Device Plugin
内核驱动、NVIDIA Container Toolkit 的安装和 RKE2 默认 Runtime 的配置可参考：
下面的资源值针对 12 GiB GPU 设计：
成功 Gang 申请 2 × 3000 MiB = 6000 MiB
资源不足 Gang 申请 2 × 9000 MiB = 18000 MiB
Queue 上限为 6000 MiB
Queue 负例申请 9000 MiB，高于 Queue 上限，但低于空闲节点约 12288 MiB 的容量
安装 Volcano 通过官方 Helm Chart 安装：
1 2 3 4 5 helm repo add volcano-sh https://volcano-sh.github.io/helm-charts helm repo update helm install volcano volcano-sh/volcano -n volcano-system --create-namespace
开启 Volcano vGPU 调度 编辑 Volcano Scheduler 配置：
1 kubectl -n volcano-system edit configmap volcano-scheduler-configmap
在第二个 Scheduler Tier 中添加如下内容：
1 2 3 4 - name: deviceshare arguments: deviceshare.VGPUEnable: true deviceshare.SchedulePolicy: binpack
重启 Volcano Scheduler：
1 kubectl -n volcano-system rollout restart deployment volcano-scheduler
安装 Volcano Device Plugin 此处安装 v1.12.0 版本：
1 2 kubectl apply -f \ https://raw.githubusercontent.com/Project-HAMi/volcano-vgpu-device-plugin/v1.12.0/volcano-vgpu-device-plugin.yml
确认节点注册资源：
1 2 3 4 export NODE_NAME=gpu-0kubectl get node "${NODE_NAME} " \ -o custom-columns='NAME:.metadata.name,NUMBER:.status.allocatable.volcano\.sh/vgpu-number,MEMORY:.status.allocatable.volcano\.sh/vgpu-memory,CORES:.status.allocatable.volcano\.sh/vgpu-cores'
示例输出：
1 2 NAME NUMBER MEMORY CORES gpu-0 10 12288 100
使用场景 验证单个 vGPU Pod 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 kubectl create namespace volcano-demo cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: volcano-vgpu-single namespace: volcano-demo spec: schedulerName: volcano restartPolicy: Never containers: - name: cuda image: nvidia/cuda:11.0.3-base-ubuntu18.04 imagePullPolicy: IfNotPresent command: - bash - -lc - | echo "===== GPU ENV =====" env | grep -E 'CUDA_DEVICE|NVIDIA_VISIBLE_DEVICES|VGPU|VOLCANO' || true echo "===== NVIDIA SMI =====" nvidia-smi sleep 3600 resources: limits: volcano.sh/vgpu-number: 1 volcano.sh/vgpu-memory: 2000 volcano.sh/vgpu-cores: 30 EOF
示例输出：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 root@test-2:~# kubectl exec -n volcano-demo volcano-vgpu-single -- \ env | grep -E 'CUDA_DEVICE|NVIDIA_VISIBLE_DEVICES|VGPU|VOLCANO' NVIDIA_VISIBLE_DEVICES=GPU-eeab99c3-99a0-e956-3036-41d06990d593 CUDA_DEVICE_MEMORY_LIMIT_0=2000m CUDA_DEVICE_SM_LIMIT=30 CUDA_DEVICE_MEMORY_SHARED_CACHE=/tmp/vgpu/31191b22-212a-4358-a5d7-8ed49788b37e.cache root@test-2:~# kubectl exec -n volcano-demo logs volcano-vgpu-single Tue Jul 14 04:37:39 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| [HAMI-core Msg(81:140697490560832:libvgpu.c:870)]: Initializing..... | 0 NVIDIA GeForce RTX 3060 On | 00000000:03:00.0 Off | N/A | | 0% 33C P8 7W / 170W | 0MiB / 2000MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ [HAMI-core Msg(81:140697490560832:multiprocess_memory_limit.c:703)]: Cleanup on exit for PID 81 [HAMI-core Msg(81:140697490560832:multiprocess_memory_limit.c:739)]: Exit cleanup complete for PID 81
Volcano 中的 Gang Scheduling Volcano 中的 Gang Scheduling（成组调度）是一种要么一起调度，要么一个都不调度的策略。
它主要解决分布式计算任务的问题。例如一个训练任务需要 4 Worker，如果集群当前只能调度 2 个 Pod，普通调度器可能先启动这 2 个 Pod，而剩下的 Pod 处于 Pending 状态。这样任务不但无法完成工作，还会一直占用 GPU、CPU 和内存。
Gang Scheduling 可以要求至少满足指定数量后才开始调度：
这表示集群能同时容纳这 4 个 Pod 时，Volcano 才会调度它们；否则整个 Job 保持 Pending 状态。
运行双 Worker Gang 通过 VolcanoJob 创建两个 Worker。每个 Worker 申请一个 vGPU 份额、3000 MiB 显存和 30% 算力，因此完整 Gang 总共需要 6000 MiB 和 60% 算力：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 cat <<EOF | kubectl apply -f - apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: vcjob-vgpu-gang namespace: volcano-demo spec: schedulerName: volcano queue: default minAvailable: 2 maxRetry: 1 tasks: - name: vgpu-worker replicas: 2 template: spec: restartPolicy: Never containers: - name: cuda image: nvidia/cuda:11.0.3-base-ubuntu18.04 imagePullPolicy: IfNotPresent command: - bash - -lc - | echo "worker: $(hostname)" echo "===== GPU ENV =====" env | grep -E 'CUDA_DEVICE|NVIDIA_VISIBLE_DEVICES|VGPU|VOLCANO' || true echo "===== NVIDIA SMI =====" nvidia-smi sleep 3600 resources: limits: volcano.sh/vgpu-number: 1 volcano.sh/vgpu-memory: 3000 volcano.sh/vgpu-cores: 30 EOF
示例输出：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 root@test-2:~# kubectl -n volcano-demo logs vcjob-vgpu-gang-vgpu-worker-0 worker: test-2 ===== GPU ENV ===== mesg: ttyname failed: Inappropriate ioctl for device NVIDIA_VISIBLE_DEVICES=void CUDA_DEVICE_MEMORY_SHARED_CACHE=/tmp/vgpu/50f68904-2a85-4581-96ba-a04bb36e98cf.cache CUDA_DEVICE_MEMORY_LIMIT_0=3000m CUDA_DEVICE_SM_LIMIT=30 ===== NVIDIA SMI ===== [HAMI-core Msg(56:140268359665472:libvgpu.c:870)]: Initializing..... Tue Jul 14 07:43:02 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3060 On | 00000000:03:00.0 Off | N/A | | 0% 33C P8 8W / 170W | 0MiB / 3000MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ [HAMI-core Msg(56:140268359665472:multiprocess_memory_limit.c:703)]: Cleanup on exit for PID 56 [HAMI-core Msg(56:140268359665472:multiprocess_memory_limit.c:739)]: Exit cleanup complete for PID 56 root@test-2:~# kubectl -n volcano-demo logs vcjob-vgpu-gang-vgpu-worker-1 mesg: ttyname failed: Inappropriate ioctl for device worker: test-2 ===== GPU ENV ===== NVIDIA_VISIBLE_DEVICES=void CUDA_DEVICE_MEMORY_SHARED_CACHE=/tmp/vgpu/e66172f2-94b9-44dc-842a-1794ea2f1d52.cache CUDA_DEVICE_MEMORY_LIMIT_0=3000m CUDA_DEVICE_SM_LIMIT=30 ===== NVIDIA SMI ===== [HAMI-core Msg(57:140009740244800:libvgpu.c:870)]: Initializing..... Tue Jul 14 07:43:03 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 595.71.05 Driver Version: 595.71.05 CUDA Version: 13.2 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3060 On | 00000000:03:00.0 Off | N/A | | 0% 33C P8 8W / 170W | 0MiB / 3000MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ [HAMI-core Msg(57:140009740244800:multiprocess_memory_limit.c:703)]: Cleanup on exit for PID 57 [HAMI-core Msg(57:140009740244800:multiprocess_memory_limit.c:739)]: Exit cleanup complete for PID 57
资源不足的情况下运行双 Worker Gang 通过 VolcanoJob 创建两个 Worker。每个 Worker 申请一个 vGPU 份额、9000 MiB 显存和 30% 算力，因此完整 Gang 总共需要 18000 MiB 和 60% 算力：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 cat <<EOF | kubectl apply -f - apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: vcjob-vgpu-gang-insufficient namespace: volcano-demo spec: schedulerName: volcano queue: default minAvailable: 2 maxRetry: 1 tasks: - name: vgpu-worker replicas: 2 template: spec: restartPolicy: Never containers: - name: cuda image: nvidia/cuda:11.0.3-base-ubuntu18.04 imagePullPolicy: IfNotPresent command: ["bash", "-lc", "nvidia-smi; sleep 3600"] resources: limits: volcano.sh/vgpu-number: 1 volcano.sh/vgpu-memory: 9000 volcano.sh/vgpu-cores: 30 EOF
创建后，可以看到两个 Pod 都处于 Pending 状态：
1 2 3 4 root@test-2:~# kubectl -n volcano-demo get pod NAME READY STATUS RESTARTS AGE vcjob-vgpu-gang-insufficient-vgpu-worker-0 0/1 Pending 0 2m52s vcjob-vgpu-gang-insufficient-vgpu-worker-1 0/1 Pending 0 2m52s
检查 Volcano Job，会发现如下提示：
1 2 3 4 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning PodGroupPending 2m31s (x2 over 2m31s) vc-controller-manager PodGroup volcano-demo:vcjob-vgpu-gang-insufficient unschedulable, reason: 1/2 tasks in gang unschedulable: pod group is not ready, 2 Pending, 2 minAvailable; Pending: 1 Schedulable, 1 Unschedulable
单独来看集群的资源可以放置一个 Worker，但因为完整的双 Worker Gang 调度无法满足，所以没有只启动其中一部分。
Volcano 中的 Queue Volcano 中的 Queue（队列）可以理解为一组作业共享的资源池和调度边界。
它主要解决多租户、多团队或不同业务之间的资源分配问题，例如：
研发队列：development
生产队列：production
AI 训练队列：training
提交到不同 Queue 的 Job，会按照 Queue 的资源配额、权重和优先级参与调度。Volcano Job 未指定 Queue 时，通常会进入 default Queue。
创建 Queue 创建一个最多允许两个 vGPU 份额、6000 MiB 和 60% 算力的 Queue：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cat <<EOF | kubectl apply -f - apiVersion: scheduling.volcano.sh/v1beta1 kind: Queue metadata: name: gpu-small-queue spec: weight: 1 reclaimable: false capability: cpu: "2" memory: "4Gi" volcano.sh/vgpu-number: "2" volcano.sh/vgpu-memory: "6000" volcano.sh/vgpu-cores: "60" EOF
Queue 上限以内的 Job 创建双 Worker 的 Gang，总共需要两个 vGPU 份额、6000 Mib 显存和 60% 算力，并指定 Queue 为 gpu-small-queue：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 cat <<EOF | kubectl apply -f - apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: vcjob-vgpu-queue-fit namespace: volcano-demo spec: schedulerName: volcano queue: gpu-small-queue minAvailable: 2 tasks: - name: vgpu-worker replicas: 2 policies: - event: TaskCompleted action: CompleteJob template: spec: restartPolicy: Never containers: - name: cuda image: nvidia/cuda:11.0.3-base-ubuntu18.04 command: ["bash", "-lc", "nvidia-smi; sleep 60"] resources: limits: volcano.sh/vgpu-number: 1 volcano.sh/vgpu-memory: 3000 volcano.sh/vgpu-cores: 30 EOF
Pod 能够正常调度运行：
1 2 3 4 root@test-2:~# kubectl -n volcano-demo get pod NAME READY STATUS RESTARTS AGE vcjob-vgpu-queue-fit-vgpu-worker-0 1/1 Running 0 46s vcjob-vgpu-queue-fit-vgpu-worker-1 1/1 Running 0 46s
查看 Queue 资源的分配情况：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 spec: capability: cpu: "2" memory: 4Gi volcano.sh/vgpu-cores: "60" volcano.sh/vgpu-memory: "6000" volcano.sh/vgpu-number: "2" dequeueStrategy: traverse parent: root reclaimable: false weight: 1 status: allocated: cpu: "0" memory: "0" pods: "2" volcano.sh/vgpu-cores: "60" volcano.sh/vgpu-memory: 6k volcano.sh/vgpu-number: "2"
超过 Queue 上限的 Job 创建双 Worker 的 Gang，总共需要 2 个 vGPU 份额、9000 Mib 显存和 60% 算力，并指定 Queue 为 gpu-small-queue：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 cat <<EOF | kubectl apply -f - apiVersion: batch.volcano.sh/v1alpha1 kind: Job metadata: name: vcjob-vgpu-queue-exceeds namespace: volcano-demo spec: schedulerName: volcano queue: gpu-small-queue minAvailable: 2 tasks: - name: vgpu-worker replicas: 2 template: spec: restartPolicy: Never containers: - name: cuda image: nvidia/cuda:11.0.3-base-ubuntu18.04 command: ["bash", "-lc", "nvidia-smi; sleep 60"] resources: limits: volcano.sh/vgpu-number: 1 volcano.sh/vgpu-memory: 4500 volcano.sh/vgpu-cores: 30 EOF
尽管 GPU 节点有 12 GiB 显存，但由于 9000 MiB 超过 Queue 上限，Pod 无法正常调度：
1 2 3 4 root@test-2:~# kubectl -n volcano-demo get pod NAME READY STATUS RESTARTS AGE vcjob-vgpu-queue-exceeds-vgpu-worker-0 0/1 Pending 0 3s vcjob-vgpu-queue-exceeds-vgpu-worker-1 0/1 Pending 0 3s