Posted Updated AI5 minutes read (About 773 words)

KServe 使用实践

KServe 是一个开源的云原生模型服务平台，能够快速在 Kubernetes 上部署和运行机器学习模型，支持多种机器学习框架、具备弹性扩容能力。

前提条件

  • Kubernetes 集群 v1.32+，此处使用 RKE2 v1.35.7+rke2r1
  • 集群有 GPU 节点，且已安装 NVIDIA 内核驱动和 NVIDIA Container Toolkit，此处使用 NVIDIA GeForce RTX 3060
  • 集群已安装 GPU Operator
  • 集群已安装 HAMi

KServe 常用 CRD

  • InferenceService：部署和管理一个模型推理服务
  • ServingRuntime：定义 Namespace 级模型运行环境
  • ClusterServingRuntime：定义集群级共享 Runtime
  • InferenceGraph：编排多个模型和路由逻辑
  • TrainedModel：管理模型版本 / 多模型
  • ClusterStorageContainer：定义模型下载 / 存储初始化方式

KServe 部署

参考文档：https://kserve.github.io/website/docs/admin-guide/kubernetes-deployment

安装依赖项

安装 Cert Manager：

1
2
3
4
5
6
7
8
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.19.2 \
  --set crds.enabled=true

KServe 需要 Gateway API / Ingress，此处使用 RKE2 Traefik 的 Gateway API 能力。

对于自建 RKE2 集群：

1
2
3
4
5
6
7
8
9
10
11
12
cat <<EOF | kubectl apply -f -
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-traefik
  namespace: kube-system
spec:
  valuesContent: |-
    providers:
      kubernetesGateway:
        enabled: true
EOF

对于 Rancher 创建的 RKE2 集群，在通过 YAML 编辑集群中添加配置：

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
...
spec:
  kubernetesVersion: v1.35.7+rke2r1
  rkeConfig:
    chartValues:
      rke2-traefik:
        providers:
          kubernetesGateway:
            enabled: true
...

创建 Gateway：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kserve-ingress-gateway
  namespace: kserve
spec:
  gatewayClassName: traefik
  listeners:
    - name: http
      protocol: HTTP
      port: 8000
      allowedRoutes:
        namespaces:
          from: All
  infrastructure:
    labels:
      serving.kserve.io/gateway: kserve-ingress-gateway
EOF

安装 KServe CRD

1
2
3
4
helm upgrade --install kserve-crd oci://ghcr.io/kserve/charts/kserve-crd \
  --namespace kserve \
  --create-namespace \
  --version v0.20.0

安装 KServe

1
2
3
4
5
6
helm upgrade --install kserve oci://ghcr.io/kserve/charts/kserve-resources \
  --namespace kserve \
  --version v0.20.0 \
  --set kserve.controller.deploymentMode=Standard \
  --set kserve.controller.gateway.ingressGateway.enableGatewayApi=true \
  --set kserve.controller.gateway.ingressGateway.kserveGateway=kserve/kserve-ingress-gateway

验证是否安装完成：

1
2
kubectl get pods -n kserve
kubectl get crd | grep serving.kserve.io

准备存储

准备存放模型的数据目录，此处使用 GPU 节点的本地目录进行存储。

在 GPU 节点创建数据目录：

1
mkdir -pv /data/kserve-models

创建测试用的 Namespace、PV 和 PVC：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: kserve-lab
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: kserve-lab-models-gpu0
spec:
  capacity:
    storage: 20Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ''
  claimRef:
    namespace: kserve-lab
    name: kserve-models
  local:
    path: /data/kserve-models
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - gpu-0
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kserve-models
  namespace: kserve-lab
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: ''
  volumeName: kserve-lab-models-gpu0
  resources:
    requests:
      storage: 20Gi
EOF

模型下载

在刚刚创建好的目录，预先下载 Qwen2.5-1.5B-Instruct 和 Qwen3-Embedding-0.6B 两个模型：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: download-modelscope
  namespace: kserve-lab
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      nodeSelector:
        kubernetes.io/hostname: gpu-0
      automountServiceAccountToken: false
      containers:
      - name: download
        image: harbor.warnerchen.com/library/python:3.10.17-alpine
        command:
        - /bin/sh
        - -c
        args:
          - |
            set -eu

            python -m pip install --no-cache-dir 'modelscope[hub]==1.39.1'

            python -u - <<'PY'
            import hashlib
            import json
            import time
            from pathlib import Path

            from modelscope import snapshot_download


            models = [
                ("Qwen/Qwen2.5-1.5B-Instruct", "qwen25-1.5b"),
                ("Qwen/Qwen3-Embedding-0.6B", "qwen3-embedding-0.6b"),
            ]

            for model_id, folder in models:
                target = Path("/models") / folder
                marker = target / "DOWNLOAD_COMPLETE.json"

                if marker.exists():
                    print("Already downloaded:", model_id, flush=True)
                    continue

                snapshot_download(
                    model_id,
                    revision="master",
                    local_dir=str(target),
                )

                assert (target / "config.json").is_file(), (
                    f"Missing config: {target}"
                )

                weights = list(target.glob("*.safetensors"))
                assert weights, f"Missing safetensors: {target}"

                hashes = {}

                for path in sorted(target.rglob("*")):
                    if (
                        path.is_file()
                        and path.suffix in (".json", ".safetensors")
                    ):
                        digest = hashlib.sha256()

                        with path.open("rb") as stream:
                            for block in iter(
                                lambda: stream.read(8 * 1024 * 1024),
                                b"",
                            ):
                                digest.update(block)

                        hashes[str(path.relative_to(target))] = (
                            digest.hexdigest()
                        )

                marker.write_text(
                    json.dumps(
                        {
                            "model_id": model_id,
                            "requested_revision": "master",
                            "downloaded_at_unix": time.time(),
                            "sha256": hashes,
                        },
                        indent=2,
                    )
                )

                print("Finished:", model_id, flush=True)
            PY
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
          limits:
            cpu: '2'
            memory: 2Gi
        volumeMounts:
        - name: models
          mountPath: /models
      volumes:
      - name: models
        persistentVolumeClaim:
          claimName: kserve-models
EOF

使用实践

通过 InferenceService 部署 vLLM

KServe 负责服务生命周期，vLLM 负责模型推理，HAMi 负责 GPU 资源调度。

创建 InferenceService：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
cat <<EOF | kubectl apply -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen-chat
  namespace: kserve-lab
  annotations:
    serving.kserve.io/deploymentMode: Standard
    storage.kserve.io/readonly: 'true'
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 1
    deploymentStrategy:
      type: Recreate
    automountServiceAccountToken: false
    volumes:
    - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 1Gi
    - name: models
      persistentVolumeClaim:
        claimName: kserve-models
        readOnly: true
    containers:
    - name: kserve-container
      image: harbor.warnerchen.com/vllm/vllm-openai:v0.24.0
      imagePullPolicy: IfNotPresent
      command:
      - vllm
      - serve
      args:
      - /mnt/models
      - --host
      - 0.0.0.0
      - --port
      - '8080'
      - --served-model-name
      - qwen-chat
      - --dtype
      - half
      - --tensor-parallel-size
      - '1'
      - --enforce-eager
      - --gpu-memory-utilization
      - '0.80'
      - --max-model-len
      - '4096'
      - --max-num-seqs
      - '2'
      - --max-num-batched-tokens
      - '1024'
      - --enable-chunked-prefill
      - --generation-config
      - vllm
      env:
      - name: HF_HUB_OFFLINE
        value: '1'
      - name: TRANSFORMERS_OFFLINE
        value: '1'
      - name: HF_HUB_DISABLE_TELEMETRY
        value: '1'
      - name: VLLM_NO_USAGE_STATS
        value: '1'
      - name: OMP_NUM_THREADS
        value: '2'
      - name: GPU_CORE_UTILIZATION_POLICY
        value: force
      ports:
      - name: http1
        containerPort: 8080
      startupProbe:
        httpGet:
          path: /health
          port: 8080
        periodSeconds: 10
        timeoutSeconds: 5
        failureThreshold: 180
      readinessProbe:
        httpGet:
          path: /health
          port: 8080
        periodSeconds: 10
        timeoutSeconds: 5
        failureThreshold: 3
      volumeMounts:
      - name: dshm
        mountPath: /dev/shm
      - name: models
        mountPath: /mnt/models
        subPath: qwen25-1.5b
        readOnly: true
      resources:
        requests:
          cpu: '1'
          memory: 3Gi
          nvidia.com/gpu: '1'
          nvidia.com/gpumem: '8192'
          nvidia.com/gpucores: '100'
        limits:
          cpu: '4'
          memory: 7Gi
          nvidia.com/gpu: '1'
          nvidia.com/gpumem: '8192'
          nvidia.com/gpucores: '100'
EOF

创建后，会生产对应的 vLLM Deployment、Service 和 HTTPRoute：

验证 vLLM 是否可用：

1
2
3
4
5
6
7
8
9
10
11
12
13
curl -fsS http://<vllm-cluster-ip>/v1/models

curl http://<vllm-cluster-ip>/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen-chat",
    "messages": [
      {"role": "system", "content": "你是一名 AI 专家。"},
      {"role": "user", "content": "用一句话解释 KServe 是什么。"}
    ],
    "temperature": 0,
    "max_tokens": 256
  }'

通过 Gateway API 访问：

1
curl -fsS -H "Host: qwen-chat-kserve-lab.example.com" http://<node-ip>/v1/models

通过 ServingRuntime 定义模型运行环境

ServingRuntime 可以将镜像、启动命令、端口与探针集中管理，每个 InferenceService 只指定 Runtime、模型位置与资源。

此处测试：用一个名叫 vllm-chat 的 ServingRuntime 定义如何启动 vLLM，然后用 InferenceService 指定加载哪个模型、分配多少资源，最后由 KServe 生成实际的 Deployment / Service 等。

创建 ServingRuntime：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
cat <<EOF | kubectl apply -f -
apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
  name: vllm-chat
  namespace: kserve-lab
spec:
  multiModel: false
  supportedModelFormats:
  - name: huggingface
    # 强制服务明确选择 vllm-chat，避免同样叫 huggingface 的模型格式选错运行时
    autoSelect: false
  containers:
  - name: kserve-container
    image: harbor.warnerchen.com/vllm/vllm-openai:v0.24.0
    imagePullPolicy: IfNotPresent
    command:
    - vllm
    - serve
    args:
    - /mnt/models
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --served-model-name
    # 将自动替换为 ISVC 名
    - '{{.Name}}'
    - --dtype
    - half
    - --tensor-parallel-size
    - '1'
    - --enforce-eager
    - --gpu-memory-utilization
    - '0.80'
    - --max-model-len
    - '4096'
    - --max-num-seqs
    - '2'
    - --max-num-batched-tokens
    - '1024'
    - --enable-chunked-prefill
    - --generation-config
    - vllm
    env:
    - name: HF_HUB_OFFLINE
      value: '1'
    - name: TRANSFORMERS_OFFLINE
      value: '1'
    - name: HF_HUB_DISABLE_TELEMETRY
      value: '1'
    - name: VLLM_NO_USAGE_STATS
      value: '1'
    - name: OMP_NUM_THREADS
      value: '2'
    - name: GPU_CORE_UTILIZATION_POLICY
      value: force
    ports:
    - name: http1
      containerPort: 8080
    startupProbe:
      httpGet:
        path: /health
        port: 8080
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 180
    readinessProbe:
      httpGet:
        path: /health
        port: 8080
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 3
    volumeMounts:
    - name: dshm
      mountPath: /dev/shm
EOF

创建 InferenceService：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
cat <<EOF | kubectl apply -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen-chat
  namespace: kserve-lab
  annotations:
    serving.kserve.io/deploymentMode: Standard
    storage.kserve.io/readonly: 'true'
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 1
    deploymentStrategy:
      type: Recreate
    automountServiceAccountToken: false
    volumes:
    - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 1Gi
    model:
      # 选择 ServingRuntime
      modelFormat:
        name: huggingface
      runtime: vllm-chat
      # 直接挂载本地 PVC，不让 KServe 从 Hugging Face 再下载
      storageUri: pvc://kserve-models/qwen25-1.5b
      resources:
        requests:
          cpu: '1'
          memory: 3Gi
          nvidia.com/gpu: '1'
          nvidia.com/gpumem: '8192'
          nvidia.com/gpucores: '100'
        limits:
          cpu: '4'
          memory: 7Gi
          nvidia.com/gpu: '1'
          nvidia.com/gpumem: '8192'
          nvidia.com/gpucores: '100'
EOF

该 InferenceService 生成的 vLLM Deployment 会使用 ServingRuntime 中定义好的配置。

KServe 使用实践

https://warnerchen.github.io/2026/08/31/KServe-使用实践/

Author

Warner Chen

Posted on

2026-08-31

Updated on

2026-08-31

Licensed under

#
Follow

Links

Categories

Tags

about1
ai13
cloud native39
coding1
google1
linux12
network1
others2
redhat1
suse69
面试1

Recents

Archives

×