Posted Updated AI9 minutes read (About 1342 words)

SGLang 使用实践

本文记录在 RTX 3060 12 GiB 上部署和调优 SGLang 的过程，包括 Python 与 Docker 启动方式，以及通过 HAMi 在 Kubernetes 中共享单张 GPU。本文参数以功能验证和调优起点为主，不代表所有模型和负载下的最佳配置。

前提条件

  • GPU 节点已安装与 SGLang 兼容的 NVIDIA 驱动，可通过 nvidia-smi 确认 GPU 能被正常识别。
  • 本文使用 uv 创建 Python 虚拟环境和安装依赖，因此 Python 安装方式需要预先安装 uv：https://docs.astral.sh/uv/#installation
  • 使用容器方式运行 SGLang 时，GPU 节点需要安装 Docker 和 NVIDIA Container Toolkit，并确保容器能够访问 NVIDIA GPU。
  • 在 Kubernetes 环境部署时，需要运用到 GPU Operator 和 HAMi。

运行 SGLang

通过 Python 运行

安装 SGLang：

1
2
3
4
5
6
7
uv venv --python 3.12

source .venv/bin/activate

uv pip install \
  sglang \
  modelscope

由于 SGLang 默认使用 FlashInfer 作为 Attention Backend，所以需要安装 CUDA Toolkit：

1
2
3
4
5
6
7
8
9
10
11
python -c "import torch; print(torch.__version__, torch.version.cuda)"
2.11.0+cu130 13.0

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
apt install -y cuda-toolkit-13-0

export CUDA_HOME=/usr/local/cuda-13.0
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"

环境验证：

1
python -m sglang.check_env

示例输出：

启动 SGLang：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
SGLANG_USE_MODELSCOPE=true \
MODEL_CACHE=/data/modelscope \
CUDA_VISIBLE_DEVICES=0 \
python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-1.5B-Instruct \
  --served-model-name Qwen2.5-1.5B-Instruct \
  --download-dir "$MODEL_CACHE" \
  --host 0.0.0.0 \
  --port 30000 \
  --tp-size 1 \
  --dtype float16 \
  --mem-fraction-static 0.80 \
  --context-length 8192 \
  --max-running-requests 16 \
  --cuda-graph-backend-decode disabled \
  --cuda-graph-backend-prefill disabled

运行成功后，会监听 30000 端口，此时可以调用 OpenAI 兼容接口：

1
2
3
4
5
6
7
8
9
10
11
curl http://localhost:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [
      {"role": "system", "content": "你是一名 Linux 和 Kubernetes 助手。"},
      {"role": "user", "content": "解释一下 Kubernetes Service 的 ClusterIP。"}
    ],
    "temperature": 0,
    "max_tokens": 256
  }'

通过 Docker 运行

SGLang 官方提供了镜像，可以直接通过 Docker 运行：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
docker run --rm \
  --name sglang \
  --runtime nvidia \
  --gpus all \
  -p 30000:30000 \
  -v /data/modelscope:/models \
  -e SGLANG_USE_MODELSCOPE=true \
  -e MODELSCOPE_CACHE=/models \
  -e CUDA_VISIBLE_DEVICES=0 \
  lmsysorg/sglang:latest-cu129 \
  sglang serve \
    --model-path Qwen/Qwen2.5-1.5B-Instruct \
    --served-model-name Qwen2.5-1.5B-Instruct \
    --download-dir /models \
    --host 0.0.0.0 \
    --port 30000 \
    --tp-size 1 \
    --dtype float16 \
    --mem-fraction-static 0.80 \
    --context-length 8192 \
    --max-running-requests 16 \
    --cuda-graph-backend-decode disabled \
    --cuda-graph-backend-prefill disabled

使用 HAMi 在单卡上运行多个 SGLang 服务

当 GPU 节点已接入 Kubernetes 集群，且安装配置内核驱动 / NVIDIA Container Toolkit / GPU Operator / HAMi 后，即可通过 HAMi 部署多个 SGLang 实例。

如果使用 GPU Operator 管理驱动和容器工具链，应关闭 GPU Operator 自带的 NVIDIA Device Plugin，由 HAMi Device Plugin 负责上报和分配 GPU 资源。

本示例环境只有一个 GPU 节点，并且节点只有一张 RTX 3060，因此两个 Pod 会共享同一张物理 GPU。多节点或单节点多卡环境中，需要配置 HAMi GPU 调度策略，或通过 nvidia.com/use-gpuuuid 将两个 Pod 固定到同一张 GPU。

Helm Chart Values 配置

GPU Operator：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
toolkit:
  enabled: true
  env:
    - name: CONTAINERD_CONFIG
      value: /var/lib/rancher/rke2/agent/etc/containerd/config.toml
    - name: CONTAINERD_SOCKET
      value: /run/k3s/containerd/containerd.sock

# GPU 节点已通过 NVIDIA 官方 Local Repo 安装驱动，此处需要禁用 GPU Operator 安装驱动
driver:
  enabled: false

# 使用 HAMi Device Plugin，此处需要禁用 GPU Operator 自带的 Device Plugin
devicePlugin:
  enabled: false

HAMi：

1
2
3
4
5
6
7
8
9
devicePlugin:
  deviceListStrategy: cdi-annotations
  # 通过 NVIDIA 官方 Local Repo 安装驱动，需要显示配置 nvidiaDriverRoot 为 /，否则 HAMi Device Plugin 会报错 libcuda.so.x.y: not found
  nvidiaDriverRoot: /
  nvidiaHookPath: /usr/local/nvidia/toolkit/nvidia-ctk

scheduler:
  kubeScheduler:
    imageTag: v1.35.6

部署 SGLang

运行两个 SGLang Deployment，每个实例使用 1 个 vGPU、5800 MiB 显存、50% 算力：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-a
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sglang-a
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: sglang-a
        engine: sglang
    spec:
      containers:
      - args:
        - --model-path
        - Qwen/Qwen2.5-1.5B-Instruct
        - --served-model-name
        - sglang-a
        - --download-dir
        - /modelscope_cache
        - --host
        - 0.0.0.0
        - --port
        - "30000"
        - --dtype
        - float16
        - --context-length
        - "4096"
        - --max-running-requests
        - "4"
        - --chunked-prefill-size
        - "2048"
        - --mem-fraction-static
        - "0.70"
        - --attention-backend
        - triton
        - --sampling-backend
        - pytorch
        - --cuda-graph-backend-decode
        - disabled
        - --cuda-graph-backend-prefill
        - disabled
        command:
        - sglang
        - serve
        env:
        - name: SGLANG_USE_MODELSCOPE
          value: "true"
        - name: MODELSCOPE_CACHE
          value: /modelscope_cache
        image: harbor.warnerchen.com/lmsysorg/sglang:latest-cu129
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /health
            port: http
            scheme: HTTP
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 5
        name: sglang
        ports:
        - containerPort: 30000
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health_generate
            port: http
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "4"
            memory: 8Gi
            nvidia.com/gpu: "1"
            nvidia.com/gpucores: "50"
            nvidia.com/gpumem: "5800"
          requests:
            cpu: "2"
            memory: 4Gi
            nvidia.com/gpu: "1"
            nvidia.com/gpucores: "50"
            nvidia.com/gpumem: "5800"
        startupProbe:
          failureThreshold: 90
          httpGet:
            path: /health
            port: http
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        volumeMounts:
        - mountPath: /modelscope_cache
          name: modelscope-cache
        - mountPath: /dev/shm
          name: shm
      volumes:
      - hostPath:
          path: /data/modelscope
          type: DirectoryOrCreate
        name: modelscope-cache
      - emptyDir:
          medium: Memory
          sizeLimit: 1Gi
        name: shm
---
apiVersion: v1
kind: Service
metadata:
  name: sglang-a
spec:
  ports:
  - name: http
    port: 30000
    protocol: TCP
    targetPort: http
  selector:
    app: sglang-a
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sglang-b
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: sglang-b
        engine: sglang
    spec:
      containers:
      - args:
        - --model-path
        - Qwen/Qwen2.5-1.5B-Instruct
        - --served-model-name
        - sglang-b
        - --download-dir
        - /modelscope_cache
        - --host
        - 0.0.0.0
        - --port
        - "30000"
        - --dtype
        - float16
        - --context-length
        - "4096"
        - --max-running-requests
        - "4"
        - --chunked-prefill-size
        - "2048"
        - --mem-fraction-static
        - "0.70"
        - --attention-backend
        - triton
        - --sampling-backend
        - pytorch
        - --cuda-graph-backend-decode
        - disabled
        - --cuda-graph-backend-prefill
        - disabled
        command:
        - sglang
        - serve
        env:
        - name: SGLANG_USE_MODELSCOPE
          value: "true"
        - name: MODELSCOPE_CACHE
          value: /modelscope_cache
        image: harbor.warnerchen.com/lmsysorg/sglang:latest-cu129
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /health
            port: http
            scheme: HTTP
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 5
        name: sglang
        ports:
        - containerPort: 30000
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health_generate
            port: http
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "4"
            memory: 8Gi
            nvidia.com/gpu: "1"
            nvidia.com/gpucores: "50"
            nvidia.com/gpumem: "5800"
          requests:
            cpu: "2"
            memory: 4Gi
            nvidia.com/gpu: "1"
            nvidia.com/gpucores: "50"
            nvidia.com/gpumem: "5800"
        startupProbe:
          failureThreshold: 90
          httpGet:
            path: /health
            port: http
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        volumeMounts:
        - mountPath: /modelscope_cache
          name: modelscope-cache
        - mountPath: /dev/shm
          name: shm
      volumes:
      - hostPath:
          path: /data/modelscope
          type: DirectoryOrCreate
        name: modelscope-cache
      - emptyDir:
          medium: Memory
          sizeLimit: 1Gi
        name: shm
---
apiVersion: v1
kind: Service
metadata:
  name: sglang-b
spec:
  ports:
  - name: http
    port: 30000
    protocol: TCP
    targetPort: http
  selector:
    app: sglang-b
  type: ClusterIP
EOF

检查运行情况：

分别对两个 SGLang 实例发起请求：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
curl http://<sglang-a-cluster-ip>:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": "你是什么大模型"
      }
    ],
    "temperature": 0,
    "max_tokens": 256
  }'

curl http://<sglang-b-cluster-ip>:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": "你是什么大模型"
      }
    ],
    "temperature": 0,
    "max_tokens": 256
  }'

通过 nvidia-smi 可以看到承载了两个 SGLang 实例：

PD 分离

SGLang 同样支持 PD 分离，通过 HAMi 可以在软件层面上将一块 GPU 分成两块 vGPU，从而进行功能性验证。

使用 HAMi 本质上还是共享同一张物理 GPU，所以只能验证功能，不能证明性能收益。

请求链路：

1
2
3
4
                         ┌── Prefill 实例
客户端 ──> Model Gateway  ┤       │
                         │       └── KV Cache 传输
                         └── Decode 实例

Mooncake

由于 RTX 3060 没有 RDMA 和 NVLink，因此这里使用 MC_FORCE_TCP=True。

部署 Prefill 实例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-prefill
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: sglang-prefill
  template:
    metadata:
      labels:
        app: sglang-prefill
    spec:
      hostIPC: true
      terminationGracePeriodSeconds: 30
      containers:
        - name: sglang
          image: harbor.warnerchen.com/lmsysorg/sglang:latest-cu129
          imagePullPolicy: IfNotPresent
          command:
            - sglang
            - serve
          args:
            - --model-path
            - Qwen/Qwen2.5-1.5B-Instruct
            - --served-model-name
            - qwen-pd
            - --host
            - 0.0.0.0
            - --port
            - "30000"
            - --disaggregation-mode
            - prefill
            - --disaggregation-transfer-backend
            - mooncake
            - --disaggregation-bootstrap-port
            - "8998"
            - --dtype
            - float16
            - --context-length
            - "2048"
            - --mem-fraction-static
            - "0.65"
            - --chunked-prefill-size
            - "512"
            - --max-prefill-tokens
            - "1024"
            - --max-running-requests
            - "2"
            - --cuda-graph-backend-decode
            - disabled
            - --cuda-graph-backend-prefill
            - disabled
            - --disable-overlap-schedule
            - --attention-backend
            - triton
            - --sampling-backend
            - pytorch
          env:
            - name: SGLANG_USE_MODELSCOPE
              value: "true"
            - name: MODELSCOPE_CACHE
              value: /modelscope_cache
            - name: MC_FORCE_TCP
              value: "True"
            - name: MC_TE_METRIC
              value: "true"
            - name: MC_TCP_ENABLE_CONNECTION_POOL
              value: "1"
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - name: http
              containerPort: 30000
              protocol: TCP
            - name: bootstrap
              containerPort: 8998
              protocol: TCP
          resources:
            requests:
              cpu: "2"
              memory: 4Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "5800"
              nvidia.com/gpucores: "50"
            limits:
              cpu: "4"
              memory: 8Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "5800"
              nvidia.com/gpucores: "50"
          startupProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 120
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - mountPath: /modelscope_cache
              name: modelscope-cache
            - mountPath: /dev/shm
              name: shm
      volumes:
        - hostPath:
            path: /data/modelscope
            type: DirectoryOrCreate
          name: modelscope-cache
        - emptyDir:
            medium: Memory
            sizeLimit: 1Gi
          name: shm
---
apiVersion: v1
kind: Service
metadata:
  name: sglang-prefill
spec:
  type: ClusterIP
  selector:
    app: sglang-prefill
  ports:
    - name: http
      port: 30000
      targetPort: http
      protocol: TCP
    - name: bootstrap
      port: 8998
      targetPort: bootstrap
      protocol: TCP
EOF

部署 Decode 实例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-decode
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: sglang-decode
  template:
    metadata:
      labels:
        app: sglang-decode
    spec:
      hostIPC: true
      terminationGracePeriodSeconds: 30
      containers:
        - name: sglang
          image: harbor.warnerchen.com/lmsysorg/sglang:latest-cu129
          imagePullPolicy: IfNotPresent
          command:
            - sglang
            - serve
          args:
            - --model-path
            - Qwen/Qwen2.5-1.5B-Instruct
            - --served-model-name
            - qwen-pd
            - --host
            - 0.0.0.0
            - --port
            - "30000"
            - --disaggregation-mode
            - decode
            - --disaggregation-transfer-backend
            - mooncake
            - --dtype
            - float16
            - --context-length
            - "2048"
            - --mem-fraction-static
            - "0.65"
            - --max-running-requests
            - "2"
            - --cuda-graph-backend-decode
            - disabled
            - --cuda-graph-backend-prefill
            - disabled
            - --attention-backend
            - triton
            - --sampling-backend
            - pytorch
          env:
            - name: SGLANG_USE_MODELSCOPE
              value: "true"
            - name: MODELSCOPE_CACHE
              value: /modelscope_cache
            - name: MC_FORCE_TCP
              value: "True"
            - name: MC_TE_METRIC
              value: "true"
            - name: MC_TCP_ENABLE_CONNECTION_POOL
              value: "1"
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - name: http
              containerPort: 30000
              protocol: TCP
          resources:
            requests:
              cpu: "2"
              memory: 4Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "5800"
              nvidia.com/gpucores: "50"
            limits:
              cpu: "4"
              memory: 8Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "5800"
              nvidia.com/gpucores: "50"
          startupProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 120
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - mountPath: /modelscope_cache
              name: modelscope-cache
            - mountPath: /dev/shm
              name: shm
      volumes:
        - hostPath:
            path: /data/modelscope
            type: DirectoryOrCreate
          name: modelscope-cache
        - emptyDir:
            medium: Memory
            sizeLimit: 1Gi
          name: shm
---
apiVersion: v1
kind: Service
metadata:
  name: sglang-decode
spec:
  type: ClusterIP
  selector:
    app: sglang-decode
  ports:
    - name: http
      port: 30000
      targetPort: http
      protocol: TCP
EOF

确认两个 SGLang 实例运行的模型一致：

1
2
curl -s http://<sglang-prefill-cluster-ip>:30000/v1/models
curl -s http://<sglang-decode-cluster-ip>:30000/v1/models

部署 Model Gateway

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-model-gateway
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: sglang-model-gateway
  template:
    metadata:
      labels:
        app: sglang-model-gateway
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: router
          image: harbor.warnerchen.com/lmsysorg/sglang:latest-cu129
          imagePullPolicy: IfNotPresent
          command:
            - python3
            - -m
            - sglang_router.launch_router
          args:
            - --pd-disaggregation
            - --prefill
            - http://sglang-prefill:30000
            - "8998"
            - --decode
            - http://sglang-decode:30000
            - --host
            - 0.0.0.0
            - --port
            - "8000"
            - --worker-startup-timeout-secs
            - "1200"
          ports:
            - name: http
              containerPort: 8000
              protocol: TCP
          resources:
            requests:
              cpu: "1"
              memory: 512Mi
            limits:
              cpu: "2"
              memory: 2Gi
          readinessProbe:
            tcpSocket:
              port: http
            periodSeconds: 5
            failureThreshold: 6
---
apiVersion: v1
kind: Service
metadata:
  name: sglang-model-gateway
spec:
  type: ClusterIP
  selector:
    app: sglang-model-gateway
  ports:
    - name: http
      port: 8000
      targetPort: http
      protocol: TCP
EOF

检查 SGLang 实例是否注册成功：

1
curl -s http://<sglang-model-gateway-cluster-ip>:8000/workers | jq

验证

对 Model Gateway 发起请求：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
curl http://<sglang-model-gateway-cluster-ip>:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen-pd",
    "messages": [
      {
        "role": "system",
        "content": "你是一名 Kubernetes 专家。"
      },
      {
        "role": "user",
        "content": "Kubernetes 里 Service 的 ClusterIP 是什么？"
      }
    ],
    "temperature": 0,
    "max_tokens": 256
  }'

NIXL

删除 Mooncake 的实例：

1
kubectl delete deployment sglang-prefill sglang-decode sglang-model-gateway

部署 Prefill 实例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-prefill
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: sglang-prefill
  template:
    metadata:
      labels:
        app: sglang-prefill
    spec:
      hostIPC: true
      terminationGracePeriodSeconds: 30
      containers:
        - name: sglang
          image: harbor.warnerchen.com/lmsysorg/sglang:latest-cu129
          imagePullPolicy: IfNotPresent
          command:
            - sglang
            - serve
          args:
            - --model-path
            - Qwen/Qwen2.5-1.5B-Instruct
            - --served-model-name
            - qwen-pd
            - --host
            - 0.0.0.0
            - --port
            - "30000"
            - --disaggregation-mode
            - prefill
            - --disaggregation-transfer-backend
            - nixl
            - --disaggregation-bootstrap-port
            - "8998"
            - --dtype
            - float16
            - --context-length
            - "2048"
            - --mem-fraction-static
            - "0.65"
            - --chunked-prefill-size
            - "512"
            - --max-prefill-tokens
            - "1024"
            - --max-running-requests
            - "2"
            - --cuda-graph-backend-decode
            - disabled
            - --cuda-graph-backend-prefill
            - disabled
            - --disable-overlap-schedule
            - --attention-backend
            - triton
            - --sampling-backend
            - pytorch
          env:
            - name: SGLANG_USE_MODELSCOPE
              value: "true"
            - name: MODELSCOPE_CACHE
              value: /modelscope_cache
            - name: SGLANG_DISAGGREGATION_NIXL_BACKEND
              value: UCX
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - name: http
              containerPort: 30000
              protocol: TCP
            - name: bootstrap
              containerPort: 8998
              protocol: TCP
          resources:
            requests:
              cpu: "2"
              memory: 4Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "5800"
              nvidia.com/gpucores: "50"
            limits:
              cpu: "4"
              memory: 8Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "5800"
              nvidia.com/gpucores: "50"
          startupProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 120
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - mountPath: /modelscope_cache
              name: modelscope-cache
            - mountPath: /dev/shm
              name: shm
      volumes:
        - hostPath:
            path: /data/modelscope
            type: DirectoryOrCreate
          name: modelscope-cache
        - emptyDir:
            medium: Memory
            sizeLimit: 1Gi
          name: shm
---
apiVersion: v1
kind: Service
metadata:
  name: sglang-prefill
spec:
  type: ClusterIP
  selector:
    app: sglang-prefill
  ports:
    - name: http
      port: 30000
      targetPort: http
      protocol: TCP
    - name: bootstrap
      port: 8998
      targetPort: bootstrap
      protocol: TCP
EOF

部署 Decode 实例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-decode
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: sglang-decode
  template:
    metadata:
      labels:
        app: sglang-decode
    spec:
      hostIPC: true
      terminationGracePeriodSeconds: 30
      containers:
        - name: sglang
          image: harbor.warnerchen.com/lmsysorg/sglang:latest-cu129
          imagePullPolicy: IfNotPresent
          command:
            - sglang
            - serve
          args:
            - --model-path
            - Qwen/Qwen2.5-1.5B-Instruct
            - --served-model-name
            - qwen-pd
            - --host
            - 0.0.0.0
            - --port
            - "30000"
            - --disaggregation-mode
            - decode
            - --disaggregation-transfer-backend
            - nixl
            - --dtype
            - float16
            - --context-length
            - "2048"
            - --mem-fraction-static
            - "0.65"
            - --max-running-requests
            - "2"
            - --cuda-graph-backend-decode
            - disabled
            - --cuda-graph-backend-prefill
            - disabled
            - --attention-backend
            - triton
            - --sampling-backend
            - pytorch
          env:
            - name: SGLANG_USE_MODELSCOPE
              value: "true"
            - name: MODELSCOPE_CACHE
              value: /modelscope_cache
            - name: SGLANG_DISAGGREGATION_NIXL_BACKEND
              value: UCX
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - name: http
              containerPort: 30000
              protocol: TCP
          resources:
            requests:
              cpu: "2"
              memory: 4Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "5800"
              nvidia.com/gpucores: "50"
            limits:
              cpu: "4"
              memory: 8Gi
              nvidia.com/gpu: "1"
              nvidia.com/gpumem: "5800"
              nvidia.com/gpucores: "50"
          startupProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 120
          readinessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          volumeMounts:
            - mountPath: /modelscope_cache
              name: modelscope-cache
            - mountPath: /dev/shm
              name: shm
      volumes:
        - hostPath:
            path: /data/modelscope
            type: DirectoryOrCreate
          name: modelscope-cache
        - emptyDir:
            medium: Memory
            sizeLimit: 1Gi
          name: shm
---
apiVersion: v1
kind: Service
metadata:
  name: sglang-decode
spec:
  type: ClusterIP
  selector:
    app: sglang-decode
  ports:
    - name: http
      port: 30000
      targetPort: http
      protocol: TCP
EOF

确认两个 SGLang 实例运行的模型一致：

1
2
curl -s http://<sglang-prefill-cluster-ip>:30000/v1/models
curl -s http://<sglang-decode-cluster-ip>:30000/v1/models

确认当前使用 NIXL：

1
2
3
4
5
6
7
kubectl logs deployment/sglang-prefill \
  | grep -oE "disaggregation_transfer_backend='(nixl|mooncake)'" \
  | head -1

kubectl logs deployment/sglang-decode \
  | grep -oE "disaggregation_transfer_backend='(nixl|mooncake)'" \
  | head -1

部署 Model Gateway

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-model-gateway
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: sglang-model-gateway
  template:
    metadata:
      labels:
        app: sglang-model-gateway
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: router
          image: harbor.warnerchen.com/lmsysorg/sglang:latest-cu129
          imagePullPolicy: IfNotPresent
          command:
            - python3
            - -m
            - sglang_router.launch_router
          args:
            - --pd-disaggregation
            - --prefill
            - http://sglang-prefill:30000
            - "8998"
            - --decode
            - http://sglang-decode:30000
            - --host
            - 0.0.0.0
            - --port
            - "8000"
            - --worker-startup-timeout-secs
            - "1200"
          ports:
            - name: http
              containerPort: 8000
              protocol: TCP
          resources:
            requests:
              cpu: "1"
              memory: 512Mi
            limits:
              cpu: "2"
              memory: 2Gi
          startupProbe:
            httpGet:
              path: /liveness
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 60
          readinessProbe:
            httpGet:
              path: /readiness
              port: http
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 6
          livenessProbe:
            httpGet:
              path: /liveness
              port: http
            periodSeconds: 15
            timeoutSeconds: 3
            failureThreshold: 5
---
apiVersion: v1
kind: Service
metadata:
  name: sglang-model-gateway
spec:
  type: ClusterIP
  selector:
    app: sglang-model-gateway
  ports:
    - name: http
      port: 8000
      targetPort: http
      protocol: TCP
EOF

检查 SGLang 实例是否注册成功：

1
curl -s http://<sglang-model-gateway-cluster-ip>:8000/workers | jq

验证

对 Model Gateway 发起请求：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
curl http://<sglang-model-gateway-cluster-ip>:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen-pd",
    "messages": [
      {
        "role": "system",
        "content": "你是一名 Kubernetes 专家。"
      },
      {
        "role": "user",
        "content": "Kubernetes 里 Service 的 NodePort 是什么？"
      }
    ],
    "temperature": 0,
    "max_tokens": 256
  }'

SGLang 使用实践

https://warnerchen.github.io/2026/08/13/SGLang-使用实践/

Author

Warner Chen

Posted on

2026-08-13

Updated on

2026-08-14

Licensed under

#
Follow

Links

Categories

Tags

about1
ai12
cloud native39
coding1
google1
linux12
network1
others2
redhat1
suse69
面试1

Recents

Archives

×