使用 FRR 测试 Calico BGP 模式

Calico 可以利用 BGP 实现节点间三层互通，而无需任何封装。因为没有了报文封装和解封的过程，所以在这种模式下网络性能最好，但需底层物理网络支持 BGP。

如果底层物理网络不支持 BGP，可以通过 FRR 工具模拟 BGP Router，使其能够使用 BGP 模式。

关于 Calico 详细解说可以参考：Calico 的三种工作模式

安装配置 FRR

准备一台 VM 安装 FRR：

apt -y install frr

开启 BGPD：

sed -i s#bgpd=no#bgpd=yes#g /etc/frr/daemons
systemctl restart frr

检查 BGPD 是否开启成功：

# 此处还没有配置 BGP，所以显示 not found
root@test-0:~# vtysh -c "show ip bgp summary"
% BGP instance not found

安装后，进行 BGP 配置：

# 进入控制台
root@test-0:~# vtysh

Hello, this is FRRouting (version 8.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

test-0# configure terminal # 配置模式
test-0(config)# router bgp 64512 # 设置 BGP Router 的 BGP AS Number
test-0(config-router)# bgp router-id 172.16.16.140 # BGP Router 的接口 IP
test-0(config-router)# no bgp ebgp-requires-policy # 关闭 EBPF Filter
test-0(config-router)# neighbor 172.16.16.142 remote-as 64512 # 配置邻居，此处为集群中的第一台节点
test-0(config-router)# neighbor 172.16.16.142 description controlplane1 # 配置描述
test-0(config-router)# neighbor 172.16.16.143 remote-as 64512 # 同上，配置集群中的第二台节点
test-0(config-router)# neighbor 172.16.16.143 description woker1
test-0(config-router)# exit
test-0(config)# exit
test-0# exit

检查配置情况，此处可以看到 State/PfxRcd 为 Active，说明已经准备好建立 BGP 连接：

root@test-0:~# vtysh -c "show ip bgp summary"

IPv4 Unicast Summary (VRF default):
BGP router identifier 172.16.16.140, local AS number 64512 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 2, using 1446 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
172.16.16.142   4      64512         0         1        0    0    0    never       Active        0 controlplane1
172.16.16.143   4      64512         0         1        0    0    0    never       Active        0 woker1

Total number of neighbors 2

配置 Calico BGP 模式

此处集群使用 RKE2，配置方式参考：https://warnerchen.github.io/2025/05/28/Calico-%E7%9A%84%E4%B8%89%E7%A7%8D%E5%B7%A5%E4%BD%9C%E6%A8%A1%E5%BC%8F/#%E8%B7%A8%E8%8A%82%E7%82%B9-Pod-%E9%80%9A%E4%BF%A1%EF%BC%88BPG-%E6%A8%A1%E5%BC%8F%EF%BC%89

配置为 BGP 模式后，创建 BGPConfiguration 和 BGPPeer：

cat <<EOF | kubectl apply -f -
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: true
  # Calico 使用的 BGP AS Number
  asNumber: 64512

---
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: peer-to-frr
spec:
  # 对应 BGP Router 的 FRR 配置信息
  asNumber: 64512
  peerIP: 172.16.16.140
EOF

创建完成后，通过 calicoctl 查看会话建立状态，global 为 Established 代表建立成功：

root@docker-test-0:~# calicoctl node status
Calico process is running.

IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+---------------+-------------------+-------+----------+-------------+
| 172.16.16.143 | node-to-node mesh | up    | 08:34:36 | Established |
| 172.16.16.140 | global            | up    | 02:36:02 | Established |
+---------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

root@docker-test-1:~# calicoctl node status
Calico process is running.

IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+---------------+-------------------+-------+----------+-------------+
| 172.16.16.142 | node-to-node mesh | up    | 08:34:36 | Established |
| 172.16.16.140 | global            | up    | 02:36:03 | Established |
+---------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

在 BGP Router 查看会话建立状态，此处可以看到 State/PfxRcd 为 1，说明会话建立成功：

root@test-0:~# vtysh -c "show ip bgp summary"

IPv4 Unicast Summary (VRF default):
BGP router identifier 172.16.16.140, local AS number 64512 vrf-id 0
BGP table version 2
RIB entries 3, using 552 bytes of memory
Peers 2, using 1446 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
172.16.16.142   4      64512        10        10        0    0    0 00:05:50            1        0 controlplane1
172.16.16.143   4      64512        10        10        0    0    0 00:05:50            1        0 woker1

Total number of neighbors 2

在 BGP Router 查看 FRR 路由表信息，可以看到存在两个 B 相关的路由，分别对应集群中两个节点的容器网段：

root@test-0:~# vtysh -c "show ip route"
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

K>* 0.0.0.0/0 [0/0] via 172.16.16.1, ens34, 00:10:16
B>* 10.42.5.192/26 [200/0] via 172.16.16.143, ens34, weight 1, 00:06:09
B>* 10.42.22.64/26 [200/0] via 172.16.16.142, ens34, weight 1, 00:06:09
C>* 172.16.16.0/24 is directly connected, ens34, 00:10:16

在 BGP Router 查看节点路由信息，同样可以看到两个容器网段的路由信息，是直接路由到对端节点的物理网络的：

root@test-0:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.16.1     0.0.0.0         UG    0      0        0 ens34
10.42.5.192     172.16.16.143   255.255.255.192 UG    20     0        0 ens34
10.42.22.64     172.16.16.142   255.255.255.192 UG    20     0        0 ens34
172.16.16.0     0.0.0.0         255.255.255.0   U     0      0        0 ens34

此时 BGP Router 可以直接与容器网段进行通信：

# Pod IP 信息
root@docker-test-0:~# kubectl get pod -owide
NAME                      READY   STATUS    RESTARTS   AGE   IP             NODE            NOMINATED NODE   READINESS GATES
busybox-f96f87d5d-bhd5b   1/1     Running   0          28m   10.42.5.213    docker-test-1   <none>           <none>
busybox-f96f87d5d-k5d22   1/1     Running   0          28m   10.42.22.100   docker-test-0   <none>           <none>

# BGP Router 进行连通信测试
root@test-0:~# ping -c 3 10.42.5.213
PING 10.42.5.213 (10.42.5.213) 56(84) bytes of data.
64 bytes from 10.42.5.213: icmp_seq=1 ttl=63 time=0.305 ms
64 bytes from 10.42.5.213: icmp_seq=2 ttl=63 time=0.252 ms
64 bytes from 10.42.5.213: icmp_seq=3 ttl=63 time=0.226 ms

--- 10.42.5.213 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2055ms
rtt min/avg/max/mdev = 0.226/0.261/0.305/0.032 ms

root@test-0:~# ping -c 3 10.42.22.100
PING 10.42.22.100 (10.42.22.100) 56(84) bytes of data.
64 bytes from 10.42.22.100: icmp_seq=1 ttl=63 time=0.200 ms
64 bytes from 10.42.22.100: icmp_seq=2 ttl=63 time=0.222 ms
64 bytes from 10.42.22.100: icmp_seq=3 ttl=63 time=0.185 ms

--- 10.42.22.100 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2027ms
rtt min/avg/max/mdev = 0.185/0.202/0.222/0.015 ms

测试集群中跨节点通信：

root@docker-test-0:~# kubectl exec -it busybox-f96f87d5d-k5d22 -- ping -c 3 10.42.5.213
PING 10.42.5.213 (10.42.5.213): 56 data bytes
64 bytes from 10.42.5.213: seq=0 ttl=62 time=1.368 ms
64 bytes from 10.42.5.213: seq=1 ttl=62 time=0.352 ms
64 bytes from 10.42.5.213: seq=2 ttl=62 time=0.313 ms

--- 10.42.5.213 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.313/0.677/1.368 ms

root@docker-test-0:~# kubectl exec -it busybox-f96f87d5d-bhd5b -- ping -c 3 10.42.22.100
PING 10.42.22.100 (10.42.22.100): 56 data bytes
64 bytes from 10.42.22.100: seq=0 ttl=62 time=0.721 ms
64 bytes from 10.42.22.100: seq=1 ttl=62 time=0.286 ms
64 bytes from 10.42.22.100: seq=2 ttl=62 time=0.307 ms

--- 10.42.22.100 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.286/0.438/0.721 ms