CCE One集群聯邦支持通過PropagationPolicy定制資源模板分發策略,以及支持通過OverridePolicy來定制資源模板分發到不同成員集群的差異化配置。
這里,我們以分發一個Nginx Deployment Web應用為例,簡要介紹如何實現應用的多集群分發。
前提條件
-
已開通CCE One集群聯邦實例,可按需給apiserver綁定公網EIP以暴露到公網訪問;
-
通過集群聯邦接入配置,獲取到聯邦apiserver訪問的KubeConfig文件;
-
聯邦中有添加成員集群,且成員集群狀態為Ready(代表已聯通);
通過kubectl命令查詢聯邦控制面中成員集群列表及狀態,可參考如下命令:
# 將如下‘/home/ctyun/.kube/cceone-fed.yaml’替換為您本地的聯邦KubeConfig文件路徑
# 集群狀態Ready=True代表已正常接入,可進行應用分發
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get cluster
NAME VERSION MODE READY AGE
cce-cceone-demo1 v1.27.8 Push True 16d
cce-cceone-demo2 v1.22.1 Push True 56d
cceone-c2 v1.22.1 Push True 55d
cceone-cluster1 v1.22.1 Push True 55d
icce-yhn-296 Push False 16d
操作指引
該操作指引以分發一個nginx deployment到成員集群為例,若需分發其他資源(包括CR),只需將nginx deployment替換為具體的資源模板即可。
需要注意的是:
-
資源模板和PropagationPolicy之間可任意創建順序不分先后,當二者匹配后即立即觸發調度分發邏輯;
-
差異化策略需要在觸發調度分發邏輯之前創建好,也即差異化策略不能是最后一個;
如下為詳細操作參考:
步驟一: 創建資源模板
執行如下命令,在聯邦控制面中創建資源模板:
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml create deployment nginx --image nginx --replicas=3
deployment.apps/nginx created
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 0/3 0 0 21s
此時,只是將K8S YAML apply到聯邦控制面,還未分發到成員集群;聯邦控制面當前也并不會調諧這些workload,也即實際看到的READY副本數會為0;
步驟二:創建差異化策略
差異化策略(OverridePolicy)模板參考如下。若不需要做集群間差異化,則可以略過該步驟。
# overridepolicy.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: OverridePolicy
metadata:
name: example
spec:
resourceSelectors:
- apiVersion: apps/v1
kind: Deployment
name: nginx # If no namespace is specified, the namespace is inherited from the parent object scope.
overrideRules:
- targetCluster:
clusterNames:
- cce-cceone-demo1
overriders:
labelsOverrider:
- operator: add
value:
cceone.propagationto: cce-cceone-demo1
- targetCluster:
clusterNames:
- cce-cceone-demo2
overriders:
labelsOverrider:
- operator: add
value:
cceone.propagationto: cce-cceone-demo2
將如上YAML文件通過如下命令apply到聯邦控制面中:
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml apply -f overridepolicy.yaml
overridepolicy.policy.karmada.io/example created
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get overridepolicy example
NAME AGE
example 21s
此時,通過命令查看nginx應用資源模板,還是未分發狀態
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 0/3 0 0 61m
步驟三:創建調度策略
調度策略(PropagationPolicy)模板參考如下,可基于實際需要調整其中的污點容忍及副本拆分策略等:
# propagationpolicy.yaml
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
name: example-policy # The default namespace is `default`.
spec:
propagateDeps: true
resourceSelectors:
- apiVersion: apps/v1
kind: Deployment
name: nginx # If no namespace is specified, the namespace is inherited from the parent object scope.
placement:
clusterAffinity: # 指定目標集群
clusterNames:
- cce-cceone-demo1
- cce-cceone-demo2
clusterTolerations: # 容忍成員集群污點,影響調度邏輯
- key: cluster.karmada.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 300
- key: cluster.karmada.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 300
replicaScheduling: # 指定應用副本在成員集群間的分發策略
replicaSchedulingType: Divided
replicaDivisionPreference: Weighted
weightPreference:
staticWeightList:
- targetCluster:
clusterNames:
- cce-cceone-demo1
weight: 1
- targetCluster:
clusterNames:
- cce-cceone-demo2
weight: 2
通過如下命令,將調度策略apply到聯邦控制面中
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml apply -f propagationpolicy.yaml
propagationpolicy.policy.karmada.io/example-policy created
ctyun@0000000g-FxXsKxFWUv:~/tmp$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get propagationpolicy example-policy
NAME CONFLICT-RESOLUTION PRIORITY AGE
example-policy Abort 0 13s
此時,通過如下命令,查看應用在多集群之間的調度情況:
# 查看應用成員Pod Ready情況
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get deployment nginx
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 3/3 3 3 10s
# 查詢應用多集群調度結果
$ kubectl --kubeconfig=/home/ctyun/.kube/cceone-fed.yaml get resourcebinding nginx-deployment -o yaml
apiVersion: work.karmada.io/v1alpha2
kind: ResourceBinding
metadata:
annotations:
policy.karmada.io/applied-placement: '{"clusterAffinity":{"clusterNames":["cce-cceone-demo1","cce-cceone-demo2"]},"clusterTolerations":[{"key":"cluster.karmada.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"cluster.karmada.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"replicaScheduling":{"replicaSchedulingType":"Divided","replicaDivisionPreference":"Weighted","weightPreference":{"staticWeightList":[{"targetCluster":{"clusterNames":["cce-cceone-demo1"]},"weight":1},{"targetCluster":{"clusterNames":["cce-cceone-demo2"]},"weight":2}]}}}'
propagationpolicy.karmada.io/name: example-policy
propagationpolicy.karmada.io/namespace: default
resourcebinding.karmada.io/dependencies: "null"
creationTimestamp: "2025-05-30T09:52:51Z"
finalizers:
- karmada.io/binding-controller
- karmada.io/binding-dependencies-distributor
generation: 2
labels:
propagationpolicy.karmada.io/permanent-id: bb4b469a-6c93-4ed7-8465-e91564eb33eb
resourcebinding.karmada.io/permanent-id: 89904db8-95a4-4820-b3b7-cea95b1c3e51
name: nginx-deployment
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: Deployment
name: nginx
uid: 86da5387-b36b-40c4-99a2-462cbc832c99
resourceVersion: "44338541"
uid: 90b31326-7a74-405e-81e3-98ac18969d9f
spec:
clusters:
- name: cce-cceone-demo2
replicas: 2
- name: cce-cceone-demo1
replicas: 1
conflictResolution: Abort
placement:
clusterAffinity:
clusterNames:
- cce-cceone-demo1
- cce-cceone-demo2
clusterTolerations:
- effect: NoExecute
key: cluster.karmada.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: cluster.karmada.io/unreachable
operator: Exists
tolerationSeconds: 300
replicaScheduling:
replicaDivisionPreference: Weighted
replicaSchedulingType: Divided
weightPreference:
staticWeightList:
- targetCluster:
clusterNames:
- cce-cceone-demo1
weight: 1
- targetCluster:
clusterNames:
- cce-cceone-demo2
weight: 2
propagateDeps: true
replicas: 3
resource:
apiVersion: apps/v1
kind: Deployment
name: nginx
namespace: default
resourceVersion: "44338446"
uid: 86da5387-b36b-40c4-99a2-462cbc832c99
schedulerName: default-scheduler
status:
aggregatedStatus: # 此處為聚合了成員集群中該Deployment對應副本Pod的部署狀態
- applied: true
clusterName: cce-cceone-demo1
health: Healthy
status:
availableReplicas: 1
generation: 1
observedGeneration: 1
readyReplicas: 1
replicas: 1
resourceTemplateGeneration: 2
updatedReplicas: 1
- applied: true
clusterName: cce-cceone-demo2
health: Healthy
status:
availableReplicas: 2
generation: 1
observedGeneration: 1
readyReplicas: 2
replicas: 2
resourceTemplateGeneration: 2
updatedReplicas: 2
conditions:
- lastTransitionTime: "2025-05-30T09:52:51Z"
message: Binding has been scheduled successfully.
reason: Success
status: "True"
type: Scheduled
- lastTransitionTime: "2025-05-30T09:52:51Z"
message: All works have been successfully applied
reason: FullyAppliedSuccess
status: "True"
type: FullyApplied
lastScheduledTime: "2025-05-30T09:52:51Z"
schedulerObservedGeneration: 2