[NCP] Cluster creation fails due to cross-AZ subnets in k8sClusterDynamic #2136
Description
What happened
: While creating an NCP K8s cluster via CB-Tumblebug using k8sClusterDynamic API, the request failed with the following error message and related logs:
cb-tumblebug | 6:50AM DBG src/core/resource/spec.go:2259 > [Get spec] ncp+kr+ci2-g3
cb-tumblebug | 6:50AM DBG src/core/resource/spec.go:2259 > [Get spec] ncp+kr+ci2-g3
cb-tumblebug | 6:50AM DBG src/core/resource/spec.go:2259 > [Get spec] ncp+kr+ci2-g3
cb-tumblebug | 6:50AM DBG src/core/resource/spec.go:2259 > [Get spec] ncp+kr+ci2-g3
cb-tumblebug | 6:50AM INF src/core/infra/provisioning.go:3248 > Found and utilize default vNet: default-shared-ncp-kr
cb-tumblebug | 6:50AM INF src/core/infra/provisioning.go:3248 > Found and utilize default vNet: default-shared-ncp-kr
cb-tumblebug | 6:50AM INF src/core/infra/provisioning.go:3274 > Found and utilize default SSHKey: default-shared-ncp-kr
cb-tumblebug | 6:50AM INF src/core/infra/provisioning.go:3299 > Found and utilize default securityGroup: default-shared-ncp-kr
cb-tumblebug | {
cb-tumblebug | "connectionName": "ncp-kr",
cb-tumblebug | "description": "",
cb-tumblebug | "name": "k8scluster01",
cb-tumblebug | "version": "1.32.6-nks.1",
cb-tumblebug | "vNetId": "default-shared-ncp-kr",
cb-tumblebug | "subnetIds": [
cb-tumblebug | "default-shared-ncp-kr",
cb-tumblebug | "default-shared-ncp-kr-01"
cb-tumblebug | ],
cb-tumblebug | "securityGroupIds": [
cb-tumblebug | "default-shared-ncp-kr"
cb-tumblebug | ],
cb-tumblebug | "k8sNodeGroupList": [
cb-tumblebug | {
cb-tumblebug | "name": "k8sng01",
cb-tumblebug | "imageId": "",
cb-tumblebug | "specId": "ncp+kr+ci2-g3",
cb-tumblebug | "rootDiskType": "",
cb-tumblebug | "rootDiskSize": "",
cb-tumblebug | "sshKeyId": "default-shared-ncp-kr",
cb-tumblebug | "onAutoScaling": "true",
cb-tumblebug | "desiredNodeSize": "1",
cb-tumblebug | "minNodeSize": "1",
cb-tumblebug | "maxNodeSize": "2",
cb-tumblebug | "label": null,
cb-tumblebug | "description": ""
cb-tumblebug | }
cb-tumblebug | ],
cb-tumblebug | "cspResourceId": "",
cb-tumblebug | "label": null,
cb-tumblebug | "systemLabel": ""
cb-tumblebug | }
cb-tumblebug | {
cb-tumblebug | "connectionName": "ncp-kr",
cb-tumblebug | "description": "",
cb-tumblebug | "name": "k8scluster01",
cb-tumblebug | "version": "1.32.6-nks.1",
cb-tumblebug | "vNetId": "default-shared-ncp-kr",
cb-tumblebug | "subnetIds": [
cb-tumblebug | "default-shared-ncp-kr",
cb-tumblebug | "default-shared-ncp-kr-01"
cb-tumblebug | ],
cb-tumblebug | "securityGroupIds": [
cb-tumblebug | "default-shared-ncp-kr"
cb-tumblebug | ],
cb-tumblebug | "k8sNodeGroupList": [
cb-tumblebug | {
cb-tumblebug | "name": "k8sng01",
cb-tumblebug | "imageId": "",
cb-tumblebug | "specId": "ncp+kr+ci2-g3",
cb-tumblebug | "rootDiskType": "",
cb-tumblebug | "rootDiskSize": "",
cb-tumblebug | "sshKeyId": "default-shared-ncp-kr",
cb-tumblebug | "onAutoScaling": "true",
cb-tumblebug | "desiredNodeSize": "1",
cb-tumblebug | "minNodeSize": "1",
cb-tumblebug | "maxNodeSize": "2",
cb-tumblebug | "label": null,
cb-tumblebug | "description": ""
cb-tumblebug | }
cb-tumblebug | ],
cb-tumblebug | "cspResourceId": "",
cb-tumblebug | "label": null,
cb-tumblebug | "systemLabel": ""
cb-tumblebug | }
cb-tumblebug | 6:50AM INF src/core/resource/k8scluster.go:201 > CreateK8sCluster
cb-tumblebug | 6:50AM ERR src/core/resource/k8scluster.go:1417 > reflect: call of reflect.Value.Interface on zero Value
cb-tumblebug | 6:50AM DBG src/core/resource/k8scluster.go:102 > [Create K8sClusterInfo] k8scluster01
cb-tumblebug | 6:50AM DBG src/core/resource/spec.go:2259 > [Get spec] ncp+kr+ci2-g3
cb-tumblebug | 6:50AM INF src/core/infra/provisioning.go:3299 > Found and utilize default securityGroup: default-shared-ncp-kr
cb-tumblebug | {
cb-tumblebug | "connectionName": "ncp-kr",
cb-tumblebug | "description": "",
cb-tumblebug | "name": "k8scluster01",
cb-tumblebug | "version": "1.32.6-nks.1",
cb-tumblebug | "vNetId": "default-shared-ncp-kr",
cb-tumblebug | "subnetIds": [
cb-tumblebug | "default-shared-ncp-kr",
cb-tumblebug | "default-shared-ncp-kr-01"
cb-tumblebug | ],
cb-tumblebug | "securityGroupIds": [
cb-tumblebug | "default-shared-ncp-kr"
cb-tumblebug | ],
cb-tumblebug | "k8sNodeGroupList": [
cb-tumblebug | {
cb-tumblebug | "name": "k8sng01",
cb-tumblebug | "imageId": "",
cb-tumblebug | "specId": "ncp+kr+ci2-g3",
cb-tumblebug | "rootDiskType": "",
cb-tumblebug | "rootDiskSize": "",
cb-tumblebug | "sshKeyId": "default-shared-ncp-kr",
cb-tumblebug | "onAutoScaling": "true",
cb-tumblebug | "desiredNodeSize": "1",
cb-tumblebug | "minNodeSize": "1",
cb-tumblebug | "maxNodeSize": "2",
cb-tumblebug | "label": null,
cb-tumblebug | "description": ""
cb-tumblebug | }
cb-tumblebug | ],
cb-tumblebug | "cspResourceId": "",
cb-tumblebug | "label": null,
cb-tumblebug | "systemLabel": ""
cb-tumblebug | }
cb-tumblebug | {
cb-tumblebug | "connectionName": "ncp-kr",
cb-tumblebug | "description": "",
cb-tumblebug | "name": "k8scluster01",
cb-tumblebug | "version": "1.32.6-nks.1",
cb-tumblebug | "vNetId": "default-shared-ncp-kr",
cb-tumblebug | "subnetIds": [
cb-tumblebug | "default-shared-ncp-kr",
cb-tumblebug | "default-shared-ncp-kr-01"
cb-tumblebug | ],
cb-tumblebug | "securityGroupIds": [
cb-tumblebug | "default-shared-ncp-kr"
cb-tumblebug | ],
cb-tumblebug | "k8sNodeGroupList": [
cb-tumblebug | {
cb-tumblebug | "name": "k8sng01",
cb-tumblebug | "imageId": "",
cb-tumblebug | "specId": "ncp+kr+ci2-g3",
cb-tumblebug | "rootDiskType": "",
cb-tumblebug | "rootDiskSize": "",
cb-tumblebug | "sshKeyId": "default-shared-ncp-kr",
cb-tumblebug | "onAutoScaling": "true",
cb-tumblebug | "desiredNodeSize": "1",
cb-tumblebug | "minNodeSize": "1",
cb-tumblebug | "maxNodeSize": "2",
cb-tumblebug | "label": null,
cb-tumblebug | "description": ""
cb-tumblebug | }
cb-tumblebug | ],
cb-tumblebug | "cspResourceId": "",
cb-tumblebug | "label": null,
cb-tumblebug | "systemLabel": ""
cb-tumblebug | }
cb-tumblebug | 6:50AM INF src/core/resource/k8scluster.go:201 > CreateK8sCluster
cb-tumblebug | 6:50AM ERR src/core/resource/k8scluster.go:1417 > reflect: call of reflect.Value.Interface on zero Value
cb-tumblebug | 6:50AM DBG src/core/resource/k8scluster.go:102 > [Create K8sClusterInfo] k8scluster01
cb-tumblebug | 6:50AM DBG src/core/resource/spec.go:2259 > [Get spec] ncp+kr+ci2-g3
cb-tumblebug | 6:50AM WRN src/core/resource/k8scluster.go:412 > Not found the Spec ncp+kr+ci2-g3 in ns default, find it from SystemCommonNs
cb-tumblebug | 6:50AM DBG src/core/resource/spec.go:2259 > [Get spec] ncp+kr+ci2-g3
cb-tumblebug | 6:50AM WRN src/core/resource/k8scluster.go:412 > Not found the Spec ncp+kr+ci2-g3 in ns default, find it from SystemCommonNs
cb-tumblebug | 6:50AM DBG src/core/resource/spec.go:2259 > [Get spec] ncp+kr+ci2-g3
cb-tumblebug | 6:50AM INF src/core/resource/k8scluster.go:422 > Use the SpecId ci2-g3 in SystemCommonNs
cb-tumblebug | 6:50AM INF src/core/resource/k8scluster.go:422 > Use the SpecId ci2-g3 in SystemCommonNs
cb-tumblebug | 6:50AM DBG src/core/resource/k8scluster.go:474 > [Via Spider] Create K8s cluster (url: http://cb-spider:1024/spider/cluster)
cb-tumblebug | 6:50AM DBG src/core/resource/k8scluster.go:475 > Request to Spider: '{ConnectionName:ncp-kr ReqInfo:{Name:d2sjdfobnvgc739p28b0 Version:1.32.6-nks.1 VPCName:d2sj1cobnvgc739p2880 SubnetNames:[d2sj1cobnvgc739p288g] SecurityGroupNames:[d2sj1lobnvgc739p28a0] NodeGroupList:[{Name:k8sng01 ImageName: VMSpecName:ci2-g3 RootDiskType: RootDiskSize: KeyPairName:d2sj1lgbnvgc739p289g OnAutoScaling:true DesiredNodeSize:1 MinNodeSize:1 MaxNodeSize:2}]}}'
cb-tumblebug | 6:50AM DBG src/core/common/client/client.go:198 > Internal Call Start Method=POST URI=http://cb-spider:1024/spider/cluster requestBody={"ConnectionName":"ncp-kr","ReqInfo":{"Name":"d2sjdfobnvgc739p28b0","NodeGroupList":[{"DesiredNodeSize":"1","ImageName":"","KeyPairName":"d2sj1lgbnvgc739p289g","MaxNodeSize":"2","MinNodeSize":"1","Name":"k8sng01","OnAutoScaling":"true","RootDiskSize":"","RootDiskType":"","VMSpecName":"ci2-g3"}],"SecurityGroupNames":["d2sj1lobnvgc739p28a0"],"SubnetNames":["d2sj1cobnvgc739p288g"],"VPCName":"d2sj1cobnvgc739p2880","Version":"1.32.6-nks.1"}}
cb-tumblebug | 6:50AM DBG src/core/resource/k8scluster.go:474 > [Via Spider] Create K8s cluster (url: http://cb-spider:1024/spider/cluster)
cb-tumblebug | 6:50AM DBG src/core/resource/k8scluster.go:475 > Request to Spider: '{ConnectionName:ncp-kr ReqInfo:{Name:d2sjdfobnvgc739p28b0 Version:1.32.6-nks.1 VPCName:d2sj1cobnvgc739p2880 SubnetNames:[d2sj1cobnvgc739p288g] SecurityGroupNames:[d2sj1lobnvgc739p28a0] NodeGroupList:[{Name:k8sng01 ImageName: VMSpecName:ci2-g3 RootDiskType: RootDiskSize: KeyPairName:d2sj1lgbnvgc739p289g OnAutoScaling:true DesiredNodeSize:1 MinNodeSize:1 MaxNodeSize:2}]}}'
cb-tumblebug | 6:50AM DBG src/core/common/client/client.go:198 > Internal Call Start Method=POST URI=http://cb-spider:1024/spider/cluster requestBody={"ConnectionName":"ncp-kr","ReqInfo":{"Name":"d2sjdfobnvgc739p28b0","NodeGroupList":[{"DesiredNodeSize":"1","ImageName":"","KeyPairName":"d2sj1lgbnvgc739p289g","MaxNodeSize":"2","MinNodeSize":"1","Name":"k8sng01","OnAutoScaling":"true","RootDiskSize":"","RootDiskType":"","VMSpecName":"ci2-g3"}],"SecurityGroupNames":["d2sj1lobnvgc739p28a0"],"SubnetNames":["d2sj1cobnvgc739p288g"],"VPCName":"d2sj1cobnvgc739p2880","Version":"1.32.6-nks.1"}}
cb-spider | [CB-SPIDER].[ERROR]: 2025-09-04 06:50:40 ClusterHandler.go:77, github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/ncp/resources.(*NcpVpcClusterHandler).CreateCluster.func2() - Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {"error":{"errorCode":400,"message":"Bad Request","details":{"error":[{"item":"257981","property":"subnetNoList","constraints":{"subnetNoList":"257981 값은 257980 중 하나여야 합니다."}}]}},"timestamp":"2025-09-04T06:50:40.783Z"}
cb-spider | [CB-SPIDER].[ERROR]: 2025-09-04 06:50:40 CommonNcpVpcFunc.go:88, github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/ncp/resources.LoggingError() - Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {"error":{"errorCode":400,"message":"Bad Request","details":{"error":[{"item":"257981","property":"subnetNoList","constraints":{"subnetNoList":"257981 값은 257980 중 하나여야 합니다."}}]}},"timestamp":"2025-09-04T06:50:40.783Z"}
cb-spider | [CB-SPIDER].[ERROR]: 2025-09-04 06:50:40 ClusterManager.go:616, github.com/cloud-barista/cb-spider/api-runtime/common-runtime.CreateCluster() - Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {"error":{"errorCode":400,"message":"Bad Request","details":{"error":[{"item":"257981","property":"subnetNoList","constraints":{"subnetNoList":"257981 값은 257980 중 하나여야 합니다."}}]}},"timestamp":"2025-09-04T06:50:40.783Z"}
cb-spider | [CB-SPIDER].[ERROR]: 2025-09-04 06:50:40 ClusterHandler.go:77, github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/ncp/resources.(*NcpVpcClusterHandler).CreateCluster.func2() - Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {"error":{"errorCode":400,"message":"Bad Request","details":{"error":[{"item":"257981","property":"subnetNoList","constraints":{"subnetNoList":"257981 값은 257980 중 하나여야 합니다."}}]}},"timestamp":"2025-09-04T06:50:40.783Z"}
cb-spider | [CB-SPIDER].[ERROR]: 2025-09-04 06:50:40 CommonNcpVpcFunc.go:88, github.com/cloud-barista/cb-spider/cloud-control-manager/cloud-driver/drivers/ncp/resources.LoggingError() - Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {"error":{"errorCode":400,"message":"Bad Request","details":{"error":[{"item":"257981","property":"subnetNoList","constraints":{"subnetNoList":"257981 값은 257980 중 하나여야 합니다."}}]}},"timestamp":"2025-09-04T06:50:40.783Z"}
cb-spider | [CB-SPIDER].[ERROR]: 2025-09-04 06:50:40 ClusterManager.go:616, github.com/cloud-barista/cb-spider/api-runtime/common-runtime.CreateCluster() - Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {"error":{"errorCode":400,"message":"Bad Request","details":{"error":[{"item":"257981","property":"subnetNoList","constraints":{"subnetNoList":"257981 값은 257980 중 하나여야 합니다."}}]}},"timestamp":"2025-09-04T06:50:40.783Z"}
cb-tumblebug | 6:50AM DBG src/core/common/client/client.go:376 > Internal Call Error Method=POST URI=http://cb-spider:1024/spider/cluster latency=983.038674 responseBody={"message":"Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {\"error\":{\"errorCode\":400,\"message\":\"Bad Request\",\"details\":{\"error\":[{\"item\":\"257981\",\"property\":\"subnetNoList\",\"constraints\":{\"subnetNoList\":\"257981 값은 257980 중 하나여야 합니다.\"}}]}},\"timestamp\":\"2025-09-04T06:50:40.783Z\"}"} status=500
cb-tumblebug | 6:50AM ERR src/core/resource/k8scluster.go:491 > Failed to Create a K8sCluster(k8scluster01) error="Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {error:{errorCode:400,message:Bad Request,details:{error:[{item:257981,property:subnetNoList,constraints:{subnetNoList:257981 값은 257980 중 하나여야 합니다.}}]}},timestamp:2025-09-04T06:50:40.783Z} (from cb-spider:1024/spider/cluster (500 Internal Server Error))"
cb-tumblebug | 6:50AM ERR src/core/resource/k8scluster.go:271 > Failed to Create a K8sCluster(k8scluster01) error="Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {error:{errorCode:400,message:Bad Request,details:{error:[{item:257981,property:subnetNoList,constraints:{subnetNoList:257981 값은 257980 중 하나여야 합니다.}}]}},timestamp:2025-09-04T06:50:40.783Z} (from cb-spider:1024/spider/cluster (500 Internal Server Error))"
cb-tumblebug | 6:50AM DBG src/core/resource/k8scluster.go:187 > [Delete K8sClusterInfo] k8scluster01
cb-tumblebug | 6:50AM DBG src/core/common/client/client.go:376 > Internal Call Error Method=POST URI=http://cb-spider:1024/spider/cluster latency=983.038674 responseBody={"message":"Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {\"error\":{\"errorCode\":400,\"message\":\"Bad Request\",\"details\":{\"error\":[{\"item\":\"257981\",\"property\":\"subnetNoList\",\"constraints\":{\"subnetNoList\":\"257981 값은 257980 중 하나여야 합니다.\"}}]}},\"timestamp\":\"2025-09-04T06:50:40.783Z\"}"} status=500
cb-tumblebug | 6:50AM ERR src/core/resource/k8scluster.go:491 > Failed to Create a K8sCluster(k8scluster01) error="Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {error:{errorCode:400,message:Bad Request,details:{error:[{item:257981,property:subnetNoList,constraints:{subnetNoList:257981 값은 257980 중 하나여야 합니다.}}]}},timestamp:2025-09-04T06:50:40.783Z} (from cb-spider:1024/spider/cluster (500 Internal Server Error))"
cb-tumblebug | 6:50AM ERR src/core/resource/k8scluster.go:271 > Failed to Create a K8sCluster(k8scluster01) error="Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {error:{errorCode:400,message:Bad Request,details:{error:[{item:257981,property:subnetNoList,constraints:{subnetNoList:257981 값은 257980 중 하나여야 합니다.}}]}},timestamp:2025-09-04T06:50:40.783Z} (from cb-spider:1024/spider/cluster (500 Internal Server Error))"
cb-tumblebug | 6:50AM DBG src/core/resource/k8scluster.go:187 > [Delete K8sClusterInfo] k8scluster01
cb-tumblebug | 6:50AM ERR src/interface/rest/server/resource/k8scluster.go:602 > failed to create K8sCluster dynamically error="Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {error:{errorCode:400,message:Bad Request,details:{error:[{item:257981,property:subnetNoList,constraints:{subnetNoList:257981 값은 257980 중 하나여야 합니다.}}]}},timestamp:2025-09-04T06:50:40.783Z} (from cb-spider:1024/spider/cluster (500 Internal Server Error))"
cb-tumblebug | 6:50AM ERR src/interface/rest/server/resource/k8scluster.go:602 > failed to create K8sCluster dynamically error="Failed to Create Cluster: failed to create cluster: Status: 400 Bad Request, Body: {error:{errorCode:400,message:Bad Request,details:{error:[{item:257981,property:subnetNoList,constraints:{subnetNoList:257981 값은 257980 중 하나여야 합니다.}}]}},timestamp:2025-09-04T06:50:40.783Z} (from cb-spider:1024/spider/cluster (500 Internal Server Error))"What you expected to happen
: Expected the cluster creation request to succeed.
How to reproduce it (as minimally and precisely as possible)
: Call the CB-Tumblebug k8sClusterDynamic API with the following request body:
{
"imageId": "default",
"specId": "ncp+kr+ci2-g3",
"connectionName": "ncp-kr",
"name": "k8scluster01",
"nodeGroupName": "k8sng01"
}Anything else we need to know?
: The current behavior of the k8sClusterDynamic API results in two subnets being created across different AZs (see Figure 1).
I started investigating the NCP console and found that a K8s cluster must be created within a single AZ.
All subnets used for the cluster (including LB subnets) must belong to the same zone (see Figure 2 and Figure 3).
It is possible to add subnets from different AZs to a VPC, but when creating a K8s cluster, only subnets within the same AZ can be included.
- Figure 1. Subnet list after calling
k8sClusterDynamicAPI (two subnets are present across different AZs)
- Figure 2. Subnet list during K8s cluster creation (AZ
KR-1selected)
- Figure 3. Subnet list during K8s cluster creation (AZ
KR-2selected)
Environment
- Source version or branch:CB-Tumblebug: working branch based on 0.11.9, cb-spider: 0.11.5
- OS: Ubuntu 22.04 LTS (running inside WSL2 on Windows 11)
Proposed solution
: Adjust the k8sClusterDynamic API so that, while other CSPs continue to create subnets across different AZs (multi-AZ), the NCP driver branches to ensure all subnets are created within the same AZ.
Any other context
: While testing with CB-Tumblebug, I deleted one subnet in AZ KR-2 using the CB-Tumblebug API and retried cluster creation via the k8sClusterDynamic API.
After removing the cross-AZ subnet, the cluster was successfully created.
- Figure 4. Cluster successfully created after removing the cross-AZ subnet (screenshot from NCP console)
