failing e2e test jobs after ControlPlaneKubeletLocalMode enabled by default #3154

neolit123 · 2025-01-29T09:01:42Z

i suspect it's

kubeadm: Promote ControlPlaneKubeletLocalMode feature gate to beta kubernetes#129716

because the other PR after is cosmetic (klog change)

https://github.com/kubernetes/kubernetes/issues?q=label%3Aarea%2Fkubeadm+is%3Aclosed

failing jobs:

https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-dryrun-latest

https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-e2e-kubeadm-kinder-dryrun-latest/1884502182036770816/build-log.txt

[etcd] Would wait for the new etcd member to join the cluster
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is not healthy after 4m0.001133273s

Unfortunately, an error has occurred:
	The HTTP call equal to 'curl -sSL http://127.0.0.1:10248/healthz' returned error: Get "http://127.0.0.1:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused

https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-external-ca-latest

https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-e2e-kubeadm-kinder-external-ca-latest/1884261091186315264/build-log.txt

I0129 09:33:28.655624     245 loader.go:402] Config loaded from file:  /etc/kubernetes/kubelet.conf
I0129 09:33:28.656811     245 loader.go:402] Config loaded from file:  /etc/kubernetes/kubelet.conf
I0129 09:33:28.657129     245 kubelet.go:337] [kubelet-start] preserving the crisocket information for the node
I0129 09:33:28.657219     245 patchnode.go:32] [patchnode] Uploading the CRI socket "unix:///run/containerd/containerd.sock" to Node "kinder-external-ca-control-plane-2" as an annotation
...
I0128 15:30:34.908571     219 round_trippers.go:632] "Response" verb="GET" url="https://172.17.0.5:6443/api/v1/nodes/kinder-external-ca-worker-1?timeout=10s" status="" milliseconds=0
I0128 15:30:35.408661     219 round_trippers.go:632] "Response" verb="GET" url="https://172.17.0.5:6443/api/v1/nodes/kinder-external-ca-worker-1?timeout=10s" status="" milliseconds=0
Get "https://172.17.0.5:6443/api/v1/nodes/kinder-external-ca-worker-1?timeout=10s": dial tcp 172.17.0.5:6443: connect: connection refused
error writing CRISocket for this node
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runKubeletWaitBootstrapPhase
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join/kubelet.go:339
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:261
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:450
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run

both cases need investigation. in one case it seems it's not reaching the kubelet and the other the apiserver.
don't seem like flakes as it failed consistently N times. these jobs are a bit uncommon, i.e. they do custom actions like dry-run/external ca.

the regular job is green:

https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-latest

cc @chrischdi

The text was updated successfully, but these errors were encountered:

neolit123 · 2025-01-29T09:29:35Z

@chrischdi and the dedicated fg=false job also started failing, oddly:
https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-control-plane-kubelet-local-mode-latest

edit: actually this one is clearer. this needs update:


# task-09-post-upgrade
/bin/bash -c set -x

IP_ADDRESS="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' kinder-control-plane-local-kubelet-mode-lb)"

CMD="docker exec kinder-control-plane-local-kubelet-mode-control-plane-1"
${CMD} grep "server: https://${IP_ADDRESS}:6443" /etc/kubernetes/kubelet.conf || exit 1

CMD="docker exec kinder-control-plane-local-kubelet-mode-control-plane-2"
${CMD} grep "server: https://${IP_ADDRESS}:6443" /etc/kubernetes/kubelet.conf || exit 1

CMD="docker exec kinder-control-plane-local-kubelet-mode-control-plane-3"
${CMD} grep "server: https://${IP_ADDRESS}:6443" /etc/kubernetes/kubelet.conf || exit 1

# Ensure exit status of 0
exit 0


++ docker inspect '--format={{ .NetworkSettings.IPAddress }}' kinder-control-plane-local-kubelet-mode-lb
+ IP_ADDRESS=172.17.0.7
+ CMD='docker exec kinder-control-plane-local-kubelet-mode-control-plane-1'
+ docker exec kinder-control-plane-local-kubelet-mode-control-plane-1 grep 'server: https://172.17.0.7:6443' /etc/kubernetes/kubelet.conf
+ exit 1
 exit status 1

neolit123 · 2025-01-29T10:03:22Z

https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-external-ca-latest

https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-e2e-kubeadm-kinder-external-ca-latest/1884261091186315264/build-log.txt

I0129 09:33:28.655624     245 loader.go:402] Config loaded from file:  /etc/kubernetes/kubelet.conf
I0129 09:33:28.656811     245 loader.go:402] Config loaded from file:  /etc/kubernetes/kubelet.conf
I0129 09:33:28.657129     245 kubelet.go:337] [kubelet-start] preserving the crisocket information for the node
I0129 09:33:28.657219     245 patchnode.go:32] [patchnode] Uploading the CRI socket "unix:///run/containerd/containerd.sock" to Node "kinder-external-ca-control-plane-2" as an annotation
...
I0128 15:30:34.908571     219 round_trippers.go:632] "Response" verb="GET" url="https://172.17.0.5:6443/api/v1/nodes/kinder-external-ca-worker-1?timeout=10s" status="" milliseconds=0
I0128 15:30:35.408661     219 round_trippers.go:632] "Response" verb="GET" url="https://172.17.0.5:6443/api/v1/nodes/kinder-external-ca-worker-1?timeout=10s" status="" milliseconds=0
Get "https://172.17.0.5:6443/api/v1/nodes/kinder-external-ca-worker-1?timeout=10s": dial tcp 172.17.0.5:6443: connect: connection refused
error writing CRISocket for this node
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runKubeletWaitBootstrapPhase
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join/kubelet.go:339
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:261
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:450
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run

external ca calls a kinder action setup-external-ca
https://github.com/kubernetes/kubeadm/blob/main/kinder/ci/workflows/external-ca-tasks.yaml#L56

it needs to be updated because it uses a naive approach to generate the same kubelet.conf on both workers and CP nodes
https://github.com/kubernetes/kubeadm/blob/main/kinder/pkg/cluster/manager/actions/setup-external-ca.go#L111

without that the kublet.conf will point to a non-existing local apiserver on worker nodes. instead it should point to lb.
the culprit is kubeadm init phase kubeconfig kubelet --control-plane-endpoint=%s --v=%d", where the CPE should be the LB.

i don't think there is a bigger issue here, i.e. we don't need to patch k/k.

edit: hmm but, --control-plane-endpoint=%s is already the lb IP according to the kinder source, but the file ends up with 172.17.0.5 which is the worker IP and there is no apiserver there at port 6443.

neolit123 · 2025-01-29T10:35:59Z

tested locally.

sudo kubeadm init phase certs ca
sudo kubeadm init phase kubeconfig all --control-plane-endpoint=foo.bar --v=5
sudo cat /etc/kubernetes/kubelet.conf | grep server
    server: https://192.168.0.101:6443

so that's a regression. we need to think how the kubelet local mode will continue to respect the user prodided clusterconfiguration.controlplaneendpoint or flag.

i will send revert PR for

kubeadm: Promote ControlPlaneKubeletLocalMode feature gate to beta kubernetes#129716

until we fix all these issues.

edit: here it is:

Revert "kubeadm: Promote ControlPlaneKubeletLocalMode feature gate to beta" kubernetes#129878

neolit123 · 2025-01-29T15:17:47Z

https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-external-ca-latest

https://storage.googleapis.com/kubernetes-ci-logs/logs/ci-kubernetes-e2e-kubeadm-kinder-external-ca-latest/1884261091186315264/build-log.txt

I0129 09:33:28.655624     245 loader.go:402] Config loaded from file:  /etc/kubernetes/kubelet.conf
I0129 09:33:28.656811     245 loader.go:402] Config loaded from file:  /etc/kubernetes/kubelet.conf
I0129 09:33:28.657129     245 kubelet.go:337] [kubelet-start] preserving the crisocket information for the node
I0129 09:33:28.657219     245 patchnode.go:32] [patchnode] Uploading the CRI socket "unix:///run/containerd/containerd.sock" to Node "kinder-external-ca-control-plane-2" as an annotation
...
I0128 15:30:34.908571     219 round_trippers.go:632] "Response" verb="GET" url="https://172.17.0.5:6443/api/v1/nodes/kinder-external-ca-worker-1?timeout=10s" status="" milliseconds=0
I0128 15:30:35.408661     219 round_trippers.go:632] "Response" verb="GET" url="https://172.17.0.5:6443/api/v1/nodes/kinder-external-ca-worker-1?timeout=10s" status="" milliseconds=0
Get "https://172.17.0.5:6443/api/v1/nodes/kinder-external-ca-worker-1?timeout=10s": dial tcp 172.17.0.5:6443: connect: connection refused
error writing CRISocket for this node
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join.runKubeletWaitBootstrapPhase
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/join/kubelet.go:339
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:261
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
	k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:450
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run

this issue seems to be that runKubeletWaitBootstrapPhase assumes there is a real kubelet running
https://github.com/kubernetes/kubernetes/blob/3bc8f01c74e80cb85e6f3813db1b410adba22bfe/cmd/kubeadm/app/cmd/phases/join/kubelet.go#L285
yet, during join dryrun, one is never started
https://github.com/kubernetes/kubernetes/blob/3bc8f01c74e80cb85e6f3813db1b410adba22bfe/cmd/kubeadm/app/cmd/phases/join/kubelet.go#L258

perhaps we should wrap the waiting

if dryrun {
  // print would wait for kubelet
} else {
  // wait
}

chrischdi · 2025-01-31T16:59:47Z

I'm planning to take a look at this next week.

/assign

chrischdi · 2025-02-03T14:08:07Z

Trying to iterate on the three issues which I call:

kinder dry-run
kinder external-ca
kinder fg-disabled is failing

1. kinder dry-run

That is easy fixable and needs to be done in k/k.

With the feature-gate disabled and when having dry-run, we:

currently return early here:
- https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/phases/join/kubelet.go#L265
and don't run runKubeletWaitBootstrapPhase which would be here:
- https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/phases/join/kubelet.go#L274

With the feature-gate enabled we directly run runKubeletWaitBootstrapPhase, so I propose to add an early return to that function too.

Example fix: chrischdi/kubernetes@65839db

2. kinder external-ca

tested locally.
sudo kubeadm init phase certs ca
sudo kubeadm init phase kubeconfig all --control-plane-endpoint=foo.bar --v=5
sudo cat /etc/kubernetes/kubelet.conf | grep server
    server: https://192.168.0.101:6443
so that's a regression. we need to think how the kubelet local mode will continue to respect the user prodided clusterconfiguration.controlplaneendpoint or flag.

i will send revert PR for

kubeadm: Promote ControlPlaneKubeletLocalMode feature gate to beta kubernetes#129716

until we fix all these issues.

edit: here it is:

Revert "kubeadm: Promote ControlPlaneKubeletLocalMode feature gate to beta" kubernetes#129878

I'm not sure if this is a regression or the wanted outcome of the feature gate instead.
In this example case kube-scheduler and kube-controller-manager also do not point to foo.bar, example:

$ kubeadm init phase certs ca
$ kubeadm init phase kubeconfig all --control-plane-endpoint=foo.bar --v=5
$ cat /etc/kubernetes/controller-manager.conf | grep server
    server: https://172.17.0.3:6443
$ cat /etc/kubernetes/scheduler.conf | grep server
    server: https://172.17.0.3:6443
$ cat /etc/kubernetes/admin.conf | grep server
    server: https://foo.bar:6443

3. kinder fg-disabled is failing

I'm now taking a look into this.

neolit123 · 2025-02-03T14:21:40Z

Example fix: chrischdi/kubernetes@65839db

makes sense.

I'm not sure if this is a regression or the wanted outcome of the feature gate instead.
In this example case kube-scheduler and kube-controller-manager also do not point to foo.bar, example:

historically the kcm and scheduler have been hardcoded to point to the local ip.
the admin.conf and kubelet.conf however received its server field value from the cpe passed by the user.
so given we are now planning to hardcode the kubelet.conf to also point to local ip, that breaks users who assumed that calling the kubelet kubeconfig phase of init on demand would give them a kubeconfig with a cpe server.

it seems to me this breaking change is inevitable, but it should be mentioned in the release note of the graduation pr.

one place where this break is kinder, like i mentioned earlier. so for the external ca workflow to pass this must be fixed here:
https://github.com/kubernetes/kubeadm/blob/main/kinder/pkg/cluster/manager/actions/setup-external-ca.go#L111
(have two types of kubeconfigs - point to local ip on cp nodes and point to lb on worker nodes)

chrischdi · 2025-02-04T08:02:09Z

Updated: kubernetes/kubernetes#129956

And added: #3157

1. kinder dry-run

Fixed in:

kubeadm: Promote ControlPlaneKubeletLocalMode feature gate to beta - second attempt kubernetes#129956

via:

kubernetes/kubernetes@e29e84e

Testing at https://github.com/neolit123/kubeadm-test/actions/runs/13131235897

2. kinder external-ca

Fixed in:

Update ControlPlaneKubeletLocalMode test for the feature gate being disabled and fix setup-external-ca in kinder #3157

via:

e9f1704

Note: in case of ControlPlaneKubeletLocalMode one way out is to set --apiserver-advertise-address when using kubeadm init phase kubeconfig kubelet.

3. kinder fg-disabled is failing

Fixed in:

kubeadm: Promote ControlPlaneKubeletLocalMode feature gate to beta - second attempt kubernetes#129956

via:

kubernetes/kubernetes@56420c6

Testing at https://github.com/neolit123/kubeadm-test/actions/runs/13130880191

chrischdi · 2025-02-06T09:34:38Z

Changes merged and jobs are still green 🎉
Should we close this?

xref:

neolit123 · 2025-02-06T10:53:36Z

thanks for the fixes

neolit123 added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. area/feature-gates labels Jan 29, 2025

neolit123 added this to the v1.33 milestone Jan 29, 2025

This was referenced Jan 29, 2025

Revert "kubeadm: Promote ControlPlaneKubeletLocalMode feature gate to beta" kubernetes/kubernetes#129878

Merged

make components on control-plane nodes point to the local API server endpoint #2271

Open

k8s-ci-robot assigned chrischdi Jan 31, 2025

This was referenced Feb 3, 2025

kubeadm: Promote ControlPlaneKubeletLocalMode feature gate to beta - second attempt kubernetes/kubernetes#129956

Merged

Update ControlPlaneKubeletLocalMode test for the feature gate being disabled and fix setup-external-ca in kinder #3157

Merged

neolit123 closed this as completed Feb 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

failing e2e test jobs after ControlPlaneKubeletLocalMode enabled by default #3154

failing e2e test jobs after ControlPlaneKubeletLocalMode enabled by default #3154

neolit123 commented Jan 29, 2025 •

edited

Loading

neolit123 commented Jan 29, 2025 •

edited

Loading

neolit123 commented Jan 29, 2025 •

edited

Loading

neolit123 commented Jan 29, 2025 •

edited

Loading

neolit123 commented Jan 29, 2025 •

edited

Loading

chrischdi commented Jan 31, 2025

chrischdi commented Feb 3, 2025

neolit123 commented Feb 3, 2025

chrischdi commented Feb 4, 2025 •

edited

Loading

chrischdi commented Feb 6, 2025 •

edited

Loading

neolit123 commented Feb 6, 2025

failing e2e test jobs after ControlPlaneKubeletLocalMode enabled by default #3154

failing e2e test jobs after ControlPlaneKubeletLocalMode enabled by default #3154

Comments

neolit123 commented Jan 29, 2025 • edited Loading

neolit123 commented Jan 29, 2025 • edited Loading

neolit123 commented Jan 29, 2025 • edited Loading

neolit123 commented Jan 29, 2025 • edited Loading

neolit123 commented Jan 29, 2025 • edited Loading

chrischdi commented Jan 31, 2025

chrischdi commented Feb 3, 2025

1. kinder dry-run

2. kinder external-ca

3. kinder fg-disabled is failing

neolit123 commented Feb 3, 2025

chrischdi commented Feb 4, 2025 • edited Loading

1. kinder dry-run

2. kinder external-ca

3. kinder fg-disabled is failing

chrischdi commented Feb 6, 2025 • edited Loading

neolit123 commented Feb 6, 2025

neolit123 commented Jan 29, 2025 •

edited

Loading

neolit123 commented Jan 29, 2025 •

edited

Loading

neolit123 commented Jan 29, 2025 •

edited

Loading

neolit123 commented Jan 29, 2025 •

edited

Loading

neolit123 commented Jan 29, 2025 •

edited

Loading

chrischdi commented Feb 4, 2025 •

edited

Loading

chrischdi commented Feb 6, 2025 •

edited

Loading