43

On minikube for windows I created a deployment on the kubernetes cluster, then I tried to scale it by changing replicas from 1 to 2, and after that kubectl hangs and my disk usage is 100%. I only have one container in my deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: first-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      run: app
  template:
    metadata:
      labels:
        run: app
    spec:
      containers:
      - name: demo
        image: ner_app
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5000

all I did was run this after the pods were successfully deployed and running

kubectl scale --replicas=2 deployment first-deployment

In another terminal I was watching the pods using

kubectl get pods --watch

But everything is unresponsive and I'm not sure how to recover from this.

When I run kubectl get pods again it gives the following message

PS D:\docker\ner> kubectl get pods
Unable to connect to the server: net/http: TLS handshake timeout

Is there a way to recover, or cancel whatever process is running?

Also my VM's are on Hyper-V for Windows 10 Pro (minikube and Docker Desktop) both have the default RAM allocated - 2048MB

The container in my pod is a machine learning process and the model it loads could be large, in the order of 200MB to 300MB

Improve this question
2
  • Closing Docker Desktop and all its related processes solved it for me. Commented Apr 22, 2023 at 16:57
  • As suggested by @Muzammil below soultion it's working as expected afer performing this command 'NO_PROXY=$NO_PROXY,192.168.49.1:6443' , in my case the docker container ip address was '192.168.49.1' and port is '6443' so just chnage that IP according to your use-case and most of the case seen that port are using same. Commented Jan 16, 2024 at 13:49

23 Answers 23

Reset to default
62

You may have some proxy problems. Try following commands:

$ unset http_proxy
$ unset https_proxy

and repeat your kubectl call.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Or add the node address to NO_PROXY environment variable. For instance, export NO_PROXY=$NO_PROXY,192.168.126.139:6443
I searched the issue, found this answer. Upvoted. Then forgot. Six months later, got the same issue, searched again, found this answer. Wish I could upvote again :-)
$ unset all_proxy is another one to add to the list as a commonly used env var. these are also sometimes all in caps, so $ unset ALL_PROXY and same for HTTP_PROXY, HTTPS_PROXY, FTP_PROXY etc
Any alternative for windows powershell ?
For anyone having similar problem for AWS EKS (after adding vpc-cni addon) NO_PROXY=$NO_PROXY;eks.amazonaws.com works.
|
28

Just happened to me on a new Windows 10 install with Ubuntu distro in WSL2. I solved the problem by running:

$ sudo ifconfig eth0 mtu 1350

or for modern Linux distros that deprecated ifconfig:

$ sudo ip link set dev eth0 mtu 1350

(BTW, I was on a VPN connection when trying the 'kubectl get pods' command)

Improve this answer

6 Comments

Thanks! I forgot about this! I was using Wireguard on Windows + WSL2. If there is any network problem usually it was MTU issue.
we had the same issue Windows + WSL2, Over wireguard we decreased the MTU to 1400. Wireguard Server was hosted at cloud provider.
Changing the MTU did the trick for me. Could you please elaborate on how MTU affects this?
@Koshur - I haven't debugged this but usually this happens when the "Do-not-fragment" bit is set on a packet that is routed through a network whose MTU is set to a lower number than the packet size.
It seems that ifconfig is deprecated. This is an alternative command: sudo ip link set dev eth0 mtu 1350
|
13

For me, the problem is that Docker ran out of memory. (EDIT: Possibly anyway; I wrote this post a while ago, and am now not so sure that is the root case, but did not write down my rationale, so idk.)

Anyway, to fix:

  1. Fully close your k8s emulator. (docker desktop, minikube, etc.)
  2. Shutdown WSL2. (wsl --shutdown) [EDIT: This step is apparently not necessary -- at least not always, since this time I skipped it, and the problem still resolved.]
  3. Restart your k8s emulator.
  4. Rerun the commands you wanted.

Sometimes it also works to simply:

  1. Right click the Docker Desktop tray-icon, press "Restart Docker", and wait a few minutes for things to restart. (sometimes this fails, with Docker Desktop saying "Docker failed to start", so I'd generally recommend the more thorough process above)
Improve this answer

2 Comments

In my case, I also had to increase WSL's memory
In my case I was running my local cluster with kind. It probably went out of memory as my browser tabs also indicated that. So I just freed some memory and restarted the container.
6

You can set up resource limits on deployments so that pods will not use the entire available resource in the node.

Improve this answer

2 Comments

Ok I will try that. How can I recover from the current situation? I came back to my workstation this morning and its still in the same state. My disk is still 100% usage and kubectl is unresponsive.
I think you should stop minikube and start it again. Does answers in stackoverflow.com/questions/56327843/… help you?
6

In my case I have my private EKS cluster and there is no 443(HTTPS) enabled in security groups.

My issue is solved after enabling the (HTTPS)443 port in security groups.

Kindly refer for AWS documentation for more details: "You must ensure that your Amazon EKS control plane security group contains rules to allow ingress traffic on port 443 from your connected network"

Improve this answer

Comments

4

You could try $ unset all_proxy to reset the socket proxy.

Also, if you're connected to a VPN, try disconnecting - it seems that can interfere with connecting to a cluster.

Improve this answer

1 Comment

I had tailscale running, stopping it solved it for me
4

i solved this problem when execute the following command

minikube delete

and then start it

minikube start --vm-driver="virtualbox"

if use this why your pods will deleted

and when run kubectl get pods

you can see this result No resources found in default namespace.

Improve this answer

Comments

3

I got the same problem. Usually, this works:

$ unset http_proxy
$ unset https_proxy

And when this⬆️ didn't work today, I just restart my docker and minikube, and it works.

Improve this answer

Comments

2

Simply closing cmd, opening again, then

minikube start

And then executing the commands again solved this issue for me. P.S: minikube start took less than a minute

Improve this answer

Comments

1

I think the other answers don't really mention or refer to the vpn and proxy documentation for minikube: https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/

The NO_PROXY variable here is important: Without setting it, minikube may not be able to access resources within the VM. minikube uses two IP ranges, which should not go through the proxy:

192.168.99.0/24: Used by the minikube VM. Configurable for some hypervisors via --host-only-cidr
192.168.39.0/24: Used by the minikube kvm2 driver.
192.168.49.0/24: Used by the minikube docker driver’s first cluster.
10.96.0.0/12: Used by service cluster IP’s. Configurable via --service-cluster-ip-rang

So adding those IP ranges to your NO_PROXY environment variable should fix the issue.

Improve this answer

Comments

1

Adding the IP address to the no_proxy list worked for me.

Obtain the IP address from ip addr output.

export no_proxy=localhost,127.0.0.1,<IP_ADDRESS>

Improve this answer

1 Comment

For those having issue with AWS EKS, NO_PROXY solves problem NO_PROXY=$NO_PROXY;eks.amazonaws.com
1

As this answer comes first on search for net-http-tls-handshake-timeout error

For those having issue with AWS EKS (and likely any K8s), NO_PROXY solves problem by adding related IP/host to environment variable. As suggested in comments for first answer.

For AWS EKS (when seeing this intermittently after vpc-cni addon upgrade) replace for specific region or single url for your use case.

NO_PROXY=$NO_PROXY;eks.amazonaws.com
Improve this answer

Comments

1

restart minikube will work.

But if you don't want to delete it

then you can just switch to other cluster and then switch back.

enter image description here

I just click other kubenete cluster (ex: docker-desktop)

and then click back to the cluster I want to run (ex: minikube)

Improve this answer

Comments

1 
minikube stop
minikube delete
minikube start

This worked for me !!!

Improve this answer

Comments

0

If you're on Linux or Mac, go to your virtualbox, and then on the toolbar choose 'Global Tools', then if you see two machines are using the same ip address, you should remove one of them. this image shows virtual box GUI

Improve this answer

Comments

0

At least for Windows 10 and 11

$PS C:\oc rollback dc/my-app

Unable to connect to the server: net/http: TLS handshake timeout

For OpenShift 4.x the problem is that for some reason you are logged-out:

$PS C:\oc status

error: You must be logged in to the server (Unauthorized)

logging in by e.g.

$oc login -u developer

resolves the problem

Improve this answer

Comments

0

Open PowerShell as an administrator and run the command "wsl --shutdown". You will see the same notification in your open Ubuntu terminal.

Open Docker Desktop.

Open a new terminal.

Run the command "minikube status" in the Ubuntu terminal.

Run the Minikube container. You can do this in Docker Desktop.

Run the command "minikube start".

That's it! You don't need to close your computer after this, and Minikube should work fine.

Improve this answer

1 Comment

And please control your Windows Subsystem for Linux. 1- Press Windows key+R to open Run dialog. type: optionalfeatures.exe and hit Enter 2- Scroll to the bottom and uncheck Windows Subsystem for Linux. Click OK.
0

I am running my cluster on virtualbox with two external etcd vm clusters and I had to start both etcd VMs. I was able to run kubectl fine afterwards.

Improve this answer

1 Comment

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
0

I met the same problem and I fix it by sync the system time of the node.

Improve this answer

Comments

0

If you are using windows than run below commands:

set varname= https_proxy
set varname= http_proxy
Improve this answer

Comments

0

In my case, this error popped out after I created a statefulset for Cassandra. I had to stop my Minikube instance and restart it with the commands:

minikube stop

minikube start --memory 5120 --cpus=4

That solved my trouble; hope to help somebody else.

Improve this answer

Comments

0

I was trying to create a secret on my Minikube cluster, which needed setting some secret

kubectl create secret generic mysql-secrets --from-literal=openmetadata-mysql-password=<password>

I got error like

error: failed to create secret Post 

    https://192.168.49.2:8443/api/v1/namespaces/default/secrets?fieldManager=kubectl-create&fieldValidation=Strict": net/http: TLS handshake timeout

First tested with

$ telnet 192.168.49.2 8443

Trying 192.168.49.2...
Connected to 192.168.49.2.
Escape character is '^]'.

So there was no issue in network,

TLS Handshake was resolved by adding the ip:port into NO_PROXY

$ export NO_PROXY=$NO_PROXY,192.168.49.2:8443

Improve this answer

Comments

0

try to increase docker resources through docker desktop and it will fix the issue

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.