Kubernetes Error: Error response from daemon: invalid mode

Recently I was trying to mount a persistence volume in a Kubernetes pod. I faced the error "Error: Error response from daemon: invalid mode". After struggling for a long time and searching internet I was able to solve this problem. Here I am writing the solution to this problem. I was trying to follow tutorial https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/ . Only difference  being that I was trying it on windows rather than Linux.

I tried with following persistance_volume.yaml file

 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "c:/jasvant/kubernetes/persistence_volume/data"

My directory on windows was present at "C:\jasvant\kubernetes\persistence_volume\data" so I replaced all '\' with '/'. But is did not work and pod container failed to start.

C:\Users\jasvant\kubernetes>kubectl get pod
NAME                                  READY   STATUS                 RESTARTS   AGE
hello-minikube-6fb6cb79cc-8drt9       1/1     Running                6          130d
kubernetes-bootcamp-dd569fc9c-bt74m   1/1     Running                5          96d
kubernetes-bootcamp-dd569fc9c-fl64v   1/1     Running                5          96d
task-pv-pod                           0/1     CreateContainerError   0          62s

The container failed to start with CreateContainerError. When I described the pod I got the following error:

C:\Users\jasvant\kubernetes>kubectl describe pod  task-pv-pod
Name:         task-pv-pod
Namespace:    default
Priority:     0
Node:         minikube/10.0.2.15
Start Time:   Wed, 05 Feb 2020 14:27:59 +0530
Labels:       
Annotations:  
Status:       Pending
IP:           172.17.0.7
Containers:
  task-pv-container:
    Container ID:
    Image:          nginx
    Image ID:
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Environment:    
    Mounts:
      /usr/share/nginx/html from task-pv-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-scccw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  task-pv-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  task-pv-claim
    ReadOnly:   false
  default-token-scccw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-scccw
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  2m14s               default-scheduler  Successfully assigned default/task-pv-pod to minikube
  Normal   Pulled     14s (x8 over 2m8s)  kubelet, minikube  Successfully pulled image "nginx"
  Warning  Failed     14s (x8 over 2m8s)  kubelet, minikube  Error: Error response from daemon: invalid mode: /usr/share/nginx/html
  Normal   Pulling    4s (x9 over 2m13s)  kubelet, minikube  Pulling image "nginx"

This error message fails to give any hint to actual problem. The root cause is that HostPath.path need to be a path on VM created by Minikube and not a directory path on windows machine. so C: was not accepted by minikube. It is not impossible to mount a windows directory onto VM and use that directory as persistence volume storage directory. By default 'C:\Users' directory from Windows machine is mounted at   /c/Users folder in VM. So "/c/Users/jasvant/kubernetes/persistence_volume/data" will work for me. But if you create any directory out of  'C:\Users' directory then it will not work.. Please refer to https://minikube.sigs.k8s.io/docs/tasks/mount/ for more information.

If you want to login into VM created by Minikube for running Kubernetes cluster. you can use "minikube ssh" command on your windows machine. It will open a ssh to VM and you can browse the file system there.