Telepresence should handle swapping deployments that change the UID

[xoyo24]

What were you trying to do?

Test telepresence with microservices-demo, by swap front-end with a local process.
I've got the demo run on gke successfully and tried this command.

telepresence --swap-deployment front-end --namespace sock-shop --expose 8079 --run npm start

What did you expect to happen?

The command to succeed.

What happened instead?

The pod failed to start with the error below.

"/usr/src/app/run.sh: line 7: can't create /etc/passwd: Read-only file system\n", 

Automatically included information

Command line: ['/usr/local/bin/telepresence', '--swap-deployment', 'front-end', '--namespace', 'sock-shop', '--expose', '8079', '--run', 'npm', 'start']
Version: 0.90
Python version: 3.7.0 (default, Jun 29 2018, 20:13:13) [Clang 9.1.0 (clang-902.0.39.2)]
kubectl version: Client Version: v1.11.1 // Server Version: v1.10.5-gke.3
oc version: (error: [Errno 2] No such file or directory: 'oc': 'oc')
OS: Darwin XiaoYang-no-MacBook-Pro.local 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64
Traceback:

Traceback (most recent call last):
  File "/usr/local/Cellar/telepresence/0.90/libexec/lib/python3.7/site-packages/telepresence/cli.py", line 87, in call_f
    return f(*args, **kwargs)
  File "/usr/local/Cellar/telepresence/0.90/libexec/lib/python3.7/site-packages/telepresence/main.py", line 76, in main
    remote_info = start_proxy(runner, args)
  File "/usr/local/Cellar/telepresence/0.90/libexec/lib/python3.7/site-packages/telepresence/proxy.py", line 222, in start_proxy
    run_id=run_id,
  File "/usr/local/Cellar/telepresence/0.90/libexec/lib/python3.7/site-packages/telepresence/remote.py", line 233, in get_remote_info
    wait_for_pod(runner, remote_info)
  File "/usr/local/Cellar/telepresence/0.90/libexec/lib/python3.7/site-packages/telepresence/remote.py", line 145, in wait_for_pod
    "Pod isn't starting or can't be found: {}".format(pod["status"])
RuntimeError: Pod isn't starting or can't be found: {'conditions': [{'lastProbeTime': None, 'lastTransitionTime': '2018-08-02T14:22:59Z', 'status': 'True', 'type': 'Initialized'}, {'lastProbeTime': None, 'lastTransitionTime': '2018-08-02T14:22:59Z', 'message': 'containers with unready status: [front-end]', 'reason': 'ContainersNotReady', 'status': 'False', 'type': 'Ready'}, {'lastProbeTime': None, 'lastTransitionTime': '2018-08-02T14:22:59Z', 'status': 'True', 'type': 'PodScheduled'}], 'containerStatuses': [{'containerID': 'docker://2321f5e077bc5a59e9e8023b4d617b5277d6775d483d1e71b8f72f80bc33454e', 'image': 'datawire/telepresence-k8s:0.90', 'imageID': 'docker-pullable://datawire/telepresence-k8s@sha256:48ed4f2469c777797a536eac62f9fe9b81370cc98ea66d9c61738a59a5249479', 'lastState': {'terminated': {'containerID': 'docker://2321f5e077bc5a59e9e8023b4d617b5277d6775d483d1e71b8f72f80bc33454e', 'exitCode': 1, 'finishedAt': '2018-08-02T14:24:35Z', 'message': "/usr/src/app/run.sh: line 7: can't create /etc/passwd: Read-only file system\n", 'reason': 'Error', 'startedAt': '2018-08-02T14:24:34Z'}}, 'name': 'front-end', 'ready': False, 'restartCount': 4, 'state': {'waiting': {'message': 'Back-off 1m20s restarting failed container=front-end pod=front-end-caee973a-7922-4e27-af11-4d17e09595b5-65784cbd8c-hzftp_sock-shop(8e752aa9-965f-11e8-a0ef-42010a8002db)', 'reason': 'CrashLoopBackOff'}}}], 'hostIP': '10.128.0.2', 'phase': 'Running', 'podIP': '10.8.1.16', 'qosClass': 'Burstable', 'startTime': '2018-08-02T14:22:59Z'}

Logs:

--context gke_telepresence-demo_us-central1-a_cluster-1 --namespace sock-shop get pod front-end-caee973a-7922-4e27-af11-4d17e09595b5-65784cbd8c-hzftp -o json
 133.8 TEL | [133] Capturing: kubectl --context gke_telepresence-demo_us-central1-a_cluster-1 --namespace sock-shop get pod front-end-caee973a-7922-4e27-af11-4d17e09595b5-65784cbd8c-hzftp -o json
 134.7 TEL | [134] Capturing: kubectl --context gke_telepresence-demo_us-central1-a_cluster-1 --namespace sock-shop get pod front-end-caee973a-7922-4e27-af11-4d17e09595b5-65784cbd8c-hzftp -o json
 135.6 TEL | [135] Capturing: kubectl --context gke_telepresence-demo_us-central1-a_cluster-1 --namespace sock-shop get pod front-end-caee973a-7922-4e27-af11-4d17e09595b5-65784cbd8c-hzftp -o json
 136.5 TEL | [136]

[ark3]

Sorry about the crash! I think something in the original deployment is conflicting with Telepresence. When we figure out what that is, we can modify Telepresence to screen that out. Can you show us the deployment you're swapping? Thank you.

[xoyo24]

@ark3 Thanks for the reply.
This is the deployment I'm trying to swamp.
https://github.com/microservices-demo/microservices-demo/blob/master/deploy/kubernetes/manifests/front-end-dep.yaml

[ark3]

I believe this is the issue: readOnlyRootFilesystem: true.

We need to figure out what Telepresence should do with the deployment's security context in general, but that particular item is incompatible with how Tel works right now. As a workaround, you could remove it; Telepresence will be able to handle that once we fix this bug. Thank you for the issue and follow-up.

[flangknecht]

@ark3 I had to figure this one out myself as the k8s cluster I'm using doesn't allow anything but read-only root fs.
I came up with the following workaround:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: telepresence
spec:
  replicas: 1  # only one replica
  template:
    metadata:
      labels:
        name: telepresence
    spec:
      initContainers:
      - name: telepresence-initial
        image: datawire/telepresence-k8s:0.90
        command: ['sh', '-c', 'cp /etc/passwd /tempvol/passwd && cp -a /usr/src/app/. /appvol/']
        volumeMounts:
        - mountPath: /appvol
          name: appdir-volume
        - mountPath: /tempvol
          name: etc-volume
      containers:
      - name: telepresence
        image: datawire/telepresence-k8s:0.90
        volumeMounts:
        - mountPath: /etc/passwd
          name: etc-volume
          subPath: passwd
        - mountPath: /usr/src/app
          name: appdir-volume
      volumes: 
        - name: etc-volume
          emptyDir: {}
        - name: appdir-volume
          emptyDir: {}

Since telepresence needs to write to places ordinary pods probably shouldn't try to write to, I think it might be a good idea to incorporate something like this into the way telepresence deploys its kubernetes-proxy.

[ark3]

Thanks for helping us identify this issue and posting your workaround. I've changed the issue title to reflect the underlying issue.

[ark3]

I believe this failure can be fixed in a straightforward manner. The failing writes to /etc come from run.sh setting up sshd depending on the current user. See #617 for some background on why the image used to run as different users. In short, this is no longer necessary. The easy fix is to move the writes from run.sh into the two Dockerfiles. Then Telepresence will not need to write to /etc at all.

[ark3]

See also this article for an example of how to build both images from one Dockerfile using a 2018-release Docker command.