Pod Not Found

[AeroNotix]

Bug Description

When trying to start mirrord exec, tool claims the pod is not found

Steps to Reproduce

 ./target/debug/mirrord exec ping some-service --pod-name some-pod-name -n some-namespace
[WARNING]: DEPRECATED - `--pod-name` is deprecated, consider using `--target instead.
Deprecated since: [28/09/2022] | Scheduled removal: [28/10/2022]
✓ layer initialized
x agent initializing...
thread '<unnamed>' panicked at 'failed to create agent in k8s: mirrord-layer: Kube failed with error `ApiError: pods "some-pod-name" not found: NotFound (ErrorResponse { status: "Failure", message: "pods \"some-pod-name\" not found", reason: "NotFound", code: 404 })`!', mirrord-layer/src/connection.rs:84:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
zsh: IOT instruction (core dumped)  ./target/debug/mirrord exec ping some-service  --pod-name some-pod-name  -n dv-menv0

Backtrace

No response

Relevant Logs

No response

Your operating system and version

Linux

Local process

ping

Local process version

No response

Additional Info

No response

[meowjesty]

Hey @AeroNotix , thanks for reporting this issue.

Could you give us more information about the environment (list of pods with kubectl get pods mainly)?

[AeroNotix]

If you're asking whether the pod exists... yes it does.

[meowjesty]

Wanted to check what's going on, as we're getting different behavior. Maybe it's something related to ping, so ping -V and more info about your Linux would also be helpful.

[aviramha]

Thanks for the report @AeroNotix and sorry for the troubles.
I assume the default namespace used by kubectl is different than the one used with -n and you're specifying a pod and not a deployment/service (you can specify deployment using --target deployment/<deploymentname>).
The only possible reason for it to fail (that I can think of right now) is maybe we have bug with the namespace + --pod-name feature and we'll re-verify it locally to make sure it works.

As a sidenote, ping wouldn't work as you probably expect it to. Ping uses raw socket in order to send the ICMP packets, and we don't support raw sockets at the moment (it'll just do it locally). The DNS resolving of the remote service should be done in a remote manner though.
Is using ping a common use case for you or you just wanted to try out mirrord? We can talk about adding such feature, but we'd love the hear more about the use case so we can generalize it and think if it fits for the project at this moment.

[eyalb181]

We've tested it and there indeed seems to be an issue with the -n argument. While we work on a fix, @AeroNotix could you try using --target/--target-namespace arguments instead? So the command would be:
./target/debug/mirrord exec ping some-service --target pod/some-pod-name -target-namespace some-namespace

[AeroNotix]

Thanks for the responses. Definitely looks like you've reproduced it. After making this issue I did continue on and get things working with the --target and --target-namespace flags.

I did also find that ping doesn't work, which is fine enough, I understand how you are hooking into things and it's entirely reasonable that not all protocols and network mechanisms work. I used ping as a simple test to see if things were working, though as it didn't I just used e.g. curl.

All in all, I can continue evaluating this tool (in comparison to others). Appreciate the quick responses and independent debugging to confirm the issue.

Will keep an eye on this as the workflow for this tool seems so far superior to the other options.

[aviramha]

@AeroNotix Thanks for the finding the bug and reporting.

Will keep an eye on this as the workflow for this tool seems so far superior to the other options.

That's great to hear. Thank you! Should you have any more issues, we're here for you :)

[AeroNotix]

Awesome, thank you all!