The multiline script fails to be uploaded to server (Windows)

[byme8]

I have the following playbook:

user: adminuser                       # default ssh user. Can be overridden by -u flag or by inventory or host definition
ssh_key: ~/.ssh/id_rsa               # ssh key
ssh_shell: /bin/bash                # shell to use for remote ssh execution, default is /bin/sh
inventory: ./css_server_setup.invertory.yml  # default inventory file. Can be overridden by --inventory flag

tasks:
  - name: copy server configs to remote server
    commands:
      - name: Update the APT package cache
        script: sudo apt-get update

      - name: Install required dependencies
        script: sudo apt-get install -y nginx wget

      - name: Install 32-bit libraries
        script: sudo apt-get install -y gcc-multilib lib32stdc++6 lib32gcc-s1

      - name: Download, extract, give premissions SteamCMD
        script: |
          mkdir -p ~/steamcmd
          wget https://steamcdn-a.akamaihd.net/client/installer/steamcmd_linux.tar.gz -O ~/steamcmd/steamcmd_linux.tar.gz
          tar -xvzf ~/steamcmd/steamcmd_linux.tar.gz -C ~/steamcmd
          chmod +x ~/steamcmd/steamcmd.sh
        condition: ! test -d ~/steamcmd

It fails with the next logs:

[css_server 13.80.253.217:22] run task "copy server configs to remote server", commands: 15
[css_server 13.80.253.217:22] completed command "Update the APT package cache" {script: /bin/sh -c 'sudo apt-get update'} (1.621s)
[css_server 13.80.253.217:22] completed command "Install required dependencies" {script: /bin/sh -c 'sudo apt-get install -y nginx wget'} (468ms)
[css_server 13.80.253.217:22] completed command "Install 32-bit libraries" {script: /bin/sh -c 'sudo apt-get install -y gcc-multilib lib32stdc++6 lib32gcc-s1'} (466ms)
[0] failed command "Download, extract, give premissions SteamCMD" on host <server ip>:22 (css_server): can't prepare script on <server ip>:22: can't upload script to <server ip>:22: failed to create remote file: file does not exist

If I rewrite the command into multiple stages:

      - name: Download, extract, give premissions SteamCMD
        script: mkdir -p ~/steamcmd
        condition: ! test -d ~/steamcmd
        
      - name: Download, extract, give premissions SteamCMD
        script: wget https://steamcdn-a.akamaihd.net/client/installer/steamcmd_linux.tar.gz -O ~/steamcmd/steamcmd_linux.tar.gz
        condition: ! test -d ~/steamcmd
        
      - name: Download, extract, give premissions SteamCMD
        script: tar -xvzf ~/steamcmd/steamcmd_linux.tar.gz -C ~/steamcmd
        condition: ! test -d ~/steamcmd
        
      - name: Download, extract, give premissions SteamCMD
        script: chmod +x ~/steamcmd/steamcmd.sh
        condition: ! test -d ~/steamcmd

it starts to work fine.

I suppose it is some weird interaction between the spot, golang on windows machine and the target linux machine because on the target machine in ~/ folder, I can see the following:

adminuser@vm:~$ ls
 Steam  '\tmp\.spot-1115342305558551936'  '\tmp\.spot-2664757902190471168'  '\tmp\.spot-6584203395433061376'  '\tmp\.spot-8180703307877217280'   css_server   steamcmd

Something like that, when the folder names are concatenated together, I saw when working on another project on Linux and creating folders with long names like that '/tmp/random stuff/another random stuff' in one attempt. I suppose the solution would be to use some "official way" to create temp folders in golang or create multiple folders in steps.

Later on, I may have a look at the source code of the spot to figure out the issue myself.

OS cat /etc/*-release

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

[umputun]

spot is already trying to enforce unix-slashes on remote hosts, but it is likely partial. I have found a few places related to the remote operations using things like filepath.Join, which is "slash-sensitive" . All those places I have found and also some others dealing with path were changed to enforce a proper / separator.

If you can build a binary from enfoce-unix-slash branch and try it, this would be nice.

I don't have a win box handy to try it for real, but if needed should be able to make one; this just will take time.

[byme8]

Cloned spot from enfoce-unix-slash.
Build it via make build.
Renamed <repo root>./.bin/spot into <repo root>./.bin/spot.exe.
Added it to PATH and got the same result.

E:\Develop\servers\cs source [main +7 ~6 -45 !]> spot /i .\css_server_setup.invertory.yml /p .\css_server_setup.spot.yml
spot
[css_server <server ip>:22] run task "copy server configs to remote server", commands: 15
[css_server <server ip>:22] completed command "Update the APT package cache" {script: /bin/sh -c 'sudo apt-get update'} (18.034s)
[css_server <server ip>:22] completed command "Install required dependencies" {script: /bin/sh -c 'sudo apt-get install -y nginx wget'} (5.212s)
[css_server <server ip>:22] completed command "Install 32-bit libraries" {script: /bin/sh -c 'sudo apt-get install -y gcc-multilib lib32stdc++6 lib32gcc-s1'} (5.073s)
failed, can't run task "copy server configs to remote server" for target "default": 1 error(s) occurred:
   [0] failed command "Download, extract, give premissions SteamCMD" on host <server ip>:22 (css_server): can't prepare script on <server ip>:22: can't upload script to <server ip>:22: failed to create remote file: file does not exist
``

[byme8]

Looks like the cause of the issue is here

The filepath.Dir(req.remoteFile) transforms back with 'windows' semantics here.

[umputun]

Good catch. I have changed all I could in remote.go to use slash-insensitive functions for the path operations (i.e. forced to use /). Give it another try pls

[byme8]

It uses correct slashes in this specific case, but now it fails in a different place.

There is a new temp file created. If we try to get it's name tmp.Name() it would contain the windows slashes. Then, when we need to get a relative path like that

It goes nuts because the path was not able to handle window slashes.
I fixed it by creating my own UnixName function and manually replacing the slashes.

func UnixName(file *os.File) string {
    return strings.ReplaceAll(file.Name(), "\\", "/")
}

It started to copy and upload files as expected.

However, during the devtest, I found another issue that breaks multi-line script execution.
When a new temp file for the script is created, the new permissions are set to it:

// make the script executable locally, upload preserves the permissions
if err = os.Chmod(tmp.Name(), 0o700); err != nil { // nolint
   return "", "", nil, ec.errorFmt("can't chmod temporary file: %w", err)
}

Later on, after upload, those permissions are copied to the remote host:

if err = remoteFh.Chmod(inpFi.Mode().Perm()); err != nil {
   return fmt.Errorf("failed to set permissions on remote file: %v", err)
}

As far as I know, on Windows, there is no way to set execute permissions to any file. As a result, during the copying, the remote file doesn't get execute permissions because the Windows file doesn't have them in the beginning,
So, I decided to hardcode the permissions to 0o700 here as well, and it started to execute the script on remote machine as expected.

I suppose I can dig more in it and make my own pull request for all of it, From what I see the affected area would be huge and it can break lots of things. So, I am not sure that it is a great idea.

Meanwhile, I noticed yet another issue. When the cond script is executed, sometimes its results are ignored, and the main script is executed every time. I am still trying to understand why it happens. Maybe it is an issue with my cond, but it is pretty simple ! test -d ~/css_server, and when I execute it manually in the console, it works as expected. I have spent a little time investigating it. Have you noticed something like that before?

[umputun]

I have tried to set up dev env on windows VM, and gave up. This thing doesn't seem to be compatible with testcontainers-go needed for tests.

However, my question is - why do we even bother with any of this? Why not run spot on win from wsl2?

[byme8]

why do we even bother with any of this? Why not run spot on win from wsl2?

I am already doing it for Ansible. Then I saw that Spot has Windows binary and decided to try. I guess it would be better to remove the Windows binary from the release page if there is no way to use it