[batch] Support uploading local files much like we upload a script file

[danking]

What happened?

The hailtop.batch client uploads a script file when the command becomes too large. This functionality frees users from thinking about the size of their commands. It's a great abstraction!

The client does not provide a mechanism to automatically upload local files or local directories which means that users must messily combine all their code and supporting data into one big file. For example,

local_in = hb.read_input('/path/to/local/script.sh')
local_data = hb.read_input('/path/to/small/local/reference.dat')

j = b.new_job()
j.command(f'bash {local_in} {local_data} {other_job.out_file} > {j.out_file}')

It need not necessarily be hb.read_input, but it does seem like a good way to re-use an extant interface. hailtop.batch should upload those files when it uploads large script files and then download them to the appropriate jobs.

Version

0.2.128

Relevant log output

No response

[danking]

@jmarshall @illusional Please feel free to comment if I didn't fully capture the request!

[cassimons]

Thanks @danking. A feature like this would let us clean up some parts of our pipelines significantly. Our current workarounds leave a bad taste in the mouth whenever I see them.