borg2: fix rclone repo URLs borg displays

[ThomasWaldmann]

As found by @bjo81:

$ borg2 transfer --repo rclone://fsn:borg2store/mail  --from-borg1 --other-repo /tmp/sb/mail
Enter passphrase for key /tmp/sb/mail: 
Enter passphrase for key ssh://None/./fsn:borg2store/mail: 

Similar:

% echo $BORG_REPO
rclone://pcloud:borg2-test2-rclone
% borg repo-info
Location: ssh://None/./pcloud:borg2-test2-rclone

borg 2.0.0b11

[ThomasWaldmann]

There are multiple issues here:

• borg somehow displays "ssh:" although it should be "rclone:"
• in a URL, // usually prefixes a host (hostname or IP or empty for localhost)
• optionally, this can be followed by :port to use a non-default port, but : is used differently for rclone
• after the //host[:port] part, it usually continues with the absolute path (beginning with a /)
• because there is no absolute path here, borg displays it using our "relative path hack" (path starting with /./ is meant to be relative)
• rclone does not uses a "host", but a "remote" and uses a special syntax: https://rclone.org/docs/#syntax-of-remote-paths

[ThomasWaldmann]

I see these ways to clean this up:

• rclone://remote/path (so we have overall valid url syntax and the remote in place of the host). rclone:///some/local/path would be a local path then (empty host/remote part). That would also fix the "None" in place of the host when borg prints the URL.
• rclone:remote:path to be closer to rclone tool syntax (but this is ambiguous with a local file called "rclone:remote:path" and also maybe not valid URL syntax - that's why borg does not use that syntax for all other protocols)

[ThomasWaldmann]

@ncw ^^

[ThomasWaldmann]

Related: #8372 (URLs with relative paths)

[ncw]

• rclone://remote/path (so we have overall valid url syntax and the remote in place of the host). rclone:///some/local/path would be a local path then (empty host/remote part). That would also fix the "None" in place of the host when borg prints the URL.

So algorithm to get the rclone remote from borg URL would be

• rclone://remote/path -> remote:path
• rclone:///path -> /path

So remove the rclone:// and change the first / to : unless the / is at the start

There are other forms of rclone remote that could be in use, probably :s3,env_auth:bucket would be the most popular - this means use the s3 config available in the environment to access bucket - no rclone config file required.

That starts to make things a bit untidy though

• rclone:///:s3,env_auth:bucket -> :s3,env_auth:bucket ?
• rclone://:s3,env_auth/bucket -> :s3,env_auth:bucket ?

• rclone:remote:path to be closer to rclone tool syntax

I think my preference would be not to munge the rclone remote as this could confuse users when they are using the rclone docs - they would have to translate to borg style.

So I think this is probably a more natural syntax (and is what restic uses)

(but this is ambiguous with a local file called "rclone:remote:path"

You'd prefix with ./ to access a local file called remote:path, so rclone:./remote:path

and also maybe not valid URL syntax - that's why borg does not use that syntax for all other protocols)

Does it need to be valid URL syntax?

[ThomasWaldmann]

@ncw Thanks for the feedback.

OK, looks like we need to specialcase rclone "URLs" in borg and borgstore at some places so we can use rclone:remote:path (or basically rclone:<whatever>).

[AfroThundr3007730]

Standardizing on URL format would be nice for consistency in the borg interface. I don't think it'd be too difficult for users to read a rclone://remote/path URL and understand that it translates to the remote:path normally used in a rclone command.

[ThomasWaldmann]

@AfroThundr3007730 yeah, that was my initial idea also, but if something doesn't fit too well in that scheme://server/path pattern, it gets rather ugly (and might be not a valid URL then anyway).

That's already a problem for our sftp and ssh URLs that need the concept of a relative path (which URLs do not have in their full form). And it's even more a problem for more complex stuff as rclone does it (see the rclone docs about how such a remote spec can be built).

[AfroThundr3007730]

Oh I use rclone to sync all my repos to Wasabi.

I'm thinking we can make the conversion work where the remote would occupy the authority portion of the URL.

wasabi:MyBucket/borg/ -> rclone://wasabi/MyBucket/borg/
s3,env_auth:bucket/   -> rclone://s3,env_auth/bucket/

Reversing that into a rclone remote just means dropping the scheme and replacing the first slash with a colon. I think this would work even with a more complex remote specification like s3,env_auth above.

The authority section of the URL, normally in the form [user[:pass]@]host[:port], could be extended a bit here to account for the remote[,remote_options] that rclone uses. I think this would be manageable since borg would just pass this to rclone for parsing anyway.

While it may not strictly follow the URL syntax, rclone encodes the authority information elsewhere in the config (or environment) anyway, leaving us with just the remote name, which frees us to reuse the authority section to hold it.

Borg doesn't have to worry about things like a port number in this case, or anything else with colons in the authority section, so no ambiguity would arise here. We could even simplify the conversion to just prepending the scheme and leaving the remote specification unmodified:

wasabi:MyBucket/borg/ -> rclone://wasabi:MyBucket/borg/
s3,env_auth:bucket/   -> rclone://s3,env_auth:bucket/

In these latter cases, particularly for S3-like remotes, the bucket looks like it's part of the authority section, but we could do away with interpreting it like a normal URL anyway. The rclone scheme would be the cue to use this alternate parsing of the URL.

Though I prefer the first method, I can see how it'd be more complex to implement properly.

[ThomasWaldmann]

@AfroThundr3007730 what i mean is that: https://rclone.org/docs/#syntax-of-remote-paths

There can be quite complex things like:

• :backend:path/to/dir
• ":http,url='https://example.com':path/to/dir"
• remote,parameter="with""quote",parameter2='with''quote':path/to/dir

So guess it would be way easier to just prefix that with rclone: for usage within borg context and just strip away that rclone: and give the rest to rclone to init the backend.

[AfroThundr3007730]

I see that the connection strings syntax gets quite complex. The prefix solution makes more sense now.