[BUG] can't install repository-s3 plugin in OpenSearch 1.2.3

[arnitolog]

I am having the issue to install repository-s3 plugin in OpenSearch 1.2.3 with the following error:
[opensearch@65554c8b2581 ~]$ bin/opensearch-plugin install repository-s3 --batch
-> Installing repository-s3
-> Downloading repository-s3 from opensearch
-> Failed installing repository-s3
-> Rolling back repository-s3
-> Rolled back repository-s3
ERROR: Invalid checksum file at https://artifacts.opensearch.org/releases/plugins/repository-s3/1.2.3/repository-s3-1.2.3.zip.sha512

[uncycler]

It affects other plugins as well.

[dblock]

@peterzhuamazon

[sandervandegeijn]

This is blocking me from upgrading to 1.2.3, with the whole log4j debacle I'd rather patch asap. :)

[peterzhuamazon]

We have identify the issue now:

The issue being that opensearch core is checking 2 spaces between the checksum and filename within .sha512.

Core requires 2spaces ./bin/opensearch-plugin install repository-s3

9601a0183412790c75103046f7024ad4a90c46fb0311726eff0d5d0306ebc82d9736438e37e038715e3f137d74f8d735f9009e994edf3d83fc22121d8569eb1c  repository-s3-1.2.3.zip

However as long as you have one space the check locally with sha512sum -c will success:

9601a0183412790c75103046f7024ad4a90c46fb0311726eff0d5d0306ebc82d9736438e37e038715e3f137d74f8d735f9009e994edf3d83fc22121d8569eb1c repository-s3-1.2.3.zip

Since we are using a new workflow for 1.2.3 signing and checksum generation, we are using 1space this time, thus cause opensearch-plugin to refuse to install the plugin by failing checksum verification.

@CEHENKLE @dblock I have not checked the code in details, but seems like it is related to this:
https://github.com/opensearch-project/OpenSearch/blob/aecc7bd005544d79c6524960a32984926ba6718a/distribution/tools/plugin-cli/src/main/java/org/opensearch/plugins/InstallPluginCommand.java#L541-L572

We will fix this with another deploy for sha512 only.

Thanks.

[peterzhuamazon]

More digging, this line specifically cause the issue:
https://github.com/opensearch-project/OpenSearch/blob/aecc7bd005544d79c6524960a32984926ba6718a/distribution/tools/plugin-cli/src/main/java/org/opensearch/plugins/InstallPluginCommand.java#L550

                final String[] fields = checksumLine.split(" {2}");

When having 1 space between checksumsha and filename, it will return an array of length 1, with the 1st entry being the entire input, not splitting correctly.

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
	at MyClass.main(MyClass.java:14)

It is exactly matching the space for 2 times in the regex " {2}", not sure of the reason of this as you can simply use " " to match all the possible spaces in between, possibly to deal with issues that some files have space within the filename?

[peterzhuamazon]

We have fixed the issue now, sorry about inconvinience @arnitolog @uncycler @ict-one-nl and thanks for reporting this bug to us.

$ ./opensearch-plugin install repository-s3
-> Installing repository-s3
-> Downloading repository-s3 from opensearch
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
'-> Installed repository-s3 with folder name repository-s3

[peterzhuamazon]

Will leave this issue open until more discussion on this.

Thanks.

[sandervandegeijn]

Great thanks!

[arnitolog]

thanks a lot!

[sandervandegeijn]

Fix confirmed

[dblock]

@peterzhuamazon The 2 spaces is/was by design. From the comment:

The supported format of the SHA-1 files is a single-line file containing the SHA-1. 
The supported format of the SHA-512 files is a single-line file containing the SHA-512 and the filename, 
separated by two spaces.

It's a convention like any other, and introducing a change is just going to make things more difficult, and this issue is now resolved, so I am going to close it. The tools need to generate something compatible with the existing code. However, if you feel strongly that this 2 spaces business is a bad idea, open an issue in OpenSearch and we can discuss it there?