Reading data from file requires `s3:ListAllMyBuckets` permission

[kxm-kstlr]

Bug Description

I cannot read the contents of an object, even though I have the s3:GetObject, as well as the s3:ListBucket permission for it. Additionally the s3:ListAllMyBuckets permission is needed, otherwise the access is forbidden.

Expected Behavior

A user who is allowed to access an object should be able to access it without additional permissions. In our scenario it is not feasible to give the
ListAllMyBuckets permission.

Environment

• s3fs-nio version: 2.0.0
• OS: Windows 10
• JDK: 8

Additional context

This behavior is caused in S3SeekableByteChannel:85 when calling getBucket. This method tries to get all buckets and then filter for the one needed, thus needing the s3:ListAllMyBuckets permission.

Proposed Solution

Remove the call of getBucket and instead retrieve the name of the bucket directly from the path object by calling getBucketName on it.
Also, I want to open a pull request with the proposed solution shortly.

[steve-todorov]

Hey,

Thanks for reporting this.

The s3:listAllMyBuckets permission is needed to comply with the Java NIO interfaces. With Java NIO you can "list all drives" -- a Windows analogy would be listing C:, D:. Using the Ss3fs-nio proider you can list all buckets.

I don't think we're using the listAllMyBuckets REST API endpoint anywhere else but to list all buckets. However I need to double check the code to be sure. If we are -- then we might need to look into ways to avoid it if possible.

[kxm-kstlr]

Hey,
thanks for the quick reply.

The issue is not related to behavior of listAllMyBuckets, but its use for retrieving data in S3SeekableByteChannel.
My change in #891 makes use of the bucket name in S3Path. This solves my issue of having to give a user the permission s3:ListAllMyBuckets to access the objects of that bucket.

[kxm-kstlr]

I just added a commit to my PR replacing the getBucket call in S3FileSystemProvider with a new method hasBucket. This will also avoid the need for the S3:ListAllMyBuckets permission in this call. Also I think it is more logical to call hasBucket to check if a bucket exists instead of calling getBucket to get all buckets and then filter for one specific bucket to find out if it exists.

[oeph]

@steve-todorov any news on this?

[steve-todorov]

Sorry, I've been a bit busy (and sick) in the past week so I haven't had a chance to look into the PR yet.
Hopefully I'll get some more time during the weekend. The change sounds reasonable but I'll have to check it out.

[oeph]

I hope you are well again.

Could you already find the time to look into this?

[steve-todorov]

@kxm-kstlr Thanks for raising the PR and for your patience.

I finally had some time to look into this in more depth.

Long story short -- the listBuckets endpoint provides metadata information about each bucket such as the createdDate. This matters, because that information is being used when implementing the Java NIO interfaces.

At the time of writing this, there is simply no way to fetch this other than pulling it from the listBuckets endpoint.
Here's some of the relevant documentation:

• Bucket object
• List Buckets response

This is partially why the tests in the CI were failing as well.

Unfortunately we will have to close this PR and the task as "wont fix", simply because of AWS S3 Rest API limitations.

P.S. We're open to revisit this should the AWS S3 API change or if another way becomes available.