Long-lasting S3 credentials

[angrybrad]

Currently, S3 credentials are short-lived and last for an hours.

This helps with security but comes at the cost of UX.

It would be nice to have the option for longer-lasting S3 credentials.

hat tip to @shennyg

[angrybrad]

I suspect the vast majority use-case for S3 credential access is users syncing assets across environments. Would be curious how much of a pain point this is once #5 is implemented.

[shennyg]

#6 would be a great start, in addition #5 would be even more helpful. Putting it on a schedule would be really great too, we do this for a few clients, every hour/day/etc sync from prod to test buckets.

[timkelty]

Yeah scheduled would be very nice

[timkelty]

@shennyg's original request noted wanting this feature because he may have devs he didn't want to give access to Console.
I think that problem is better solved by #6, so you'd be able to invite a use and just give them permissions to generate temporary credentials.

I think the only way you'd absolutely need long lasting creds, is if you had to use them in another app.

[shennyg]

Yes agree #6 is a better solution.

[matthewstick]

I keep having timeouts on the S3 sync, because the file uploads on a site is larger. I need longer than an hour. Would be nice if creds were maybe 4 hours default. Or 24 hours. 1 hours is a bit extreme.

[jasonmccallister]

Hi @matthewstick, this is a limitation forced by AWS that we can’t extend. We are working on a feature to make this available in a UI but there are some details we need to flush out first.

I’ll make sure we talk about this at our next Cloud meeting.

[michtio]

I'd also advocate for having at least one environment where we can choose for "long-lasting credentials". Since Cloud aims on the slightly larger projects, these projects are mostly ran by a multidisciplinary team.

Especially during the development phase, we would love to keep people in sync with the assets. We run a little git hook, that on pull simply runs ./craft index-assets/all so assets stay in sync, we don't want to constantly pull in a lot of assets on our local environment, this usually ends up in things being completely out of sync.

this is less related to images, but more to documents, we have quite some projects where "File Managers" are a big thing, this is all synced with e.g. a Typesense, if those assets get "out of sync" our developers simply end with different result sets, and the test usually becomes flagged as "failed" while this is a false positive.

Not all people in our team are confident working with the CLI or aws-cli to sync down/up for that matter. The current workflow actually works contra-productive for our team.

For now we install the S3 plugin, and run our own development bucket, but this means it needs to be setup, and stay alive, bringing an additional cost for us, but on the go-live we can't forget to sync, so it adds overhead to the team.

On the other hand when a Support Requests gets into the team, and there is a bug with the assets, we need to be able to mimic the production environment, this usually entails "sync production to dev" and go from there, but we do want to use that "state" to test on, testing also means uploading files and triggering events and sorts, we feel less confident (even tho it's also S3) to know these tests actually don't run on Craft Cloud, but on a side "S3"

We also have a QA process that runs, everyone knows how to spin up DDEV locally (QA platform actually triggers that command, so no manual CLI work is necessary) but QA'ers that need to test the whole day, also need to test "Do file uploads work, do they behave as expected" and these teams are running into error when trying to upload with the Craft Cloud credentials, as we can't even upload files locally into Craft Cloud.

Only the viewing rights don't make a lot of sense, as this means switching (and/or removing) .env variables, which we don't want to do basically.

So long-lastig credentials for multidisciplinary teams make sense, and is actually a must in most of our cases.

We can approach this in a few ways:

Make it possible to choose (put the responsibility into the dev/devops) if we want long-lastig or short rotating credentials, don't allow this on production obviously, there the rotating credentials actually add an extra security layer on top.

Or besides offering the 2/3 environments, offer a "dev assets" bucket, that we can easily hook into, that always have long-lasting credentials. This could be optionally hooked into, or could be an add-on on top of cloud for teams that actually need it (opt-in through Craft Console).

[michtio]

Obviously the above, even tho a dev-bucket or another option is viable, a "sync between" environments feature would finish it all of ;)

[jamesmacwhite]

I'd advocate for long lasting S3 credentials for being able to backup assets through another service. However if #66 is implemented, then this becomes less of an issue for that context, but let's say for example, you want to snapshot your assets on a weekly basis or regularly enough that can you can just pull down a gzip of them and extract locally, rather than having to use the short life credentials i.e export env vars and run AWS CLI.