feat[file]: add s3 functionality

[m-mattia-m]

• Configure S3 endpoint, bucket, base path, credentials
• Upload file to S3
• Get file from S3
• delete file from S3

PS: Please bear in mind that I've never used NestJS before, so I'm not sure if everything is best practice, but it works. Don't hesitate to tell me what I should change or update.

[stonith404]

Thank you @m-mattia-m :) I’ll try to review the PR as soon as I can. Since I’ll be on vacation next week and have a lot going on, it may take me some time to review and test it fully.

[stonith404]

Sorry for my late response, I have much going on.

I noticed an issue with how chunked file uploads are being handled. Currently, only the first chunk of a file is successfully uploaded to S3. Pingvin Share relies on chunked uploads, where the frontend splits a file into multiple parts and uploads each part sequentially.

With your current implementation, only the initial chunk is uploaded. You can verify this by trying to upload a file larger than the chunk size (default is 10 MB).

[stonith404]

Additionally, I would suggest to create a separate S3 file service as the methods in the file service get a bit too complicated when both S3 and local file storage are handled.

[m-mattia-m]

No worries :)

I can see what you mean by "a bit too complicated" :D. I will create a separate service for the S3 functionality. Yes you are right, large files are failing during the upload. I've never tested it with large files which I should have done. I will implement this feature in a few weeks but I have also much going on :).

[stonith404]

@m-mattia-m Sure, take your time :)

[m-mattia-m]

@stonith404 FYI: I fixed the chunking problem and split the file service into a local and a S3 service. Consider that I need to refactor these changes, but I will do that in a few days.
If you have time, I would appreciate it when you could test the changes if you can find any issues. :)

[m-mattia-m]

@stonith404 I updated the code a bit, and I hope it's fine. Don't hesitate to tell me what I should change or update. :)

[stonith404]

Thanks for taking the time to implement my feedback :)

Most things are now working as expected, but I’ve noticed a few remaining issues. Please take a look at my additional comments.

[m-mattia-m]

I noticed two things that I will change/implement in a few weeks when I have more time.

• When you try to open the preview for a PDF that is larger than 10Mb (chunk size), an error occurs (it downloads the file (without the correct name and type extension) instead of displaying the content in the PDF viewer).
• If you change the storage provider during runtime, the backend searches in the new location where it can't find these files that were uploaded before the storage change, so I will add a location attribute in the database: s3, local to each file.

[stonith404]

Thanks for the changes!

Sure, take your time. I think it would be sufficient to add the provider information to the share, instead of adding it to every file.

When these two issues are fixed it should finally be ready to merge :)

[m-mattia-m]

I fixed the two issues I mentioned before. Now everything should work :).

[stonith404]

I made some minor changes like formatting the code, adding the Prisma migration and removing storageProvider from the dto as the client doesn't need this property.

For me it looks good now. Should I merge it?

[m-mattia-m]

Thank you and that would be great. :)

[stonith404]

Great, thanks again for your contribution!

[GitNees]

hi @m-mattia-m I just filed a bug report as I can use S3 buckets and upload, but impossible to delete the files from the shares.

[m-mattia-m]

For anyone's information: Bug Report: S3 shares are not deleted on S3 compatible buckets #705