Skip to content

Feat/monthly reports #508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

landonshumway-ia
Copy link
Collaborator

@landonshumway-ia landonshumway-ia commented Jan 30, 2025

CSG has requested that we implement automated monthly transaction reporting. This required several updates to increase the number of transactions that could be handled within a single report. Previously we were passing in the full csv report string to the email-service lambda payload, which was prone to a 6MB file size limit, now we are storing these reports in a compressed zip format in S3, and passing in the S3 key to the email service so it can pull down the file and attach the zip file to the email directly. This adds the needed infrastructure to support monthly reporting in a scaleable manner.

Requirements List

  • No new requirements, this change is backwards compatible with the current system

Description List

  • Added event bridge rule to trigger transaction reporter lambda on the first day of every month
  • Added transaction reporting S3 bucket to store compiled csv report files in a compressed format
  • Update email-notification-service node lambda to pull down report files from S3
  • Fix formatting for reporting email body text to support markdown

Testing List

  • yarn test:unit:all should run without errors or warnings
  • yarn serve should run without errors or warnings
  • yarn build should run without errors or warnings
  • Python tests added for monthly reporting code verification
  • Verification of monthly report generation for over 100k transactions in sandbox environment.

Closes #493

@landonshumway-ia landonshumway-ia marked this pull request as ready for review January 31, 2025 20:59
Copy link
Collaborator

@jusdino jusdino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really good! A few questions:

- comments describing purpose of .gz files
- remove unneeded cors policies from transaction report bucket
Copy link
Collaborator

@jusdino jusdino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly there...

Copy link
Collaborator

@jusdino jusdino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! @jlkravitz , this is ready for you.

Copy link
Collaborator

@jlkravitz jlkravitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good. A couple questions/comments!

Given that we don't have hard requirements for how admins
will access these reports outside of email messages, this
removes the unnecessary complexity of storing .gz files.
Copy link
Collaborator

@jlkravitz jlkravitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks for clarifying the tests. @isabeleliassen Good to merge.

@isabeleliassen
Copy link
Collaborator

@jlkravitz still some unresolved conversations I think!

@landonshumway-ia
Copy link
Collaborator Author

@isabeleliassen those have now been resolved. This should be ready to go. Thanks.

@isabeleliassen isabeleliassen self-requested a review February 6, 2025 21:20
@isabeleliassen isabeleliassen merged commit 5556649 into csg-org:development Feb 6, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automated Monthly CSV Reports
4 participants