High volume email not woking properly with SES

[AhtishamShahid]

Investigation: Goal Reminder Email Delivery Drop After Migration from Braze to SES

Description:

We are currently migrating the goal reminder email system from Braze to SES (using Django Email via edx-ace). After switching to SES, we observed a 50% drop in email delivery success rates, with no clear errors or logs explaining the issue. Despite thorough reviews of SES and edx-ace logs, the root cause remains unidentified.

Key Actions & Findings

1. Initial Approach:

   • Migrated emails to SES (edx-ace with Django Email backend).
   • Issue: 50% of emails failed to trigger events, with no error logs.
   • No errors related to SES were logged .
   • Added info logs are working as expected.

2. Troubleshooting Steps:

   • Bypassed edx-ace: Sent emails directly via boto3 (AWS SDK).
   • Result: Email events worked as expected, confirming the issue lies within the edx-ace/SES integration.
   • After this implementation, there might be some issue with django-ses package.
   • This implementation caused a drop in the sent emails by around 25%, which is still a significant drop.

Data Analysis

Metric	SES	Braze
Total Logs	99,952	99,734
Avg. Process Time	0.254 seconds	0.074 seconds
Max Time	14.724 seconds	12.802 seconds
Min Time	0.128 seconds	0.047 seconds

• Braze is 70% faster on average (0.180 seconds difference).
• SES shows no signs of throttling (350k weekly emails, no surge-related errors).

Open Questions

1. Should we prioritize further investigation into the edx-ace/SES integration issue?
   • I have no idea how to do that because there are no errors and info logs are all there.
   • AWS analytics are correct according to event data.
   • Some code performance improvements can be done in management command but they are not related to this issue.
2. Would creating a dedicated SES backend for edx-ace resolve the problem, based on the boto3 workaround? -
   • Yes, and we should create the new backend for SES.
   • At the moment we are using django-email -> django-ses flow .

Relevant Links

• Management command to send goal reminder email
• [Boto3 Workaround PR] (fix: bypass edx ace for sending goal reminder email using ses #36148)

Next Steps

We need feedback on whether to:

1. Dig deeper into the root cause of the edx-ace/SES integration issue.
2. Proceed with implementing a new SES backend for edx-ace based on the boto3 workaround.

Please share your thoughts and suggestions!

[ormsbee]

FYI @OmarIthawi, since this relates to edx-ace.

[ormsbee]

Would creating a dedicated SES backend for edx-ace resolve the problem, based on the boto3 workaround?

FWIW, this sounds like the most straightforward thing to me, particularly if it's a small amount of code and it allows us to drop django-ses as a dependency along the way.

[OmarIthawi]

@AhtishamShahid @ormsbee I don't have any insights on this issue. It appears to be a bug with the Django SES package.

If so, converting the send_email_using_ses to a standard ACE plugin would make sense. It sounds like it's a one file.

Here's few things that I would recommend checking for:

• Receiver Name length of dropped emails: it could be that long names are dropped silently
• Receiver Email length of dropped emails: it could be that long emails are dropped silently
• Language used in the name and subject: if all English emails arrive but no Arabic email arrive, that's probably an encoding issue
• Set tracing on the django-ses and see where the emails are being dropped