OOM kill with multiple RAM limits

[Fertigwaren]

Hey,
first of all - we recently swapped from mailhog to mailpit and are pretty happy with it, thank you for your work!

Sadly we are running into an oom kill error.
We deploy mailpit to our k8s cluster while using the official latest mailpit docker image.
The deployment isn't doing anything fancy, simply mounting a persistent storage for the database file and setting up ingress.

We are able to receive simple mails without any issues but as soon as we include some attachments the application crashes with status code 137 (oom kill).

terminated
Reason:Reason: OOMKilled - exit code: 137

To be a bit more precise: the attachments are mostly images and pdf's ranging from 1mb to 5mb up to 5 attachments per mail.
The mails are getting sent through our laravel backend.

After encountering it the first time i tried to raise the RAM limits for the container.
First from 500mb to 1gb following with a raise to 2gb.
Even with 2gb of RAM the container kept running out of memory when receiving the mails mentioned above.
When idling it pretty much uses no memory at all.

After crashing mailpit is able to display the mail and also knows about the attachments and its sizes but can't display the image or document itself.
Mailpit does also crash when clicking these broken attachments, after that the attachments are usually healthy again which is even more baffling to me.

We use the following environment variables:

  - name: MP_MAX_MESSAGES
    value: '200'
  - name: MP_SMTP_AUTH_ACCEPT_ANY
    value: '1'
  - name: MP_SMTP_AUTH_ALLOW_INSECURE
    value: '1'
  - name: MP_DATABASE
    value: '/data/mailpit_local_database.db'
  - name: MP_VERBOSE
    value: 'false'

Do you guys see what im missing? Im pretty much clueless at this point.
Thanks!

[axllent]

@Fertigwaren To be honest I've never encountered an OOM error with Mailpit, and it should easily handle large attachments. I have a database that is around 4GB in size, and contains emails with 30+MB attachments, and memory remains lower than 150MB.

What I will need to check though is if I limit ram, as I'm not sure if there are any sort sudden spikes with large attachments (I need to get back to you on this in the next day or so).

But... I have a suspicion it may relate to your/data mount. Can you please tell me what that mount is? If it's a network mount (eg: NFS) then we may have the cause (https://mailpit.axllent.org/docs/configuration/email-storage/).

[axllent]

To follow on from my previous message - I believe you did in fact have hit memory limits (and nothing to do with the database store). To put things into context, Mailpit, unlike a regular mail server, doesn't just store a message, it processes and compresses the message before storing it in a database ~ all of which is far more memory intensive. This is usually very light weight, however as large attachments are introduced, so does the temporary memory requirements to process and store these. Go (which Mailpit is written in) has a built-in garbage collector (to free unused memory), however when multiple messages are received in sequence, the overall memory usage can increase quite significantly until the memory is freed again.

An email containing a single 27MB PDF attachment is approximately 39MB in size (ie: email encoding). This is received by Mailpit's SMTP server, and then:

1. The message is parsed and broken down to extract information from the headers, information about the attachments, as well as other information
2. The raw email is then compressed (ZSTD) to minimize database storage
3. The compressed email is then converted to a hexadecimal string for the SQL query to insert into the database
4. The message is stored in the database

After further debugging I can conclude that steps 2 & 3 are the main consumers of RAM in this process. Step 2 is what it is and won't change - this makes a significant difference to the size and performance of the database, and ZSTD is one of the least memory-intensive, fast and good compression techniques available (more below). Step 5, however, is currently only needed due to the optional rqlite database connection (storing binary data in the database needs to be manually converted to a SQL query using hexadecimal encoding).

I have found that conditionally doing step 3 (ie: only when rqlite is used) does have an impact on the memory usage, as converting a compressed binary to a hexadecimal string, and then converting that string into a SQL query uses a fair bit of additional RAM.

I am not saying that this completely resolve your issue, but it should help somewhat. I was able to to run three SMTP threads pumping the 40MB message into an instance of Mailpit which was limited to 1.5GB without a crash (300 messages), something I was not able to do prior to my change. It is very difficult to actually gauge the exact difference, but I think it makes an approximate 20% difference. This can of course differ depending on the size of the message, number of attachments, and the type of attachments.

The other thing I have done is to reduce the ZSTD compression from the default setting (3) to the fastest (1), meaning the fastest compression with the least memory. With this setting my test machine barely got over 1GB of ram while processing 300x 40MB messages (10.9GB database size). Using the default ZSTD compression (3) I saw it spike regularly to about 1.2GB with the same test, and which also resulted in approximately the same database size (so the additional compression did not actually make much difference to size).

I am aware my tests are not very scientific, but I am noticing a marked improvement when dealing with large messages like these.

I have just pushed the changes to the develop branch, meaning it is now included in the axllent/mailpit:edge docker image. Would you please test that image to see if this helps your OOM issue? You will likely still need 1.5-2GB for your use-case (to be sure), but these changes should help keep it well within those thresholds.

[Fertigwaren]

Thanks for the insanely detailed answer, really appreciate it.
I did some testing with the edge branch image but it sadly wasn't the magical fix i was hoping for.

Mailpit is not crashing anymore after increasing my limit to about 4GB RAM.
Obviously we won't be running the 4GB on a daily basis but your first comment already gave me another hint.

For some reason, my brain didn't register that our mounted AWS EBS storage is actually network-based...
I'm going to setup a local sqlite db in the same namespace and check the RAM usage afterwards.
I'll probably also check the difference in usage between edge and latest.

Going to update this thread with my findings.

[axllent]

That is unfortunate (that it didn't resolve your issue), I was hoping to at least reduce the memory enough while processing large emails given your estimated requirements. I guess it also depends on several factors I mentioned earlier, plus I don't really know know the volume you're sending & processing at the point where it maxes out.

I've been experimenting more to see if there are other optimisations I can make, but alas, I can't find any obvious wins. It definitely revolves around the message parsing, editing (adding/editing headers), and of course compression and storage - however those are all necessary parts of the application.

Networked filesystems are an issue though, especially since Mailpit is currently (hard) configured to use WAL (wite-ahead logging) which improves DB performance a lot, not to mention the other benefits listed on that page. You are not the first person to be caught out by the networked filesystem limitation though, so I am contemplating adding a configuration option to disable WAL. That should technically allow a network filesystem to be used, although I'm not sure that would be a good idea - SQLite isn't an ideal database for that scenario (i/o latency), and the risk is also that users may try open the same database via multiple instances, complain about poor performance, or experience other issues (eg: network dropouts etc) is also a factor.

Let me know your what your find once you have had a chance to test, and then we can discuss more 👍

[axllent]

I have been doing some more accurate benchmarking to get a better understanding of the potential affects of the ZSTD compression on RAM, trialling three different scenarios: without compression (at all), fastest compression (level 1), and standard compression (level 3). Just ignore the level 22 tests, I added them later for reference. The results are quite interesting.

For the main test I sent 100x 37.5 MB raw emails containing a single 27.4MB PDF attachment to Mailpit to tracking the memory usage. Once complete I also clicked on about 10 different emails to render them in the web UI, meaning Mailpit is then extracting the raw message from the database, plus I also opened the attachment.

In the following results, level 0 means no compression:

Compression	Time to ingest	DB size	RAM after ingestion	RAM after opening
0 (disabled)	1m16.883s	3.7GB	270MB	270MB
1 (fastest)	1m14.900s	3.6GB	570MB	1GB
3 (default)	1m14.455s	3.6GB	714MB	1.4GB
22 (max)	7m22.685s	2.7GB	1,515MB	1.98GB

The interesting points to note here are:

• Using no compression takes slightly longer to ingest into Mailpit compared to compressed (including the time to compress), most likely because of the larger SQL queries. It's a very difficult thing to prove though, and in the end the difference is negligible (a combined total of 2 seconds for all 100 messages).
• Opening uncompressed messages & attachments did not increase the currently reserved memory usage at all
• There was not much database size difference when comparing the three, however that is likely due do with fact that the PDF is already optimized, so I wasn't expecting much for this test case vs: something like large text-heavy emails or those with compressible attachments.
• RAM usage doubles with the fastest ZSTD compression, triples with level 3 (default).
• Relative RAM usage will double when compressed messages are opened.

Please note that the RAM values I provided are almost constant ceiling values (it went up to those values and remained there for the duration of the respective part of each test). Go's internal garbage collector (which automatically frees up unused/unreserved memory) will ultimately free up memory at some point, however that's not your issue - your issue is the RAM ceiling reaching your VPS limits ~ which these tests investigate.

I did another similar test with 2,500 regular/normal emails - all with varying sizes, many with small attachments, but also many with just text & html content. Here the compression definitely made a difference to the database size (compared to no compression):

Compression	Time to ingest	DB size	RAM after ingest
0	16.1s	147.5MB	50MB
1	14.3s	99.5 MB	290MB
3	14.6s	97.1MB	485MB
22	22.4s	88.6MB	1.2GB

Note that I didn't bother opening any for this test as they are tiny in comparison, and while I'm sure if I clicked enough it would have a minor impact, the garbage collector would also kick in and would recycle the memory anyway (so it may even go down). This test was trying to test a "normal" use situation and was added for reference.

So with this in mind, I think it's clear I do need to add an option to disable compression entirely for use-cases such as yours. The database size isn't an issue, and it should definitely greatly reduce your memory requirements. This also ties in to #448, although I'm skeptical how much difference this will make to CPU usage because ZSTD compression is fast, and what one is trading off in compression is likely replaced with database CPU, read & writes.

*Edit: Just out of curiosity and for future reference I added level 22 (max) compression levels for reference.

[Fertigwaren]

Hey, I finally found some time for testing and to get back to you.

As you suggested in your first reply, I switched from using persistent network storage for SQLite to an rqlite instance running in the same Kubernetes namespace. I used 35mb of attachments split into 2 and 3 files for my tests.

Since allocating 2GB of RAM to Mailpit, it hasn’t crashed. However, anything below that still results in an OOM kill. I also tested both the latest and edge builds, but the difference in my specific case was minimal.

The edge branch consumed right around 1.6gb of ram during spikes.

In comparison the latest branch spiked at around 1.67gb.

To be fair 2GB is still quite a lot but it's fine in our environment and i wouldn't mind leaving it running like that.

I'd also be down to test out different compression levels if you plan to implement them as a configurable runtime option.
If there is anything else you want me test out i'm more than willing to, otherwise you can close this issue.

Thanks again for your help!

[axllent]

Thank you. Based on your feedback I believe the real difference between the two was that SQLite was running remotely, so the part of the RAM being saved was because the database was remote, and part of it the faster compression. It may have been the difference needed to push it over the "memory limit edge".....

I am still doing various tests, but once I am complete there will be two changes:

1. Message compression will be set to the fastest zstd compression (1) instead of 3 (as per the current edge build). Compression is enabled by default (as it was already), but the compression gain between levels 1 and 3 isn't significant enough for emails in general but does consume less RAM with 1.
2. There will be a new option to disable message compression altogether (for new messages). This will have by far the largest impact on your RAM, but will mean your database is somewhat bigger - basically this is the option you will want.

I'm still considering whether the message compression level will be a user-configurable value (with 0 turning it off), or a hardcoded 1 with a separate option to disable compression - I'm leaning now towards the configurable value as this provides more flexibility, and either way there is the addition of at least one more command flag (I don't like having any unnecessary ones in there).

I am also investigating a separate option to disable WAL journaling for local databases (ie: not rqlite). WAL (Write Ahead Logging) greatly improves SQLite performance and crash recovery, but as a trade-off does not work with network filesystems. That's not to say that SQLite is an ideal database for any network filesystem as all read & writes are prone to network latency, however it should technically work I read. mileage may vary, so I'm very hesitant to say "it will work flawlessly with NFS".

[axllent]

I have just released v1.23.0 which contains two separate options that relate to your issue.

1. The ability to disable message compression in the database (--compression 0 flag or env MP_COMPRESSION=0) - this should definitely resolve your OOM issues as the benchmarks have shown. You will end up with a larger database, however that shouldn't be an issue at all (unless you're planning to store 4 billion emails :-)). Please note that this setting only impacts new messages after you apply the setting, not those already stored in the database.
2. The ability to disable Write-Ahead Logging (WAL) support in SQLite (--disable-wal flag or env MP_DISABLE_WAL=true) . WAL improves database performance, however also makes SQLite incompatible with network file systems. This option should allow you to use your original NFS mounted volumes for storage. Please note that storing SQLite databases over the network is always prone to latency and other issues (see caveats and considerations).

Please let me know how it goes as I'd love to see how it compares to your previous benchmarks.