Illegal Instruction (Core Dump) #110
Comments
|
Looks like out of memory failure at first glance. I guess your account has got a lot of emails? The app by design keeps the entire Does it fail only if the Does it fail soon after app starting or after a while? Can you try running the app in console this way for now: It will print extended logs to I'm going to enable configurable |
|
Can you also try the app with another account that doesn't have a lot of emails, if you have such account? |
|
To recap, in your case the app fails after 189.19 seconds (~ 3 minutes) since default node memory limit (~ 1GB) is to low to hold the entire emails database. Can you try running the app using the following way and let me know how it goes? This will set memory limit to 4GB for the main process which holds the emails database. If that won't be enough I will be ready to provide a solution for increasing memory limit for the renderer processes too. |
|
Besides I didn't initially notice this stuff |
|
@internalkernel can you try this build? This build got improved local database serialization. If it still fails please run the app the following way and send log file to the email specified here in Plus change |
|
Sorry about the slow reply here... and thanks for your attention! I'll try to address everything in order. Indeed it works fine when Local Store is disabled. Running the app with:
results in a out of memory error:
The second suggestion to increase max_memory helped:
It ran for a good hour or two, indexed many of the messages and then crashed with the same error as before, i.e. The new build you linked (thanks by the way), unfortunately didn't make it past the login window initially. I deleted all my settings for the app and installed clean and it is currently working. I'll update with more progress later... Thanks! |
The provided build is supposed to be backward compatible (means no breaking changes have been made). Did you happen to keep the log file with the failed login window action? |
|
I did, it was brief... In hindsight, it might have been parsing the current database? I did keep the old profile, I can always restore and try to run it again.
2.3.3 is running solid, about a 1/3rd of the emails are indexed so far... Thanks! |
|
Would be great to see the lines after these (it's begining of the signing process)
It does parse the database used/created by the previous app version, but I did care about backward compatibility, see read method implementation. Besides that new build loads the database serialized in JSON from the buffer using https://www.npmjs.com/package/oboe instead of just parsing a long string by calling JSON.parse. Anyway, I'm going to double check the backward compatibility before releasing a new version. |
|
That was the end of the log, the app stalled at that point... I gave it about a minute or two, I could let it sit longer if that's helpful... |
Please do if that's possible. I guess https://www.npmjs.com/package/oboe could potentially parse the data slower than regular By the way, what is your |
|
Adjusted timeouts to allow for a 10 min window, it crashed after about 3 min... The auto-login feature is turned on. Link to the full log: https://pastebin.com/zwvkdDF0 I then tried to go back to the newer profile, but the app crashed almost immediately upon launching. Here's that log: https://pastebin.com/fg2afAre The good news is that I haven't seen another crash, not since running the 2.3.3 build and increasing max_memory to 3072. Database.bin was about 500MBs. both profiles were comparable. |
|
Thanks for the thoughtful report, it helps to make the app work better.
I see that database file got loaded successfully in 141911ms (~142 sesonds ~ 2.4 minutes which is very slow, hope new serialization in 2.3.3 works faster) but I will need to look into the why the sign-in process finally failed:
Can you try disabling the full-text search feature and give it another try (either via UI or manually in If the bootstrap/initial fetch got completed using the 2.3.3 version (icon stopped blinking in green) does the app work well after the restart with full-text search feature enabled? Asking since the app sends almost the entire database to the indexing process on start and so the interaction between processes would be blocked in the same way.
Thanks for this info. I also see that you got 14165 emails saved in the local database ( Going to try filling my database with fake data making its size close to 14165 emails and debug the stuff then. |
No need to do this as I tested full-text search bootstrapping on 20k mails database and noticed that issue is in something else. So going to provide updated build. |
|
@internalkernel I just have updated the build here. The build comes with the following changes:
|
|
Thanks again for the new build and happy to help here... Do I win a prize for having the most emails? Beyond a bloated database... :) Gotta love project management, 90% of what I do feels like babysitting. After installing the updated build, I tried loading the profile from 2.3.2. The app sits on the "Loading Database for 300 seconds" message, then crashed after about 3-4 min. Not sure if that's helpful, since the profile database was part of a previous crash. And, full log from that run: https://pastebin.com/F1uj579u I'll try the new build on a fresh profile, the other 2.3.3 build would index everything but crash upon reopening the database. I'll see if that has changed this time around as well. In case it's worth mentioning, I noticed that the message count in the Database view never exceeded 6522 emails - even though the Log output is counting upwards of 15K. Thanks again for your time and effort here... it is much appreciated! |
|
Unfortunately, there are no errors in this log which makes it harder to figure the cause of the crash. But it's clear that a database file has been successfully loaded and the app then switched to the accounts view screen (next screen after the login form). So 2.3.3 is backward compatible with previous versions as database got loaded. There is a need to focus on making 2.3.3 work with a large loaded database. For now, I can just suggest disabling the full-text search feature.
Do you observe 6522 value on |
|
Disabling Full Text Search and everything runs smooth... 2.3.3 runs fine once that's disabled and I tried the old 2.3.2 profile with Full Text Search disabled and that launched fine after importing the database as well. The 6522 was on the All Mail Folder, but only in 2.3.2 during the first index (before the crash). It shows the complete email list in 2.3.3. with the correct count. Thanks once again! |
|
Ok, so I will be improving full-text search bootstrapping on large datasets. Even though 2.3.3 is not released yet it's better to use it on large datasets as it has already got some improvements. |
@internalkernel can you enable full-text search feature and then run updated build? If still fails try changing I tested the updated build on ~50k emails database and it worked well. The previous build was failing on the same database. |
|
On the new build, with Full Text Search enabled... I'm getting the Java Heap out of memory errors, it does take awhile to get there though - at the moment max-old-space is set to 4096, it takes close to 6-8 hours before it hits that point. As of this morning, I disabled Full Text Search to see if it makes a difference in the out of memory errors. All other aspects appear to be working as intended, the database view shows all emails and the interface is quick / responsive. |
|
Would be great to monitor the app for a while so we could detect which process eagerly and incrementally eating the memory. Such monitoring can be accomplished by running the following commands:
Then after the crash please send both log files to the email specified here in the
My guess is that it's either The most recent build is uploaded here. It doesn't include new fixes yet but only some dependencies and tutanota's webclient updating. |
|
Installed the updated build and launched as you suggested (attached logs in the email sent). In the last run, I only used the Database view once to check the amount of indexed emails. I haven't used it to extensively yet, if that makes a difference with memory usage. |
|
I've detected some pattern looking into the provided logs. Please, for now, disable the If you want you can backup somewhere the |
|
Updated build is here. Steps to perform:
Specified memory limits look huge but let's monitor the real values running new build. The bootstrap fetch process is going to take roughly 3-6 hours for your ~18k messages, not because the app is slow but because it has to respect rate limiting set by protonmail.
You can use the database view mode extensively or just keep in open while the database is being bootstrapped (new messages will get there dynamically by portions of ~500 messages each). |
|
I think you got it this time... I was able to run the app all day without crashing, and memory usage remained stable as well. I clicked around the Database view on several occasions attempting to use it as much as possible as well. Also noticed that the Database view seemed to finish downloading the message store as well. Thanks once again! |
Because the full logs you provided last time helped a lot as well as your willingness to cooperate in investigating the case. Based on the logs it took 2 hours and 24 minutes to complete the database bootstrap fetch process. The app fetched 18252 email messages during that time. Bootstrapping is a one-off/initial action which means that after it got completed the app will only reactively fetch small database patches. The main app's process takes 1992MB based on the final log line. I think it's higher than the default nodejs's memory limit which means most likely you will have to run the app like this I will think how to make the max mem of the main process configurable without a need to pass the argument but taking the config params from Going to publish a new release soon. It will consume a little less memory since I've recently fixed a memory leaking issue in https://github.com/vladimiry/electron-rpc-api. Closing as resolved. |
|
Implemented improvements releases with v2.3.3 build. |
Hi and thanks for a great app... I recently switched to email-securely from protonmail-desktop but I'm having problems running the app.
It continuously crashes with an Illegal Instruction and Core Dump. I've increased logging but there doesn't seem to be anything useful in there. The only error I see, is if I launch securely from a terrminal - the Illegal Instruction occurs on the command line but there's no evidence of errors in the actual log. In fact the log file isn't created unless I set output to info. I'm running Ubuntu 18.10 & KDE 5.15.
Command Line output:
`
<--- Last few GCs --->
[10038:0x21d53cdbd000] 189186 ms: Mark-sweep 1088.5 (1533.4) -> 1088.5 (1481.4) MB, 26.0 / 0.0 ms (average mu = 0.975, current mu = 0.001) last resort GC in old space requested
[10038:0x21d53cdbd000] 189212 ms: Mark-sweep 1088.5 (1481.4) -> 1088.5 (1459.9) MB, 26.6 / 0.0 ms (average mu = 0.948, current mu = 0.001) last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x0aa1b4aa29d9
1: fromString(aka fromString) [0x284e8e39ec39] [buffer.js:341] [bytecode=0x3001a242361 offset=53](this=0x12445d2026f1 ,0x15a0e7e8b711 <Very long string[340707643]>,0x0aa1b4aca891 <String[4]: utf8>)
2: from [0x1d53f4b15f31] [buffer.js:202] [bytecode=0x3001a241f61 offset=11](this=0x151bc08043f1 <JSFunction Buffer (sfi = 0x1d53f4b0c639)>,0x15...
Illegal instruction (core dumped)
`
Log File Output:
[2019-02-19 09:03:51.586] [debug] [WEBVIEW:protonmail] [lib/api] resolveService() queueing rate limited method: "Contact.get" [2019-02-19 09:03:51.586] [verbose] [WEBVIEW:protonmail] [lib/api] resolveService() calling rate limited method: Contact.get {"waitTime":0,"rateLimitedMethodsCallCount":348} [2019-02-19 09:03:51.955] [debug] [WEBVIEW:protonmail] [lib/api] resolveService() queueing rate limited method: "Contact.get" [2019-02-19 09:03:51.955] [verbose] [WEBVIEW:protonmail] [lib/api] resolveService() calling rate limited method: Contact.get {"waitTime":0,"rateLimitedMethodsCallCount":349} [2019-02-19 09:03:52.452] [debug] [WEBVIEW:protonmail] [lib/api] resolveService() queueing rate limited method: "Contact.get" [2019-02-19 09:03:52.453] [verbose] [WEBVIEW:protonmail] [lib/api] resolveService() calling rate limited method: Contact.get {"waitTime":0,"rateLimitedMethodsCallCount":350} [2019-02-19 09:03:52.849] [info] [WEBVIEW:protonmail] [api/build-db-patch] api:buildDbPatch() <[account.component][0]> bootstrapDbPatch() fetched 349 contacts [2019-02-19 09:03:52.849] [verbose] [WEBVIEW:protonmail] [api/build-db-patch] api:buildDbPatch() <[account.component][0]> bootstrapDbPatch() start fetching folders [2019-02-19 09:03:52.849] [debug] [WEBVIEW:protonmail] [lib/api] resolveService() queueing rate limited method: "Label.query" [2019-02-19 09:03:52.849] [verbose] [WEBVIEW:protonmail] [lib/api] resolveService() calling rate limited method: Label.query {"waitTime":0,"rateLimitedMethodsCallCount":351} [2019-02-19 09:03:53.236] [info] [WEBVIEW:protonmail] [api/build-db-patch] api:buildDbPatch() <[account.component][0]> bootstrapDbPatch() fetched 54 folders [2019-02-19 09:03:53.237] [verbose] [WEBVIEW:protonmail] [api/build-db-patch] api:buildDbPatch() <[account.component][0]> bootstrapDbPatch() construct initial database patch [2019-02-19 09:03:53.245] [verbose] [WEBVIEW:protonmail] [api/build-db-patch] api:buildDbPatch() <[account.component][0]> bootstrapDbPatch() trigger initial storing [2019-02-19 09:03:53.245] [info] [WEBVIEW:protonmail] [api/build-db-patch] api:buildDbPatch() <[account.component][0]> persist() start [2019-02-19 09:03:53.257] [info] [electron-main/api/endpoints-builders/database] dbPatch() [2019-02-19 09:03:53.382] [info] [electron-main/api/endpoints-builders/database] patchMetadata() [2019-02-19 09:03:53.382] [verbose] [electron-main/api/endpoints-builders/database] dbPatch() {"entitiesModified":true,"metadataModified":false,"modified":true} [2019-02-19 09:03:53.382] [info] [electron-main/database] saveToFile() [2019-02-19 09:03:53.383] [verbose] [preload: database-indexer] [index] dbIndexerNotification.next, action.type: ipc_main_api_db_indexer_notification_actions:Index [2019-02-19 09:03:53.383] [info] [preload: database-indexer] [index] action.Index() Received mails to remove/add: 0/0 [2019-02-19 09:03:53.383] [info] [electron-main/api/endpoints-builders/database] dbIndexerOn() action.type: ipc_main_api_db_indexer_on:ProgressState [2019-02-19 09:03:53.383] [verbose] [electron-main/api/endpoints-builders/database] dbIndexerOn() ProgressState.status: {"indexing":true} [2019-02-19 09:03:53.384] [info] [electron-main/database] dump() [2019-02-19 09:03:53.384] [info] [electron-main/database] iterateAccounts() [2019-02-19 09:03:53.397] [info] [electron-main/api/endpoints-builders/database] dbIndexerOn() action.type: ipc_main_api_db_indexer_on:ProgressState [2019-02-19 09:03:53.397] [verbose] [electron-main/api/endpoints-builders/database] dbIndexerOn() ProgressState.status: {"indexing":false}Thanks for your time!
The text was updated successfully, but these errors were encountered: