Exhausting memory

[vitaaliy]

From time to time Windows Task Monitor shows again saw-like RAM consumption... Then amount of used memory gradually increases up to the physical limit: 8 Gb... blocking any computer activity. Several Firefox processes run in parallel, size of one eats the main part of the memory available.
Bookmarks: 84082
Favicons to fetch: 83992 (99.9%)
Folders: 15677
Separators: 3
Oddities: 0
Closing and reopening of FF helps... for several hours. With BSP2 disabled all works normally.

[andreygursky]

Hi @vitaaliy, this sounds like #49.

While it cannot be really solved now, @aaFn has kindly implemented a workaround. Please try to switch "Favicon fetching" to "Pause" in BSP2 preferences.

[vitaaliy]

Thank you, Andrey, but I switched favicon fetching to "Pause" long, long ago. Now I think about disabling favicon fetching completely and forget about them as of horrible dream. Really do they deserve such sacrifices? I am grateful to aaFn for this BSP2 addon - I cannot imagine how I would work with my bulky bookmarks collection without it. Comparing to the main BSP2 effect - favicons are like bells and whistles at its side...

[aaFn]

Hello @vitaaliy , strange, can you check again on the options page and tell me ? With favicon fetching set to pause, there shouldn't be any memory saw problem,
The only difference with disabling it are:

• that what is already in memory is kept, while with "disabled", it is forgotten
• you can refresh manually a favicon and it will work, as it is a voluntary action.

Let me know, if you confirm it is set, I might have a bug lying around ...

[vitaaliy]

Okay @aaFn - I have just enabled BSP2, took a screenshot of BSP2 options and rerun FF. Now it is all quite and memory consumption is about 3 Gb. Probably the devil will show himself tomorrow. I will keep you informed.

[vitaaliy]

In the afternoon the saw started...
, but then disappeared, and the memory level now is 5.8 Gb.

[aaFn]

Ok, were you doing anything to bookmarks at that time, or do have you another add-on related to bookmarks installed ? Or did you restart BSP2 in any way ?

Asking because there is no reason that BSP2 does anything by itself in the background, it should only be because notifications are sent to it by FF (like modified bookmark), either originated by the user or by another add-on acting on bookmarks.

[vitaaliy]

No another add-on rivalry to BSP2 ;). Yessss... when new FF version arrives I try to find there such intrinsic functionality - but fail each time. All my bookmark activity is in adding new, deleting, moving and sorting. Sometimes I get some BM subtree lost and planted somewhere - God knows where - and at such moments I ask BSP2 to find this escapee - it finds it always :) - and I move it to its rightful place.
Now I see no saw activity. Memory used is stable about 5 Gb.

[aaFn]

Ok, let me know if it starts again, I re-checked in the code and so far saw nothing obvious that would start anything like that by itself.

[vitaaliy]

O.K. I will complain here if any trouble happens :)

[vitaaliy]

Longly awaited trouble showed itself. BSP2 was enabled for about 10 hours. BM activity was negligible if any. Most part of the time FF was left to itself. No memory saw. But the amount of memory used gradually increased until it almost exhausted all RAM available. FF has about 10 processes, and one of them used about 6 Gb. Processor usage was about 1%. Certain HDU activity. At this state reaction to the keyboard or mouse was very reluctant and delayed. With considerable efforts, I disabled BSP2 and reloaded FF. It displayed the diagnostic message about a breakdown. Now it works normally, BSP2 disabled (alas!), memory used is about 4 Gb. No delays, no memory saw...
Of course, I am not sure that the cause of these misbehavior lies in the BSP2. And besides my drawback is the huge amount of opened tabs... maybe more than 200...
Any hints?

[aaFn]

Hello @vitaaliy , I made an experiment on my side with about 30000 bookmarks and fetching favicons intensively.
That lasted for 6 hours, and at the end of the 6 hours, one process was around 2 GB in FF.

I opened a tab with the about:memory URL and this gave me the following results:

• The Main Process pid was around 2 GB -> this the one !
• One of the Web Extension pid was 500 MB -> I could determine this was BSP2, for both the sidebar and the background, and all looked normal in there.

Details of "Main process" are showing a lot of strings and classes which are taking a lot of space in the Main process; plus a big heap. See below:

2,211,922,900 B (100.0%) -- explicit
├──1,509,569,568 B (68.25%) -- js-non-window
│  ├──1,402,184,144 B (63.39%) -- zones
│  │  ├──1,271,322,704 B (57.48%) -- zone(0x238ac952000)
│  │  │  ├────632,475,320 B (28.59%) ++ strings
│  │  │  ├────557,393,504 B (25.20%) -- realm([System Principal], DevTools (Module loader))
│  │  │  │    ├──556,582,416 B (25.16%) -- classes

├────502,673,040 B (22.73%) ── heap-unclassified

While things seem normal for BSP2, given it has around 30000 objects ..

WebExtensions (pid 208)
Explicit Allocations

534,093,824 B (100.0%) -- explicit
├──428,383,568 B (80.21%) -- window-objects
│  ├──216,611,936 B (40.56%) -- top(moz-extension://cf381681-36b0-41e7-bd9c-226aa4373319/sidebar/panel.html, id=4294967315)
│  │  ├──211,104,832 B (39.53%) -- active/window(moz-extension://cf381681-36b0-41e7-bd9c-226aa4373319/sidebar/panel.html)
│  │  │  ├──204,303,280 B (38.25%) ++ dom
│  │  │  ├────4,342,320 B (00.81%) ++ js-realm(moz-extension://cf381681-36b0-41e7-bd9c-226aa4373319/sidebar/panel.html)
│  │  │  ├────2,447,408 B (00.46%) ++ layout
│  │  │  ├───────11,360 B (00.00%) ── property-tables
│  │  │  └──────────464 B (00.00%) ── bindings
│  │  └────5,507,104 B (01.03%) ++ js-zone(0x2003b13b000)
│  ├──137,241,328 B (25.70%) -- top(moz-extension://cf381681-36b0-41e7-bd9c-226aa4373319/_generated_background_page.html, id=4294967312)
│  │  ├──128,198,752 B (24.00%) -- js-zone(0x20039a12000)
│  │  │  ├──124,343,112 B (23.28%) ++ strings
│  │  │  ├────2,854,952 B (00.53%) ++ shapes

So I am concluding that there is a form of "memory leak" in the main FF process :-(

It looks I cannot do much about it, apart trying to avoid situations which are creating those memory leaks ...

• In your case, you told me above that favicon fetching was paused
• You also told me you had a lot of tabs
• So I guess you are switching very frequently from tab to tab, as you navigate
• Switching from tab to tab triggers an event in BSP2, so as to try to catch favicons from tabs to accelerate things, exactly like the Native bookmark sidebar is doing
• This triggers a browser.bookmarks.search(), and then if the URI in the tab is not a data: ready made URI, and we are not in pause favicon fetching, a favicon fetch.
• I am presuming the browser.bookmarks.search() could be at origin of the memory leak ...

So in 2.0.41 I am changing the code so as to not do browser.bookmarks.search() if we are in pause favicon fetching and if the tab does not hold a data: ready made URI (data: is very rare).

Should be out tomorrow. Hope this will help ...

[aaFn]

Meanwhile, if you could yourself do an about:memory when you are in a situation with lots of memory eaten, that would be interesting to look at the result ...

[vitaaliy]

Last two days RAM never exhausted completely... Now the system eats 6.45 Gb. No exaggerated saw seen. FF processes eat memory considerably, but nothing outstanding: 200 - 600 Mb. I attach about:memory report here. But before that, when things were really tough one of the FF processes really ate outstanding memory amount: several Gigs...
memory-report.json.gz
Thank you for your attention and labor :)

[aaFn]

Indeed, all looks good on the BSP2 side in your capture, I see around 600 MB, with good proportion of DOM, which is normal given the number of bookmarks you have.
And Main process is a little bit more, but nothing exagerated in there as you say ..

We would need to see, when you have a FF eating a lot, if it is like the case I could provoke above.
If that happens again and you are able to make an about:memory capture, I would be quite interested.

[vitaaliy]

Here it came... Gross memory saw, Then almost all memory been eaten. I managed to take two pictures:
and:
. But comp's activity approached to almost zero... I planned to fix "about:memory" (at that moment the heaviest FF process has grown up to 6 Gigs probably, but all my actions were almost frozen, and I failed... Memory consumption became swiftly lower... and all FF processes were again looking almost pretty - I even decided not to take the last measurement. Now all is again calm as still Pacific. Total memory expenditure - about 3.8 Gb. No saw. Such is life...