Sidebar "wait background loading...", will not populate, other issues

[ttggzz]

Hello, Sorry for this long post and possible redundancy. Hadn't used bsp2 for a couple of weeks, loaded in side bar and will not go past "Wait background load..." with a spinning icon. The space below in the sidebar is blank.

BSP2 @ 2.0.43, I am on Manjaro Linux, up to date and running Firefox 63.0.3 64 bit. I saw this "Stuck Prepare tree display", don't know if it's related but wanted to verify before trying anything:
#65

I found this issue "load problem":
#50
and same:
#52
And per instructions retrieved this:
Load local store duration: 57 ms
I didn't go any further with the instructions because I am at a later version than cited in those threads, plus the other issues noted below.

I had the issue earlier of FFx library not being in sync with bsp2 due to me disabling bsp2. I did go through the process of allowing it to sync [was done when I came back 6 hours later] and have not disabled bsp2 since. If I recall I had 65,000+ bookmarks and 15,000 or so folders/separators.

On a possibly related note, I read this "exhausting memory" thread.
#64
I did not have favicon fetching paused nor "dual save" checked. I have been experiencing memory overload with Firefox open for a long time. 8GB available, will slowly creep up to 7+ if I let it. I've been just restarting FFx to reel it in.

I paused the favicon fetching and checked dual save. I did not look prior to doing this, but now the "statistics" are all at zero. Refreshing the page does not change anything, and the bsp2 sidebar still will not populate.

Of note: The last time I tried bsp2 a few weeks back, I'd do a search and a huge number of links would show up, none of which had the search term. As an experiment I opened the library and did the same search, retrieved the expected small number of links. Then did an experiment, added a unique bookmark link. BSP2 again showed massive search results [no matches that I could see], the library search showed just the one unique link. At that time I abandoned BSP2 but kept it installed, just now trying again.

Finally, from one of the above threads, what is "memory saw"? I did a search with various other words and can't seem to find anything.

thanks for your time and any info!

[aaFn]

Hello @ttggzz, this is strange, it looks like things are taking a very long time to read from FF API.

I did notice lately that this FF bug https://bugzilla.mozilla.org/show_bug.cgi?id=1396364
has become largely worse than before. One of my VM's with a test of 80000 bookmarks was taking 20 minutes to load in the past, and in my recent tests, after several hours it was still running .. and not finished.
I had to reduce my number to around 20000 (easy for me since it was a test set of bookmarks, not a real list ...) so that it comes back to acceptable times.

I suspect you are hitting that also. I just created version 2.0.44 with one more trace in it to detect such cases.

I would ask that you execute the below, which could also act in case there is a corruption somewhere else:

• In the options panel, set the Revert to old behavior of loading full bookmark tree from FF API at each FF restart. Slows down bookmark tree availability option
• In the main addon panel, disable BSP2
• Show the browser console (Ctrl+Shift+J)
• Empty it (click on top left trash bin icon)
• In the main addon panel, enable BSP2
• Wait for some time and copy here the contents of the browser console (please copy the full contents = Ctrl-A to select all, then Ctrl-C to get it in the clipboard, then Ctrl-V here to put the full clipboard in your post)
• Then come back to the options panel and unset the option, to come back to normal mode.

And let's see what you get ...

[ttggzz]

Thanks aaFn, I followed the procedure. I left the browser sitting inactive for 6 hours before copying content. I had upgraded to 2.0.44 before running the procedure, hopefully this did not taint the results.

Load saved state..

background.js:110:5
Read saved tree info..

background.js:110:5
Read saved tree..

background.js:110:5
Read migration state..

background.js:110:5
structureVersion: -img16-bnlist-spfldr

background.js:110:5
disableFavicons_option: false

background.js:110:5
pauseFavicons_option: true

background.js:110:5
Load local store duration: 20 ms

background.js:110:5
No install event background.js:2047:3
(Loading from FF API)

background.js:110:5
TypeError: chromeWin.gBrowserInit is undefined[Learn More]
ext-webNavigation.js:128:1
(FF API) Tree load duration: 31792 ms

background.js:110:5
Tree build duration: 286 ms

background.js:110:5
Save duration: 1407 ms

background.js:110:5
TypeError: chromeWin.gBrowserInit is undefined[Learn More]
ext-webNavigation.js:128:1

[ttggzz]

I unchecked the option in bsp2 and did a test. The library loaded in the sidebar almost instantly. For the couple of seconds while it was loading I believe it said "loading from bkgnd" or something like that, whereas before when it wouldn't load and had the spinning icon it said "wait background loading".

I did a test search in the library with 6 matches, running the same search in bsp2 came up with the same results [before bsp2 was returning a hundred or more results, none of which matched]. Looking good!

Was I correct in pausing the favicon fetching and checking dual save, do you think that was the culprit of ram usage as mentioned in issue #64? I don't know yet if there's any difference, but like I said if I let Firefox run for a long time without restarting the ram usage would slowly creep up until the computer froze.

The statistics will now populate, this was a screenshot:

[aaFn]

Hello @ttggzz , so I guess something was corrupted from past.
That was cleared out by forcing the reload from the FF API with the option.
I see that the FF API returned in around 32s so all goes fine.

You can disable dual save, it is a security in case things go wrong, but it consume more local storage space and I do not believe you need it (or let me know).

On favicon fetching, I see you still need to fetch nearly all of them.
This is going to take a long time, and yes we observed it was related to the garbage collector not acting swiftly in the past with @vitaaliy.
Hopefully what I did in 2.0.43 would have lowered it, Latest news I have from @vitaaliy is that it was ok for now.

So I would suggest you activate favicon fetching by bits ... like let it run for 1 hour, then pause it, and verify if memory is ok, let time for things to calm down ... And you can restart like that if you see nothing going wrong .. and just pause as soon as you see high memory levels in FF, to let FF garbage collector time to act.

Let me know how things are going.

[ttggzz]

Hello aaFn, ram usage didn't seem too bad so I left the fetching on all day yesterday. Should I be ticking "pause" favicon fetching if I close the browser? See below.

After Firefox being open all day I then started closing/opening in the evening due to elevated ram usage because I had other browsers open [spiked up to close to max]. I am seeing now, I guess I didn't quite catch your instructions, I was supposed to experiment with pausing the fetching not close the browser, sorry :) But, it may have been too much anyway, with 3 browsers open.

Not sure but I believe during this opening/closing cycle the "favicons to fetch" stayed where it was prior to closing then continued to drop and at final closing was at about 70%. I had been observing the percentage diminishing throughout the day.

Today when I open Firefox, "favicons to fetch" is back up to 95%. I reloaded the bsp2 add-on page, same. That's where I was wondering if fetching needs to be paused before closing.
thanks for any info
PS: I updated to 2.0.45 sometime last night
PPS: in case it's relevant, recalling I had disabled dual save per your suggestion, I believe that was earlier in the day.

[aaFn]

Hello @ttggzz , normally you shouldn't need to pause indeed, but given there can be high memory phenomenons, which can corrupt such volumes, it is probably wiser to set to pause some time before you close, to let things calm down, and make sure there is no corruption that could occur.

[ttggzz]

OK, turn for the worse. Firefox had crashed and set favicons to fetch back to 100% from 60%, so I re-ticked the "dual save" and forged ahead. I had been periodically ticking "pause" on the fetching, waiting 10 minutes or so, then restarting Firefox. Favicons were maybe at 50%, I just did the restarting procedure mentioned now I'm back to "wait, background loading..." in the sidebar, and statistics looks like this, which I don't think I've seen before:

On another note, I have another add-on, All Tabs Helper, which I'd noticed after FFx had been running for a long time the drop down for ATH [shows all the tabs in that window] would not populate. However, you could click "all" in the toolbar and the list would populate. I began to notice whenever ATH would reach this state, the "statistics" in bsp2 would also not populate. It would be a blank box. Just thought I'd mention it. Once FFx was restarted both the add-ons would go back to normal behavior.
thanks.
PS: before posting I restarted the computer which had not been done in maybe a week, then opened the browser. Wanted to be sure I had a clean slate, so to speak.

[aaFn]

Ok, so dual save was made for cases like that indeed, let's hope it plays the trick.
thank you for keeping me posted.

[ttggzz]

Sorry, but I think you might have missed my current status, maybe it got buried in the backstory details in my last post? I had paused fetching in order to restart, was running dual save due to previous loss of all favicons [was at 50%]. Then, when I restarted Firefox [after rebooting computer], bsp2 seems to basically not be starting, the "statistics" box is empty except for [NaN%] next to favicons to fetch [which I don't believe I've seen before], and I'm back to "wait, background loading" in the sidebar.

Square one.

addendum: my distro has updated Firefox to 64.0, which I just installed. Same results as above.

[aaFn]

Ok, yes I missed the point sorry.
I guess best is to redo the proc at top. Then if all restarts, simply pause favicon fetching and let it there.

• In the options panel, set the Revert to old behavior of loading full bookmark tree from FF API at each FF restart. Slows down bookmark tree availability option
• Also set Pause favicon fetching
• In the main addon panel, disable BSP2
• In the main addon panel, enable BSP2
• It should start, wait for some time
• Then come back to the options panel and unset the Revert option, to come back to normal mode, but leave pause on.

If the above does not work, simplest to reset al is to nuinstall and reinstall BSP2. And set pause to on.

Hope this will go better.

[ttggzz]

Are you suggesting I forego favicon fetching all together? The thought had crossed my mind earlier, all that means is I will not see the website favicon ID in bsp2's library, correct? I will add, my computer feels quite brisk as far as performance goes with bsp2 currently not functioning. With 2 browsers running ram usage stays below 4GB, before it would just keep climbing until freeze or FFx crash.
thanks

[aaFn]

Hello @ttggzz, yes I am suggesting that you stop favicon fetching, and when you want you can give it a go for 1 hour for example, but since this is what is triggering high memory levels in the Main FF process as we could observe with @vitaaliy, this needs to be very limited usage.

All we can see with @vitaaliy is that this appears to be the garbagge collector who does not free things fast enough, and this is happening when things are saved to local storage too often. I already spaced a lot those saves, but obviously this depends on the machine. The Pause is just doing a bigger time between the saves related to fetching. Until I can find something to monitor GC state, or provoke it (it is not available today to add-ons, or to monitor levels of memory which are not yet freed ....)

By the way, not sure what you mean by "2 browsers" .. do you mean 2 FF windows ? That should be ok.

[ttggzz]

I re-installed, was surprised to see how quickly 87,000+ items populated sidebar, probably within a couple of minutes. I have paused fetching, I will do without until that "garbage collecting" is worked out. Memory usage still hovering around 4GB with Firefox and Palemoon open, 9 windows split between the 2 with maybe 150/50 tabs open... a small session! Old laptop performing well [10 year old core 2 duo maxed out to 8GB ram]. Thank you for your efforts. re: "*2 browsers*" Was checking out Palemoon a long time ago prior to some big changes going on in Firefox but decided not to migrate. I now keep gmail open in it because I don't like being logged into youtube where I spend a lot of time researching and entertaining. I like to avoid google as much as possible, maybe it doesn't matter about youtube but being logged out helps psychologically :) I also use Palemoon for banking, insurance, stuff like that, keep minimal add-ons to be sure I can log in properly. I'll use google chrome occasionally for movie streaming. I have several Firefox profiles that can be opened simultaneously, my main one is the biggie where I have bsp2 installed. I also have a smaller business one and several test profiles, one of which has no changes to it, for the stray website that doesn't seem to want to open properly, see if it'll work there.

…

On Wed, Dec 12, 2018 at 7:15 AM /PRTL\ ***@***.***> wrote: test On Wed, Dec 12, 2018 at 3:09 AM aaFn ***@***.***> wrote: > Hello @ttggzz <https://github.com/ttggzz>, yes I am suggesting that you > stop favicon fetching, and when you want you can give it a go for 1 hour > for example, but since this is what is triggering high memory levels in the > Main FF process as we could observe with @vitaaliy > <https://github.com/vitaaliy>, this needs to be very limited usage. > > All we can see with @vitaaliy <https://github.com/vitaaliy> is that this > appears to be the garbagge collector who does not free things fast enough, > and this is happening when things are saved to local storage too often. I > already spaced a lot those saves, but obviously this depends on the > machine. The Pause is just doing a bigger time between the saves related to > fetching. Until I can find something to monitor GC state, or provoke it (it > is not available today to add-ons, or to monitor levels of memory which are > not yet freed ....) > > By the way, not sure what you mean by "2 browsers" .. do you mean 2 FF > windows ? That should be ok. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#72 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AgSEthLFWhYwLNyDBz9KG21SIe4AjFnhks5u4LmegaJpZM4ZFTDq> > . >

[aaFn]

Thank you for the explanations. Makes sense.
Let's cross fingers now ...

[aaFn]

No news = good news ?