On the doorstep sat his old woman, With the same broken wash-tab before her

[vitaaliy]

Hi aaFn, hi all,
Just feeling like the Old Man from the famous Russian folk tale "Fisherman and little fish". Things went worse probably when I installed FF v. 60.0.1. 64 bit. FF loading became very tiresome... too long, chewing almost all 8 Gb of RAM in the saw way: increase - decrease (while I remain in the standard FF bookmarks mode). When I need the help of BSP-2 and switch it on - the thing goes into endless loop - Loading bookmarks. 5... 10 minutes pass and nothing happens. The only way out I found - delete BSP-2 from Extensions and then reload it anew. So it works only one time after installation. Any hints?

[aaFn]

Hello @vitaaliy , not sure ..
While you are on the standard FF bookmark, BSP2 front end is not running, so this saw thing shouldn't happen.

Now from our last discussion on #5, I get it that you were somewhere still in the process of migrating all icons to 16x16 mode. I suppose you didn't have the occasion to let it run alone during a night, or this is crippling because things are too frequent and queueing .. so somehow something was still triggering it, or there is too much in queue and that takes a long time to dequeue ?? It shouldn't, but supposing something was still happening, the best plan B thing to do was indeed what you did = uninstall and reinstall.
This has cleared all local storage and so cut any background migration task of favicons.

Now, what do you mean by

it works only one time after installation

?

Is it that the "saw phenomenon" is happening again ?
FF 60 certainly did things in the bad direction on my system also, but that was solved with 60.0.1. So that's not it.

I can only think that now you are in the long loop of getting the favicons from fresh, which is long because of the sheer number of bookmarks you have. By the way, I was never sure of why you didn"t have the saw phenomenon in the first place when you initially loaded BSP2 ... that memory problem is inherent to the fetch() and storage things as I could see it = they are creating that memory problem when used intensively, and I didn't find a way to get rid of that so far, I could only decrease it by spacing the requests .. but I guess on your system they are still too close to each other

The best way to cut it out is to:

• go to the options page of BSP2
• tick the Disable favicon fetching option
• stop and restart BSP2 (like twice Ctrl-Q, or click twice on the button in the toolbar if you still have it)
• stop firefox to remove any queued storage and fetch requests
• restart firefox after it has been completely stopped (check on the Windows task list to make sure ..)

That should stop the crazy saw thing .. The only drawback is you won't have the fancy favicons, but you do not care as the main usage of BSP2 you have is to search your bookmarks and lost subtree parts .. :-)

Let me know how that goes, or more details if still problems. Maybe activate the traces and show here what you get in the trace textarea ... and also show the contents of the Browser console (Ctrl+Shift+J).

[vitaaliy]

Now, what do you mean by
it works only one time after installation
?

Alas, it is the key predicament. The life cycle of my BSP2 installation is from the moment of its downloading and using to find the bookmark needed. If I return to the standard FF bookmarks the second try to switch BSP2 on plunges into the endless loop loading the bookmarks. It means that if I really try to use BSP2 once more the whole magic should be repeated: [deleting - downloading - using until the standard bookmarks call]. So, you see, I am far from the latest bells and whistles about favicons.
Deep distress, really...

[aaFn]

In one of your cycles, once BSP2 is started just after install, can you do what I mentioned above, in that specific order:

1. go to the options page of BSP2
2. tick the Disable favicon fetching option
3. tick the Activate traces option
4. Open the browser console (Ctrl+Shift+J)
5. stop and restart BSP2 (like twice Ctrl-Q, or click twice on the button in the toolbar if you still have it)

and post here what you see in the trace window, and in the browser console ?

[vitaaliy]

Opening the browser console gives the following endless sequence:
code:1:1104
Console was cleared.code:1
code:1:1104
Console was cleared.code:1
code:1:1104
Console was cleared.code:1
code:1:1104 …

5. Stopping and restarting BSP2 led to:
   PlatformOs: win
   Setting Windows variations

Browser Console was going on printing that same dull sequence. Though other BC windows were displaying a lot of diagnostic info. I attach the output at "Logging" stream. Where to find "Traces" results is not clear...
Logging.txt

[aaFn]

Thank you. Strange, first time I ever see that sort of error in the browser console, it doesn't seem to make any interpretable sense.

The only thing I can deduce from the above is that it appears the load from local store is not completing when restarting .. and FF API is not returning to BSP2. This is very early in the BSP2 start process.
I wonder if you do not have a cycle in your bookmark tree ...

Can I ask one more thing then:

1. Recycle again by uninstall / reinstall
2. Open the browser console (Ctrl+Shift+J)
3. Make sure that in the browser console, the following traces are activated (click on the little down arrow at right of each label to verify / set the checkmarks):
   • Net: Errors and Warnings (not XHR, not Log)
   • CSS: Errors only (not Warnings, not Reflows)
   • JS: all checked
   • Security: all checked
   • Logging: all checked
   • Server: all checked
4. Do a right click on a bookmark in BSP2, and click on "Refresh favicon"
5. Add a new bookmark anywhere in the tree with BSP2 (like right click + copy / paste an existing one)
6. Do a folder Open / folder Close in BSP2
7. Stop BSP2 by going back to native bookmarks

Send me the result of the browser console. If there is a cycle in the bookmarks tree, I believe when saving the things to local storage something should yell .. and the three actions above from 4 to 6 should each trigger a save.

.......
Note: I have a Win10 VM with FF 60.0.1 and about 75000 bookmarks, and all goes fine (just slow because this is a VM - do not pay attention to the load duration, I froze the VM that night and restarted it this morning, its more like 20 minutes usually) ..
Here is the trace for reference .. Between "**" the first line that is missing from your trace, and which appears when local storage load is complete:

PlatformOs: win
Setting Windows variations
**structureVersion: -img16**
disablefavicons_option: false
Load local store duration: 828 ms
BSP2 version: 2.0.20
Tree load duration: 46323053 ms
Tree build duration: 6429 ms
Display duration: 43092 ms
fontFamily = '"Segoe UI"'
fontSize   = '12px'
Stats:
------
Bookmarks:  59222
Folders:    15184
Separators: 1
Oddities:   0
--------------------
Save duration: 71014 ms

[vitaaliy]

No wonder. All my life I stumble upon exotic software system errors. That is my karma. Probably I had to inform you in advance ;). Now to your prescriptions.
Experiment May, 26.zip

[aaFn]

Thank you @vitaaliy . Nothing really significant in all this :-(

Let's go the detailed way if you agree .. I will make the bet there is something wrong in your bookmark structure .. and this is not showing until we reload it from local storage when BSP2 is starting a 2nd time. The only thing I believe can make things wrong is if you have "duplicate" bookmarks, so I will look for it.

Here below you will find a zip file of a debug version. Traces are force-activated from start, more traces have been added, and favicons are force-disabled.

Here are instructions of what I propose you to do, if you can run that on your side in that exact order, and give me the results:

1. In FF, uninstall BSP2 if you have it installed.
2. Stop FF, wait for the process to disappear.
3. Download the file below and unzip its contents to some temporary directory = it contains the debug version of BSP2 I mentioned above.
4. Start FF
5. Open the Browser console (Ctrl+Shift+J)
6. Make sure as before the following traces are activated by clicking on the little down arrow at right of each label to verify / set the checkmarks:
   • Net: Errors and Warnings checked (but not XHR, not Log)
   • CSS: Errors only checked (but not Warnings, not Reflows)
   • JS: all checked
   • Security: all checked
   • Logging: all checked
   • Server: all checked
7. Empty the browser console (click on the trash bin)
8. Open the Add-ons manager
9. Click on the Parameter/config wheel ("Tools for all add-ons" when you hover the mouse on it), and inside that, click on "Debug add-ons"
10. A new window opens, with a "Load Temporary Add-on" button at top left
11. Click on that button, a dialog appears .. navigate to the directory where you unziped the trace version of BSP2, select "manifest.json" and press "Open"
12. The trace version of BSP2 will be loaded and will start
13. Wait for it to load the bookmarks and show you the bookmarks tree.
14. There should not be any memory phenomenon since favicon retrieval is disabled.
15. Look at the trace textarea, copy its contents (Ctrl-A to select all + Ctrl-C to copy) and send it back here.
16. Look at the browser console, copy its contents and send it back here.
17. Empty the browser console (trash bin ..)
18. Press Ctrl-Q to remove the add-on
19. Press Ctrl-Q to make the add-on reappear
20. Wait for it to restart and provide the bookmarks tree (or maybe hang .. we'll see)
21. Then look at the trace textarea, copy its contents (Ctrl-A to select all + Ctrl-C to copy) and send back here that second run trace.
22. Look at the browser console, copy its contents and send back here also that second run trace.

And let's see if we get something more meaningful from all that ... Thank you for your patience.

Here is the file ->
BSP2-trace.zip

[vitaaliy]

The only thing I believe can make things wrong is if you have "duplicate" bookmarks, so I will look for it.
Maybe it could save us a gigantic labor you propose. Yesssss... mea culpa, mea culpa, mea máxima culpa. I know that my structure does have duplicate bookmarks. I have done them with my own hands. Sometimes it looked more handy to have identical bookmarks at different branches of the tree structure. The native bookmarks FF machinery never objected it, Show Parent Folder 2.1.1 By Alice0775 - neither. Can it help you in pinpointing the problem? If no - I will follow the plan you proposed.

[aaFn]

:-)
Yes, that explains .. Well, can you please still execute the protocol above and send the result ?
That trace version should identify them and give us the extent of what you did.
You should retrieve your own hands labor :-D

If you can paste the results here, that should then give us fuel on if there is a way to support it. I need to see the structure of what you have done ... and if you can explain a little more what was your intention in doing that ?

...............
The problem is that in order to accelerate things and get around FF API limitations / slowness (see #5 and #2), I had to now since 2.0.18 build an internal tree structure representing exactly the bookmark tree.

So far so good, however, when saving that to disk to retrieve it later and accelerate loading in a future release (I am going step by step ...), this gets serialized in a json form by FF.
Json serialization does not like cycles nor complex graphes ... which are bound to happen if there are duplicates, i.e. the same unique id appearing twice or more.
Saving is ok as I observed, but reloading them .. wrecks things up in FF or in the add-on memory at least.

...............
Also, I need to warn you ... in FF, the bookmark id is really meant to be unique. If it appears several times, some operations should fail or work awkwardly .. which could explain the parts of tree "disappearing" as I believe you were explaining in one post in the past.

For example, in FF API, there is only one parentId for each bookmark, see https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/bookmarks/BookmarkTreeNode.
It also only has one index.

So if the same bookmark (i.e. same unique id) appears several times under different parents / different places (same parent or not), this can only have "unpredictible" effects ... working fine in display, but when you do operations on them, like moving them around or deleting (or searching ...), you get at best partial results, at worst things which become orphaned (no parent) or even worse, cannot be manipulated anymore.

[aaFn]

Also, if you can explain to me how you created those duplicates by hand ?

[vitaaliy]

Hi aaFn,
"Also, if you can explain to me how you created those duplicates by hand ?"

• Very simple: I drag the current link from the FF address field and drop it straight into the chosen folder in the bookmarks sidebar. Due the large and complex structure of my B-tree it takes too long and needs many clicks if I use the routine procedure following all parent edges of the chosen node.

"and if you can explain a little more what was your intention in doing that ?"

• Sure. Hierarchical natural classification is oriented strictly at fixed viewpoint. Consider two hierarchies:

1. Bob IS male, Alice IS female, male IS human, female IS human
2. Bob IS engineer, Alice IS programmer, engineer IS profession, programmer IS profession.

If I need to fetch all attributes of Bob - in the given example I should scrutinize all present classification trees. It is too unwieldy. The best solution would be the complete graph (semantic net - in Artificial Intelligence) where each object would have unic representation - its designation. All properties and relations of this object would be localized around it.
FF does not work with semantic nets - it gives the primitive bookmarks tree. That is why one has to plunge both given classifications into one tree - so our heroes: Bob and Alice would be represented more than once.

FF authors gave us the tag mechanism. It is very unwieldy for large and complex graph structures. It takes too much efforts to classify with tags at each object insertion. But logical expressions (and, or, not) at retrieval stage are not provided.

Yes... I remember strange behavior of my bookmarks - when some nodes from time to time changed their location. Not disappeared - but changed - without any evident reason. It is the situation where I use BSP2 to find the escapee.

If my practice of nodes' proliferation is against the rules and is the source of strange effects and errors - I
would repair my structure if some method would help to find these nodes.

Now to your scenario.
May'27.zip

[aaFn]

Thank you.

So good news :-) = you do not have "duplicates" in the meaning I was intending, i.e. the same true structural object with same internal unique id appearing twice or more.
Your "duplicates" are simply textual duplicates, they have no structural impact on the tree, it remains a tree .. even though their contents is the same, they are different object in the tree.

Bad news = I do not understand what is making your second run not working on 2.0.19 .. all the more that here with my trace version, it appears the 2nd run is working .. do you confirm ?

If yes, the only difference being the disable favicons forced from start, then once you disabled favicons, your 2.0.19 version should also work well ... I can think of more spacing and a delay mechanism in 2.0.20 to reduce more the effects, or give you time to switch off the option and recycle ..

[vitaaliy]

You are right - my "duplicates" really are textual clones.
"Bad news = I do not understand what is making your second run not working on 2.0.19 .. all the more that here with my trace version, it appears the 2nd run is working .. do you confirm ?"

• Alas... I was acting strictly according to your prescriptions without deep understanding of all peculiarities. That is why I copied your commands one after another and gave my confirmations. If you need to check anything - let us use the same method: You give the new plan, and I strictly follow it.

As for me - I made the following experiment. I deleted your debug-version and loaded current BSP2 2.0.19. It began to work right away while "Disable favicon fetching" field being not marked. I let it build the B-tree, then switched to the native FF B-marks, marked "Disable favicon fetching" field, closed and opened FF, then switched BSP2 on (with marked DFF field) - and it started that same endless Loading bookmarks process. What is interfering? Does favicons fetching interferes? I can endure the slowness of its reaction, but periodic deleting and reinstalling is really tiresome...

[aaFn]

Ok, from the traces, I can see that the trace version is working fine the second time.
Not sure why 2.0.19 is not working fine on the second time .. All the more if DFF is set, favicon fetching shouldn't interfere.

So one more experiment:

• You'll find a new trace version at bottom of this comment.
• It does not enable DFF from start, but it keeps the traces, and it introduces an hysteresis of 1 minute after end of tree load before starting to fetch favicons. And it also has more spacing between the saves to local store to try to decrease further the memory saw phenomenon.
• Can you re-try your experiment above with that version instead (so load temporary add-on each time ..), giving me the contents of the trace textarea of each run ? Make sure you switch to native FF Bookmarks within less than a minute after B-tree is displayed.

Then, uninstall the trace version, stop FF ..; restart FF

• Re-load the trace version below (temporary) ... but let it go ... I.e. do not touch DFF, do not stop it, wait 1 minute after the B-tree is displayed, watch the memory while favicons are fetched, and let me know if the memory saw effect is getting better or not ..
• When you're tired of watching memory, get the trace textarea to send it here, switch to native bookmarks, close / reopen FF, reload that trace version (temporary), and verify that B-tree is coming after 25s or so.
• Again get textarea traces, send here .. and you can uninstall the trace version

Let's see what you get on those 4 traces .. if this is working better, I will publish it offcially as 2.0.20 (without the traces) so that your 2.0.19 gets upgraded and hopefully you can work better.

Thank you.
BSP2-trace2.zip

[aaFn]

By the way, how many FF windows do you have open ? If more than one, can you make sure that only one has BSP2, and all the others have the native bookmarks ?