Feature Request: search bookmarks with a regexp input string !

[stanpetit]

Hi aaFn !

Your extension works really great.
The drag'n drop behavior is efficient and precise, great work !

But the main aim remains "search the needle in a haystack" ...

Would it be possible to ask :

• don't search in url
• search in tags
  with a couple of check-boxes next to the search-bar, like : [ search: ... ] T[x] U[ ]

Would it be possible to support queries with a regexp input ? e.g. :

      /[^#]+(#([^#]+)+/     that should match these bookmarks labeled :
         talking about #Bookmark-search-plus-2 with #aaFn
         wow #this_extension works really great #aaFn
         blah blah blah this #person &talks about #89 

or    /(\bp\:[^\:]+)/     that should match "p:Guy Name" :
         blah blah blah this p:Guy Name: talks about t:that
where p:xxx yyy: is a person ... that I have referenced as such ... sure, I ref'd him ! 
but I can't remember so many names ! getting old, better take a nap in the haystack ...

And would it be possible to rename bookmarks with a regexp input ?

And would it be possible to apply a bunch of tags to bookmarks with a regexp input ?

... In fact I am using the bookmarks to "journalize" some events and take refs to "authors" or persons relationship, dates. It happens to be faster to stay in Firefox for digging, then extracting the referenced stuff by tags or quick markups in labels.

You were complaining about non-modal windows : I find this behavior very nice because I can still copy and paste from an opened page while creating/editing a bookmark, adding valuable meaningful info that is not part of the page title, and that will describe better the content that was useful for my research within the page.

Before the "Web Extension time", there was a very powerful Firefox extension that was a cute sqlite browser over the Firefox bookmarks database.
I cant find anything as powerful inside Firefox today ...

Hopefully, your extension is a good help to quick access !
Thanks again,
Best, Stan.

[aaFn]

Hello @stanpetit, these are quite interesting considerations :-)

For now, I am using the builtin search API https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/bookmarks/search, so:

• search on URL or Title only would be possible
• search (or write ..) on tags is impossible .. in fact we do not have access to tags from the Web Extension API -> https://bugzilla.mozilla.org/show_bug.cgi?id=1225916
• regexpt searches are not possible with it

I do have the intention to add an alternate search method, cf. #28 where we would prefix the search string with something ot scope the search.
I would need to find a notation which is compatible with regexp I guess .. or a prefix to indicate folder scope or regexp, but then we wouldn't be able to combine.
Any suggestion ?

By the way, if you read the definition of the search API I pointed at above, you will see that if you put n words (separated by blank), all of them must match, but not necessarily in the specified order:

Search terms are space-delimited and may be enclosed in quotes to allow multiple-word phrases to be searched against.
Each search term matches if it matches any substring in the bookmark's URL or title.
Matching is case-insensitive. For a bookmark to match the query, all the query's search terms must match.

Note that I the enclosed in quotes thing does not seem to be working .. or I am not using the right quotes in the search box (the string is passed as is to the API).

On rename, need to think more how we would like to do that, obviously that would be on title only, so we could use the s/.../.../ classical form in sed for example (http://www.grymoire.com/Unix/Sed.html#uh-1) or sub (old, new) somewhat like in awk (http://www.grymoire.com/Unix/Awk.html#uh-46).

You are correct that non modal allows copy paste and this is good indeed :-)

[stanpetit]

Dear @aaFn

I do have the intention to add an alternate search method, cf. #28 where we would prefix the search string with something to scope the search.
I would need to find a notation which is compatible with regexp I guess .. or a prefix to indicate folder scope or regexp, but then we wouldn't be able to combine.

In python, I use something like

':r:f#some_regex_pattern'    two options '(:[rf])?', unordered,
                             followed by a comment symbol '#'
for options: as_regexp, as_folder  +  "string" that may be a regex_pattern

js:    $1      $2 (?=X)   $3       '(?=X)':condition: $2 is followed by 'X'
/^(:[rf])?(:[rf])?(?=#)#(.+)$/  insensitive

tests :                  should match :
:r:f#regex_pattern       m: opt:$1, opt:$2, string:$3    r|f, r|f
:f#regex_pattern         m:         opt:$2, string:$3    r|f
#regex_pattern           m:                 string:$3    no option
:r:fregex_pattern        no match
regex_pattern            no match

n_opts === n_groups  ex:  (:[abc])?(:[abc])?(:[abc])?#
                     for three options, a, b, c, unordered

Also, the general shell command line options, short syntax, are '-o' for '--option', 
maybe it would be a better choice ... I use ':o' by personal convenience.

Could this opt_symbol be 'opted' in the addon params ? 
Your addon params are nice and wise, makes it very user-friendly, thanks a lot !

But in fact, I wouldn't bother about any funky syntax, that should be left to the users unbeliviable whishes ... and do not mix with the usual regexp syntax, to let the user achive what he needs ... in the simplest conditions : KISS ...
Use two check boxes ! [search_field ...] F[ ] R[ ], aside to the search_field !
And please use a dedicated pop_up_window for the regexp search : the search field is limited in size by the Firefox UI side-panel, and regexp char_size (width) can easely grow far above the max_panel_size, and therefore become a pain to read as a whole ... put some opt_syntax before the regex_pattern and you even loose as many chars as the options length ...
Here is an addon that allows regexp search within current web page : RegexSearch.

about tags ... :

The bookmarks, history, downloads, tags ... are stored and managed by a Sqlite database,
the file resides at
%APPDATA%\Mozilla\Firefox\Profiles\YOUR_PROFILE_DIR\places.sqlite.

The SQL language has the clause : WHERE ... LIKE '...' :

/* FORMAT AS MARKDOWN LINK : concat_operator is ||
 * SELECT   '['||moz_places.title||']('||moz_places.url||')'
 */
PRAGMA case_sensitive_like = false ;
SELECT moz_places.title, moz_places.url 
FROM   moz_places, moz_bookmarks 
WHERE  moz_bookmarks.fk = moz_places.id 
AND    moz_places.title     LIKE '% _sv data%'    -- matches ... csv|tsv data...
AND    moz_places.title     LIKE '%lua%'          -- and matching ...lua...
AND    moz_places.title NOT LIKE '%github%' ;     -- not matching ...github...


two symbols are provided :  SQL :  RE        note: ESCAPE '_', '%', to match 
zero or more chars          %   :  *               the symbols literals ...
one          char           _   :  . 

                       -- REGEXP equiv        algo in the SQL engine
    LIKE '%data%'      -- /^.+(?=data).+$/    positive match, appended to result-set
NOT LIKE '%github%'    -- /^.+(?=github).+$/  positive match, excluded from result-set

Everything in the API relies on top of the Sqlite engine. The only syntax the API gives you access to is :
'word1' AND 'word2' NOT 'word3', that you refer as 'words in quotes',
the "search" API capabilities are the consequences of the SQL clauses LIKE, NOT LIKE.

Though, you should have the full content ( or a prepared bunch ) of the bookmarks, inside a dictionnary in memory, like { n:[ bm_id, bm_titel, bm_url, bm_type ], ... }.
Then you can apply a regexp on the dictionnary's members fields, bm_titel or bm_url, and return the matching bm_id at dict[n], for any positive result in the loop.

The whole dictionnary in memory may be quite expensive and slow down Firefox ... but this way you can apply a regexp search : outside of the API, at the cost of extra memory usage.

Sure you can achieve the same job, faster than the API, if you can get an access to the Sqlite engine. I dont know if it is possible.

I am using the sqlite-manager addon to play with the bookmarks database.

Also, note that the next two queries do not yeld the same result :

PRAGMA case_sensitive_like = false ;
SELECT moz_places.title, moz_places.url 
FROM   moz_places, moz_bookmarks            -- filtered by "bookmarks"
WHERE  moz_bookmarks.fk = moz_places.id 
AND    moz_places.title     LIKE '%data%' ;

PRAGMA case_sensitive_like = false ;
SELECT moz_places.title, moz_places.url 
FROM   moz_places                           -- not filtered : includes "history" ...
WHERE  moz_places.title     LIKE '%data%' ;

Best, Stan.

EDIT: Sorry, I closed by mistake ... I dont know how to undo this !

[aaFn]

I reopened.
Will read in details during the week-end, but on the sqlite database file places.sqlite, a WebExtension add-on cannot access it by itself. The user needs to explicitely give it to the add-on, by a drag and drop or another method ... so to my knowledge there is no bypass to the API's.

[stanpetit]

Hey @aaFn : more for the week-end ...

Storage is the stuff to implement onto.
The painful stuff is their introductory warning ... but to use the roots of it may be safe anyway.
The db layout they present is from 2009, I saw other stuff even older and the basis look quite the same.
I wouldn't bet 10K on the tags topic after your reference to https://bugzilla.mozilla.org/show_bug.cgi?id=1225916, in a long term I mean ...

When reading some of your issues listed, I noted that you wanted to get the whole data in memory. Did I picture this on my own ? ( I cant remember where ).

In this case, the Mozilla database engine offers to create temporary tables within itself.
If serialized as backup.sqlite and reloaded in the SQLite memory-db, this would be the fastest access to all the BSP stuff self-storage as a simple indices tables.
Then a sequential SELECT over each entry ID would let apply the regexp on each content, title or URI.
It will be slower than with the search method of the API, but not that much : the js regexp engine, with auto-compiled expressions, is running at low level in the JIT compiler, very fast as the db engine does !
And if 'words' chunks already exist in the regexp string, some SQL query can per-filter the result-set to reduce the set on which to apply the regexp searches. ( like above )

At the same time, you can use the SELECT ... WHERE ... LIKE ... [ AND [NOT] LIKE ... [ ...[...] ] ] that is far more powerful that the search method of the API.
The SELECT ... [...] are to be written programmatically from a few text patterns.
SQL is straightforward, very structured language, so the very few various syntactical possibilities can be handled more easily than with a syntax-tree parser à la Lex-Yac, or Sed-Awk, I mean :
SELECT ... WHERE ... LIKE ... [ AND [NOT] LIKE ... ...], inside a temporary table of indices, is only two cases : TRUE (no symbol) or FALSE. The user writes the search-string with ONE extra symbol : '!' for NOT (for example).
SQL|regexp provide '_'|'.' for one_char, '%'|'*' for zero_to_many, that can be part of the 'quoted_words' the user searches :

for the case where the user uses extended SQL syntax : 
quotes are needed to handle spaces in "w_rds or _xpre%ions"    SQL
                                      "w.rds or .xpre*ions"    regexp
                 1                        2            
/((?:'|")?[% \w]+)+?((?:'|") *\!? *(?:'|"))?/ global, 

tested on : "a__ z%e%" "rtyuiop" ! "q__ s%d%"  "fghjklm" 

$1: opt quote, word, or group of chars with symbols, 
$2: quote, spaces with optional NOT, opt quote   

kill any quoted 'word or expression' at boundary in all g1[n], 
should yield two lists, to be weaved together : 
seq = [ g1[0], g2[0], g1[1], g2[1] ...] in a loop

resume ^(?:'|")\s+(?:'|")$       to       LIKE
resume ^(?:'|")\s*!\s*(?:'|")$   to   NOT LIKE

Make the simple SQL sandwich of the concatenated sequence in order : seq[0] seq[1] ... seq[n], and it's done : you have all the SQL power on this side : the search bar. The API doesn't implement this, I cant figure why not !
The few symbols, regexp-like or SQL-like can be handled aside to each-other without any problem : [ %\w] becomes [\*\.%\w] note: '_' is part of \w + a few code to stick to SQL, it resumes as three cases or so, in two syntax variants, as far as it seems.

And the regexp-search implementation is just a sequential access to strings, referenced in a temporary index table, with an opportunistic filter.

But at the present time I didn't find the doc explaining the raw access to the database with SQL directives, it is warped again inside another *?GrrR! ... API that sucks, I mean it gets to complicated object code again !

See You, Stan.

[aaFn]

Hello @stanpetit, I already tried to get access to Javascript Code modules (https://developer.mozilla.org/en-US/docs/Mozilla/JavaScript_code_modules) in the past, to get around problems with the bookmarks API, but that doesn't seem to be available to WebExternsion add-ons.
Only to priviledged scope code ... which would be kind of normal given the power they make available.

So it does not look like BSP2 can use those API's you point at.

In memory = yes, all the tree is now in memory.
This is how I got around the slow performance of the Bookmark API. 1st time run can be long depending on the number of bookmarks and tags because of that bug https://bugzilla.mozilla.org/show_bug.cgi?id=1396364,
but then all is stored in memory and in a local store of my own (which is now binary as I get it since FF 66 :-) ... so should be faster than before).

Note, this a tree, or more exactly a graph because I have in parallel a hashtable entry of each node also, besides the root, to get fast access to them.

So it is particularly fit for recursive algortihms. But not for relational DB's or SQL ... SELECT and so are not very good to traverse graphes.

With that structure, I can implement now my own search ... Using regexp is not a worry, it is just a filtering method like another .. Moreover, there appears to be in standard in Javascript and MDN what we need for it -> https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions
What do you think ot it ?

Meanwhile I have been thinking on a simple thing, to switch between different search modes: I could turn the magnifying glass sympol in the search box to a button, opening a drop down list, and changing the button image to show the mode it is in.
Right now, if you right click on it, it opens a small menu to open BSP2 in a web page .. which by the way should solve your problem of limited width of the search field .. :-)

Then the drop down could have the following sections

• Search fields (radio choices, only one can be selected)
  • search on title + url (default)
  • search on title only
  • search on url only
• Search scope (radio choices, only one can be selected)
  • search all (default)
  • search recursively from current folder, or from parent of current bookmark
• Match method (radio choices, only one can be selected)
  • simple match of multiple words (default) = like today
  • regexp match

Then I could have combinations of background color, border and glass symbol to visually show in the button which choices are selected in the drop down.

What do you think ?

[stanpetit]

Hi @aaFn. wow : so reactive !

It is sad that you cannot access the SQL layer.
In fact, from the previous posts, maybe you can picture that SQL is much more simple than regexp : with the SQL LIKE clause, we can perform a sequence of consecutive search statements on raw unordered words. This is what the search method of the API offers, that you implement onto.

With the addition of the SQL NOT, and the _ % operators, it is nearly as powerful as regexp ( in precision ) with a very simple, user friendly syntax. And more than this : it executes these compound searches in one optimized shot, very fast, whatever the order of the multiple words or expressions you pass in.

Also SQL truly adheres the set theory logic and offers it at a very 'low conceptual cost' : performing the same with regex can be so challenging and would discourage me ( and many others ) except for some specific well defined cases.

Regexp search is very precise for a search on 'one expression on characters'.
This way, it will sure be great for refining a scope into a previous 'natural' search that returns too many results.

Though, the main issue is still the lack of a complete implementation of the very obvious SQL clause LIKE ... / NOT LIKE ..., plus the _ % operators inside 'quoted expressions'.

In this sense, it would be very legitimate to request the Mozilla-Team to provide an alternate, extended search method that you could use directly in your search-field, with your existing mechanism, at (almost) no cost (previous post).

On their side it should be quite simple to implement this, with one more parameter to the search method itself : you already provide it a list of words or a string, that they split into a list. The additional parameter would be a list, or string, of zeros and one for LIKE or NOT LIKE, one for each word or 'quoted expression', not more (almost).
The '_' (1 char) and '%' (zero-or-many chars) operators inside 'quoted expressions' are just part of the SQL queries syntax itself ... (almost) zero effort to implement, just ask them : 'let it be' ( please ... )

As the developer of an extension, you may be heard more than I would be.
If you open a request, I can plead the cause If you ask me to ...

SQL and Regexp are complementary.
And it should begin with a complete implementation of the LIKE or NOT LIKE on the API side to get the full power : Regexp cant substitute to the lack of SQL.

About the regexp

Sure https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions is the js regexps man page.

If you are not familiar with them, there are some valuable tools :

• The site of RegexBuddy (a commercial app) which has a very detailed and extensive documentation on the regex syntax, including it's 'many flavors' : above chart, choose the JS lingo in the drop-down.
• regex101, nice for documented checking ! choose the JS lingo, left panel.
• txt2re, may look as a brain twister ...
• lehollandaisvolant that uses regulex : visualizing is not a gadget !

In fact, the only tricky stuff is that we can not mimic in a simple syntax the SQL behavior of unordered words search, which are so convenient : how could we remember the precise order of an input, furthermore if we didn't write by ourselves : do you rename any of your bookmarks, any time ?

the expr:    /.*dev.*you.*/    should match a whole string in the terms order : 

"As the developer of an extension, you may be heard"        matched
"you may be heard as the developer of an extension "    not matched

but:    /.*(?:dev|you).*(?:dev|you).*/    would match both of them.

also, no easy method to negate a group of chars : this is the main pitfall

Filtering results on negative assertion is complementary, and should be done 
with a simple js text search in a second shot. Regexp is not useful here.
In fact, Regexp is mostly useful for characters layout patterns.  

You can try this at regex101.
To keep yours, find SAVE AND SHARE/Update Regex on top-left.

About the user interface

Right now, if you right click on it, it opens a small menu to open BSP2 in a web page .. which by the way should solve your problem of limited width of the search field .. :-)

You are a smart dude !

What do you think ?

You realy ask me ? Sounds nice !
I guess it already written by the time I am writing now ...

Best, Stan.

[stanpetit]

Hi aaFn !
Sorry I was away : my daughter has had a bad accident (and a lots of luck too), so I left everything to go to their place (we live separated), where the life is so different, and many things were to be done ... I have forgotten everything else.

I hope that you are fine : sorry to read #94

Best,
Stan.

[aaFn]

Sorry to read that. Hope things are better now for you. On my side I was swamped with my work and travels, so couldn't start looking at coding this yet. Just made the bare minimum on recent releases to survive. Hopefully getting some vacations will help .. :-)
Yes, demand is sometimes closer to exigence. Thank you for caring.

[aaFn]

Hello @stanpetit , hope this will find you well.

So this is out, in 2.0.65 !
Let me know how this works for you, and if you find any bug.

Note: with FF 68, there is a bug on add-on updates, it keeps in cache some old version of add-on contents.
So once your BSP2 is updated to 2.0.65, close FF, then reopen it.

You will see immediately that something changed: the magnifying glass has become a colored active button, and clicking on it, you can change the field to search on, the scope (all or restricted to a folder), and the match method, with words (classical method) or regex :-)

Enjoy ..