Please provide source for JS code in sphinx/search/*.py

[mitya57]

In most Python files in sphinx/seach/*.py, there is an embedded minified JavaScript code. While it looks tiny, it is actually quite large (for example, the version in ru.py contains over 14000 symbols).

In distributions like Debian and Fedora, this is considered non-free code, and we can't ship it (see also previous discussions in #1434). Also, it is completely impossible to understand what is happening in that code and modify it.

As far as I understand, each such code is made of two parts (please correct me if my guess is wrong):

• Some common code (which is the same in all .py files);
• The language-specific part which is somehow generated from the "stopwords" list.

It would be very nice if the following things are provided:

• A non-minified version of the "shared" part of that code;
• A script that can generate the "language-specific" part;
• A script and/or README file that describes how to join the parts together and minify them.

[shimizukawa]

@shibukawa please take a look this ticket.

[shibukawa]

Hi,

Natural language processing code is generated by the following tool. This was modified from the original snowball code generator:

https://github.com/shibukawa/snowball/

This tool is written in C. Original tool can generate C and Java, and my modified tool can generates Python and JSX(static altJS) code in addition to original C/Java. Generated Python scripts are in another pypi package:

https://pypi.python.org/pypi/snowballstemmer

And generated JSX code is in the following npm package:

https://www.npmjs.com/package/snowball-stemmer.jsx

And when I modified Sphinx code, I wrote description at the end of dev guide:

http://sphinx-doc.org/devguide.html#debugging-tips

I think bundling Snowball code generator is not needed (because it is a big tool and not special for Sphinx). And there have been the document describe it. So the task I have to do is only bundling non-minified JS code, I think.

Is it ok for you?

[mitya57]

@shibukawa: Yes, having the non-minified JS code somewhere will be fine for us.

[shimizukawa]

I received the non-minified JS files from @shibukawa. I'll take over this issue.

Just memo: amount of non-minified 16 language JS files is 1.2MB (as tgz: 128KB).

[mitya57]

I will be also fine with having a script to generate them in the tarball, even if the script will be using some external software — as an alternative to shipping non-minified files.

[mitya57]

@shimizukawa Any progress on this issue? This is currently blocking Sphinx 1.3 from being uploaded to Debian (and as I have already said, there are multiple ways to make us happy).

[shimizukawa]

@mitya57 Sorry, I kept you waiting. Now I occupied my day job... Maybe I can make a time at end of this April.

[mitya57]

@shimizukawa No problem. Can you please send me the “non-minified JS files that you received from @shibukawa” mentioned above, for the time being? (You can find my email in Sphinx commits log).

[shimizukawa]

@mitya57 this is. https://dl.dropboxusercontent.com/u/284189/sphinx-issue-1784-snowballstemmer-raw-js-files.zip

[mitya57]

Thanks!

[mitya57]

@shimizukawa Any update on this issue? Are you going to fix it, or should we go back to repacking tarball in Debian (as we did in past)?

[shimizukawa]

@mitya57 I made a change for the issue. Please can you try the stable code?

[mitya57]

@shimizukawa Much better now, thanks a lot!

Can you please also tell me which commandline you used to generate the full JS file? (I realize that I need snowballstemmer, but I don't know anything about how to use it).

[shimizukawa]

Well, ... as @shibukawa mentioned at #1784 (comment) , generating full JS command is here (I guess..): https://www.npmjs.com/package/snowball-stemmer.jsx

@shibukawa Is it correct?

[mitya57]

OK, thanks again!

[mitya57]

FWIW I have proposed a pull request to merge @shibukawa's generators into mainstream snowball, see snowballstem/snowball#24. This will allow Sphinx to use the original project's code, not the fork.

[Naereen]

I think this is still an issue, now the latest sphinx version seems to require the non-minified-js/*-stemmer.js files, but they are not installed automatically on PATH/sphinx/search/ if sphinx is installed with pip from the live git branch (dunno if the stable pypi package has the same issue).

Please find attached a log of the issue I just encountered (I add to download manually the non-minified-js/*-stemmer.js file from here).
sphinx-err.log

[shimizukawa]

Now #2114 has been merged to fix missing *-stemmer.js files. @Naereen could you check it?

[Naereen]

Seems fixed to me. Thanks.