Fix/joss review 2021 08 30 (#37)

* fix: use LICENSE rather than COPYING for the license file.

* add: contribution guidelines

These guidelines have been based on the guidelines used by the atom
and gnome-todo projects and been adapted to inscripits.

* chg: refined documentation.

* add: more examples for annotation profiles.

* chg: improved examples.

* fix: documentation on annotation rules.

* wip: improved statement of need.

* add: improved statement of need based on the reviewer comments

* fix: grammar, spelling and linking erros.

* chg: extended guidelines.

* fix: spacing and phrasing of non-specialized tools.

* fix: codefactor issues.

* chg: do not consider conf.py for codefactor.io

* chg: do not consider conf.py for codefactor.io

Author: AlbertWeichselbraun

CONTRIBUTING.md

@@ -0,0 +1,54 @@
+# Contributing to Inscriptis
+
+First off, thank you for considering contributing to inscriptis. 
+There are many ways how you can contribute to the project and these guidelines aim at supporting you in doing so.
+
+1. [Reporting bugs and seeking support](#reporting-bugs-and-seeking-support)
+2. [Suggesting enhancements](#suggesting-enhancements)
+3. [Pull requests](#pull-requests) (contributing code)
+4. [Python style guide](#python-style-guide)
+
+
+## Reporting bugs and seeking support
+
+Bugs and support requests are tracked as GitHub issues.
+
+To create an effective and high quality ticket, please include the following information in your
+ticket:
+
+ 1. **Use a clear and descriptive title** for the issue to identify the problem. This also helps other users to quickly locate bug reports that affect them.
+ 2. **Describe the exact steps necessary for reproducing the problem** including at least information on
+    - the affected URL
+    - the command line parameters or function arguments you used
+ 3. What would have been the **expected behavior**?
+ 4. Describe the **observed behavior**.
+ 5. Provide any additional information which might be helpful in reproducing and/or fixing this issue. 
+
+
+## Suggesting enhancements
+
+Enhancements are also tracked as GitHub issues and should contain the following information:
+
+ 1. **A clear and descriptive title** helps other people to identify enhancements they like, so that they can also add their thoughts and suggestions.
+ 2. **Provide a step-by-step description** of the suggested enhancement.
+ 3. **Describe the current behavior** and **explain which behavior you expected to see instead** and why.
+
+
+## Pull requests
+
+1. Ensure that your code complies with our [Python style guide](#python-style-guide).
+2. Write a unit test that covers your new code and put it into the `./tests` directory.
+3. Execute `tox .` in the project's root directory to ensure that your code passes the static code analysis, coding style guidelines and security checks.
+4. In addition, please document any new API functions in the Inscriptis documentation.
+
+
+## Python style guide
+
+Inscriptis code should comply to
+- the [PEP8 Style Guide for Python Code](https://www.python.org/dev/peps/pep-0008/), and
+- to the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)
+
+Please also ensure that 
+1. functions are properly documented with docstrings that comply to the Google Python Style Guide, and
+2. any new code is covered by unit tests.
+

COPYING renamed to LICENSE

@@ -32,7 +32,10 @@ inscriptis -- HTML to text conversion library, command line client and Web servi
 
 A python based HTML to text conversion library, command line client and Web
 service with support for **nested tables**, a **subset of CSS** and optional
-support for providing an **annotated output**.
+support for providing an **annotated output**. 
+
+Inscriptis is particularly well suited for applications that require high-performance, high-quality (i.e., layout-aware) text representations of HTML content, and will aid knowledge extraction and data science tasks conducted upon Web data.
+
 Please take a look at the
 `Rendering <https://github.com/weblyzard/inscriptis/blob/master/RENDERING.md>`_
 document for a demonstration of inscriptis' conversion quality.
@@ -47,6 +50,38 @@ This document provides a short introduction to Inscriptis.
 
 .. contents:: Table of contents
 
+Statement of need - why inscriptis?
+===================================
+
+1. Inscriptis provides a **layout-aware** conversion of HTML that more closely resembles the rendering obtained from standard Web browsers and, therefore, better preserves the spatial arrangement of text elements. 
+
+   Conversion quality becomes a factor once you need to move beyond simple HTML snippets. Non-specialized approaches and less sophisticated libraries do not correctly interpret HTML semantics and, therefore, fail to properly convert constructs such as itemizations, enumerations, and tables.
+
+   Beautiful Soup's `get_text()` function, for example, converts the following HTML enumeration to the string `firstsecond`.
+
+   .. code-block:: HTML
+   
+      <ul>
+        <li>first</li>
+        <li>second</li>
+      <ul>
+
+
+   Inscriptis, in contrast, not only returns the correct output
+   
+   .. code-block::
+   
+      * first
+      * second
+
+   but also supports much more complex constructs such as nested tables and also interprets a subset of HTML (e.g., `align`, `valign`) and CSS (e.g., `display`, `white-space`, `margin-top`, `vertical-align`, etc.) attributes that determine the text alignment. Any time the spatial alignment of text is relevant (e.g., for many knowledge extraction tasks, the computation of word embeddings and language models, and sentiment analysis) an accurate HTML to text conversion is essential.
+
+2. Inscriptis supports `annotation rules <#annotation-rules>`_, i.e., user-provided mappings that allow for annotating the extracted text based on structural and semantic information encoded in HTML tags and attributes used for controlling structure and layout in the original HTML document. These rules might be used to
+
+   - provide downstream knowledge extraction components with additional information that may be leveraged to improve their respective performance.
+   - assist manual document annotation processes (e.g., for qualitative analysis or gold standard creation). ``Inscriptis`` supports multiple export formats such as XML, annotated HTML and the JSONL format that is used by the open source annotation tool `doccano <https://github.com/doccano/doccano>`_.
+   - enabling the use of ``Inscriptis``  for tasks such as content extraction (i.e., extract task-specific relevant content from a Web page) which rely on information on the HTML document's structure.
+
 
 Installation
 ============
@@ -125,11 +160,8 @@ The inscript.py command line client supports the following parameters::
     -v, --version         display version information
    
 
-Examples
---------
-
 HTML to text conversion
-~~~~~~~~~~~~~~~~~~~~~~~
+-----------------------
 convert the given page to text and output the result to the screen::
 
   $ inscript.py https://www.fhgr.ch
@@ -144,7 +176,7 @@ convert HTML provided via stdin and save the output to output.txt::
 
 
 HTML to annotated text conversion
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+---------------------------------
 convert and annotate HTML from a Web page using the provided annotation rules::
 
   $ inscript.py https://www.fhgr.ch -r ./examples/annotation-profile.json
@@ -188,10 +220,11 @@ yields the following JSONL output
 The provided list of labels contains all annotated text elements with their
 start index, end index and the assigned label.
 
+
 Annotation postprocessors
-~~~~~~~~~~~~~~~~~~~~~~~~~
+-------------------------
 Annotation postprocessors enable the post processing of annotations to formats
-that are suitable for you particular application. Post processors can be
+that are suitable for your particular application. Post processors can be
 specified with the `-p` or `--postprocessor` command line argument::
 
   $ inscript.py https://www.fhgr.ch \
@@ -233,7 +266,7 @@ Currently, inscriptis supports the following postprocessors:
 
 .. figure:: https://github.com/weblyzard/inscriptis/raw/master/docs/paper/images/annotations.png
    :align: left
-   :alt: Annotations extracted from the Wikipedia entry for Chur wht the `--postprocess html` postprocessor.
+   :alt: Annotations extracted from the Wikipedia entry for Chur with the `--postprocess html` postprocessor.
 
    Snippet of the rendered HTML file created with the following command line options and annotation rules:
 
@@ -293,6 +326,97 @@ The service also supports a version call::
   $ curl http://localhost:5000/version
 
 
+Example annotation profiles
+===========================
+
+The following section provides a number of example annotation profiles illustrating the use of Inscriptis' annotation support.
+The examples present the used annotation rules and an image that highlights a snippet with the annotated text on the converted web page, which has been 
+created using the HTML postprocessor as outlined in Section `annotation postprocessors <#annotation-postprocessors>`_.
+
+Wikipedia tables and table metadata
+-----------------------------------
+
+
+The following annotation rules extract tables from Wikipedia pages, and annotate table headings that are typically used to indicate column or row headings.
+
+.. code-block:: json
+
+   {
+      "table": ["table"],
+      "th": ["tableheading"],
+      "caption": ["caption"]
+   }
+
+The figure below outlines an example table from Wikipedia that has been annotated using these rules.
+
+.. figure:: https://github.com/weblyzard/inscriptis/raw/master/docs/images/wikipedia-chur-table-annotation.png
+   :alt: Table and table metadata annotations extracted from the Wikipedia entry for Chur.
+
+
+References to entities, missing entities and citations from Wikipedia
+---------------------------------------------------------------------
+
+This profile extracts references to Wikipedia entities, missing entities and citations. Please note that the profile isn't perfect, since it also annotates `[ edit ]` links.
+
+.. code-block:: json
+
+   {
+      "a#title": ["entity"],
+      "a#class=new": ["missing"],
+      "class=reference": ["citation"]
+   }
+
+The figure shows entities and citations that have been identified on a Wikipedia page using these rules.
+
+.. figure:: https://github.com/weblyzard/inscriptis/raw/master/docs/images/wikipedia-chur-entry-annotation.png
+   :alt: Metadata on entries, missing entries and citations extracted from the Wikipedia entry for Chur.
+
+
+
+
+
+Posts and post metadata from the XDA developer forum
+----------------------------------------------------
+
+The annotation rules below, extract posts with metadata on the post's time, user and the user's job title from the XDA developer forum.
+
+.. code-block:: json
+
+   {
+       "article#class=message-body": ["article"],
+       "li#class=u-concealed": ["time"],
+       "#itemprop=name": ["user-name"],
+       "#itemprop=jobTitle": ["user-title"]
+   }
+
+The figure illustrates the annotated metadata on posts from the XDA developer forum.
+
+.. figure:: https://github.com/weblyzard/inscriptis/raw/master/docs/images/xda-posts-annotation.png
+   :alt: Posts and post metadata extracted from the XDA developer forum.
+
+
+
+Code and metadata from Stackoverflow pages
+------------------------------------------
+The rules below extracts code and metadata on users and comments from Stackoverflow pages.
+
+.. code-block:: json
+
+   {
+      "code": ["code"],
+      "#itemprop=dateCreated": ["creation-date"],
+      "#class=user-details": ["user"],
+      "#class=reputation-score": ["reputation"],
+      "#class=comment-date": ["comment-date"],
+      "#class=comment-copy": ["comment-comment"]
+   }
+
+Applying these rules to a Stackoverflow page on text extraction from HTML yields the following snippet:
+
+.. figure:: https://github.com/weblyzard/inscriptis/raw/master/docs/images/stackoverflow-code-annotation.png
+   :alt: Code and metadata from Stackoverflow pages.
+
+
 Advanced topics
 ===============
 
@@ -354,7 +478,7 @@ The following options are available for fine tuning inscriptis' HTML rendering:
    parameter `indentation='extended'` to also use indentation for tags such as
    `<div>` and `<span>` that do not provide indentation in their standard
    definition. This strategy is the default in `inscript.py` and many other
-   tools such as lynx. If you do not want extended indentation you can use the
+   tools such as Lynx. If you do not want extended indentation you can use the
    parameter `indentation='standard'` instead.
 
 2. **Overwriting the default CSS definition:** inscriptis uses CSS definitions
@@ -380,10 +504,11 @@ The following options are available for fine tuning inscriptis' HTML rendering:
       config = ParserConfig(css=css)
       parser = Inscriptis(html_tree, config)
       text = parser.get_text()
-   
+
 
 Changelog
 =========
 
 A full list of changes can be found in the
 `release notes <https://github.com/weblyzard/inscriptis/releases>`_.
+

README.rst

@@ -37,15 +37,17 @@
 extensions = ['sphinx.ext.autodoc',
     'sphinx.ext.viewcode',
     'sphinx.ext.githubpages',
-    'sphinx.ext.napoleon']
+    'sphinx.ext.napoleon',
+    'myst_parser']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
-source_suffix = ['.rst', '.md']
+source_suffix = {'.rst': 'restructuredtext',
+                 '.md': 'markdown'}
 
 # The master toctree document.
 master_doc = 'index'
@@ -193,5 +195,3 @@
 
 # A list of files that should not be packed into the epub file.
 epub_exclude_files = ['search.html']
-
-

docs/conf.py

@@ -0,0 +1 @@
+../CONTRIBUTING.md

docs/contributing.md

@@ -13,8 +13,9 @@ Contents:
 .. toctree::
    :maxdepth: 2
 
-   installation
+   Documentation <README>
    benchmarking
+   contributing
    inscriptis-module-documentation