linking of pygetpapers docanalysis and amilib to create a corpus browser and analyzer

[petermr]

Ideas:

• pygetpapers implicitly creates a corpus of results; make this explicit
• corpora can come from:
  • pygetpapers (EPMC, ?OpenAlex)
  • existing coherent collections of documents (e.g. IPCC, UNFCCC but certainly many more)
• corpora should be normalized, sectioned and id-numbered down to at least paragraph level (and sometimes sentences). Normalization includes:
  • separate directories for each corpus component (because we generate additional info from each)
  • normalization should be to a standard HTML schema (e.g. figures and tables should have captions, tables should have thead tbody, etc.)
• tools should be on a pipeline, not monolithic. Intermediate results should be output to file for checking and restarting. Even "raw" HTML may need 2-3 pipeline steps (e.g. normalization, id-numbering). We may need a graphical pipeline tool.
• Move corpus functionality from amilib into docanalysis. amilib should not be the top-level tool but used as an engine and library to support the others
• Legacy2HTML (e.g. PDF) can be done with amilib before offering to docanalysis.

Our USPs include:

• simplicity of use of pygetpapers (possibly with an optional GUI component to assemble the query)
• full-text retrieval and analysis, especially text-based

<please comment, add more>

[petermr]

current amilib functionality by module:

Module	Function
ami_args.py	Abstract class for argparse option
ami_bib.py	Bibliographic support
ami_corpus.py	create, normalize, search, tranform a corpus
ami_csv.py	CSV utilities
ami_dict.py	Ami Dictionary
ami_graph.py	(stub)
ami_html.py	large collection for manipulating HTML
ami_integrate.py	miscellaneous conversions
ami_nlp.py	(stubs)
ami_pdf_libs.py	large PDF2HTML , includes pdfplumber
ami_svg.py	(stub) mainly convenience routiines
ami_util.py	lowlevel numeric/geom utilities
amidriver.py	headless browser
amix.py	entry point for amilib main
bbox.py	bounding box manipulation
constants.py	(stub)
dict_args.py	dictionary options in argparse
file_lib.py	file utilities
headless_lib.py	messy utility routines (may be obsolete)
html_args.py	HTML options in argparse
html_extra.py	stub obsolete?
html_generator.py	messy ?obsolete
html_marker.py	markup HTML (messy)
pdf_args.py	PDF options in argprase
search_args.py	search options in argparse
util.py	many scattered utilities
wikimedia.py	download and conversion of Wikimedia
xml_lib.py	convenience routies (lxml wrappers)

[petermr]

Proposal:

• transfer corpus functionality to docanalysis. @petermr to help maintain while @ShweataNHegde is busy
• amilib continues to serve various functionalities:
  • PDF conversion
  • HTML management
  • datatables support
• downloading:
  • OpenAlex?
  • headless HTML?