List of smaller extraction bugs (text & metadata)

[adbar]

I have mostly tested trafilatura on a set of English, German and French web pages I had run into by surfing or during web crawls. There are definitely further web pages and cases in other languages for which the extraction doesn't work so far.

Corresponding bug reports can either be filed as a list in an issue like this one or in the code as XPath expressions in xpaths.py (see BODY_XPATH and COMMENTS_XPATH lists).

Thanks!

[adbar]

Extraction bugs in text and metadata can be listed here as in adbar/htmldate#8 where issues specifically related to dates should be reported.

• https://web.archive.org/web/20211229181034/https://research.checkpoint.com/2021/a-deep-dive-into-doublefeature-equation-groups-post-exploitation-dashboard/
• https://web.archive.org/web/20220217023331/https://thehill.com/homenews/senate/594044-sen-lujan-to-return-to-senate-in-time-to-vote-for-supreme-court-nominee
• https://web.archive.org/web/20220425123536/https://www.ansa.it/sito/notizie/politica/2022/04/25/25-aprile-sala-la-contestazione-alla-nato-e-folle_d63f0569-d81d-4831-8988-cfa6961c96cb.html (mentioned in Removal of useless new line and carriage return characters in Html headings and paragraphs #199)

For details see below.

[cheezman34]

Words are getting smashed together on this page:

https://research.checkpoint.com/2021/a-deep-dive-into-doublefeature-equation-groups-post-exploitation-dashboard/

I looked into the extraction code here a bit. The date here is inside a span, which gets stripped, and then the date becomes the tail of the header. All of the whitespace (which includes a newline) gets lost, and then the tail is just directly appended to the header. I'm not sure if the best strategy to fix would be to include a space between the tail and text of the header node when they get extracted, or maybe to look for newlines in the text and somehow respect them. It looks like some of the lxml stuff just strips whitespace automatically when you access "text" and "tail" attributes.

I didn't dig into this one, but I'm guessing it's something similar as the first case. The webpage relies on whitespace that gets stripped by the extraction algorithm.

[adbar]

Yes, I think the issues in the document you mention are related to deleted <span> sections.

[karlkovaciny]

Hey, this is a great library. I was ready to subscribe to a service just to get what this does for me.

For the 30th page I extracted, https://thehill.com/homenews/senate/594044-sen-lujan-to-return-to-senate-in-time-to-vote-for-supreme-court-nominee, Trafilatura 1.0.0 returned only 150 chars of text:

© Greg Nash
Luján planning return to Senate in time to vote for Supreme Court nominee
By Olafimihan Oshin - 02/13/22 12:54 PM EST
Skip to main content

I downloaded the HTML source (lujan.txt) and confirmed it does have the article text in it (starting with "Sen. Ben Ray Luján").

I decided to try the external fallback "Readability". I started Python in my trafilatura container and ran this code:

import lxml
with open('lujan.html') as f:
     doc = parse(f).getroot()
     x = trafilatura.external.try_readability(doc, "file:///lujan.html")
     print(lxml.etree.tostring(x, pretty_print=True, encoding="unicode"))

But that just gave me a bunch of XML/JavaScript that didn't even have the main text in it.

Perhaps a fallback could be added that when extracted text is small and there are large continuous blocks of unextracted text, to grab those instead?

[adbar]

Hi @karlkovaciny, the cutting-edge version from the repository is slightly better, it outputs the article but still includes garbled javascript. That's definitely a case to watch for.

EDIT: for the archived version of the page I now get the same problem as you.

[adbar]

Suggested in #208:

• Text below the article: https://web.archive.org/web/20220513095359/https://www.vivereancona.it/2022/05/13/ubriaco-cammina-lungo-la-flaminia-quando-vede-arrivare-i-soccorsi-si-getta-nei-cespugli-e-fugge/2100180067/
• Few text lost in false positives (other articles): https://web.archive.org/web/20220513100231/https://www.vallesabbianews.it/notizie-it/%28Ro%C3%A8-Volciano%29-Davide-Nedrotti-campione-del-mondo-60595.html
• Text above and below: https://web.archive.org/web/20220513100330/https://www.quinewsfirenze.it/firenze-terremoto-chianti.htm