[Performance] Xpath with regex

[deedy5]

There is a way to speed up xpath expressions: https://lxml.de/xpathxslt.html#regular-expressions-in-xpath

expr = BODY_XPATH[0]
expr2 = expr, but using regex

import re
from time import time
import requests
from lxml import html

url = "https://www.morgenpost.de/berlin/article408569802/berlin-polizei-feuerwehr-news-blaulicht-aktuell-2103-unfall-radfahrerin-taxi-schwere-verletzungen.html"
resp = requests.get(url)
tree = html.fromstring(resp.text)

expr = """.//*[self::article or self::div or self::main or self::section][
    @class="post" or @class="entry" or
    contains(@class, "post-text") or contains(@class, "post_text") or
    contains(@class, "post-body") or contains(@class, "post-entry") or contains(@class, "postentry") or
    contains(@class, "post-content") or contains(@class, "post_content") or
    contains(@class, "postcontent") or contains(@class, "postContent") or contains(@class, "post_inner_wrapper") or
    contains(@class, "article-text") or contains(@class, "articletext") or contains(@class, "articleText")
    or contains(@id, "entry-content") or
    contains(@class, "entry-content") or contains(@id, "article-content") or
    contains(@class, "article-content") or contains(@id, "article__content") or
    contains(@class, "article__content") or contains(@id, "article-body") or
    contains(@class, "article-body") or contains(@id, "article__body") or
    contains(@class, "article__body") or @itemprop="articleBody" or
    contains(translate(@id, "B", "b"), "articlebody") or contains(translate(@class, "B", "b"), "articlebody")
    or @id="articleContent" or contains(@class, "ArticleContent") or
    contains(@class, "page-content") or contains(@class, "text-content") or
    contains(@id, "body-text") or contains(@class, "body-text") or
    contains(@class, "article__container") or contains(@id, "art-content") or contains(@class, "art-content")][1]"""
expr2 = """(.//article | .//div | .//main | .//section)[
    re:test(@class,
        '^post$|^entry$|post-text|post_text|post-body|post-entry|postentry|post-content|post_content|postcontent|
        post_inner_wrapper|article-text|articletext|entry-content|article-content|article__content|article-body|
        article__body|articlebody|articlecontent|page-content|text-content|body-text|article__conteiner|art-content',
    'i') or
    re:test(@id,
        'entry-content|article-content|article__content|article-body|
        article__body|articlebody|articlecontent|body-text|art-content',
    'i') or
    re:test(@itemprop, 'article-body', 'i')
    ][1]"""

t0 = time()
r = tree.xpath(expr)
text = r[0].text_content()
print(time() - t0, len(text))
print(text)

t0 = time()
r = tree.xpath(expr2, namespaces={'re':"http://exslt.org/regular-expressions"})
text = r[0].text_content()
print(time() - t0, len(text))
print(text)

expr time = 0.013
expr2 time = 0.008

If you don't mind, it is possible to rewrite all complex xpath expressions in the library.
There should be a noticeable performance gain.

[deedy5]

Tried xpath with regex on several sites.
In general, the idea is bad, because such expressions cannot be debugged in the browser.