Skip to content

Leading whitespace in sitemaps isn't stripped #491

Open
@JustAnotherArchivist

Description

@JustAnotherArchivist

wpull does not strip the leading whitespace between the opening <loc> and the actual location, thus constructing a relative URL with spaces instead. The trailing whitespace is getting stripped. ArchiveBot job 72ckueskvsyzq8444tfso85g0 extracted URLs like https://www.windband.ch/%20%20%20%20%20%20%20%20%20%20%20%20https:/windband.ch/angebot from the following https://www.windband.ch/sitemap.xml:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">
    <url>
        <loc>
            https://windband.ch
        </loc>
        <xhtml:link rel="alternate" hreflang="fr" href="https://windband.ch/fr"/>
        <xhtml:link rel="alternate" hreflang="it" href="https://windband.ch/it"/>
        <xhtml:link rel="alternate" hreflang="de-CH" href="https://windband.ch"/>
        <xhtml:link rel="alternate" hreflang="x-default" href="https://windband.ch"/>
    </url>
    <url>
        <loc>
            https://windband.ch/angebot
        </loc>
        <xhtml:link rel="alternate" hreflang="fr" href="https://windband.ch/fr/angebot"/>
        <xhtml:link rel="alternate" hreflang="it" href="https://windband.ch/it/angebot"/>
        <xhtml:link rel="alternate" hreflang="de-CH" href="https://windband.ch/angebot"/>
        <xhtml:link rel="alternate" hreflang="x-default" href="https://windband.ch/angebot"/>
    </url>
...

This has happened on other ArchiveBot jobs before. (Side note, the extra <xhtml:link> are not getting extracted.)

It's worth mentioning that XML parsers must pass the whitespace through and that, while there is no solid standard on the sitemap.xml format, the closest thing makes no mention of whitespace.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions