Historically accumulated comprehensive PSL generation and/or dissemination

[ibnesayeed]

Many public suffixes, both public and private, have been added to the list, but some of them were removed from the list at later date. While this is great to have an up-to-date list to assess the live web, but when we need to process historical URLs those removed suffixes cause issues. It would be really helpful to maintain an add-only list (in addition to the existing one) from which no historically valid suffixes are ever removed. Albeit, such deletions can be marked with an inline note containing date of removal and reason.

One prominent use-case for such a historically accumulated PSL comes from web archives like the Wayback Machine of the Internet Archive and many other web archives around the world.

In the past I have tried two approaches to generate such an accumulated list:

• From the Wayback Machine: I downloaded all the historical captures of the published PSL file from the Wayback Machine, merged them all, removed any comments and empty lines, and sorted them to create a list of unique suffixes.
• From the Git history of this repo: I checked out every version of the list file from this repo iteratively and merged them to create the most comprehensive PSL (which resulted in more suffixes than the ones collected from the Wayback Machine).

While these approaches work for the most part, they lose a great deal of context and annotations. There is also the risk of including wrong or misspelled prefixes in such automatically generated comprehensive lists which might have been added by mistake in a revision, but were reverted later.

It would be helpful to provision one or more of the following, if practical:

• Include scripts to automatically generate comprehensive lists from the repo history while considering exclusions of any known bad entries.
• Maintaining such a comprehensive list manually with necessary comments and annotations.
• Disseminating the alternate comprehensive PSL from a well-known URL.

[simon-friedberger]

Can you elaborate what kind of processing you want to do on historical URLs? I don't quite understand why this would be useful.

[ibnesayeed]

@simon-friedberger I assumed web archiving as a practice and as a research discipline is a common knowledge now, but I understand that it might not be part of the day-to-day experience of everyone. To give you a little bit of background, I am the research lead of the Wayback Machine at the Internet Archive, where we have been archiving the web for almost three decades now. In addition to simply playing back historical versions of web pages, we produce all kinds of summaries and reports on the historical web and many web researchers interact with our APIs and services to conduct their researches too. In many cases it is important for us to realize various parts of a hostname and draw boundaries around registered domain and subdomains.

Now, imagine we (or any external researchers) want to assess how many web pages are archived for each domain name in each year or to find out the average lifespan of web pages by domain name. Removals like #2327 mess the statistics up for historical captures of web pages. For example, before this PR was merged, hostnames like ws-dl.blogspot.co.uk and ibnesayeed.blogspot.co.uk would be treated as independent domains, but after the PR was merged, all such blogs would be consolidated as subdomains of blogspot.co.uk. Similarly, if a TLD is removed from the list, because it is not in practice anymore, all their historical captures might be treated as captures of invalid domains (if a tool is relying on PSL for hostname parsing).

[dnsguru]

What you are asking makes more sense now with the context of what your objective is.

Thank you for your hard work on an important internet resource. We work hard on this internet resource and appreciate what goes into it.

We encourage you perhaps consider a process to download and archive a PSL daily and incorporate it into your process going forward. Not clear on what options for calling back any historical day.

A variety of different projects here do things with the PSL like turn it into zone files or json - those might have some cached versions from along their journeys.

[simon-friedberger]

Sounds reasonable, do watch out that sometimes domains are intentionally removed and creating just one list that applies to all the past sites is probably also not correct.

Is there something you would need from us that the git history of this repository does not provide?

[ibnesayeed]

Is there something you would need from us that the git history of this repository does not provide? -- @simon-friedberger

I guess, some automated means or a curated list of commits to avoid this:

do watch out that sometimes domains are intentionally removed and creating just one list that applies to all the past sites is probably also not correct -- @simon-friedberger

In the original post I said this:

There is also the risk of including wrong or misspelled prefixes in such automatically generated comprehensive lists which might have been added by mistake in a revision, but were reverted later.

[dnsguru]

@simon-friedberger has a good point... we have, from time to time, had people reverse a request when it did not accomplish for them what they had hoped, or where they had not RTFM and broke their website. Probably best to omit those. These unfortunately are not labelled as such.

[simon-friedberger]

If somebody registers blogspot again they might not want it to be a public suffix. I don't see a way around having a list for each point in time. When you look at a site which you captured three months ago your best bet is to grab the PSL from three months ago.

I'm afraid there is nothing we can do. I believe a curated list would not accomplish much, our tooling is good enough that we almost never publish an incorrect list, at least I am not aware of that happening and when people change their mind it's just not obvious what the result should be, that depends on the details of how the data is being used.