Introduce a variant of url.searchParams that operates according to URL rules instead of <form> rules

[domenic]

Problem

See background in #18 and #478.

URLSearchParams was designed, not to hold URL query data, but instead to hold application/x-www-form-urlencoded data, i.e. the data that is sent to a server when submitting a HTML <form>.

Unfortunately, it was misnamed URLSearchParams instead of ApplicationXWWWFormURLEncodedParams. And, even more unfortunately, a property named searchParams was added to the URL class, which is an instance of the URLSearchParams class. Any attempts to use the searchParams class will give misleading information about the URL. And any attempts to manipulate it will change the contents of your URL's query string in unintended ways, converting values from a query string serialization (of the type produced by the URL parser) into an application/x-www-form-urlencoded serialization.

Some examples of how url.searchParams does not allow faithful introspection into the URL record:

const urlA = new URL('http://localhost:9999/segment?foo=bar/baz? boo');
const urlB = new URL('http://localhost:9999/segment?foo=bar%2Fbaz%3F%20boo');

// Not equal:
console.log(urlA.href); // "http://localhost:9999/segment?foo=bar/baz?%20boo"
console.log(urlB.href); // "http://localhost:9999/segment?foo=bar%2Fbaz%3F%20boo"

console.log(urlA.search); // "?foo=bar/baz?%20boo"
console.log(urlB.search); // "?foo=bar%2Fbaz%3F%20boo"

// Equal:
console.log(urlA.searchParams.get("foo")); // "bar/baz? boo"
console.log(urlB.searchParams.get("foo")); // "bar/baz? boo"

// Equal, but both different from search:
console.log(urlA.searchParams.toString()); // "foo=bar%2Fbaz%3F+boo"
console.log(urlB.searchParams.toString()); // "foo=bar%2Fbaz%3F+boo"

Some examples of how using url.searchParams for mutation will cause unintended changes to your URL record:

const url = new URL('http://httpbin.org/anything?a=~');

console.log(url.href);   // "http://httpbin.org/anything?a=~"
console.log(url.search); // "?a=~"

// This should be a no-op, but it is not:
url.searchParams.set("a", url.searchParams.get("a"));

console.log(url.href); // "http://httpbin.org/anything?a=%7E"

const url = new URL('http://httpbin.org/anything?a=~');

console.log(url.href);   // "http://httpbin.org/anything?a=~"
console.log(url.search); // "?a=~"

// This should not change the value of a, but it does:
url.searchParams.set("b", "d");

console.log(url.href); // "http://httpbin.org/anything?a=%7E&b=d"

const url = new URL('http://httpbin.org/anything?a=b c');

console.log(url.href);   // "http://httpbin.org/anything?a=b%20c"
console.log(url.search); // "?a=b%20c"

// This should be a no-op (sorting a single-element set), but it is not:
url.searchParams.sort();

console.log(url.href); // "http://httpbin.org/anything?a=b+c"

Solution

In #478 (comment) I proposed four solutions to this problem. In response, @ricea (Chromium) and @achristensen07 (WebKit) indicated they were "in favor of maintaining the status quo". I interpret this as meaning that any changes to either the URL query string parser/serializer, or the application/x-www-form-urlencoded parser/serializer, or the URLSearchParams class and url.searchParams member, are not on the table.

Given these constraints, it seems the only thing we could do is propose a new non-breaking addition to the API. As such, I propose a URLQueryParams class and a corresponding url.queryParams member, which are identical to URLSearchParams and url.searchParams, except that they use the URL parsing/serialization rules instead of the application/x-www-form-urlencoded rules. (Alternate names include url.realSearchParams or url.searchParams2.)

With that added, we could effectively deprecate url.searchParams (i.e., state loudly in the spec and MDN that using it will give unreliable results and mess up your URLs), and note that URLSearchParams is useful for representing <form> serialization, but not useful for manipulating URL search parameters.

(Optionally, we might want to define url.query / location.query / workerLocation.query as aliases for the corresponding .search properties, to fully align on the "query" naming and obsolete the "search" naming. But that's separable.)

[annevk]

The whole parameter-based format came from forms and is reflected in URLs when you use GET. Where does the parameter-based format that does not originate from forms come from?

[mpriour]

@annevk The non form format comes from APIs and application level routers.

[domenic]

Where does the parameter-based format that does not originate from forms come from?

Are you asking where the rules in https://url.spec.whatwg.org/#query-state came from? I mean, you wrote them down :). I'd presume from one of the RFCs.

[annevk]

I'm saying that never had an official key-value format. As far as I know application/x-www-form-urlencoded is the only thing that does something with & and =. I think there might be some servers that also do something with ; though, but not sure how official that ever was.