HTML filters occasionally lost effectiveness #42

Crystal-RainSlide · 2018-05-15T11:07:46Z

Prerequisites

I verified that this is not a filter issue
This is not a support issue or a question
I performed a cursory search of the issue tracker to avoid opening a duplicate issue
I tried to reproduce the issue when...
- uBlock Origin is the only extension
- uBlock Origin with default lists/settings
- using a new, unmodified browser profile
I am running the latest version of uBlock Origin
I checked the documentation to understand that the issue I report is not a normal behavior

Description

An ad script in https://www.baidu.com/s?wd=%E7%9A%AE%E9%9E%8B

<script id="ecomScript">
...
<script>

I tried to remove it with www.baidu.com##^script#ecomScript, but I found that only script:contains() is supported.

Then I took some time with the code and got 'setAdsHeight' in the js outline from Firefox's dev tools, then everything works fine, but the time has gone.

Certainly CSS Selector don't support this, but adblocks may support it for simplifing the rules&filtering.

A specific URL where the issue occurs

All website equipped with <script foo="bar">

Steps to Reproduce

Descripted↑

Expected behavior:

example.com##script#id

Actual behavior:

Nothing happens & desired filter don't work

Your environment

uBlock Origin version: Newest
Browser Name and version: Firefox Developer Edition 61.0
Operating System and version: Windows 7 Customed.

The text was updated successfully, but these errors were encountered:

Crystal-RainSlide · 2018-05-15T11:15:25Z

One other thing...

I mistaked the outdated fork source of fang5566/uBlock , which is the offical Chinese (中文) introduction&wiki , and posted uBlock-LLC/uBlock#1767 .

Then I got a ban at https://github.com/gorhill/uBlock/issues/new for that misposted issue.

WHY???

gorhill · 2018-05-15T11:16:48Z

There is no ban on https://github.com/gorhill/uBlock/issues/new, it's just reserved to contributors. Here is the proper issue tracker.

gorhill · 2018-05-15T11:18:33Z

I answered to you on uBlockAdmin (which is a scammy fork by the way):

When I view-source your URL https://www.baidu.com/s?wd=%E7%9A%AE%E9%9E%8B, I get:

<!DOCTYPE html>


No script in there.

gorhill · 2018-05-15T11:21:53Z

Duh, sorry, I didn't realize I could scroll. I will investigate.

gorhill · 2018-05-15T11:24:10Z

Ok, again, when I view-source your URL https://www.baidu.com/s?wd=%E7%9A%AE%E9%9E%8B, I can't find any instance of ecomScript in the source.

ghost · 2018-05-15T11:25:33Z

ecomScript is already gone. I tested on script id="head_script" and cannot reproduce.

gorhill · 2018-05-15T11:27:01Z

@Crystal-RainSlide HTML filtering will only match what is in seen in view-source:, not what is dynamically added afterward.

gorhill · 2018-05-15T11:31:09Z

@gwarser Yes, cofirmed it works fine with www.baidu.com##^script#head_script, the script element is present in view-source:, and I confirmed it was properly removed by the above filter, so the syntax works, it's just a matter of filtering what is present in the source data as per view-source:.

ghost · 2018-05-15T11:33:33Z

When I tested this earlier ecomScript was present, but disappeared after refresh or two. This and my assumption that content of view-source should be filtered, misleaded me to think that something is not working. But now I'm pretty sure all is working as it should.

@Crystal-RainSlide why you think that only script:contains() is supported? Plese, provide detailed steps to reproduce.

uBlock-user · 2018-05-15T13:06:49Z

When I tested this earlier ecomScript was present, but disappeared after refresh or two.

Still there on my end - https://i.imgur.com/xZr5QIe.jpg Can't reproduce though, it blocks just as expected.

uBlock-user · 2018-05-15T13:13:29Z

script:contains() is only supported on Firefox-legacy branch, you must have installed that build instead of WebExtension build.

gorhill · 2018-05-15T13:17:58Z

##script:contains() is deprecated syntax, it is internally converted to ##^script:has-text() -- true for either Firefox legacy or webext.

Crystal-RainSlide · 2018-05-15T23:47:27Z

Sorry for my half day long offline.

About `script#ecomScript` 's disappearing

Whether things like business ads, ecomScript will appear depends on Baidu's advertising strategy. 皮鞋 means leather shoes, a kind of commodity, so this search word would cause ecomScript appearing in my test. But Baidu's ad market is mainly in China, so do the advertisers. So ecomScript may disappear in your tests for oversea customers are less likely to click ads.

About reproduction

I also failed to reproduce it after Firefox update automatically. But I got some other results:

↑ I've stayed with www.baidu.com##^#content_right for a long time, but it lost effectiveness when I'm reproducing.

So it seems like a stability problem but not an unsupported syntax... I'll go on with more tests.

Crystal-RainSlide · 2018-05-16T01:41:58Z

DONE, reproduced on my

Firefox Developer Edition 61.0

Steps

(0. Open the The logger to verify whether HTML filters work?)

Pick an element which will definitely appear (but not body) and block it with ^. Pick an element is for obviousness. I used www.baidu.com##^#head this time.
Open https://www.baidu.com/s?wd=%E7%9A%AE%E9%9E%8B in a tab. That element should be removed aright.
Right click on that tab and duplicate it.
Check the duplicated tab. That element should appear again.

Then, Chrome... Note that I'm using Cent Browser, not offical Chrome.

ghost · 2018-05-16T08:40:22Z

I can reproduce once out of ~10 attempts. This happens when page is loaded from cache (I suppose, because there is no response header):

uBlock-user · 2018-05-16T08:59:01Z

~~www.baidu.com##^#head doesn't work on my end, I can reproduce every time and #head appears in the view-source of both tabs -- original and duplicate.~~

ghost · 2018-05-16T09:29:58Z

Weird thing - I can easily reproduce on https://github.com/ with github.com##^#user\[login\], but it's very hard with github.com##^form

//edit:

May be because of this:

Race Cache with Network: When we detect that disk IO may be slow, we send a network request in parallel, and we use the first response that comes back. For users with slow spinning disks and a low latency network, the result would be faster loads. (Firefox 59)

ghost · 2018-05-16T09:57:49Z

gorhill/uBlock#3618

ghost · 2018-05-16T11:22:24Z

I tried to set request header Cache-Control: no-cache, must-revalidate and this does not help.
Probably something from this https://bugzilla.mozilla.org/buglist.cgi?quicksearch=cache%20webrequest&list_id=14151937

gorhill · 2018-05-16T11:26:06Z

I can reproduce every time and #head appears in the view-source of both tabs -- original and duplicate.

view-source: is to be used to find out what can be filtered, not what has been filtered. Firefox bypasses uBO when requesting a page via view-source:.

uBlock-user · 2018-05-16T11:50:51Z

You're right, I just tested ##.cr-content with baidu.com##^.cr-content and it works as expected.

gorhill · 2018-05-16T11:53:45Z

Then, Chrome... Note that I'm using Cent Browser, not offical Chrome.

HTML filtering is not supported on Chromium-based browsers, it's missing the proper API.

gorhill · 2018-05-16T12:43:16Z

I can reproduce sporadically when using the "duplicate tab" trick. When the issue occurs, the HTML filter is not reported in the logger. I will keep investigating.

ghost · 2018-05-16T13:10:03Z

Actually Cache-Control: no-cache, must-revalidate (set by https://addons.mozilla.org/firefox/addon/header-editor/) may help, but only after browser restart.

gorhill · 2018-05-16T13:29:23Z

Yes, I was looking at this: https://bugzilla.mozilla.org/show_bug.cgi?id=1376932.

I confirm that when the issue occurs, uBO's onHeadersReceived listener is not being called at all, Firefox bypasses uBO, hence uBO can't do its job.

uBlock-user · 2018-05-16T14:12:13Z

So this is a browser bug ?

gorhill · 2018-05-16T14:41:39Z

Yes, but I need to provide a workaround given the seriousness of it: one main use of HTML filtering is to remove unwanted specific inline script tag, and the issue here means this could lead to unwanted inline script code being executed. Beside, this should also solve the issue described in https://bugzilla.mozilla.org/show_bug.cgi?id=1376932 -- NoScript is being mentioned but uBO does also suffers from it (also uMatrix: gorhill/uMatrix#893).

Crystal-RainSlide mentioned this issue May 15, 2018

[Syntax Support] example.com##script#id uBlock-LLC/uBlock#1767

Closed

gorhill added the unable to reproduce cannot reproduce the issue label May 15, 2018

uBlock-user added the invalid not a uBlock issue label May 15, 2018

uBlock-user closed this as completed May 15, 2018

Crystal-RainSlide changed the title ~~[Syntax Support] example.com##script#id~~ HTML filters (example.com##^.badstuff) occasionally lost effectiveness (reproducing...) May 15, 2018

ghost removed the unable to reproduce cannot reproduce the issue label May 16, 2018

ghost reopened this May 16, 2018

uBlock-user removed the invalid not a uBlock issue label May 16, 2018

uBlock-user changed the title ~~HTML filters (example.com##^.badstuff) occasionally lost effectiveness (reproducing...)~~ HTML filters (example.com##^.badstuff) occasionally lost effectiveness May 16, 2018

uBlock-user changed the title ~~HTML filters (example.com##^.badstuff) occasionally lost effectiveness~~ HTML filters occasionally lost effectiveness May 16, 2018

uBlock-user added the Firefox specific to Firefox label May 16, 2018

gorhill closed this as completed in gorhill/uBlock@1c8df4a May 16, 2018

uBlock-user added the fixed issue has been addressed label May 16, 2018

ghost mentioned this issue May 16, 2018

HTML Filtering + Back button gorhill/uBlock#3618

Closed

gorhill mentioned this issue Sep 18, 2018

Error loading from the disk cache pages obtained by blocking remote fonts. #229

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTML filters occasionally lost effectiveness #42

HTML filters occasionally lost effectiveness #42

Crystal-RainSlide commented May 15, 2018 •

edited

Loading

Crystal-RainSlide commented May 15, 2018

gorhill commented May 15, 2018 •

edited

Loading

gorhill commented May 15, 2018 •

edited

Loading

gorhill commented May 15, 2018

gorhill commented May 15, 2018

ghost commented May 15, 2018

gorhill commented May 15, 2018

gorhill commented May 15, 2018 •

edited

Loading

ghost commented May 15, 2018 •

edited by ghost

Loading

uBlock-user commented May 15, 2018 •

edited

Loading

uBlock-user commented May 15, 2018

gorhill commented May 15, 2018 •

edited

Loading

Crystal-RainSlide commented May 15, 2018

Crystal-RainSlide commented May 16, 2018

ghost commented May 16, 2018 •

edited by ghost

Loading

uBlock-user commented May 16, 2018 •

edited

Loading

ghost commented May 16, 2018 •

edited by ghost

Loading

ghost commented May 16, 2018

ghost commented May 16, 2018 •

edited by ghost

Loading

gorhill commented May 16, 2018 •

edited

Loading

uBlock-user commented May 16, 2018

gorhill commented May 16, 2018

gorhill commented May 16, 2018

ghost commented May 16, 2018

gorhill commented May 16, 2018

uBlock-user commented May 16, 2018

gorhill commented May 16, 2018

HTML filters occasionally lost effectiveness #42

HTML filters occasionally lost effectiveness #42

Comments

Crystal-RainSlide commented May 15, 2018 • edited Loading

Prerequisites

Description

A specific URL where the issue occurs

Steps to Reproduce

Expected behavior:

Actual behavior:

Your environment

Crystal-RainSlide commented May 15, 2018

gorhill commented May 15, 2018 • edited Loading

gorhill commented May 15, 2018 • edited Loading

gorhill commented May 15, 2018

gorhill commented May 15, 2018

ghost commented May 15, 2018

gorhill commented May 15, 2018

gorhill commented May 15, 2018 • edited Loading

ghost commented May 15, 2018 • edited by ghost Loading

uBlock-user commented May 15, 2018 • edited Loading

uBlock-user commented May 15, 2018

gorhill commented May 15, 2018 • edited Loading

Crystal-RainSlide commented May 15, 2018

About script#ecomScript 's disappearing

About reproduction

Crystal-RainSlide commented May 16, 2018

Steps

ghost commented May 16, 2018 • edited by ghost Loading

uBlock-user commented May 16, 2018 • edited Loading

ghost commented May 16, 2018 • edited by ghost Loading

ghost commented May 16, 2018

ghost commented May 16, 2018 • edited by ghost Loading

gorhill commented May 16, 2018 • edited Loading

uBlock-user commented May 16, 2018

gorhill commented May 16, 2018

gorhill commented May 16, 2018

ghost commented May 16, 2018

gorhill commented May 16, 2018

uBlock-user commented May 16, 2018

gorhill commented May 16, 2018

Crystal-RainSlide commented May 15, 2018 •

edited

Loading

gorhill commented May 15, 2018 •

edited

Loading

gorhill commented May 15, 2018 •

edited

Loading

gorhill commented May 15, 2018 •

edited

Loading

ghost commented May 15, 2018 •

edited by ghost

Loading

uBlock-user commented May 15, 2018 •

edited

Loading

gorhill commented May 15, 2018 •

edited

Loading

About `script#ecomScript` 's disappearing

ghost commented May 16, 2018 •

edited by ghost

Loading

uBlock-user commented May 16, 2018 •

edited

Loading

ghost commented May 16, 2018 •

edited by ghost

Loading

ghost commented May 16, 2018 •

edited by ghost

Loading

gorhill commented May 16, 2018 •

edited

Loading