[quantizer] Visualize layer-wise qerror

[jinevening]

What

Let's visualize quantization error in the quantized model

Why

To observe which quantized layer harms accuracy.

How

Input: fp32 circle, q circle, test data

Steps

1. Fake quantize q circle
2. Dump intermediate FMs of fp32 model with test data
3. Dump intermediate FMs of fq model with test data
4. Compare intermediate FMs and compute qerror
5. Generate DOT graph

Nodes with higher qerror will be more red.

Sample (MobileNet V2 full uint8)

full graph (visible on web)

[mhs4670go]

I think it's a good idea!

Let me ask some questions:)

1. After observing the harms, where can we use them?

2. Is there any chance to have a difference between the qerror with fake quant and the one with normal quant?

3. Above graph seems that qerror is not accumulated to the bottom layers. So, even if the qerror of upper layers are reduced, isn't there any possibility that the result of below layers will be worse?

[jinevening]

Thanks for good questions!

where can we use them?

I can think of two possible use cases.

1. Model design: Model developers can concentrate on sensitive layers of a model. Those layers may be replaced with other types of layers, or quantized with different parameters.
2. Mixed-precision quantization: We can quantize sensitive layers with higher bits e.g., int16 rather than uint8.

Diff between qerror with fake quant and normal quant

Of course there exist differences. Using quant kernels (instead of fake quant) can be another option, but it is difficult to implement a wide coverage of quant kernels.

Fake quant is used in other ML frameworks (TF, PyTorch), and our experiments show that fake-quant kernels can mimic quant kernels quite accurately.

Qerror accumulation

We compare the same points in fp32 model and quant model, so qerrors are in fact accumulated. We can observe in which points qerrors are increased.

I think that in general, the reduction of qerror in any layer will reduce the qerror of the final result. There can be counter-examples, but they may be very rare cases.

[lemmaa]

/cc @Samsung/one-vscode

[jinevening]

I'm finding a good name for this module

Candidates: visqerror (visualize qerror), or visq

Interface will be like below.

$ visqerror --fp32_circle sample.circle \
  --q_circle sample.q.circle \
  --data test.h5 \
  --peir_output peir_output.dot \
  --mse_output mse_output.dot

The above command will compute peir and mse between sample.circle and sample.q.circle, and generate qerror dot graphs peir_output.dot and mse_output.dot.

Any suggestion for naming or interface is welcome!

CC @lemmaa @seanshpark @mhs4670go

[seanshpark]

visq looks nice for module name in compiler.
for the tool name, visqerror or visqerr looks OK.

interface looks OK :)

[lemmaa]

I like both names. But, the visqerror feels uniquely like something snakes wriggling. :)

Anyway, when I first saw the picture of #9694 (comment), I felt something like this. It would be interesting to have a name that conveys this feeling. :)

[seanshpark]

1 + 5 -> RarelyWellDone -> in Korean; "거의 잘 된적이 없는..." -_-;

[seanshpark]

Had a short offline talk with @jinevening ,

• best UI would be to generate(?) a html file and drop to a browser and see the graph
• looking inside, we can generate some model file (in JSON?) and use (1) browser as viewer (2) some eletron app as a viewer (3) netron-visq, modified netron like netron-circle
• we can also think of ONE-vscode too
• using HTML-JS-CSS would take time so we can go with in parallel to DOT file viewing.

[lemmaa]

... (3) netron-visq, modified netron like netron-circle

I hope so. :)

[seanshpark]

With this scenario,

1. user creates .json file with visq command line
2. visq will create(or mayby just copy from somewhere) html and the data .json file
3. with something like --view-in-browser option, visq will invoke the browser with html file
4. browser will show the quantization error graph

I'v done some test with loading a local file from local .html in the browser.

• we can use FileReader class ONLY with <input> tag, that is, user MUST select the file in HTML view. --> this seems odd with above scenario
• we can NOT use XMLHttpRequest as browser rejects by security reasons
• we can put as json object in .js file and use it from the rendering logic

  const data = { ... };

We can use nodejs as local file server but seems getting too far...

[seanshpark]

Found an issue when we use .circle file for the model and .json for the error data.

• reading a local file in the javascript is impossible cause of security reasons (mentioned above)
• cannot read .circle

To resolve this...

• convert .circle file also to .js in form of .json ? --> done this in ONE-vscode and it's not good
• use nodejs as local web server
• use electron app
• define simple .json form of both model .circle and error .json
• ...