Flaky: test_fusion.test_fusion

[haojin2]

Flaky test on 1.6.x branch: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-gpu/detail/PR-16873/1/pipeline

======================================================================

FAIL: test_fusion.test_fusion

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest

    self.test(*self.arg)

  File "/work/mxnet/tests/python/gpu/../unittest/common.py", line 177, in test_new

    orig_test(*args, **kwargs)

  File "/work/mxnet/tests/python/gpu/test_fusion.py", line 219, in test_fusion

    check_binary_ops()

  File "/work/mxnet/tests/python/gpu/test_fusion.py", line 175, in check_binary_ops

    check_fused_symbol(a/b, a=arr1, b=arr2)

  File "/work/mxnet/tests/python/gpu/test_fusion.py", line 62, in check_fused_symbol

    np.testing.assert_allclose(orig.asnumpy(), fused.asnumpy(), rtol=rtol[dtype], atol=atol[dtype])

  File "/usr/local/lib/python3.5/dist-packages/numpy/testing/_private/utils.py", line 1515, in assert_allclose

    verbose=verbose, header=header, equal_nan=equal_nan)

  File "/usr/local/lib/python3.5/dist-packages/numpy/testing/_private/utils.py", line 841, in assert_array_compare

    raise AssertionError(msg)

AssertionError: 

Not equal to tolerance rtol=0.01, atol=0.001



Mismatch: 5.56%

Max absolute difference: 640.

Max relative difference: 0.01087

 x: array([[-1.0059e+00, -2.3379e+00, -1.0029e+00, -8.2109e+00, -7.8662e-01,

        -1.5000e+00],

       [-6.1250e+00, -5.9102e+00, -1.2055e+01, -2.1387e+00, -4.3867e+00,...

 y: array([[-1.0059e+00, -2.3379e+00, -1.0029e+00, -8.2109e+00, -7.8662e-01,

        -1.4990e+00],

       [-6.1250e+00, -5.9102e+00, -1.2055e+01, -2.1387e+00, -4.3867e+00,...

-------------------- >> begin captured stdout << ---------------------

Checking fusion of relu

Checking fusion of sigmoid

Checking fusion of softsign

Checking fusion of exp

Checking fusion of expm1

Checking fusion of log

Checking fusion of log10

Checking fusion of log2

Checking fusion of log1p

Checking fusion of degrees

Checking fusion of radians

Checking fusion of sin

Checking fusion of cos

Checking fusion of tan

Checking fusion of arcsin

Checking fusion of arccos

Checking fusion of arctan

Checking fusion of sinh

Checking fusion of cosh

Checking fusion of tanh

Checking fusion of arcsinh

Checking fusion of arctanh

Checking fusion of sqrt

Checking fusion of rsqrt

Checking fusion of cbrt

Checking fusion of rcbrt

Checking fusion of square

Checking fusion of squeeze

Checking fusion of zeros_like

Checking fusion of ones_like

Checking fusion of flatten

Checking fusion of round

Checking fusion of rint

Checking fusion of fix

Checking fusion of floor

Checking fusion of ceil

Checking fusion of trunc

Checking fusion of sign

Checking fusion of reciprocal

Checking fusion of abs

Checking fusion of gamma

Checking fusion of gammaln

Checking fusion of erf

Checking fusion of negative

Checking fusion of arccosh

Checking fusion of erfinv

Checking fusion of Activation(act_type='relu')

Checking fusion of Activation(act_type='sigmoid')

Checking fusion of Activation(act_type='tanh')

Checking fusion of Activation(act_type='softrelu')

Checking fusion of Activation(act_type='softsign')

Checking fusion of Cast(dtype='float16')

Checking fusion of Cast(dtype='float32')

Checking fusion of Cast(dtype='float64')

Checking fusion of Cast(dtype='int32')

Checking fusion of reshape

Checking fusion of expand_dims

Checking fusion of clip

Checking fusion of smooth_l1



--------------------- >> end captured stdout << ----------------------

-------------------- >> begin captured logging << --------------------

common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=691654171 to reproduce.

--------------------- >> end captured logging << ---------------------

[ptrendx]

I noticed this and changed this test from testing a/b (which is super unstable if b is close to 0) to a/(b+1) in #16836.

[ptrendx]

@haojin2 Do you agree that this (changing a/b to a/(b+1)) fixes the flakiness and we can close this issue?

[haojin2]

@ptrendx If you have verified that this change could make this test pass more than 500 trials then I'm for it.
Note: A test that succeeds with 99% chance could only pass 500 trials consecutively with < 1% of chance.

[ptrendx]

I ran that part of this test 10000 times after the change without problem. Closing.