Check failed: !ndinputs.back()->is_none()

[lightingghost]

I came across with an error of Check failed: !ndinputs.back()->is_none() node_58 0. The full code is too complicated to post here. I am trying to get a minimum sample to reproduce this error. Briefly, the model is the input variable going through a gluon Block, and some ndarray ops are used to calculate the loss.

Something I have tried:

1. I tried to call hybridize() for the Block, and the error disappears.
2. It seems to be related to the function below when calculating the loss:

    def binarize(self, x, i):
        rand = mx.nd.random_uniform(low=0, high=1,
                                        shape=self.rand_shape[i],
                                        ctx=self.model.ctx)
        sample = mx.nd.broadcast_lesser_equal(rand, mx.nd.sigmoid(x))
        sample_shape = [-1] + list(self.model.data_shape[i])[1:]
        return sample.reshape(sample_shape)

If I replace the function with

    def binarize(self, x, i):
        return x

The error disappears.

Besides, I found there is tojson() method for symbol, but not for ndarray. Is there a way to visualize the computation graph so that I can know what operation of node_58 is?

Any help will be appreciated.

[lightingghost]

Here is a minimum sample to reproduce this error

import mxnet as nd
from mxnet import nd, sym, autograd
from mxnet import gluon as g
import pdb

inp = nd.ones((3, 2))
class Test(g.nn.HybridBlock):
    def __init__(self):
        super(Test, self).__init__()
        self.fc1 = g.nn.Dense(20, use_bias=False)
        self.fc2 = g.nn.Dense(30, use_bias=False)
        
    def hybrid_forward(self, F, x):
        A = self.fc1(x)
        X = self.fc2(x)
        A = A.reshape((-1, 4, 5))
        X = X.reshape((-1, 5, 6))
        A = 0.5 <= A.sigmoid()
        s = F.batch_dot(A, X)
        return s
    
pdb.set_trace()
test = Test()
#test.hybridize()
test.initialize()
with autograd.record():
    r = test(inp)
    l = r.sum()
    
l.backward()

As stated earlier, calling hybridize() will make the error disappear.

Besides, a slight change from batch_dot to dot will make the error disappear.

[lightingghost]

It seem this error is related to the FullyConnected operator. The following code will reproduce the error:

x = nd.ones((3, 2))
y = nd.ones((2, 20))
y.attach_grad()
z = g.Parameter('weight', shape=(20, 2))
z.initialize(ctx=mx.cpu())
with autograd.record():
    #a = nd.dot(x, z.data(), transpose_b=True)
    a = nd.FullyConnected(x, z.data(), None, num_hidden=20, no_bias=True)
    a = a.reshape((-1, 4, 5))
    b = nd.ones((3, 5, 6))
    a = 0.5 <= a
    e = nd.batch_dot(a, b)
    f = e.sum()
f.backward()

But changing FullyConnected to dot will be fine. Further diving into the imperative.cc shows the heap address of the output of FullyConnected was used to store the output of reshape and then sum, making the backward function unable to find the output of FullyConnected.

As a workaround, I can write my own Dense block which uses dot instead of FullyConnected, but I think this problem still worth investigation. Btw, why does Dense use FullyConnected instead of dot initially? Doing the multiplication and addition in a single operator?

Another thing is, the info property in NDArray has type of dmlc::any, which is very hard to debug in lldb, is there a method you suggest that can see the content in info?

Thanks

[TobiasSkovgaardJepsen]

I appear to have encountered this error as well. In my case the error initially failed silently because I am adding two losses, but it turns out only one of them is used during backpropagation. Using just the other loss I get:

MXNetError: [11:35:53] src/imperative/imperative_utils.cc:58: Check failed: !ndinputs.back()->is_none() node_588342 0

Stack trace returned 7 entries:
[bt] (0) 0   libmxnet.so                         0x0000000123133b90 libmxnet.so + 15248
[bt] (1) 1   libmxnet.so                         0x000000012313393f libmxnet.so + 14655
[bt] (2) 2   libmxnet.so                         0x000000012468ee74 MXNDListFree + 587044
[bt] (3) 3   libmxnet.so                         0x00000001246848e3 MXNDListFree + 544659
[bt] (4) 4   libmxnet.so                         0x00000001245d8f2e MXAutogradBackwardEx + 1022
[bt] (5) 5   _ctypes.cpython-36m-darwin.so       0x000000010fdbf427 ffi_call_unix64 + 79
[bt] (6) 6   ???                                 0x00007ffee1aeb630 0x0 + 140732684744240

Unlike @lightingghost, I could not fix this by using a custom Dense layer that uses F.dot in hybrid_forward.