This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
mx.io.NDArrayIter cant pad when size is large #16996
Closed
Description
Description
When input data is smaller than batch size, sometime it errors out with:
Traceback (most recent call last):
File "test.py", line 7, in <module>
for batch in dataiter:
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 230, in __next__
return self.next()
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 682, in next
data = self.getdata()
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 764, in getdata
return self._batchify(self.data)
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 751, in _batchify
second_data = self._getdata(data_source, end=pad)
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 707, in _getdata
]]) for x in data_source
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/io/io.py", line 707, in <listcomp>
]]) for x in data_source
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 705, in __getitem__
return self._slice(key.start, key.stop)
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 1338, in _slice
start, stop, _ = _get_index_range(start, stop, self.shape[0])
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 3080, in _get_index_range
raise IndexError('Slicing stop %d exceeds limit of %d' % (stop, length))
IndexError: Slicing stop 190 exceeds limit of 10
To Reproduce
import numpy as np
import mxnet as mx
data = np.arange(40).reshape((10,2,2))
dataiter = mx.io.NDArrayIter(data=data, batch_size=200, last_batch_handle='pad')
for batch in dataiter:
print(batch.data[0].asnumpy().shape)
Using a batch size of 20 succeeds, but larger sizes fail.
Steps to reproduce
This is using the 1.6.0 branch