Hitting a max size value with large numbers of entites #374
Description
Diode version
v1.4.0
Deployment type
docker compose
Deployment type version
Docker version 26.1.3, build b72abbb
Diode SDK type
diode-sdk-python
Diode SDK version
v1.1.0
Diode NetBox Plugin version
v1.2.0
NetBox version
v4.2.9
Steps to reproduce
Create a large number of entities somehow, in my case it is a list of length 27905 of Device, Interface, and IP_Address entities.
Expected behavior
Entities are processed, perhaps by modifying a server side configuration to enable large ingest sizes?
Observed behavior
Traceback (most recent call last):
File "/root/py3.12/lib64/python3.12/site-packages/netboxlabs/diode/sdk/client.py", line 250, in ingest
return self._stub.Ingest(request, metadata=self._metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py3.12/lib64/python3.12/site-packages/grpc/_interceptor.py", line 277, in __call__
response, ignored_call = self._with_call(
^^^^^^^^^^^^^^^^
File "/root/py3.12/lib64/python3.12/site-packages/grpc/_interceptor.py", line 332, in _with_call
return call.result(), call
^^^^^^^^^^^^^
File "/root/py3.12/lib64/python3.12/site-packages/grpc/_channel.py", line 440, in result
raise self
File "/root/py3.12/lib64/python3.12/site-packages/grpc/_interceptor.py", line 315, in continuation
response, call = self._thunk(new_method).with_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py3.12/lib64/python3.12/site-packages/grpc/_channel.py", line 1198, in with_call
return _end_unary_response_blocking(state, call, True, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py3.12/lib64/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "grpc: received message larger than max (6485629 vs. 4194304)"
debug_error_string = "UNKNOWN:Error received from peer ipv4:10.1.251.45:8080 {created_time:"2025-07-01T11:52:07.387824697+00:00", grpc_status:8, grpc_message:"grpc: received message larger than max (6485629 vs. 4194304)"}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/oli_testing/nornir_code/./device_to_diode.py", line 1195, in <module>
main()
File "/root/oli_testing/nornir_code/./device_to_diode.py", line 1180, in main
response = client.ingest(entities=entities)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/py3.12/lib64/python3.12/site-packages/netboxlabs/diode/sdk/client.py", line 259, in ingest
raise DiodeClientError(err) from err
netboxlabs.diode.sdk.exceptions.DiodeClientError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "grpc: received message larger than max (6485629 vs. 4194304)"
debug_error_string = "UNKNOWN:Error received from peer ipv4:10.1.251.45:8080 {created_time:"2025-07-01T11:52:07.387824697+00:00", grpc_status:8, grpc_message:"grpc: received message larger than max (6485629 vs. 4194304)"}"