`{badmatch,{error,modified}}` in ra_log_reader

[mkuratczyk]

Describe the bug

Under some circumstances, ra_log_reader can crash like this:

crasher:
  initial call: ra_server_proc:init/1
  pid: <0.1495.0>
  registered_name: '%2F_qq-2'
  exception error: no match of right hand side value
                   {'EXIT',
                       {{badmatch,{error,modified}},
                        [{ra_log_reader,'-segment_sparse_read/3-fun-1-',3,
                             [{file,"src/ra_log_reader.erl"},{line,361}]},
                         {ra_log_reader,
                             '-segment_sparse_read/3-lists^foldl/2-1-',3,
                             [{file,"src/ra_log_reader.erl"},{line,358}]},
                         {ra_log_reader,sparse_read,3,
                             [{file,"src/ra_log_reader.erl"},{line,207}]},
                         {ra_log,sparse_read,2,
                             [{file,"src/ra_log.erl"},{line,505}]},
                         {ra_server,log_read,2,
                             [{file,"src/ra_server.erl"},{line,2388}]},
                         {ra_server_proc,handle_log_effect,4,
                             [{file,"src/ra_server_proc.erl"},{line,2134}]},
                         {ra_server_proc,handle_effect,5,
                             [{file,"src/ra_server_proc.erl"},{line,1433}]},
                         {lists,foldl_1,3,
                             [{file,"lists.erl"},{line,2151}]}]}}

Reproduction steps

I don't have the exact steps at this point. It happened when running tests which involved AMQP-1.0 consumers that released and rejected some of the messages.

Full logs:
rabbit-3@carrot.log.txt
rabbit-2@carrot.log.txt
rabbit-1@carrot.log.txt

Expected behavior

no crash

Additional context

No response

[mkuratczyk]

I can't find a consistent way of reproducing it, but these steps did reproduce it twice:

# single node is sufficient
make run-broker

# declare a queue called `dlq`
 omq amqp --queues quorum -T /queues/dlq -x 0 -D 0

# declare a dead-lettering policy from `qq` to `dlq`
rabbitmqadmin-ng declare policy --name dlq --pattern 'qq.*' --definition '{"dead-letter-exchange":"","dead-letter-routing-key"
:"dlq"}' --apply-to queues

# create a backlog
omq amqp -y 0 -t /queues/qq --queues quorum -c 300 -C 200000 -s 5000

# consume the backlog rejecting the messages
omq amqp -x 0 -T /queues/qq --consumer-credits 2000 --amqp-reject-rate 100