Description
Observed behavior
Upgrading NATS from version 2.9.15 to version 2.10.25 has caused errors in Jetstream persistence. The first time I executed version 2.10.25 apparently there were no problems, at least the log was clean. But when I stopped NATS and executed it a second time, I got warnings about errors in the stream and finally the consumer was incorrect. I have tested it on 3 servers configured the same way and the error is the same with this stream.
This stream is empty and is configured with Interest based retention. There is another empty stream with Limits based retention and had no problems.
I also tried publishing messages in the stream in v2.9.15 so that it is not empty when updating and in that case there were no problems.
Then I tried purging the stream to leave it empty again, and there were no problems there either.
However, if I consume the new messages with the existing durable consumer and it becomes empty again, the error returns on update.
This problem seems to be similar #5412
Full log here: nats-server.log
[INF] Starting restore for stream '$G > CONNECTOR'
[WRN] Filestore [CONNECTOR] Stream state outdated, last block has additional entries, will rebuild
[WRN] Filestore [CONNECTOR] Recovering stream state from index errored: prior state file
[INF] Restored 0 messages for stream '$G > CONNECTOR' in 2ms
...
[INF] Recovering 1 consumers for stream - '$G > CONNECTOR'
...
[WRN] Detected consumer '$G > CONNECTOR > Connector-SequentialSender' ack floor 5588 is ahead of stream's last sequence 0
This is the status of the stream and the consumer BEFORE the warning:
C:\>nats stream info
[basic] ? Select a Stream CONNECTOR
Information for Stream CONNECTOR created 2024-08-01 13:24:58
Subjects: CONNECTOR.ERPNOTIFICATIONS
Replicas: 1
Storage: File
Options:
Retention: Interest
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 2m0s
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: false
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: 1.0 GiB
Maximum Age: 60d0h0m0s
Maximum Message Size: unlimited
Maximum Consumers: unlimited
State:
Messages: 0
Bytes: 0 B
First Sequence: 5,589
Last Sequence: 5,588 @ 2024-10-22 16:28:40 UTC
Active Consumers: 1
C:\>nats consumer info
[basic] ? Select a Stream CONNECTOR
[basic] ? Select a Consumer Connector-SequentialSender
Information for Consumer CONNECTOR > Connector-SequentialSender created 2024-10-10T15:37:53+02:00
Configuration:
Name: Connector-SequentialSender
Delivery Subject: _INBOX.m23kQ0r_x9h6iDLA6zDx-g
Filter Subject: CONNECTOR.ERPNOTIFICATIONS
Deliver Policy: New
Ack Policy: Explicit
Ack Wait: 4m30s
Replay Policy: Instant
Max Ack Pending: 1
Flow Control: false
State:
Last Delivered Message: Consumer sequence: 5,603 Stream sequence: 5,588
Acknowledgment Floor: Consumer sequence: 5,603 Stream sequence: 5,588
Outstanding Acks: 0 out of maximum 1
Redelivered Messages: 0
Unprocessed Messages: 0
Active Interest: No interest
And AFTER the warning. The sequence of the stream has been reset to 0:
C:\>nats stream info
[basic] ? Select a Stream CONNECTOR
Information for Stream CONNECTOR created 2024-08-01 13:24:58
Subjects: CONNECTOR.ERPNOTIFICATIONS
Replicas: 1
Storage: File
Options:
Retention: Interest
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 2m0s
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: false
Limits:
Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: 1.0 GiB
Maximum Age: 60d0h0m0s
Maximum Message Size: unlimited
Maximum Consumers: unlimited
State:
Messages: 0
Bytes: 0 B
First Sequence: 0
Last Sequence: 0
Active Consumers: 1
C:\>nats consumer info
[basic] ? Select a Stream CONNECTOR
[basic] ? Select a Consumer Connector-SequentialSender
Information for Consumer CONNECTOR > Connector-SequentialSender created 2024-10-10T15:37:53+02:00
Configuration:
Name: Connector-SequentialSender
Delivery Subject: _INBOX.m23kQ0r_x9h6iDLA6zDx-g
Filter Subject: CONNECTOR.ERPNOTIFICATIONS
Deliver Policy: New
Ack Policy: Explicit
Ack Wait: 4m30s
Replay Policy: Instant
Max Ack Pending: 1
Flow Control: false
State:
Last Delivered Message: Consumer sequence: 5,603 Stream sequence: 5,588
Acknowledgment Floor: Consumer sequence: 5,603 Stream sequence: 5,588
Outstanding Acks: 0 out of maximum 1
Redelivered Messages: 0
Unprocessed Messages: 0
Active Interest: No interest
Expected behavior
The sequence should not have been reset and the consumer should remain valid.
Server and client version
Tested with version 2.10.25 and 2.10.26 and NATS CLI
Host environment
Running a single-node NATS JetStream in Windows with this configuration:
max_payload: 8MB
port: 4222
monitor_port: 8222
#Logging
debug: true
trace: true
trace_verbose: true
logfile_size_limit: 30MB
logfile_max_num: 3
log_file: "nats-server.log"
jetstream {
max_file_store: 30GB
max_memory_store: 10GB
store_dir: data
}
Steps to reproduce
No response