consume msg failed: [error="Local: Timed out"

[charleskakumanu]

Hello,

Facing below issue:
[WARN] [kafka/kafka_consumer.go:139] ["consume msg failed"] [topic=] [groupID=] [error="Local: Timed out"]

Order of logs i see:
[server/server.go:134] ["request receive"]
[util/milvus_dial_option.go:98] ["tls two success"]
[reader/etcd_op.go:151] ["success to create etcd client"]
[util/channel_mapping.go:58] [ChannelMapping] [sourceCnt=0] [targetCnt=0] [averageCnt=1]
[reader/etcd_op.go:777] ["current time"] [ts=2025/04/03 06:45:08.087 +00:00]
[reader/etcd_op.go:566] ["get all collection data"] [count=49]
[reader/etcd_op.go:710] ["get all partition data"] [partition_num=97]
** [reader/etcd_op.go:815] ["fail to get db name for collection"] [collection_id=]
[INFO] [reader/etcd_op.go:825] ["fail to get db name for collection, maybe the db has been dropped"] [collection_id=]**

the last 2 logs statements multiple times and then below

[2025/04/03 06:45:07.622 +00:00] [INFO] [util/milvus_dial_option.go:98] ["tls two success"]
[2025/04/03 06:45:08.089 +00:00] [INFO] [util/milvus_dial_option.go:98] ["tls two success"]
[2025/04/03 06:45:08.550 +00:00] [INFO] [util/milvus_dial_option.go:98] ["tls two success"]
[2025/04/03 06:45:08.993 +00:00] [INFO] [util/milvus_dial_option.go:98] ["tls two success"]
[2025/04/03 06:45:09.445 +00:00] [INFO] [util/milvus_dial_option.go:98] ["tls two success"]
[2025/04/03 06:45:09.912 +00:00] [INFO] [util/milvus_dial_option.go:98] ["tls two success"]
[2025/04/03 06:45:10.356 +00:00] [INFO] [util/milvus_dial_option.go:98] ["tls two success"]
[2025/04/03 06:45:10.796 +00:00] [INFO] [util/milvus_dial_option.go:98] ["tls two success"]

[reader/target_client.go:181] ["not found the database"]
[reader/etcd_op.go:833] ["the collection info has been dropped in the source and target"] [collection_name=]
[reader/etcd_op.go:815] ["fail to get db name for collection"] ---------------> multiple logs again
[WARN] [reader/target_client.go:181] ["not found the database"]
[INFO] [writer/channel_writer.go:87] ["new channel writer"] [droppedObjs="{"database":{},"collection":{},"partition":{}}"]
[msgdispatcher/manager.go:67] ["create new dispatcherManager"]
[INFO] [msgdispatcher/dispatcher.go:96] ["creating dispatcher..."]
[INFO] [msgdispatcher/manager.go:178] ["dispatcherManager is running..."]
[INFO] [kafka/kafka_client.go:70] ["init kafka Config "]
[INFO] [msgstream/mq_msgstream.go:124] ["Msg Stream state"] [can_produce=true]
[INFO] [kafka/kafka_consumer.go:221] ["get latest msg ID "]
[INFO] [msgstream/mq_msgstream.go:214] ["Successfully create consumer"]
[INFO] [msgdispatcher/dispatcher.go:133] ["asConsumer successfully"]
[INFO] [msgdispatcher/dispatcher.go:161] ["add new target"]
[INFO] [msgdispatcher/manager.go:115] ["add main dispatcher"]
[INFO] [msgdispatcher/dispatcher.go:189] ["get signal"]
[INFO] [msgdispatcher/dispatcher.go:212] ["handle signal done"]
[INFO] [msgdispatcher/client.go:114] ["register done"]
[INFO] [msgdispatcher/dispatcher.go:217] ["begin to work"]
[INFO] [reader/etcd_op.go:566] ["get all collection data"] [count=49]
[INFO] [reader/replicate_channel_manager.go:167] ["has added dropped collection"] [ids="[]"]
[INFO] [reader/collection_reader.go:273] ["exist collection"]
[INFO] [reader/replicate_channel_manager.go:269] ["success to get the collection info in the target instance"]
[INFO] [kafka/kafka_client.go:70] ["init kafka Config "]
[INFO] [msgstream/mq_msgstream.go:124] ["Msg Stream state"] [can_produce=true]
[INFO] [kafka/kafka_consumer.go:221] ["get latest msg ID "]
[INFO] [msgstream/mq_msgstream.go:214] ["Successfully create consumer"]
[INFO] [msgstream/mq_msgstream.go:224] ["start to close mq msg stream"]
[INFO] [kafka/kafka_consumer.go:245] ["close consumer "]
[INFO] [reader/replicate_channel_manager.go:735] ["create a replicate handler"]
[INFO] [reader/replicate_channel_manager.go:434] ["start read channel in the manager"]
[reader/replicate_channel_manager.go:1286] ["start read channel in the handler"]
[reader/replicate_channel_manager.go:445] ["start read the source channel"]
[INFO] [reader/collection_reader.go:248] ["the collection is not in the watch list"]
[INFO] [msgdispatcher/manager.go:67] ["create new dispatcherManager"]
[INFO] [msgdispatcher/dispatcher.go:96] ["creating dispatcher..."]
[INFO] [kafka/kafka_client.go:70] ["init kafka Config "]
[INFO] [msgstream/mq_msgstream.go:124] ["Msg Stream state"] [can_produce=true]
[INFO] [msgdispatcher/manager.go:178] ["dispatcherManager is running..."]
[WARN] [reader/etcd_op.go:529] ["the collection isn't existed"]
[reader/etcd_op.go:688] ["not found the collection name, maybe the db or collection has benn deleted"] [collection_id=]
[INFO] [server/cdc_impl.go:1569] ["database is dropped"] [database=] [collection=]

many below logs
[WARN] [reader/etcd_op.go:529] ["the collection isn't existed"]
[INFO] [reader/etcd_op.go:688] ["not found the collection name, maybe the db or collection has benn deleted"]
[INFO] [server/cdc_impl.go:1569] ["database is dropped"] [database=]
[INFO] [reader/collection_reader.go:320] ["the collection is not in the watch list"]
[INFO] [reader/etcd_op.go:751] ["the partition info is filtered"] [partition=]
[INFO] [reader/etcd_op.go:742] ["partition state is not created/dropped or partition name is default"] [partition_name=_default] [state=PartitionCreated]

[INFO] [writer/replicate_message_manager.go:126] ["new replicate message handler"]

and it's the same below log for hours
[WARN] [kafka/kafka_consumer.go:139] ["consume msg failed"] [topic=] [groupID=] [error="Local: Timed out"]

[SimFG]

This log will not affect the operation

[charleskakumanu]

This log will not affect the operation

Got it. Nice!
Do we need to load the collection for it to be replicated in the target cluster? @SimFG

[SimFG]

@charleskakumanu It's unnecessary.

[charleskakumanu]

@charleskakumanu It's unnecessary.

thank you