bulker: Add configurable retention time for retry topic #25
Conversation
| @@ -196,10 +196,16 @@ e.g. if `BULKER_KAFKA_TOPIC_PREFIX=some.prefix.`, then a full topic name could b | |||
|
|
|||
| ### `BULKER_KAFKA_TOPIC_RETENTION_HOURS` | |||
|
|
|||
| *Optional, default value: `168` (7 days)* | |||
| *Optional, default value: `48` (2 days)* | |||
There was a problem hiding this comment.
bulker/kafkabase/kafka_config.go
Line 25 in c0fb023
|
|
||
| Main topic retention time in hours. | ||
|
|
||
| ### `BULKER_KAFKA_TOPIC_SEGMENT_HOURS` | ||
|
|
||
| *Optional, default value: `24` (1 day)* |
There was a problem hiding this comment.
bulker/kafkabase/kafka_config.go
Line 26 in c0fb023
kafkabase/kafka_config.go
Outdated
| @@ -28,6 +28,7 @@ type KafkaConfig struct { | |||
| KafkaFetchMessageMaxBytes int `mapstructure:"KAFKA_FETCH_MESSAGE_MAX_BYTES" default:"1048576"` | |||
|
|
|||
| KafkaRetryTopicSegmentBytes int `mapstructure:"KAFKA_RETRY_TOPIC_SEGMENT_BYTES" default:"104857600"` | |||
| KafkaRetryTopicRetentionHours int `mapstructure:"KAFKA_RETRY_TOPIC_RETENTION_HOURS" default:"168"` | |||
There was a problem hiding this comment.
Lines 203 to 207 in c0fb023
There was a problem hiding this comment.
Thank you!
Actually 7 days is too high value for that. When doing retries, retry consumer re-produces messages that are not ready yet. There is no point of having that retention higher than KafkaTopicRetentionHours
There was a problem hiding this comment.
Got it, I used the default value from the docs. What value would you recommend for this? Should we set it to match the KafkaTopicRetentionHours value (2 days)?"
There was a problem hiding this comment.
Done, I've updated the default value to 2 days.
dmeremyanin
force-pushed
the
retry-topic-retention
branch
from
1ac144e to
cac1100
Compare
Overview
This PR includes the following updates related to Kafka env variables:
BULKER_KAFKA_TOPIC_SEGMENT_HOURS- added a description for the environment variable in the documentation to clarify its purpose and default value (24h).BULKER_KAFKA_TOPIC_RETENTION_HOURSin the documentation - fixed the default value for the environment variable to reflect the actual setting.KAFKA_RETRY_TOPIC_RETENTION_HOURS- updated the code to properly use the environment variable, which was documented but not used in the code.