[pkg/stanza][fileconsumer] - finalize archive's implementation #39901
Conversation
VihasMakwana
requested review from
djaglowski,
andrzej-stencel and
a team
as code owners
VihasMakwana
force-pushed
the
archive-implementation
branch
from
91f3ea8 to
496796f
Compare
VihasMakwana
changed the title
VihasMakwana
force-pushed
the
archive-implementation
branch
from
496796f to
ecce69e
Compare
|
@djaglowski @andrzej-stencel Looking forward to hear your thoughts on this approach 😄 |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
cc: @andrzej-stencel |
Co-authored-by: Andrzej Stencel <[email protected]>
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
receiver/filelogreceiver/README.md
Outdated
| @@ -65,6 +65,7 @@ Tails and parses logs from files. | |||
| | `ordering_criteria.sort_by.format` | | Relevant if `sort_type` is set to `timestamp`. Defines the strptime format of the timestamp being sorted. | | |||
| | `ordering_criteria.sort_by.ascending` | | Sort direction | | |||
| | `compression` | | Indicate the compression format of input files. If set accordingly, files will be read using a reader that uncompresses the file before scanning its content. Options are ``, `gzip`, or `auto`. `auto` auto-detects file compression type. Currently, gzip files are the only compressed files auto-detected, based on ".gz" filename extension. `auto` option is useful when ingesting a mix of compressed and uncompressed files with the same filelogreceiver. | | |||
| | `polls_to_archive` | | This settings control the number of poll cycles to store on disk, rather than being discarded. By default, the receiver will purge the record of readers for existed for 3 generations. Refer [archiving](#archiving) and [polling](../../pkg/stanza/fileconsumer/design.md#polling) for more details. | | |||
There was a problem hiding this comment.
Should mention what the default value is.
| @@ -221,6 +222,10 @@ Here is some of the information the file log receiver stores: | |||
|
|
|||
| Exactly how this information is serialized depends on the type of storage being used. | |||
|
|
|||
| ### Archiving | |||
|
|
|||
| If `polls_to_archive` setting is used in conjunction with `storage` setting, file offsets older than three poll cycles are stored on disk rather than being discarded. This feature enables the receiver to remember file for a longer period and also aims to use limited amount of memory. | |||
There was a problem hiding this comment.
What happens when polls_to_archive is used but storage is not set? Should describe this in the docs too.
| @@ -221,6 +222,10 @@ Here is some of the information the file log receiver stores: | |||
|
|
|||
| Exactly how this information is serialized depends on the type of storage being used. | |||
|
|
|||
| ### Archiving | |||
|
|
|||
| If `polls_to_archive` setting is used in conjunction with `storage` setting, file offsets older than three poll cycles are stored on disk rather than being discarded. This feature enables the receiver to remember file for a longer period and also aims to use limited amount of memory. | |||
There was a problem hiding this comment.
In my view, this documentation does not explain to the user why they should use it. Can you describe a specific scenario when using this option changes the behavior of the receiver to the benefit of the user? This section should describe the scenario and what the behavior is without and with this option set.
There was a problem hiding this comment.
@andrzej-stencel I've added a scenario. Can you take a look?
receiver/filelogreceiver/README.md
Outdated
| @@ -65,6 +65,7 @@ Tails and parses logs from files. | |||
| | `ordering_criteria.sort_by.format` | | Relevant if `sort_type` is set to `timestamp`. Defines the strptime format of the timestamp being sorted. | | |||
| | `ordering_criteria.sort_by.ascending` | | Sort direction | | |||
| | `compression` | | Indicate the compression format of input files. If set accordingly, files will be read using a reader that uncompresses the file before scanning its content. Options are ``, `gzip`, or `auto`. `auto` auto-detects file compression type. Currently, gzip files are the only compressed files auto-detected, based on ".gz" filename extension. `auto` option is useful when ingesting a mix of compressed and uncompressed files with the same filelogreceiver. | | |||
| | `polls_to_archive` | | This settings control the number of poll cycles to store on disk, rather than being discarded. By default, the receiver will purge the record of readers for existed for 3 generations. Refer [archiving](#archiving) and [polling](../../pkg/stanza/fileconsumer/design.md#polling) for more details. | | |||
There was a problem hiding this comment.
Can we mark this option as experimental, so that we can possibly change it without breaking the beta stability guarantees of the component?
I feel this implementation may be not the ultimate solution and would like to reserve the right to change it in the future.
There was a problem hiding this comment.
Yeah, we can mark this experimental.
Just to confirm, you are suggesting we use this feature under a feature gate. Right?
There was a problem hiding this comment.
Hm no, I wasn't thinking about a feature gate. This feature can be available for use as is, let's just mention in the docs that it's experimental.
There was a problem hiding this comment.
Okay got it. Thanks!!
This PR finalizes the archive implementation and adds test case.
It updates following things:
exclude_older_thanis enabled #32727 (comment) for detailed discussion on this topic.handleUnmatchedFileshelper to create readers once archive matching is done.Relates: #38056