Add Option to run Incremental Export From Bigtable (Timestamp Filtering)

[maheshgoyal15]

Description:

This pull request introduces a new feature to enable incremental data exports from Google Cloud Bigtable based on timestamp filtering. This addresses a specific customer requirement for efficiently extracting daily data subsets for transfer to a non-production Bigtable instance.

Problem:

The customer needs to export data from their production Bigtable database on a daily basis, filtering it based on a timestamp criterion. Currently, the existing export functionality lacks the ability to perform incremental exports based on timestamps, forcing them to perform full table scans or implement complex custom solutions.

Solution:

This PR implements the following changes to enable incremental exports based on timestamps:

• Introduce Timestamp Filtering Mechanism:
  • Adds two new command-line options: --startTimestamp and --endTimestamp.
  • These options allow users to specify a time range in UTC format (YYYY-MM-DDTHH:MM:SSZ) to filter the data during export.
  • The export process will then only retrieve and export rows that fall within the specified time range.
• Leverage Bigtable Timestamp Filtering Capabilities:
  • Utilizes Bigtable's TimestampRangeFilter to efficiently retrieve only the required data.
  • This ensures that only the data within the specified time range is scanned and exported, minimizing resource consumption and export time.
• Implementation Details:
  • The BigtableToAvro.Options interface has been extended to include getStartTimestamp() and getEndTimestamp() methods.
  • A timestampConverter function is implemented to convert the provided UTC timestamp strings into microseconds.
  • A filterProvider ValueProvider is used to construct the RowFilter based on the provided start and end timestamps.
  • The BigtableIO.Read operation now includes a withRowFilter(filterProvider) call to apply the timestamp filter.
  • Validation is added to make sure if one time stamp is provided, the other one must be provided as well.
• Code Snippet (RowFilter Construction):

ValueProvider<RowFilter> filterProvider = new DualInputNestedValueProvider<>(
        options.getStartTimestamp(),
        options.getEndTimestamp(),
        (TranslatorInput<String, String> input) -> {
            String startTimestamp = input.getX();
            String endTimestamp = input.getY();

            boolean hasStart = startTimestamp != null && !startTimestamp.isEmpty();
            boolean hasEnd = endTimestamp != null && !endTimestamp.isEmpty();

            // Check if exactly one timestamp is provided (which is invalid)
            if ((hasStart && !hasEnd) || (!hasStart && hasEnd)) {
                throw new IllegalArgumentException(
                        "Both startTimestamp and endTimestamp must be provided together, or neither should be provided.");
            }

            // If neither timestamp is provided, return null (no filter)
            if (!hasStart && !hasEnd) {
                return null;
            }

            // Convert timestamps to microseconds
            Long startMicros = timestampConverter.apply(startTimestamp);
            Long endMicros = timestampConverter.apply(endTimestamp);

            // Build the timestamp filter
            com.google.cloud.bigtable.data.v2.models.Filters.TimestampRangeFilter filterBuilder =
                    FILTERS.timestamp().range();

            if (startMicros != null) {
                filterBuilder.startClosed(startMicros);
            }
            if (endMicros != null) {
                filterBuilder.endOpen(endMicros);
            }

            return filterBuilder.toProto();
        });

read = read.withRowFilter(filterProvider);

[ron-gal]

It's very difficult to see the diffs =\ is there anything we can do about it? seems formatting is wrong...

[ron-gal]

was this done on purpose?

[maheshgoyal15]

R: @svetakvsundhar

[ron-gal]

Please also update the integration test to use the filter, and maybe add unit tests to test edge cases like start without end etc

[ron-gal]

2/25?

[ron-gal]

something in this line seems broken. If not, please remove the whitespace after "Exporting".

[ron-gal]

missing text in the beginning

[svetakvsundhar]

Thanks @maheshgoyal15 ! Could you add some e2e tests here to ensure this WAI?

[maheshgoyal15]

@ron-gal There were few delays but I am able to incorporate all the recommendation you provided. Can you take a look at this PR