Xmin saving intermediate stream state after a failed connection attempt

[prateekmukhedkar]

Connector Name

source-postgres

Connector Version

2.0.29-dev-xmin

What step the error happened?

During the sync

Revelant information

Expected behavior: with xmin sync, we expect that stream state containing the high watermark is only written after a successful sync, that is when the job succeeds. This is because we are not ordering the query table results by xmin and hence we have to process all the rows before committing the state message. (Or in other words we cannot emit intermediate state during xmin sync)

Actual: I noticed that after a failed attempt we wrote an intermediate state for the stream. This deviates from the expected behavior. Logs are attached.

Relevant log output

No response

Contribute

• Yes, I want to contribute

[prateekmukhedkar]

Connection sync logs.

The state that was written after the first failed attempt:
(note that even though the namespace says otherwise i was syncing the 200m_users.users table containing 200M rows)

[
  {
    "streamDescriptor": {
      "name": "users",
      "namespace": "20m_users"
    },
    "streamState": {
      "num_wraparound": 0,
      "xmin_raw_value": 17384,
      "xmin_xid_value": 17384
    }
  }
]