Clean state in HashJoinExecutor by watermark

[yuhao-su]

as title

[jon-chuang]

Oh, actually, how to do this? Since the Join state key is a HashKey, so one can't easily clean by scanning a watermark range.

[yuhao-su]

Oh, actually, how to do this? Since the Join state key is a HashKey, so one can't easily clean by scanning a watermark range.

You are right. Cleaning the state in storage is easier.

However, to clean the operator cache, we should do more"

1. we are using a hash table with join key as key, we should used a ordered data structure.
2. we are using HashKey as the key in JoinHashMap. We should make sure the event time order is consistent with the HashKey order (not hard to do that).

[jon-chuang]

Cleaning the state in storage is easier.

Oh yes, indeed, we will use the join key columns rather than their hash as the key for the StateTable. So if the first column is watermark column, then we can clean the StateTable.

Well, perhaps we do not need to worry about hashjoin operator cache, as LRU can take care of the state cleaning in this case.