source: introduce mock source for unit testing

[xx01cyx]

Is your feature request related to a problem? Please describe.

Currently, there is no way to write unit tests for streaming source in source executor.

Describe the solution you'd like

A MockSource could be introduced, which mocks the behavior of external source.

Describe alternatives you've considered

No response

Additional context

No response

[tabVersion]

Then what is the expected behavior? I think datagen is what you are looking for.

[TennyZhuang]

Then what is the expected behavior? I think datagen is what you are looking for.

datagen is not friendly for UT.

[github-actions]

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.

[xx01cyx]

Due to our code design, if we are to construct a mock source for source executor, we must pass a Receiver (or sth. alike) into connector properties (which impls Serialization & Deserialization) before the construction of the source executor. Via the offline discussion with @tabVersion , there are 2 solutions for now:

1. Pass an HTTP endpoint (not good to include such network stuff in unit testing)
2. Construct the Receiver beforehand and pass its memory address (seems hack)

Wonder if there's any better solution for this.

cc. @xxchan @fuyufjh @BugenZhao @TennyZhuang

[fuyufjh]

To test the streaming operators, I think "append-only table source" is enough, which allows you to insert values to form an append-only stream.

Can you please explain more about the motivation for this issue?

By the way, Tao has once proposed to introduce an HTTP endpoint to allow users to POST data (e.g. JSON) into RisingWave. It sounds similar, but it's a user-facing feature rather than designed for testing.

[fuyufjh]

Just guess: If you are thinking about testing watermark, the append-only table source should support it after #6952 is done.

[xx01cyx]

To test the streaming operators, I think "append-only table source" is enough, which allows you to insert values to form an append-only stream.

Can you please explain more about the motivation for this issue?

By the way, Tao has once proposed to introduce an HTTP endpoint to allow users to POST data (e.g. JSON) into RisingWave. It sounds similar, but it's a user-facing feature rather than designed for testing.

This is just for testing source executor, and is not for testing watermark. For now, we use datagen for source executor ut, which generates data in a background task asynchronously following a predefined pattern. I'm actually thinking of a foreground way of generating source data, and this issue limits the scope to unit testing.

Actually, datagen is working well in source executor ut (though not perfect). Maybe we can keep this as it is, and instead of implementing the mock source, we could broaden the scope beyond unit testing by introducing the HTTP source.