[FLINK-37120][cdc-connector] Add ending split chunk first to avoid TaskManager oom

[beryllw]

Add ending split chunk first to avoid TaskManager oom

[beryllw]

CI：https://github.com/beryllw/flink-cdc/actions/runs/12779452404
@leonardBang Could you please help me retrigger the CI?

[beryllw]

Adding the ending split chunk first changes the snapshot read result order. I will fix the unit test.

[leonardBang]

THanks @beryllw for the contribution, the idea makes sense to me, but we need to consider the compatibility, a configuration is recommended

[beryllw]

THanks @beryllw for the contribution, the idea makes sense to me, but we need to consider the compatibility, a configuration is recommended

If data consistency is not an issue, are there any compatibility concerns we need to address?

[lvyanquan]

What‘s more, could you add this optimization to MongoDBChunkSplitter?

[beryllw]

What‘s more, could you add this optimization to MongoDBChunkSplitter?

Sure, i will check MongoDBChunkSplitter.

[beryllw]

What‘s more, could you add this optimization to MongoDBChunkSplitter?

#3704 (comment)

I overlooked this issue #3704, good idea. Maybe support AssignStrategy in base module is better? Do we implement this in a future PR or in this PR?

[lvyanquan]

I missed this issue #3704, good idea. Maybe support AssignStrategy in base module is better? Do we implement this in a future PR or in this PR?

My previous PR provided both ascending_order and descending_order order configurations, but I think it would be better to place both start and end unbounded chunks at the beginning, so you can go ahead.

I think if there is no impact on restarting from the previous state, there is no compatibility issue, but adding a parameter to control it would be safer.

[lvyanquan]

I think adding a test in MySqlSourceITCase to test the case of restoring from failure will be better.

[beryllw]

I think adding a test in MySqlSourceITCase to test the case of restoring from failure will be better.

agree.

[lvyanquan]

LGTM.
I'm worried if this parameter is a bit complicated, what about naming it scan.incremental.assign-max-chunk-first.enabled to avoid using ending? As chunk is a concept in snapshot phase.
WDYT @beryllw @leonardBang?

[beryllw]

LGTM. I'm worried if this parameter is a bit complicated, what about naming it scan.incremental.assign-max-chunk-first.enabled to avoid using ending? As chunk is a concept in snapshot phase. WDYT @beryllw @leonardBang?

What about scan.incremental.snapshot.assign-ending-chunk-first.enabled, when the write throughput is low, the ending chunk is not equal to the maximum chunk.

[beryllw]

@leonardBang @lvyanquan PTAL

[lvyanquan]

Thanks @beryllw for this update, since you've added this feature to jdbc common module, it seems like that this config option was not added in DynamicTableSourceFactory and was not passed to Source.

[beryllw]

https://github.com/beryllw/flink-cdc/actions/runs/13258656371?pr=7
@leonardBang Please help me retrigger the CI.

[beryllw]

CI https://github.com/beryllw/flink-cdc/actions/runs/13276446389.

[lvyanquan]

Thanks @beryllw for this contribution and it looks good to me.

I think you can create one jira to make the default value of this option to true after one or two major versions and testing in product environment.

[lvyanquan]

And you might need to do some rebase to fix the ci failure.

[beryllw]

And you might need to do some rebase to fix the ci failure.

Sure.

[beryllw]

https://github.com/beryllw/flink-cdc/actions/runs/13387743918
@leonardBang Please help me retrigger the CI.

[leonardBang]

Thanks @beryllw for the contribution, I left two comments

[leonardBang]

Thanks @beryllw for the great work, LGTM

[lvyanquan]

Hi, @beryllw. Could you check the failed cases? If some are unrelated to the change, please add some description.

[beryllw]

Looks like unrelated to this change, here is the CI beryllw#7, and I also manually tested the failed tests(except for MySqlConnectorITCase).

[beryllw]

@leonardBang Is there anything else that needs to be completed? I'm concerned about potential conflicts later on.Could you please help me move this forward?

[leonardBang]

@leonardBang Is there anything else that needs to be completed? I'm concerned about potential conflicts later on.Could you please help me move this forward?

Hey, @beryllw Thank you for your reminder, I'll help investigate the failure case and merge your PR.

[leonardBang]

@leonardBang Is there anything else that needs to be completed? I'm concerned about potential conflicts later on.Could you please help me move this forward?

I guess the failed test cases in SqlServer and MongoDB may be related to our change, as some test case may assume the legacy split order will be assign first, could you check them?

[beryllw]

I guess the failed test cases in SqlServer and MongoDB may be related to our change, as some test case may assume the legacy split order will be assign first, could you check them?

Sure, I will recheck the failed test cases in SqlServer and MongoDB.

[leonardBang]

Rebase this PR to master to obtain fixed patches, let's see what happens next @beryllw

[leonardBang]

CI passed, merging....

[beryllw]

Rebase this PR to master to obtain fixed patches, let's see what happens next @beryllw

Sorry, I've been too busy to check earlier. Thanks for your help.

[leonardBang]

Rebase this PR to master to obtain fixed patches, let's see what happens next @beryllw

Sorry, I've been too busy to check earlier. Thanks for your help.

You're welcome @beryllw , no need to say sorry, the community is just our interest, our own work is the first priority ^^