Skip to content

Add ddl to create an iceberg data source #16652

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Apr 7, 2025
Merged

Add ddl to create an iceberg data source #16652

merged 11 commits into from
Apr 7, 2025

Conversation

buhtr
Copy link
Collaborator

@buhtr buhtr commented Apr 2, 2025

Changelog entry

FQ: Add ability to create an external data sources for iceberg tables.

Changelog category

  • New feature

Description for reviewers

The complete description is located on a wiki page.

@buhtr buhtr requested review from a team as code owners April 2, 2025 09:16
@buhtr buhtr requested a review from uzhastik April 2, 2025 09:16
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 2, 2025 •
edited
Loading

⚪ 2025-04-02 09:16:40 UTC Pre-commit check linux-x86_64-release-asan for 964a712 has started.
⚪ 2025-04-02 09:16:44 UTC Artifacts will be uploaded here
⚪ 2025-04-02 09:19:37 UTC ya make is running...
🟡 2025-04-02 10:50:41 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14008 13870 0 66 33 39

⚪ 2025-04-02 10:52:00 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-02 11:04:33 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
191 (only retried tests) 115 0 35 7 34

⚪ 2025-04-02 11:04:44 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-04-02 11:16:35 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
106 (only retried tests) 40 0 33 1 32

🟢 2025-04-02 11:16:42 UTC Build successful.
🟡 2025-04-02 11:17:16 UTC ydbd size 3.8 GiB changed* by +641.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 22506ce merge: 964a712 diff diff %
ydbd size 4 123 873 368 Bytes 4 124 530 224 Bytes +641.5 KiB +0.016%
ydbd stripped size 1 424 942 728 Bytes 1 425 088 776 Bytes +142.6 KiB +0.010%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 2, 2025 •
edited
Loading

⚪ 2025-04-02 09:17:02 UTC Pre-commit check linux-x86_64-relwithdebinfo for 964a712 has started.
⚪ 2025-04-02 09:17:10 UTC Artifacts will be uploaded here
⚪ 2025-04-02 09:20:07 UTC ya make is running...
🟡 2025-04-02 10:44:36 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
28446 25789 0 3 2541 113

⚪ 2025-04-02 10:46:59 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-04-02 11:03:10 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
161 (only retried tests) 57 0 0 0 104

🟢 2025-04-02 11:03:17 UTC Build successful.
🟡 2025-04-02 11:03:40 UTC ydbd size 2.2 GiB changed* by +376.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 22506ce merge: 964a712 diff diff %
ydbd size 2 350 323 080 Bytes 2 350 708 960 Bytes +376.8 KiB +0.016%
ydbd stripped size 491 558 304 Bytes 491 598 496 Bytes +39.2 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 2, 2025

🟢 2025-04-02 09:17:21 UTC The validation of the Pull Request description is successful.

CyberROFL
CyberROFL previously approved these changes Apr 2, 2025
View reviewed changes
GrigoriyPA
GrigoriyPA reviewed Apr 3, 2025
View reviewed changes
kardymonds
kardymonds previously approved these changes Apr 3, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 4, 2025 •
edited
Loading

⚪ 2025-04-04 09:25:10 UTC Pre-commit check linux-x86_64-release-asan for 402eea5 has started.
⚪ 2025-04-04 09:25:24 UTC Artifacts will be uploaded here
⚪ 2025-04-04 09:28:14 UTC ya make is running...
⚫ 2025-04-04 10:17:46 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 4, 2025 •
edited
Loading

⚪ 2025-04-04 09:26:08 UTC Pre-commit check linux-x86_64-relwithdebinfo for 402eea5 has started.
⚪ 2025-04-04 09:26:11 UTC Artifacts will be uploaded here
⚪ 2025-04-04 09:29:05 UTC ya make is running...
⚫ 2025-04-04 10:17:47 UTC Check cancelled

uzhastik
uzhastik previously approved these changes Apr 4, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 4, 2025 •
edited
Loading

⚪ 2025-04-04 10:21:11 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7543708 has started.
⚪ 2025-04-04 10:21:36 UTC Artifacts will be uploaded here
⚪ 2025-04-04 10:25:14 UTC ya make is running...
🟡 2025-04-04 11:40:51 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
28471 25821 0 1 2541 108

⚪ 2025-04-04 11:43:35 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-04-04 11:52:51 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
150 (only retried tests) 52 0 0 0 98

🟢 2025-04-04 11:52:58 UTC Build successful.
🟡 2025-04-04 11:53:17 UTC ydbd size 2.2 GiB changed* by +300.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 23449f8 merge: 7543708 diff diff %
ydbd size 2 351 269 696 Bytes 2 351 577 416 Bytes +300.5 KiB +0.013%
ydbd stripped size 491 735 744 Bytes 491 779 264 Bytes +42.5 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

GrigoriyPA
GrigoriyPA previously approved these changes Apr 4, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 4, 2025 •
edited
Loading

⚪ 2025-04-04 10:23:34 UTC Pre-commit check linux-x86_64-release-asan for 7543708 has started.
⚪ 2025-04-04 10:23:50 UTC Artifacts will be uploaded here
⚪ 2025-04-04 10:26:57 UTC ya make is running...
🟡 2025-04-04 12:03:37 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14024 13884 0 73 33 34

⚪ 2025-04-04 12:04:52 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-04 12:18:15 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
187 (only retried tests) 95 0 39 22 31

⚪ 2025-04-04 12:18:27 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-04-04 12:30:38 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
120 (only retried tests) 50 0 33 7 30

🟢 2025-04-04 12:30:47 UTC Build successful.
🟡 2025-04-04 12:31:19 UTC ydbd size 3.8 GiB changed* by +494.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 23449f8 merge: 7543708 diff diff %
ydbd size 4 125 423 960 Bytes 4 125 930 432 Bytes +494.6 KiB +0.012%
ydbd stripped size 1 425 352 232 Bytes 1 425 477 800 Bytes +122.6 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@gridnevvvit
Copy link
Member

gridnevvvit commented Apr 4, 2025 •
edited
Loading

не могу найти end-to-end где выполняется запрос CREATE через ExecuteQuery и он успешно работает (или с ошибкой)

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 6, 2025 •
edited
Loading

⚪ 2025-04-06 11:24:22 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1766297 has started.
⚪ 2025-04-06 11:25:04 UTC Artifacts will be uploaded here
⚪ 2025-04-06 11:28:20 UTC ya make is running...
⚫ 2025-04-06 11:30:37 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 6, 2025 •
edited
Loading

⚪ 2025-04-06 11:24:22 UTC Pre-commit check linux-x86_64-release-asan for 1766297 has started.
⚪ 2025-04-06 11:24:46 UTC Artifacts will be uploaded here
⚪ 2025-04-06 11:27:36 UTC ya make is running...
⚫ 2025-04-06 11:30:38 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 6, 2025 •
edited
Loading

⚪ 2025-04-06 11:31:05 UTC Pre-commit check linux-x86_64-relwithdebinfo for 72ae5d0 has started.
⚪ 2025-04-06 11:31:19 UTC Artifacts will be uploaded here
⚪ 2025-04-06 11:34:09 UTC ya make is running...
🟡 2025-04-06 12:52:35 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
28478 25800 0 5 2554 119

⚪ 2025-04-06 12:54:56 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-06 13:05:12 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
177 (only retried tests) 72 0 1 0 104

⚪ 2025-04-06 13:05:23 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-04-06 13:15:42 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
146 (only retried tests) 44 0 1 0 101

🟢 2025-04-06 13:15:49 UTC Build successful.
🟡 2025-04-06 13:16:10 UTC ydbd size 2.2 GiB changed* by +296.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a23c2f6 merge: 72ae5d0 diff diff %
ydbd size 2 354 469 984 Bytes 2 354 773 624 Bytes +296.5 KiB +0.013%
ydbd stripped size 492 040 672 Bytes 492 080 096 Bytes +38.5 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 6, 2025 •
edited
Loading

⚪ 2025-04-06 11:31:06 UTC Pre-commit check linux-x86_64-release-asan for 72ae5d0 has started.
⚪ 2025-04-06 11:31:10 UTC Artifacts will be uploaded here
⚪ 2025-04-06 11:34:06 UTC ya make is running...
🟡 2025-04-06 13:04:28 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14019 13874 0 77 32 36

⚪ 2025-04-06 13:05:39 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-06 13:19:01 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
191 (only retried tests) 100 0 41 17 33

⚪ 2025-04-06 13:19:09 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-04-06 13:31:18 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
122 (only retried tests) 55 0 35 1 31

🟢 2025-04-06 13:31:25 UTC Build successful.
🟡 2025-04-06 13:31:55 UTC ydbd size 3.8 GiB changed* by +494.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a23c2f6 merge: 72ae5d0 diff diff %
ydbd size 4 131 699 200 Bytes 4 132 205 608 Bytes +494.5 KiB +0.012%
ydbd stripped size 1 427 182 056 Bytes 1 427 307 560 Bytes +122.6 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 6, 2025 •
edited
Loading

⚪ 2025-04-06 13:47:18 UTC Pre-commit check linux-x86_64-release-asan for 3550133 has started.
⚪ 2025-04-06 13:47:34 UTC Artifacts will be uploaded here
⚪ 2025-04-06 13:50:24 UTC ya make is running...
🟡 2025-04-06 15:02:34 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14019 13892 0 71 22 34

⚪ 2025-04-06 15:03:59 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-06 15:16:37 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
177 (only retried tests) 111 0 35 2 29

⚪ 2025-04-06 15:16:45 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-04-06 15:28:41 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
94 (only retried tests) 32 0 33 2 27

🟢 2025-04-06 15:28:48 UTC Build successful.
🟡 2025-04-06 15:29:18 UTC ydbd size 3.8 GiB changed* by +494.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a23c2f6 merge: 3550133 diff diff %
ydbd size 4 131 699 200 Bytes 4 132 205 608 Bytes +494.5 KiB +0.012%
ydbd stripped size 1 427 182 056 Bytes 1 427 307 560 Bytes +122.6 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 6, 2025 •
edited
Loading

⚪ 2025-04-06 13:47:18 UTC Pre-commit check linux-x86_64-relwithdebinfo for 3550133 has started.
⚪ 2025-04-06 13:47:24 UTC Artifacts will be uploaded here
⚪ 2025-04-06 13:50:20 UTC ya make is running...
🟢 2025-04-06 14:53:17 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
28478 25815 0 0 2552 111

🟢 2025-04-06 14:55:35 UTC Build successful.
🟡 2025-04-06 14:55:52 UTC ydbd size 2.2 GiB changed* by +296.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a23c2f6 merge: 3550133 diff diff %
ydbd size 2 354 469 984 Bytes 2 354 773 624 Bytes +296.5 KiB +0.013%
ydbd stripped size 492 040 672 Bytes 492 080 096 Bytes +38.5 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 6, 2025 •
edited
Loading

⚪ 2025-04-06 16:13:53 UTC Pre-commit check linux-x86_64-release-asan for 1009e2e has started.
⚪ 2025-04-06 16:14:19 UTC Artifacts will be uploaded here
⚪ 2025-04-06 16:17:47 UTC ya make is running...
🟡 2025-04-06 17:34:17 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
14019 13848 0 104 33 34

⚪ 2025-04-06 17:35:29 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-04-06 17:48:08 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
213 (only retried tests) 140 0 36 10 27

⚪ 2025-04-06 17:48:17 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-04-06 18:00:22 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
100 (only retried tests) 40 0 33 1 26

🟢 2025-04-06 18:00:29 UTC Build successful.
🟡 2025-04-06 18:00:58 UTC ydbd size 3.8 GiB changed* by +494.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a23c2f6 merge: 1009e2e diff diff %
ydbd size 4 131 699 200 Bytes 4 132 205 608 Bytes +494.5 KiB +0.012%
ydbd stripped size 1 427 182 056 Bytes 1 427 307 560 Bytes +122.6 KiB +0.009%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 6, 2025 •
edited
Loading

⚪ 2025-04-06 16:13:53 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1009e2e has started.
⚪ 2025-04-06 16:13:57 UTC Artifacts will be uploaded here
⚪ 2025-04-06 16:16:51 UTC ya make is running...
🟡 2025-04-06 17:22:32 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
28478 25813 0 3 2556 106

⚪ 2025-04-06 17:25:08 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-04-06 17:35:37 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
155 (only retried tests) 55 0 0 0 100

🟢 2025-04-06 17:35:45 UTC Build successful.
🟡 2025-04-06 17:36:04 UTC ydbd size 2.2 GiB changed* by +296.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a23c2f6 merge: 1009e2e diff diff %
ydbd size 2 354 469 984 Bytes 2 354 773 624 Bytes +296.5 KiB +0.013%
ydbd stripped size 492 040 672 Bytes 492 080 096 Bytes +38.5 KiB +0.008%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

GrigoriyPA
GrigoriyPA reviewed Apr 7, 2025
View reviewed changes
ydb/core/kqp/ut/federated_query/common/common.h
@@ -28,5 +28,6 @@ namespace NKikimr::NKqp::NFederatedQueryTest {
NYql::IDatabaseAsyncResolver::TPtr databaseAsyncResolver = nullptr,
std::optional<NKikimrConfig::TAppConfig> appConfig = std::nullopt,
std::shared_ptr<NYql::NDq::IS3ActorsFactory> s3ActorsFactory = nullptr,
const TKikimrRunnerOptions& options = {});
const TKikimrRunnerOptions& options = {},
NYql::ISecuredServiceAccountCredentialsFactory::TPtr credentialsFactory = nullptr);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Может быть добавить credentialsFactory в TKikimrRunnerOptions? Там потенциально ещё очень много параметров надо прокинуть, в аргументах их все держать мне кажется будет не очень удобно

@buhtr buhtr merged commit fca7a28 into ydb-platform:main Apr 7, 2025
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GrigoriyPA GrigoriyPA GrigoriyPA left review comments

@gridnevvvit gridnevvvit gridnevvvit approved these changes

@uzhastik uzhastik uzhastik approved these changes

@CyberROFL CyberROFL CyberROFL left review comments

@kardymonds kardymonds kardymonds left review comments

@vitalyisaev2 vitalyisaev2 Awaiting requested review from vitalyisaev2

@evanevanevanevannnn evanevanevanevannnn Awaiting requested review from evanevanevanevannnn

Assignees
No one assigned
Labels
new-feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@buhtr @gridnevvvit @CyberROFL @GrigoriyPA @uzhastik @kardymonds