Skip to content

[ch] optimize master_commit_red_percent query #6580

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

izaitsevfb
Copy link
Contributor

perf:

./clickhouse_query_perf.py --query master_commit_red_percent  --perf --times 1 --base HEAD

+------+----------+-----------+-------------+---------------+-----------+------------+-------------+--------------+
| Test | Avg Time | Base Time | Time Change | % Time Change |  Avg Mem  |  Base Mem  |  Mem Change | % Mem Change |
+------+----------+-----------+-------------+---------------+-----------+------------+-------------+--------------+
|  0   |   1164   |    5585   |    -4421    |      -79      | 160555724 | 7544036697 | -7383480973 |     -98      |
|  1   |   2992   |    5483   |    -2491    |      -45      | 516092615 | 7858494950 | -7342402335 |     -93      |
+------+----------+-----------+-------------+---------------+-----------+------------+-------------+--------------+

Removes expensive join, improves the pre-filtering utilizing a materialized view.

Testing

The new query in some cases returns slightly different results for broken_trunk_red metrics (although results for all_red metric are identical), which leads me to conclude that partitioning:

 ROW_NUMBER() OVER(
            PARTITION BY j.name,
            j.head_sha
            ORDER BY
                j.run_attempt DESC
        ) AS row_num

is not stable (missing workflow_name? each row can have multiple pushes?)

but overall the results are close enough (ε < 0.01) and further debugging and fixing could be made in a separate PR.

Copy link

vercel bot commented Apr 25, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
torchci ✅ Ready (Inspect) Visit Preview Apr 25, 2025 11:39pm

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 25, 2025
@izaitsevfb izaitsevfb requested review from huydhn and clee2000 April 25, 2025 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants