Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for storing logs in table #17598

Merged
merged 26 commits into from
Mar 29, 2025
Merged

Conversation

dqhl76
Copy link
Collaborator

@dqhl76 dqhl76 commented Mar 12, 2025

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

close: #17495

Enable this feature by changing config:

[log.persistentlog]
on = true
SELECT * FROM persistent_system.query_log WHERE query_id = 'd4a567c5-2e81-4a4a-b724-aee96ed60a93' ORDER BY timestamp LIMIT 3;

-[ RECORD 1 ]-----------------------------------
   timestamp: 2025-03-26 09:21:02.749640
        path: databend_query::servers::http::v1::http_query_handlers: http_query_handlers.rs:385
      target: databend_query::servers::http::v1::http_query_handlers
   log_level: INFO
  cluster_id: test_cluster
     node_id: Ov1oLXihzCVpOmkuF4ySd
warehouse_id: NULL
    query_id: d4a567c5-2e81-4a4a-b724-aee96ed60a93
      fields: {"message":"http query new request(from databend-client-rust/0.12.0): HttpQueryRequest { session_id: None, session: Some(HttpSessionConf { catalog: None, database: Some(\"default\"), role: Some(\"account_admin\"), secondary_roles: None, keep_server_session_secs: None, settings: Some({}), txn_state: None, need_sticky: false, need_keep_alive: false, last_server_info: None, last_query_ids: [], internal: None }), sql: \"select * from t\", pagination: PaginationConf { wait_time_secs: 10, max_rows_in_buffer: 5000000, max_rows_per_page: 10000 }, string_fields: true, stage_attachment: None }"}
-[ RECORD 2 ]-----------------------------------
   timestamp: 2025-03-26 09:21:02.754336
        path: databend_query::servers::http::v1::query::http_query: http_query.rs:493
      target: databend_query::servers::http::v1::query::http_query
   log_level: INFO
  cluster_id: test_cluster
     node_id: Ov1oLXihzCVpOmkuF4ySd
warehouse_id: NULL
    query_id: d4a567c5-2e81-4a4a-b724-aee96ed60a93
      fields: {"message":"create query","node_id":"Ov1oLXihzCVpOmkuF4ySd","query_id":"d4a567c5-2e81-4a4a-b724-aee96ed60a93","session_id":"11dc7dd0-b28e-433a-b888-eb0708d4329f","sql":"select * from t"}
-[ RECORD 3 ]-----------------------------------
   timestamp: 2025-03-26 09:21:02.754643
        path: databend_query::servers::http::v1::query::execute_state: execute_state.rs:349
      target: databend_query::servers::http::v1::query::execute_state
   log_level: INFO
  cluster_id: test_cluster
     node_id: Ov1oLXihzCVpOmkuF4ySd
warehouse_id: NULL
    query_id: d4a567c5-2e81-4a4a-b724-aee96ed60a93
      fields: {"message":"http query prepare to plan sql"}

3 rows read in 0.118 sec. Processed 118 rows, 5.22 KiB (997.3 rows/s, 44.16 KiB/s)

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Mar 12, 2025
@dqhl76 dqhl76 marked this pull request as ready for review March 19, 2025 09:18
@dqhl76 dqhl76 requested a review from zhang2014 March 19, 2025 09:31
@dqhl76 dqhl76 marked this pull request as draft March 20, 2025 01:54
@dqhl76 dqhl76 marked this pull request as ready for review March 27, 2025 05:34
@zhang2014 zhang2014 marked this pull request as draft March 28, 2025 07:20
@dqhl76 dqhl76 marked this pull request as ready for review March 28, 2025 09:29
@zhang2014 zhang2014 merged commit bbbca20 into databendlabs:main Mar 29, 2025
218 of 223 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: Persist logs to a table
3 participants