connection closed before message completed

[xujpxm]

Environment

What version are you running? Etc.

Sentry Version: 24.12.1 self hosted ,with distributed clickhouse

Steps to Reproduce

Snuba ERROR cause CPU > 100%

Expected Result

what reason caused this error and how to fix? Thanks!

Actual Result

What actually happened. Maybe a screenshot/recording? Maybe some logs?
{"timestamp":"2025-01-26T10:56:07.447671Z","level":"ERROR","fields":{"message":"Strategy(error sending request for url (http://10.238.141.13:8123/?load_balancing=in_order&insert_distributed_sync=1&query=INSERT+INTO+errors_dist+FORMAT+JSONEachRow): connection closed before message completed\n\nCaused by:\n connection closed before message completed\n\nStack backtrace:\n 0: anyhow::error::<impl core::convert::From for anyhow::Error>::from\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.82/src/backtrace.rs:27:14\n 1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Resultcore::convert::Infallible,E>>::from_residual\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/result.rs:2010:27\n 2: rust_snuba::strategies::clickhouse::batch::BatchFactory::new_batch::{{closure}}\n at ./rust_snuba/src/strategies/clickhouse/batch.rs:134:32\n 3: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17\n 4: tokio::loom::std::unsafe_cell::UnsafeCell::with_mut\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9\n 5: tokio::runtime::task::core::Core<T,S>::poll\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30\n 6: tokio::runtime::task::harness::poll_future::{{closure}}\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19\n 7: <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panic/unwind_safe.rs:272:9\n 8: std::panicking::try::do_call\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:557:40\n 9: std::panicking::try\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:520:19\n 10: std::panic::catch_unwind\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panic.rs:358:14\n 11: tokio::runtime::task::harness::poll_future\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18\n 12: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27\n 13: tokio::runtime::task::harness::Harness<T,S>::poll\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15\n 14: tokio::runtime::task::raw::poll\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5\n 15: tokio::runtime::task::raw::RawTask::poll\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18\n 16: tokio::runtime::task::LocalNotified::run\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9\n 17: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:639:22\n 18: tokio::runtime::coop::with_budget\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5\n 19: tokio::runtime::coop::budget\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5\n 20: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9\n 21: tokio::runtime::scheduler::multi_thread::worker::Context::run\n 22: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21\n 23: tokio::runtime::context::scoped::Scoped::set\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9\n 24: tokio::runtime::context::set_scheduler::{{closure}}\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26\n 25: std::thread::local::LocalKey::try_with\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/local.rs:283:12\n 26: std::thread::local::LocalKey::with\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/local.rs:260:9\n 27: tokio::runtime::context::set_scheduler\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:17\n 28: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9\n 29: tokio::runtime::context::runtime::enter_runtime\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16\n 30: tokio::runtime::scheduler::multi_thread::worker::run\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5\n 31: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45\n 32: <tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21\n 33: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17\n 34: tokio::loom::std::unsafe_cell::UnsafeCell::with_mut\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9\n 35: tokio::runtime::task::core::Core<T,S>::poll\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:30\n 36: tokio::runtime::task::harness::poll_future::{{closure}}\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19\n 37: <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panic/unwind_safe.rs:272:9\n 38: std::panicking::try::do_call\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:557:40\n 39: std::panicking::try\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:520:19\n 40: std::panic::catch_unwind\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panic.rs:358:14\n 41: tokio::runtime::task::harness::poll_future\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18\n 42: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27\n 43: tokio::runtime::task::harness::Harness<T,S>::poll\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15\n 44: tokio::runtime::task::raw::RawTask::poll\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18\n 45: tokio::runtime::task::UnownedTask::run\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9\n 46: tokio::runtime::blocking::pool::Task::run\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9\n 47: tokio::runtime::blocking::pool::Inner::run\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17\n 48: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}\n at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13\n 49: std::sys::backtrace::rust_begin_short_backtrace\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/sys/backtrace.rs:154:18\n 50: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/mod.rs:538:17\n 51: <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panic/unwind_safe.rs:272:9\n 52: std::panicking::try::do_call\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:557:40\n 53: std::panicking::try\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:520:19\n 54: std::panic::catch_unwind\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panic.rs:358:14\n 55: std::thread::Builder::spawn_unchecked::{{closure}}\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/thread/mod.rs:537:30\n 56: core::ops::function::FnOnce::call_once{{vtable.shim}}\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/ops/function.rs:250:5\n 57: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/alloc/src/boxed.rs:2454:9\n 58: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/alloc/src/boxed.rs:2454:9\n 59: std::sys::pal::unix::thread::Thread::new::thread_start\n at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/sys/pal/unix/thread.rs:105:17\n 60: \n 61: __clone)"},"target":"rust_snuba::consumer"}