langgraph-checkpoint-postgres (psycopg.OperationalError: sending query and params failed: SSL error: bad length) encountered across multiple version

[shubhamnegikellton]

Checked other resources

• This is a bug, not a usage question. For questions, please use GitHub Discussions.
• I added a clear and detailed title that summarizes the issue.
• I read what a minimal reproducible example is (https://stackoverflow.com/help/minimal-reproducible-example).
• I included a self-contained, minimal example that demonstrates the issue INCLUDING all the relevant imports. The code run AS IS to reproduce the issue.

Example Code

from psycopg import Connection
from psycopg_pool import ConnectionPool
from psycopg.rows import dict_row
from langgraph.checkpoint.postgres import PostgresSaver

connection_kwargs = {"autocommit": True, "prepare_threshold": 0}

async with AsyncConnectionPool(conninfo=conninfo, max_size=20, kwargs=connection_kwargs) as pool:
           graph = create_react_agent(
                llm,
                build_tools,
                messages_modifier=_modify_messages,
                checkpointer=AsyncPostgresSaver(pool),  # type:ignore[arg-type]
            )

            async for event in graph.astream_events(
                {"messages": [("human", search_params.question)]},
                config={"configurable": {"thread_id": conversation_id, "recursion_limit": 20}},
                stream_mode="values",
                version="v2",
            ):

Error Message and Stack Trace (if applicable)

INSERT INTO checkpoints ( thread_id, checkpoint_ns, checkpoint_id, parent_checkpoint_id, checkpoint, metadata ) 
VALUES ( ? ) ON CONFLICT ( thread_id, checkpoint_ns, checkpoint_id ) DO 
UPDATE SET checkpoint = EXCLUDED.checkpoint, metadata = EXCLUDED.metadata

psycopg.OperationalError: sending query and params failed: SSL error: bad length
 File "/app/app/search/ai_models.py", line 315, in chat
    async for event in graph.astream_events(
  File "/app/venv/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 1386, in astream_events
    async for event in event_stream:
  File "/app/venv/lib/python3.12/site-packages/langchain_core/tracers/event_stream.py", line 1012, in _astream_events_implementation_v2
    await task
  File "/app/venv/lib/python3.12/site-packages/langchain_core/tracers/event_stream.py", line 967, in consume_astream
    async for _ in event_streamer.tap_output_aiter(run_id, stream):
  File "/app/venv/lib/python3.12/site-packages/langchain_core/tracers/event_stream.py", line 203, in tap_output_aiter
    async for chunk in output:
  File "/app/venv/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1832, in astream
    async with AsyncPregelLoop(
               ^^^^^^^^^^^^^^^^
  File "/app/venv/lib/python3.12/site-packages/langgraph/pregel/loop.py", line 1035, in __aexit__
    return await asyncio.shield(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 754, in __aexit__
    raise exc_details[1]
  File "/usr/local/lib/python3.12/contextlib.py", line 737, in __aexit__
    cb_suppress = await cb(*exc_details)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/venv/lib/python3.12/site-packages/langgraph/pregel/executor.py", line 200, in __aexit__
    raise exc
  File "/app/venv/lib/python3.12/site-packages/langgraph/pregel/loop.py", line 957, in _checkpointer_put_after_previous
    await cast(BaseCheckpointSaver, self.checkpointer).aput(
  File "/app/venv/lib/python3.12/site-packages/langgraph/checkpoint/postgres/aio.py", line 270, in aput
    await cur.execute(
  File "/app/venv/lib/python3.12/site-packages/ddtrace/contrib/dbapi_async.py", line 136, in execute
    return await self._trace_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/venv/lib/python3.12/site-packages/ddtrace/contrib/dbapi_async.py", line 105, in _trace_method
    return await method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/venv/lib/python3.12/site-packages/psycopg/cursor_async.py", line 97, in execute
    raise ex.with_traceback(None)
psycopg.OperationalError: sending query and params failed: SSL error: bad length
SSL SYSCALL error: EOF detected

Description

Faced the below issue with langgraph-checkpoint-postgres:

psycopg.OperationalError: sending query and params failed: SSL error: bad length
SSL SYSCALL error: EOF detected

NOTE:
I have tried with multiple langgraph-checkpoint-postgres i.e 2.0.11, 2.0.9, 2.0.13, 2.0.15

System Info

langchain = "0.3.11"
langchain-community = "0.3.11"
langchain-experimental = "0.3.3"
langchain-openai = "0.2.12"
langchain-postgres = "0.0.12"
langgraph = "0.2.58"
psycopg = { extras = ["binary"], version = "3.2.3" }
psycopg-pool = "3.2.3"
sqlalchemy = { version = "2.0.36", extras = ["asyncio"] }
sqlmodel = "0.0.22"
asyncpg = "0.30.0"
langgraph-checkpoint-postgres = "2.0.15"

[antonioalegria]

I have the exact same issue.

Some times, and I can't figure out exactly when to properly reproduce it, the following is logged:

2025-03-06 14:00:53 | W |              psycopg | error ignored terminating <psycopg.AsyncPipeline [BAD] at 0xffff482543e0>: the connection is lost
2025-03-06 14:00:53 | W |         psycopg.pool | discarding closed connection: <psycopg.AsyncConnection [BAD] at 0xffff6408b080>

And then there is a failure:

SSL error: bad length
SSL SYSCALL error: EOF detected
Traceback (most recent call last):
  File "/app/dojo/common/asyncio.py", line 39, in wrap_event_iterator_with_keep_alive
    event = event_task.result()
            ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 1389, in astream_events
    async for event in event_stream:
  File "/usr/local/lib/python3.12/site-packages/langchain_core/tracers/event_stream.py", line 1012, in _astream_events_implementation_v2
    await task
  File "/usr/local/lib/python3.12/site-packages/langchain_core/tracers/event_stream.py", line 967, in consume_astream
    async for _ in event_streamer.tap_output_aiter(run_id, stream):
  File "/usr/local/lib/python3.12/site-packages/langchain_core/tracers/event_stream.py", line 203, in tap_output_aiter
    async for chunk in output:
  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 2227, in astream
    async with AsyncPregelLoop(
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/loop.py", line 1109, in __aexit__
    return await exit_task
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 754, in __aexit__
    raise exc_details[1]
  File "/usr/local/lib/python3.12/contextlib.py", line 737, in __aexit__
    cb_suppress = await cb(*exc_details)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/executor.py", line 206, in __aexit__
    raise exc
  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/loop.py", line 1023, in _checkpointer_put_after_previous
    await prev
  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/loop.py", line 1025, in _checkpointer_put_after_previous
    await cast(BaseCheckpointSaver, self.checkpointer).aput(
  File "/usr/local/lib/python3.12/site-packages/langgraph/checkpoint/postgres/aio.py", line 261, in aput
    await cur.executemany(
  File "/usr/local/lib/python3.12/site-packages/psycopg/cursor_async.py", line 132, in executemany
    raise ex.with_traceback(None)
psycopg.OperationalError: sending prepared query failed: SSL error: bad length
SSL SYSCALL error: EOF detected

[vbarda]

any chance you might be running out of disk space? saw this https://stackoverflow.com/questions/63028407/psycopg2-databaseerror-ssl-error-bad-length, not 100% sure if this is relevant

[antonioalegria]

At least on my side I don't think that's it. This only happens sometimes, not on all requests...

The log messages below always precede the psycopg.OperationalError. Those two messages appear and then, when after that a request comes and is processed it blows up like this in the middle of processing when it's storing the checkpoint.

2025-03-06 14:00:53 | W |              psycopg | error ignored terminating <psycopg.AsyncPipeline [BAD] at 0xffff482543e0>: the connection is lost
2025-03-06 14:00:53 | W |         psycopg.pool | discarding closed connection: <psycopg.AsyncConnection [BAD] at 0xffff6408b080>

If at least we could handle this issue, then at least we could mitigate it. Like this some requests fail because it blows up in the middle with no way to deal with it.

[vbarda]

@antonioalegria any chance you have really large messages in react agent? potentially tool messages with large content?

[antonioalegria]

Yes, there is that chance, definitely.

[cloudlessdreams]

Yes, there is that chance, definitely.

I'm also exploring this idea.. great chat btw

[antonioalegria]

Do you need any more info?