Generic camera stops working after some time, usually several hours #58425

adorobis · 2021-10-25T16:16:46Z

The problem

Generic camera component stops working, usually after several hours after system restart.

What version of Home Assistant Core has the issue?

core-2021.10.6

What was the last working version of Home Assistant Core?

core-2021.9.7

What type of installation are you running?

Home Assistant Core

Integration causing the issue

Generic Camera

Link to integration documentation on our website

https://www.home-assistant.io/integrations/generic/

Example YAML snippet

camera:
  - platform: generic
    still_image_url: http://10.144.1.107/image/jpeg.cgi
    username: !secret camera1_username
    password: !secret camera1_password
  - platform: generic
    name: Dashboard 2 Camera
    still_image_url: http://10.144.1.51:2323/?cmd=getCamshot&password=xxxx
  - platform: generic
    name: Dashboard Camera
    still_image_url: http://10.144.1.139:2323/?cmd=getCamshot&password=yyyy

Anything in the logs that might be useful for us?

2021-10-25 15:55:06 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
    resp = await self._request_handler(request)
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/aiohttp/web_app.py", line 499, in _handle
    resp = await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/homeassistant/components/http/forwarded.py", line 98, in forwarded_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/homeassistant/components/http/request_context.py", line 24, in request_context_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/homeassistant/components/http/ban.py", line 78, in ban_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/homeassistant/components/http/auth.py", line 144, in auth_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/homeassistant/components/http/view.py", line 136, in handle
    result = await result
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/homeassistant/components/camera/__init__.py", line 591, in get
    return await self.handle(request, camera)
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/homeassistant/components/camera/__init__.py", line 609, in handle
    image = await _async_get_image(
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/homeassistant/components/camera/__init__.py", line 164, in _async_get_image
    image_bytes = await camera.async_camera_image(
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/homeassistant/components/generic/camera.py", line 146, in async_camera_image
    response = await async_client.get(
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/httpx/_client.py", line 1740, in get
    return await self.request(
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/httpx/_client.py", line 1494, in request
    response = await self.send(
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/httpx/_client.py", line 1586, in send
    response = await self._send_handling_auth(
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/httpx/_client.py", line 1616, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/httpx/_client.py", line 1655, in _send_handling_redirects
    response = await self._send_single_request(request, timeout)
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/httpx/_client.py", line 1699, in _send_single_request
    ) = await transport.handle_async_request(
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/httpx/_transports/default.py", line 281, in handle_async_request
    ) = await self._pool.handle_async_request(
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 219, in handle_async_request
    async with self._connection_acquiry_lock:
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/httpcore/_backends/base.py", line 76, in __aenter__
    await self.acquire()
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/httpcore/_backends/anyio.py", line 104, in acquire
    await self._lock.acquire()
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/anyio/_core/_synchronization.py", line 119, in acquire
    self.acquire_nowait()
  File "/usr/local/share/homeassistant/lib/python3.8/site-packages/anyio/_core/_synchronization.py", line 146, in acquire_nowait
    raise RuntimeError('Attempted to acquire an already held Lock')
RuntimeError: Attempted to acquire an already held Lock

Additional information

I have 3 cameras, 2 are provided by Fully Kiosk Browser android app, 1 is a D-Link IP camera. All 3 stop working at the same time and multiple of above error messages are written to the HA log.
Never had similar issue on earlier versions of HA.
The installation is on TrueNAS jail, in python venv. Python version is 3.8.6

probot-home-assistant · 2021-10-25T16:16:49Z

generic documentation
generic source
_{^{(message by IssueLinks)}}

adorobis · 2021-10-27T12:41:01Z

Short update, I've downgraded the PyTurboJPEG module to version 1.5.2 and it seems to solve the issue. At least since 26 hour now the cameras work ok and no errors are logged. So clearly something wrong with this module in version 1.6.1 which has been introduced in one of the recent HA updates (#56571)

adorobis · 2021-10-28T18:46:23Z

Unfortunately I called it a success too early. The issue is still there... So I reinstalled the 1.6.2 version hoping someone will pick this issue up.

adorobis · 2021-11-17T08:28:28Z

I have the latest release of HA: 2021.11.4 and the issue is still there. The difference is that in the logs I can see a lot of messages:
ERROR (MainThread) [homeassistant.components.generic.camera] Error getting new camera image from Generic Camera: All connection attempts failed
After restarting HA it is working again for a while (usually around 10 hours)

mikosoft83 · 2021-11-18T08:39:53Z

I have the same issue, three http image fetching cameras and all of them worked fine and just stopped working all of a sudden. I tried ssl verify on and off without success. The cameras show first image after restart then all subsequent image fetches fail.

lfrlucas · 2021-12-19T23:43:11Z

Same issue here. After some hours generic camera stops working. Only recovers with HA restart.

lfrlucas · 2021-12-20T16:28:21Z

My logs:
2021-12-20 00:17:28 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
resp = await request_handler(request)
File "/usr/local/lib/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
resp = await handler(request)
File "/usr/local/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 98, in forwarded_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 78, in ban_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 181, in auth_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 137, in handle
result = await result
File "/usr/src/homeassistant/homeassistant/components/camera/init.py", line 629, in get
return await self.handle(request, camera)
File "/usr/src/homeassistant/homeassistant/components/camera/init.py", line 668, in handle
stream = await camera.handle_async_mjpeg_stream(request)
File "/usr/src/homeassistant/homeassistant/components/proxy/camera.py", line 268, in handle_async_mjpeg_stream
return await async_get_still_stream(
File "/usr/src/homeassistant/homeassistant/components/camera/init.py", line 260, in async_get_still_stream
await write_to_mjpeg_stream(img_bytes)
File "/usr/src/homeassistant/homeassistant/components/camera/init.py", line 241, in write_to_mjpeg_stream
await response.write(
File "/usr/local/lib/python3.9/site-packages/aiohttp/web_response.py", line 514, in write
await self._payload_writer.write(data)
File "/usr/local/lib/python3.9/site-packages/aiohttp/http_writer.py", line 116, in write
self._write(chunk)
File "/usr/local/lib/python3.9/site-packages/aiohttp/http_writer.py", line 76, in _write
raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport

adorobis · 2021-12-21T11:58:20Z

Interestingly, since HA restart yesterday (running 2021.12.0) and despite below errors in the log the camera stream is still working. Let's see for how long:

2021-12-21 12:24:41 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/forwarded.py", line 98, in forwarded_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/ban.py", line 78, in ban_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/auth.py", line 181, in auth_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/view.py", line 137, in handle
    result = await result
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/camera/__init__.py", line 629, in get
    return await self.handle(request, camera)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/camera/__init__.py", line 668, in handle
    stream = await camera.handle_async_mjpeg_stream(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/ring/camera.py", line 141, in handle_async_mjpeg_stream
    await stream.close()
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/haffmpeg/core.py", line 158, in close
    await self._loop.run_in_executor(None, _close)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/haffmpeg/core.py", line 153, in _close
    self._proc.stdin.write(b"q")
BrokenPipeError: [Errno 32] Broken pipe

This error repeats irregularly not sure what is triggering it.

uvjustin · 2021-12-25T17:59:12Z

The posts from before Nov and earlier (the posts from Dec don't appear to be related) may be from bugs in the httpx library. They are also causing similar issues in philips js: #49801 .
aiohttp is more robust but unfortunately it doesn't support digest authentication directly.
A workaround is to avoid using mjpeg streams since constant new connections to the camera for still images may overload your cameras and also contribute to a connection pool issue.

richardkuw · 2022-01-03T20:25:58Z

My generic cameras also stop displaying in Lovelace after a random length of time with a lock related error, generally within a few hours of starting HA core. Restarting fixes it for a bit, rolling back to a November release fixes it entirely. The traceback is similar to the one in the beginning of this thread except that I am using the latest version of HA:

Jan  3 12:58:20 hass 0251a0b04476[687]: 2022-01-03 12:58:20 ERROR (MainThread) [aiohttp.server] Error handling request
Jan  3 12:58:20 hass 0251a0b04476[687]: Traceback (most recent call last):
Jan  3 12:58:20 hass 0251a0b04476[687]:   File "/usr/local/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
Jan  3 12:58:20 hass 0251a0b04476[687]:     resp = await request_handler(request)
...
Jan  3 12:58:20 hass 0251a0b04476[687]:   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_synchronization.py", line 119, in acquire
Jan  3 12:58:20 hass 0251a0b04476[687]:     self.acquire_nowait()
Jan  3 12:58:20 hass 0251a0b04476[687]:   File "/usr/local/lib/python3.9/site-packages/anyio/_core/_synchronization.py", line 148, in acquire_nowait
Jan  3 12:58:20 hass 0251a0b04476[687]:     raise RuntimeError('Attempted to acquire an already held Lock')
Jan  3 12:58:20 hass 0251a0b04476[687]: RuntimeError: Attempted to acquire an already held Lock

Restoring a backup from November fixes the problem. It started for me with 2021.12.1. I should note that I am chromecasting a Lovelace dashboard containing six picture-entity cards that show the cameras to a Google display device, pretty much 24x7. Maybe that accelerates the issue for me.

I see the following changed in /usr/local/lib/python3.9/site-packages/anyio/_core/_synchronization.py at that point. This change was listed as "Fixed race condition when cancelling acquire() in Lock or Semaphore" on the anyio github:
agronholm/anyio#388

115   async def acquire(self) -> None:
        """Acquire the lock."""
        await checkpoint_if_cancelled()
        try:
            self.acquire_nowait()
        except WouldBlock:
            task = get_current_task()
            event = Event()
            token = task, event
            self._waiters.append(token)
            try:
                await event.wait()
            except BaseException:
                if not event.is_set():
                    self._waiters.remove(token)
>> THESE TWO LINES WERE ADDED IN 2021.12 <<
                elif self._owner_task == task:
                    self.release()

                raise

So perhaps my particular issue is a side effect of that change. I don't pretend to know how this code works, but there was an additional lock related change in anyio that seems like it might be relevant. I don't think it has made its way into HA yet, I wonder if it might help:
Deadlock in anyio.Lock agronholm/anyio#398

uvjustin · 2022-01-28T07:45:30Z

The PRs above should be included in the newest beta. If anyone here is on the newest beta, please let us know if the issue has been resolved.

uvjustin · 2022-02-06T05:51:41Z

Does this issue still happen in 2022.2?

adorobis · 2022-02-06T08:45:40Z

My system is up for 20 hours and so far so good. Probably need to observe a little longer before closing it.

adorobis · 2022-02-06T08:49:02Z

I've checked logs from previous day and the following error is there:

2022-02-05 11:31:03 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/forwarded.py", line 98, in forwarded_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/ban.py", line 79, in ban_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/auth.py", line 219, in auth_middleware
    return await handler(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/http/view.py", line 137, in handle
    result = await result
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/camera/__init__.py", line 748, in get
    return await self.handle(request, camera)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/camera/__init__.py", line 787, in handle
    stream = await camera.handle_async_mjpeg_stream(request)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/homeassistant/components/ring/camera.py", line 147, in handle_async_mjpeg_stream
    await stream.close()
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/haffmpeg/core.py", line 158, in close
    await self._loop.run_in_executor(None, _close)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/share/homeassistant/lib/python3.9/site-packages/haffmpeg/core.py", line 153, in _close
    self._proc.stdin.write(b"q")
BrokenPipeError: [Errno 32] Broken pipe

Haven't been looking at cameras at that time and the restart was for a different reason.

uvjustin · 2022-02-06T10:16:06Z

There's nothing in that log from generic camera - that problem seems like it's coming from ring camera.

mikosoft83 · 2022-02-06T11:06:09Z

I'm going to update HA now so I'll be monitoring it for a couple of coming days and will report back

adorobis · 2022-02-06T11:16:54Z

There's nothing in that log from generic camera - that problem seems like it's coming from ring camera.

Indeed, I haven't noticed that :)

adorobis · 2022-02-08T10:15:43Z

The system is up for two days now and generic cameras still work ok. Seems that the issue is solved as of HA 2022.2.0

mikosoft83 · 2022-02-14T11:51:38Z

Well, my problem isn't solved. I updated to 2022.2. I'm picking up a local generic camera (from HA addon Valetudo mapper) and it stops working after some time. Every time I reload generic cameras it works for some time and then stops. I was also checking the logs but there's nothing in the logs at all.

Before certain update (don't remember which) this camera worked without any issue every time.

probot-home-assistant bot added the integration: generic label Oct 25, 2021

This was referenced Jan 22, 2022

Update version requirements for httpcore and anyio #64711

Merged

Bump httpx to 0.21.3 and pin requirements for httpcore, anyio, and h11 #64822

Merged

MJPEG transport timeout after few hours #64737

Closed

uvjustin mentioned this issue Feb 6, 2022

Generic camera images via URL hang system #63647

Closed

adorobis closed this as completed Feb 8, 2022

github-actions bot locked and limited conversation to collaborators Mar 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generic camera stops working after some time, usually several hours #58425

Generic camera stops working after some time, usually several hours #58425

adorobis commented Oct 25, 2021

probot-home-assistant bot commented Oct 25, 2021

adorobis commented Oct 27, 2021

adorobis commented Oct 28, 2021

adorobis commented Nov 17, 2021

mikosoft83 commented Nov 18, 2021

lfrlucas commented Dec 19, 2021

lfrlucas commented Dec 20, 2021

adorobis commented Dec 21, 2021 •

edited

Loading

uvjustin commented Dec 25, 2021

richardkuw commented Jan 3, 2022

uvjustin commented Jan 28, 2022

uvjustin commented Feb 6, 2022

adorobis commented Feb 6, 2022

adorobis commented Feb 6, 2022

uvjustin commented Feb 6, 2022

mikosoft83 commented Feb 6, 2022

adorobis commented Feb 6, 2022

adorobis commented Feb 8, 2022

mikosoft83 commented Feb 14, 2022

Generic camera stops working after some time, usually several hours #58425

Generic camera stops working after some time, usually several hours #58425

Comments

adorobis commented Oct 25, 2021

The problem

What version of Home Assistant Core has the issue?

What was the last working version of Home Assistant Core?

What type of installation are you running?

Integration causing the issue

Link to integration documentation on our website

Example YAML snippet

Anything in the logs that might be useful for us?

Additional information

probot-home-assistant bot commented Oct 25, 2021

adorobis commented Oct 27, 2021

adorobis commented Oct 28, 2021

adorobis commented Nov 17, 2021

mikosoft83 commented Nov 18, 2021

lfrlucas commented Dec 19, 2021

lfrlucas commented Dec 20, 2021

adorobis commented Dec 21, 2021 • edited Loading

uvjustin commented Dec 25, 2021

richardkuw commented Jan 3, 2022

uvjustin commented Jan 28, 2022

uvjustin commented Feb 6, 2022

adorobis commented Feb 6, 2022

adorobis commented Feb 6, 2022

uvjustin commented Feb 6, 2022

mikosoft83 commented Feb 6, 2022

adorobis commented Feb 6, 2022

adorobis commented Feb 8, 2022

mikosoft83 commented Feb 14, 2022

adorobis commented Dec 21, 2021 •

edited

Loading