Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

StackExchanger watcher regularly restarts after connection reset by peer errors #980

Open
benoit74 opened this issue Jun 3, 2024 · 1 comment
Open

StackExchanger watcher regularly restarts after connection reset by peer errors #980

benoit74 opened this issue Jun 3, 2024 · 1 comment
Labels
awaits-reprocase Needs a new case to gather more info bug question

Comments

@benoit74
Copy link
Collaborator

benoit74 commented Jun 3, 2024

StackExchanger watcher regularly restarts after connection reset by peer errors

It looks like intermittent issue on an upstream (archive.org? wasabi?)

Sample occurence:

Common labels: {"cluster":"kiwix-prod","container":"zimfarm-watcher","flags":"F","job":"zimfarm/zimfarm-watcher","namespace":"zimfarm","pod":"watcher-deployment-54cf77666f-k79zm","service_name":"zimfarm-watcher"}
Line limit: "1000 (379 displayed)"
Total bytes processed: "33.4  kB"


2024-06-03 08:56:03.893	[2024-06-03 06:56:03,762::INFO] Getting list of SE domains
2024-06-03 08:56:07.699	[2024-06-03 06:56:07,699::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:01:18.439	[2024-06-03 07:01:18,439::ERROR] FAILED. An error occurred: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:19.269	[2024-06-03 07:01:18,439::ERROR] ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:19.269	Traceback (most recent call last):
2024-06-03 09:01:19.269	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:01:19.269	    response = self._make_request(
2024-06-03 09:01:19.269	               ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.269	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:01:19.269	    raise new_e
2024-06-03 09:01:19.269	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:01:19.269	    self._validate_conn(conn)
2024-06-03 09:01:19.269	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:01:19.269	    conn.connect()
2024-06-03 09:01:19.269	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:01:19.270	    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:01:19.270	                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:01:19.270	    ssl_sock = ssl_wrap_socket(
2024-06-03 09:01:19.270	               ^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:01:19.270	    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:01:19.270	               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:01:19.270	    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:01:19.270	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:01:19.270	    return self.sslsocket_class._create(
2024-06-03 09:01:19.270	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:01:19.270	    self.do_handshake()
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:01:19.270	    self._sslobj.do_handshake()
2024-06-03 09:01:19.270	ConnectionResetError: [Errno 104] Connection reset by peer
2024-06-03 09:01:19.270	
2024-06-03 09:01:19.270	During handling of the above exception, another exception occurred:
2024-06-03 09:01:19.270	
2024-06-03 09:01:19.270	Traceback (most recent call last):
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 564, in send
2024-06-03 09:01:19.270	    resp = conn.urlopen(
2024-06-03 09:01:19.270	           ^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
2024-06-03 09:01:19.270	    retries = retries.increment(
2024-06-03 09:01:19.270	              ^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 470, in increment
2024-06-03 09:01:19.270	    raise reraise(type(error), error, _stacktrace)
2024-06-03 09:01:19.270	          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
2024-06-03 09:01:19.270	    raise value.with_traceback(tb)
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:01:19.270	    response = self._make_request(
2024-06-03 09:01:19.270	               ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:01:19.270	    raise new_e
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:01:19.270	    self._validate_conn(conn)
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:01:19.270	    conn.connect()
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:01:19.270	    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:01:19.270	                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:01:19.270	    ssl_sock = ssl_wrap_socket(
2024-06-03 09:01:19.270	               ^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:01:19.270	    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:01:19.270	               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:01:19.270	    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:01:19.270	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:01:19.270	    return self.sslsocket_class._create(
2024-06-03 09:01:19.270	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:01:19.270	    self.do_handshake()
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:01:19.270	    self._sslobj.do_handshake()
2024-06-03 09:01:19.270	urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:19.270	
2024-06-03 09:01:19.270	During handling of the above exception, another exception occurred:
2024-06-03 09:01:19.270	
2024-06-03 09:01:19.270	Traceback (most recent call last):
2024-06-03 09:01:19.270	  File "/usr/local/bin/watcher", line 583, in entrypoint
2024-06-03 09:01:19.270	    sys.exit(runner.run())
2024-06-03 09:01:19.270	             ^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/bin/watcher", line 494, in run
2024-06-03 09:01:19.270	    self.check_and_go()
2024-06-03 09:01:19.270	  File "/usr/local/bin/watcher", line 427, in check_and_go
2024-06-03 09:01:19.270	    last_modified = get_last_modified_for(url)
2024-06-03 09:01:19.270	                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/bin/watcher", line 68, in get_last_modified_for
2024-06-03 09:01:19.270	    with requests.head(url, allow_redirects=True) as resp:
2024-06-03 09:01:19.270	         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 100, in head
2024-06-03 09:01:19.270	    return request("head", url, **kwargs)
2024-06-03 09:01:19.270	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
2024-06-03 09:01:19.270	    return session.request(method=method, url=url, **kwargs)
2024-06-03 09:01:19.270	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
2024-06-03 09:01:19.270	    resp = self.send(prep, **send_kwargs)
2024-06-03 09:01:19.270	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
2024-06-03 09:01:19.270	    r = adapter.send(request, **kwargs)
2024-06-03 09:01:19.270	        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:01:19.270	  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 579, in send
2024-06-03 09:01:19.270	    raise ConnectionError(err, request=request)
2024-06-03 09:01:19.270	requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:01:55.223	[entrypoint] exposing s3_url secret as S3_URL
2024-06-03 09:01:55.356	[entrypoint] exposing zimfarm_username secret as ZIMFARM_USERNAME
2024-06-03 09:01:55.357	[entrypoint] exposing zimfarm_password secret as ZIMFARM_PASSWORD
2024-06-03 09:02:16.063	[2024-06-03 07:02:16,063::INFO] 
2024-06-03 09:02:16.063	  __                                         _       _
2024-06-03 09:02:16.063	 / _| __ _ _ __ _ __ ___      __      ____ _| |_ ___| |__   ___ _ __
2024-06-03 09:02:16.063	| |_ / _` | '__| '_ ` _ \ ____\ \ /\ / / _` | __/ __| '_ \ / _ \ '__|
2024-06-03 09:02:16.063	|  _| (_| | |  | | | | | |_____\ V  V / (_| | || (__| | | |  __/ |
2024-06-03 09:02:16.063	|_|  \__,_|_|  |_| |_| |_|      \_/\_/ \__,_|\__\___|_| |_|\___|_|
2024-06-03 09:02:16.063	
2024-06-03 09:02:16.063	
2024-06-03 09:02:16.063	[2024-06-03 07:02:16,063::INFO] Testing S3 credentials
2024-06-03 09:02:40.652	[2024-06-03 07:02:40,652::INFO] Testing Zimfarm credentials with https://api.farm.openzim.org/v1…
2024-06-03 09:02:42.048	[2024-06-03 07:02:42,048::INFO] Starting watcher:
2024-06-03 09:02:42.048	  with zimfarm username: stackwatcher
2024-06-03 09:02:42.048	  using cache: s3.us-west-1.wasabisys.com
2024-06-03 09:02:42.048	  with bucket: org-kiwix-stackexchange
2024-06-03 09:02:42.048	[2024-06-03 07:02:42,048::INFO] Getting list of SE domains
2024-06-03 09:02:44.595	[2024-06-03 07:02:44,594::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:03:30.530	[2024-06-03 07:03:30,530::ERROR] FAILED. An error occurred: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:03:30.549	[2024-06-03 07:03:30,530::ERROR] ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:03:30.549	Traceback (most recent call last):
2024-06-03 09:03:30.549	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:03:30.549	    response = self._make_request(
2024-06-03 09:03:30.549	               ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.549	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:03:30.549	    raise new_e
2024-06-03 09:03:30.549	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:03:30.549	    self._validate_conn(conn)
2024-06-03 09:03:30.549	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:03:30.549	    conn.connect()
2024-06-03 09:03:30.549	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:03:30.549	    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:03:30.549	                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.549	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:03:30.549	    ssl_sock = ssl_wrap_socket(
2024-06-03 09:03:30.549	               ^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.549	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:03:30.549	    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:03:30.550	               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:03:30.550	    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:03:30.550	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:03:30.550	    return self.sslsocket_class._create(
2024-06-03 09:03:30.550	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:03:30.550	    self.do_handshake()
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:03:30.550	    self._sslobj.do_handshake()
2024-06-03 09:03:30.550	ConnectionResetError: [Errno 104] Connection reset by peer
2024-06-03 09:03:30.550	
2024-06-03 09:03:30.550	During handling of the above exception, another exception occurred:
2024-06-03 09:03:30.550	
2024-06-03 09:03:30.550	Traceback (most recent call last):
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 564, in send
2024-06-03 09:03:30.550	    resp = conn.urlopen(
2024-06-03 09:03:30.550	           ^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
2024-06-03 09:03:30.550	    retries = retries.increment(
2024-06-03 09:03:30.550	              ^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 470, in increment
2024-06-03 09:03:30.550	    raise reraise(type(error), error, _stacktrace)
2024-06-03 09:03:30.550	          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
2024-06-03 09:03:30.550	    raise value.with_traceback(tb)
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:03:30.550	    response = self._make_request(
2024-06-03 09:03:30.550	               ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:03:30.550	    raise new_e
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:03:30.550	    self._validate_conn(conn)
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:03:30.550	    conn.connect()
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:03:30.550	    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:03:30.550	                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:03:30.550	    ssl_sock = ssl_wrap_socket(
2024-06-03 09:03:30.550	               ^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:03:30.550	    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:03:30.550	               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:03:30.550	    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:03:30.550	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:03:30.550	    return self.sslsocket_class._create(
2024-06-03 09:03:30.550	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:03:30.550	    self.do_handshake()
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:03:30.550	    self._sslobj.do_handshake()
2024-06-03 09:03:30.550	urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:03:30.550	
2024-06-03 09:03:30.550	During handling of the above exception, another exception occurred:
2024-06-03 09:03:30.550	
2024-06-03 09:03:30.550	Traceback (most recent call last):
2024-06-03 09:03:30.550	  File "/usr/local/bin/watcher", line 583, in entrypoint
2024-06-03 09:03:30.550	    sys.exit(runner.run())
2024-06-03 09:03:30.550	             ^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/bin/watcher", line 490, in run
2024-06-03 09:03:30.550	    self.check_and_go()
2024-06-03 09:03:30.550	  File "/usr/local/bin/watcher", line 427, in check_and_go
2024-06-03 09:03:30.550	    last_modified = get_last_modified_for(url)
2024-06-03 09:03:30.550	                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/bin/watcher", line 68, in get_last_modified_for
2024-06-03 09:03:30.550	    with requests.head(url, allow_redirects=True) as resp:
2024-06-03 09:03:30.550	         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 100, in head
2024-06-03 09:03:30.550	    return request("head", url, **kwargs)
2024-06-03 09:03:30.550	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
2024-06-03 09:03:30.550	    return session.request(method=method, url=url, **kwargs)
2024-06-03 09:03:30.550	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
2024-06-03 09:03:30.550	    resp = self.send(prep, **send_kwargs)
2024-06-03 09:03:30.550	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.550	  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
2024-06-03 09:03:30.551	    r = adapter.send(request, **kwargs)
2024-06-03 09:03:30.551	        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:03:30.551	  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 579, in send
2024-06-03 09:03:30.551	    raise ConnectionError(err, request=request)
2024-06-03 09:03:30.551	requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:04:19.656	[entrypoint] exposing s3_url secret as S3_URL
2024-06-03 09:04:19.657	[entrypoint] exposing zimfarm_username secret as ZIMFARM_USERNAME
2024-06-03 09:04:19.658	[entrypoint] exposing zimfarm_password secret as ZIMFARM_PASSWORD
2024-06-03 09:04:22.099	[2024-06-03 07:04:22,098::INFO] 
2024-06-03 09:04:22.298	  __                                         _       _
2024-06-03 09:04:22.298	 / _| __ _ _ __ _ __ ___      __      ____ _| |_ ___| |__   ___ _ __
2024-06-03 09:04:22.298	| |_ / _` | '__| '_ ` _ \ ____\ \ /\ / / _` | __/ __| '_ \ / _ \ '__|
2024-06-03 09:04:22.298	|  _| (_| | |  | | | | | |_____\ V  V / (_| | || (__| | | |  __/ |
2024-06-03 09:04:22.298	|_|  \__,_|_|  |_| |_| |_|      \_/\_/ \__,_|\__\___|_| |_|\___|_|
2024-06-03 09:04:22.298	
2024-06-03 09:04:22.298	
2024-06-03 09:04:22.298	[2024-06-03 07:04:22,099::INFO] Testing S3 credentials
2024-06-03 09:04:27.539	[2024-06-03 07:04:27,539::INFO] Testing Zimfarm credentials with https://api.farm.openzim.org/v1…
2024-06-03 09:04:27.651	[2024-06-03 07:04:27,651::INFO] Starting watcher:
2024-06-03 09:04:27.651	  with zimfarm username: stackwatcher
2024-06-03 09:04:27.651	  using cache: s3.us-west-1.wasabisys.com
2024-06-03 09:04:27.651	  with bucket: org-kiwix-stackexchange
2024-06-03 09:04:27.651	[2024-06-03 07:04:27,651::INFO] Getting list of SE domains
2024-06-03 09:04:28.826	[2024-06-03 07:04:28,826::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:10:17.017	[2024-06-03 07:10:17,017::ERROR] FAILED. An error occurred: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:10:17.534	[2024-06-03 07:10:17,017::ERROR] ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:10:17.534	Traceback (most recent call last):
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:10:17.534	    response = self._make_request(
2024-06-03 09:10:17.534	               ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:10:17.534	    raise new_e
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:10:17.534	    self._validate_conn(conn)
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:10:17.534	    conn.connect()
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:10:17.534	    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:10:17.534	                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:10:17.534	    ssl_sock = ssl_wrap_socket(
2024-06-03 09:10:17.534	               ^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:10:17.534	    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:10:17.534	               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:10:17.534	    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:10:17.534	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:10:17.534	    return self.sslsocket_class._create(
2024-06-03 09:10:17.534	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:10:17.534	    self.do_handshake()
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:10:17.534	    self._sslobj.do_handshake()
2024-06-03 09:10:17.534	ConnectionResetError: [Errno 104] Connection reset by peer
2024-06-03 09:10:17.534	
2024-06-03 09:10:17.534	During handling of the above exception, another exception occurred:
2024-06-03 09:10:17.534	
2024-06-03 09:10:17.534	Traceback (most recent call last):
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 564, in send
2024-06-03 09:10:17.534	    resp = conn.urlopen(
2024-06-03 09:10:17.534	           ^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
2024-06-03 09:10:17.534	    retries = retries.increment(
2024-06-03 09:10:17.534	              ^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 470, in increment
2024-06-03 09:10:17.534	    raise reraise(type(error), error, _stacktrace)
2024-06-03 09:10:17.534	          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 38, in reraise
2024-06-03 09:10:17.534	    raise value.with_traceback(tb)
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
2024-06-03 09:10:17.534	    response = self._make_request(
2024-06-03 09:10:17.534	               ^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
2024-06-03 09:10:17.534	    raise new_e
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-06-03 09:10:17.534	    self._validate_conn(conn)
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
2024-06-03 09:10:17.534	    conn.connect()
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
2024-06-03 09:10:17.534	    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
2024-06-03 09:10:17.534	                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
2024-06-03 09:10:17.534	    ssl_sock = ssl_wrap_socket(
2024-06-03 09:10:17.534	               ^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
2024-06-03 09:10:17.534	    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
2024-06-03 09:10:17.534	               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
2024-06-03 09:10:17.534	    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
2024-06-03 09:10:17.534	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/ssl.py", line 455, in wrap_socket
2024-06-03 09:10:17.534	    return self.sslsocket_class._create(
2024-06-03 09:10:17.534	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/ssl.py", line 1042, in _create
2024-06-03 09:10:17.534	    self.do_handshake()
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/ssl.py", line 1320, in do_handshake
2024-06-03 09:10:17.534	    self._sslobj.do_handshake()
2024-06-03 09:10:17.534	urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:10:17.534	
2024-06-03 09:10:17.534	During handling of the above exception, another exception occurred:
2024-06-03 09:10:17.534	
2024-06-03 09:10:17.534	Traceback (most recent call last):
2024-06-03 09:10:17.534	  File "/usr/local/bin/watcher", line 583, in entrypoint
2024-06-03 09:10:17.534	    sys.exit(runner.run())
2024-06-03 09:10:17.534	             ^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/bin/watcher", line 490, in run
2024-06-03 09:10:17.534	    self.check_and_go()
2024-06-03 09:10:17.534	  File "/usr/local/bin/watcher", line 427, in check_and_go
2024-06-03 09:10:17.534	    last_modified = get_last_modified_for(url)
2024-06-03 09:10:17.534	                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/bin/watcher", line 68, in get_last_modified_for
2024-06-03 09:10:17.534	    with requests.head(url, allow_redirects=True) as resp:
2024-06-03 09:10:17.534	         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 100, in head
2024-06-03 09:10:17.534	    return request("head", url, **kwargs)
2024-06-03 09:10:17.534	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
2024-06-03 09:10:17.534	    return session.request(method=method, url=url, **kwargs)
2024-06-03 09:10:17.534	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
2024-06-03 09:10:17.534	    resp = self.send(prep, **send_kwargs)
2024-06-03 09:10:17.534	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.534	  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
2024-06-03 09:10:17.534	    r = adapter.send(request, **kwargs)
2024-06-03 09:10:17.534	        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-03 09:10:17.535	  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 579, in send
2024-06-03 09:10:17.535	    raise ConnectionError(err, request=request)
2024-06-03 09:10:17.535	requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2024-06-03 09:11:02.072	[entrypoint] exposing s3_url secret as S3_URL
2024-06-03 09:11:02.074	[entrypoint] exposing zimfarm_username secret as ZIMFARM_USERNAME
2024-06-03 09:11:02.075	[entrypoint] exposing zimfarm_password secret as ZIMFARM_PASSWORD
2024-06-03 09:11:19.567	[2024-06-03 07:11:19,567::INFO] 
2024-06-03 09:11:19.567	  __                                         _       _
2024-06-03 09:11:19.567	 / _| __ _ _ __ _ __ ___      __      ____ _| |_ ___| |__   ___ _ __
2024-06-03 09:11:19.567	| |_ / _` | '__| '_ ` _ \ ____\ \ /\ / / _` | __/ __| '_ \ / _ \ '__|
2024-06-03 09:11:19.567	|  _| (_| | |  | | | | | |_____\ V  V / (_| | || (__| | | |  __/ |
2024-06-03 09:11:19.567	|_|  \__,_|_|  |_| |_| |_|      \_/\_/ \__,_|\__\___|_| |_|\___|_|
2024-06-03 09:11:19.567	
2024-06-03 09:11:19.567	
2024-06-03 09:11:19.567	[2024-06-03 07:11:19,567::INFO] Testing S3 credentials
2024-06-03 09:11:35.117	[2024-06-03 07:11:35,116::INFO] Testing Zimfarm credentials with https://api.farm.openzim.org/v1…
2024-06-03 09:11:35.275	[2024-06-03 07:11:35,275::INFO] Starting watcher:
2024-06-03 09:11:35.275	  with zimfarm username: stackwatcher
2024-06-03 09:11:35.275	  using cache: s3.us-west-1.wasabisys.com
2024-06-03 09:11:35.275	  with bucket: org-kiwix-stackexchange
2024-06-03 09:11:35.275	[2024-06-03 07:11:35,275::INFO] Getting list of SE domains
2024-06-03 09:11:37.900	[2024-06-03 07:11:37,899::INFO] Grabbing online versions and comparing with S3…
2024-06-03 09:21:05.560	[2024-06-03 07:21:05,463::INFO] All synced up.
2024-06-03 09:21:05.560	[2024-06-03 07:21:05,560::INFO] Done.

Not sure something is needed to be done since the system recover nicely ; maybe just avoid to restart completely the process.
@benoit74 benoit74 added bug question awaits-reprocase Needs a new case to gather more info labels Jun 3, 2024
@benoit74
Copy link
Collaborator Author

benoit74 commented Jun 3, 2024

List of all restarts in the last 30 days (not filtered by cause, some of them are expected due to updates typically)

2024-05-07 17:29:05
2024-05-13 10:38:41
2024-05-14 09:56:57
2024-05-14 11:17:08
2024-05-14 12:42:25
2024-05-14 13:52:54
2024-05-14 14:08:04
2024-05-15 04:08:13
2024-05-27 04:30:34
2024-05-28 08:12:46
2024-05-29 08:14:52
2024-05-30 08:16:38
2024-05-31 08:19:12
2024-06-01 08:20:42
2024-06-01 08:22:34
2024-06-01 08:24:48
2024-06-01 08:34:15
2024-06-01 08:44:18
2024-06-01 08:45:28
2024-06-01 08:55:54
2024-06-03 09:02:16
2024-06-03 09:04:22
2024-06-03 09:11:19

@benoit74 benoit74 mentioned this issue Jun 3, 2024
Closed
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaits-reprocase Needs a new case to gather more info bug question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@benoit74