Skip to content

Commit 1b74803

Browse files
authored
gh-93162: Add ability to configure QueueHandler/QueueListener together (GH-93269)
Also, provide getHandlerByName() and getHandlerNames() APIs. Closes #93162.
1 parent c6f6ede commit 1b74803

8 files changed

+325
-31
lines changed

Doc/library/logging.config.rst

+70
Original file line numberDiff line numberDiff line change
@@ -661,6 +661,76 @@ it with :func:`staticmethod`. For example::
661661
You don't need to wrap with :func:`staticmethod` if you're setting the import
662662
callable on a configurator *instance*.
663663

664+
.. _configure-queue:
665+
666+
Configuring QueueHandler and QueueListener
667+
""""""""""""""""""""""""""""""""""""""""""
668+
669+
If you want to configure a :class:`~logging.handlers.QueueHandler`, noting that this
670+
is normally used in conjunction with a :class:`~logging.handlers.QueueListener`, you
671+
can configure both together. After the configuration, the ``QueueListener`` instance
672+
will be available as the :attr:`~logging.handlers.QueueHandler.listener` attribute of
673+
the created handler, and that in turn will be available to you using
674+
:func:`~logging.getHandlerByName` and passing the name you have used for the
675+
``QueueHandler`` in your configuration. The dictionary schema for configuring the pair
676+
is shown in the example YAML snippet below.
677+
678+
.. code-block:: yaml
679+
680+
handlers:
681+
qhand:
682+
class: logging.handlers.QueueHandler
683+
queue: my.module.queue_factory
684+
listener: my.package.CustomListener
685+
handlers:
686+
- hand_name_1
687+
- hand_name_2
688+
...
689+
690+
The ``queue`` and ``listener`` keys are optional.
691+
692+
If the ``queue`` key is present, the corresponding value can be one of the following:
693+
694+
* An actual instance of :class:`queue.Queue` or a subclass thereof. This is of course
695+
only possible if you are constructing or modifying the configuration dictionary in
696+
code.
697+
698+
* A string that resolves to a callable which, when called with no arguments, returns
699+
the :class:`queue.Queue` instance to use. That callable could be a
700+
:class:`queue.Queue` subclass or a function which returns a suitable queue instance,
701+
such as ``my.module.queue_factory()``.
702+
703+
* A dict with a ``'()'`` key which is constructed in the usual way as discussed in
704+
:ref:`logging-config-dict-userdef`. The result of this construction should be a
705+
:class:`queue.Queue` instance.
706+
707+
If the ``queue`` key is absent, a standard unbounded :class:`queue.Queue` instance is
708+
created and used.
709+
710+
If the ``listener`` key is present, the corresponding value can be one of the following:
711+
712+
* A subclass of :class:`logging.handlers.QueueListener`. This is of course only
713+
possible if you are constructing or modifying the configuration dictionary in
714+
code.
715+
716+
* A string which resolves to a class which is a subclass of ``QueueListener``, such as
717+
``'my.package.CustomListener'``.
718+
719+
* A dict with a ``'()'`` key which is constructed in the usual way as discussed in
720+
:ref:`logging-config-dict-userdef`. The result of this construction should be a
721+
callable with the same signature as the ``QueueListener`` initializer.
722+
723+
If the ``listener`` key is absent, :class:`logging.handlers.QueueListener` is used.
724+
725+
The values under the ``handlers`` key are the names of other handlers in the
726+
configuration (not shown in the above snippet) which will be passed to the queue
727+
listener.
728+
729+
Any custom queue handler and listener classes will need to be defined with the same
730+
initialization signatures as :class:`~logging.handlers.QueueHandler` and
731+
:class:`~logging.handlers.QueueListener`.
732+
733+
.. versionadded:: 3.12
664734

665735
.. _logging-config-fileformat:
666736

Doc/library/logging.handlers.rst

+6
Original file line numberDiff line numberDiff line change
@@ -1051,7 +1051,13 @@ possible, while any potentially slow operations (such as sending an email via
10511051
want to override this if you want to use blocking behaviour, or a
10521052
timeout, or a customized queue implementation.
10531053

1054+
.. attribute:: listener
10541055

1056+
When created via configuration using :func:`~logging.config.dictConfig`, this
1057+
attribute will contain a :class:`QueueListener` instance for use with this
1058+
handler. Otherwise, it will be ``None``.
1059+
1060+
.. versionadded:: 3.12
10551061

10561062
.. _queue-listener:
10571063

Doc/library/logging.rst

+13
Original file line numberDiff line numberDiff line change
@@ -1164,6 +1164,19 @@ functions.
11641164
This undocumented behaviour was considered a mistake, and was removed in
11651165
Python 3.4, but reinstated in 3.4.2 due to retain backward compatibility.
11661166

1167+
.. function:: getHandlerByName(name)
1168+
1169+
Returns a handler with the specified *name*, or ``None`` if there is no handler
1170+
with that name.
1171+
1172+
.. versionadded:: 3.12
1173+
1174+
.. function:: getHandlerNames()
1175+
1176+
Returns an immutable set of all known handler names.
1177+
1178+
.. versionadded:: 3.12
1179+
11671180
.. function:: makeLogRecord(attrdict)
11681181

11691182
Creates and returns a new :class:`LogRecord` instance whose attributes are

Lib/logging/__init__.py

+21-3
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2001-2019 by Vinay Sajip. All Rights Reserved.
1+
# Copyright 2001-2022 by Vinay Sajip. All Rights Reserved.
22
#
33
# Permission to use, copy, modify, and distribute this software and its
44
# documentation for any purpose and without fee is hereby granted,
@@ -18,7 +18,7 @@
1818
Logging package for Python. Based on PEP 282 and comments thereto in
1919
comp.lang.python.
2020
21-
Copyright (C) 2001-2019 Vinay Sajip. All Rights Reserved.
21+
Copyright (C) 2001-2022 Vinay Sajip. All Rights Reserved.
2222
2323
To use, simply 'import logging' and log away!
2424
"""
@@ -38,7 +38,8 @@
3838
'exception', 'fatal', 'getLevelName', 'getLogger', 'getLoggerClass',
3939
'info', 'log', 'makeLogRecord', 'setLoggerClass', 'shutdown',
4040
'warn', 'warning', 'getLogRecordFactory', 'setLogRecordFactory',
41-
'lastResort', 'raiseExceptions', 'getLevelNamesMapping']
41+
'lastResort', 'raiseExceptions', 'getLevelNamesMapping',
42+
'getHandlerByName', 'getHandlerNames']
4243

4344
import threading
4445

@@ -885,6 +886,23 @@ def _addHandlerRef(handler):
885886
finally:
886887
_releaseLock()
887888

889+
890+
def getHandlerByName(name):
891+
"""
892+
Get a handler with the specified *name*, or None if there isn't one with
893+
that name.
894+
"""
895+
return _handlers.get(name)
896+
897+
898+
def getHandlerNames():
899+
"""
900+
Return all known handler names as an immutable set.
901+
"""
902+
result = set(_handlers.keys())
903+
return frozenset(result)
904+
905+
888906
class Handler(Filterer):
889907
"""
890908
Handler instances dispatch logging events to specific destinations.

Lib/logging/config.py

+83-9
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright 2001-2019 by Vinay Sajip. All Rights Reserved.
1+
# Copyright 2001-2022 by Vinay Sajip. All Rights Reserved.
22
#
33
# Permission to use, copy, modify, and distribute this software and its
44
# documentation for any purpose and without fee is hereby granted,
@@ -19,15 +19,17 @@
1919
is based on PEP 282 and comments thereto in comp.lang.python, and influenced
2020
by Apache's log4j system.
2121
22-
Copyright (C) 2001-2019 Vinay Sajip. All Rights Reserved.
22+
Copyright (C) 2001-2022 Vinay Sajip. All Rights Reserved.
2323
2424
To use, simply 'import logging' and log away!
2525
"""
2626

2727
import errno
28+
import functools
2829
import io
2930
import logging
3031
import logging.handlers
32+
import queue
3133
import re
3234
import struct
3335
import threading
@@ -563,7 +565,7 @@ def configure(self):
563565
handler.name = name
564566
handlers[name] = handler
565567
except Exception as e:
566-
if 'target not configured yet' in str(e.__cause__):
568+
if ' not configured yet' in str(e.__cause__):
567569
deferred.append(name)
568570
else:
569571
raise ValueError('Unable to configure handler '
@@ -702,6 +704,21 @@ def add_filters(self, filterer, filters):
702704
except Exception as e:
703705
raise ValueError('Unable to add filter %r' % f) from e
704706

707+
def _configure_queue_handler(self, klass, **kwargs):
708+
if 'queue' in kwargs:
709+
q = kwargs['queue']
710+
else:
711+
q = queue.Queue() # unbounded
712+
rhl = kwargs.get('respect_handler_level', False)
713+
if 'listener' in kwargs:
714+
lklass = kwargs['listener']
715+
else:
716+
lklass = logging.handlers.QueueListener
717+
listener = lklass(q, *kwargs['handlers'], respect_handler_level=rhl)
718+
handler = klass(q)
719+
handler.listener = listener
720+
return handler
721+
705722
def configure_handler(self, config):
706723
"""Configure a handler from a dictionary."""
707724
config_copy = dict(config) # for restoring in case of error
@@ -721,26 +738,83 @@ def configure_handler(self, config):
721738
factory = c
722739
else:
723740
cname = config.pop('class')
724-
klass = self.resolve(cname)
725-
#Special case for handler which refers to another handler
741+
if callable(cname):
742+
klass = cname
743+
else:
744+
klass = self.resolve(cname)
726745
if issubclass(klass, logging.handlers.MemoryHandler) and\
727746
'target' in config:
747+
# Special case for handler which refers to another handler
728748
try:
729-
th = self.config['handlers'][config['target']]
749+
tn = config['target']
750+
th = self.config['handlers'][tn]
730751
if not isinstance(th, logging.Handler):
731752
config.update(config_copy) # restore for deferred cfg
732753
raise TypeError('target not configured yet')
733754
config['target'] = th
734755
except Exception as e:
735-
raise ValueError('Unable to set target handler '
736-
'%r' % config['target']) from e
756+
raise ValueError('Unable to set target handler %r' % tn) from e
757+
elif issubclass(klass, logging.handlers.QueueHandler):
758+
# Another special case for handler which refers to other handlers
759+
if 'handlers' not in config:
760+
raise ValueError('No handlers specified for a QueueHandler')
761+
if 'queue' in config:
762+
qspec = config['queue']
763+
if not isinstance(qspec, queue.Queue):
764+
if isinstance(qspec, str):
765+
q = self.resolve(qspec)
766+
if not callable(q):
767+
raise TypeError('Invalid queue specifier %r' % qspec)
768+
q = q()
769+
elif isinstance(qspec, dict):
770+
if '()' not in qspec:
771+
raise TypeError('Invalid queue specifier %r' % qspec)
772+
q = self.configure_custom(dict(qspec))
773+
else:
774+
raise TypeError('Invalid queue specifier %r' % qspec)
775+
config['queue'] = q
776+
if 'listener' in config:
777+
lspec = config['listener']
778+
if isinstance(lspec, type):
779+
if not issubclass(lspec, logging.handlers.QueueListener):
780+
raise TypeError('Invalid listener specifier %r' % lspec)
781+
else:
782+
if isinstance(lspec, str):
783+
listener = self.resolve(lspec)
784+
if isinstance(listener, type) and\
785+
not issubclass(listener, logging.handlers.QueueListener):
786+
raise TypeError('Invalid listener specifier %r' % lspec)
787+
elif isinstance(lspec, dict):
788+
if '()' not in lspec:
789+
raise TypeError('Invalid listener specifier %r' % lspec)
790+
listener = self.configure_custom(dict(lspec))
791+
else:
792+
raise TypeError('Invalid listener specifier %r' % lspec)
793+
if not callable(listener):
794+
raise TypeError('Invalid listener specifier %r' % lspec)
795+
config['listener'] = listener
796+
hlist = []
797+
try:
798+
for hn in config['handlers']:
799+
h = self.config['handlers'][hn]
800+
if not isinstance(h, logging.Handler):
801+
config.update(config_copy) # restore for deferred cfg
802+
raise TypeError('Required handler %r '
803+
'is not configured yet' % hn)
804+
hlist.append(h)
805+
except Exception as e:
806+
raise ValueError('Unable to set required handler %r' % hn) from e
807+
config['handlers'] = hlist
737808
elif issubclass(klass, logging.handlers.SMTPHandler) and\
738809
'mailhost' in config:
739810
config['mailhost'] = self.as_tuple(config['mailhost'])
740811
elif issubclass(klass, logging.handlers.SysLogHandler) and\
741812
'address' in config:
742813
config['address'] = self.as_tuple(config['address'])
743-
factory = klass
814+
if issubclass(klass, logging.handlers.QueueHandler):
815+
factory = functools.partial(self._configure_queue_handler, klass)
816+
else:
817+
factory = klass
744818
props = config.pop('.', None)
745819
kwargs = {k: config[k] for k in config if valid_ident(k)}
746820
try:

Lib/logging/handlers.py

+1
Original file line numberDiff line numberDiff line change
@@ -1424,6 +1424,7 @@ def __init__(self, queue):
14241424
"""
14251425
logging.Handler.__init__(self)
14261426
self.queue = queue
1427+
self.listener = None # will be set to listener if configured via dictConfig()
14271428

14281429
def enqueue(self, record):
14291430
"""

0 commit comments

Comments
 (0)