Introducing Annotated Logger: A Python package to aid in adding metadata to logs

Annotated logger: structured metadata for Python logs

GitHub’s Vulnerability Management team has released Annotated Logger, a Python package that simplifies attaching metadata to log messages. The team built the tool to make filtering logs in Splunk easier: several of their Python projects grew complex over time, and while their JSON-formatted logs already supported extra fields, consistently adding fields like the deployed Git branch or a CVE identifier required repeated manual work or hand-maintained dictionaries of shared context.

How it works

At its core, Annotated Logger is a decorator. Apply @annotate_logs() to a function and the package handles logging automatically when execution completes. More usefully, the decorator can inject a pre-configured logger object into the decorated function, with any custom fields already attached. The feature set expanded as the package moved from an internal helper into a standalone project used across multiple repositories.

@annotate_logs()
def foo():
    return True

>>> foo()
{"created": 1733176439.5067494, "levelname": "DEBUG", "name": "annotated_logger.8fcd85f5-d47f-4925-8d3f-935d45ceeefc", "message": "start", "action": "__main__:foo", "annotated": true}
{"created": 1733176439.506998, "levelname": "INFO", "name": "annotated_logger.8fcd85f5-d47f-4925-8d3f-935d45ceeefc", "message": "success", "action": "__main__:foo", "success": true, "run_time": "0.0", "annotated": true}
True

Configuration and usage

Start by installing the package with pip install annotated-logger. A fuller example follows, showing several capabilities together.

import os
from annotated_logger import AnnotatedLogger
al = AnnotatedLogger(
    name="annotated_logger.example",
    annotations={"branch": os.environ.get("BRANCH", "unknown-branch")}
)
annotate_logs = al.annotate_logs

@annotate_logs()
def split_username(annotated_logger, username):
    annotated_logger.annotate(username=username)
    annotated_logger.info("This is a very important message!", extra={"important": True})
    return list(username)

>>> split_username("crimsonknave")
{"created": 1733349907.7293086, "levelname": "DEBUG", "name": "annotated_logger.example.c499f318-e54b-4f54-9030-a83607fa8519", "message": "start", "action": "__main__:split_username", "branch": "unknown-branch", "annotated": true}
{"created": 1733349907.7296104, "levelname": "INFO", "name": "annotated_logger.example.c499f318-e54b-4f54-9030-a83607fa8519", "message": "This is a very important message!", "important": true, "action": "__main__:split_username", "branch": "unknown-branch", "username": "crimsonknave", "annotated": true}
{"created": 1733349907.729843, "levelname": "INFO", "name": "annotated_logger.example.c499f318-e54b-4f54-9030-a83607fa8519", "message": "success", "action": "__main__:split_username", "branch": "unknown-branch", "username": "crimsonknave", "success": true, "run_time": "0.0", "count": 12, "annotated": true}
['c', 'r', 'i', 'm', 's', 'o', 'n', 'k', 'n', 'a', 'v', 'e']
>>>
>>> split_username(1)
{"created": 1733349913.719831, "levelname": "DEBUG", "name": "annotated_logger.example.1c354f32-dc76-4a6a-8082-751106213cbd", "message": "start", "action": "__main__:split_username", "branch": "unknown-branch", "annotated": true}
{"created": 1733349913.719936, "levelname": "INFO", "name": "annotated_logger.example.1c354f32-dc76-4a6a-8082-751106213cbd", "message": "This is a very important message!", "important": true, "action": "__main__:split_username", "branch": "unknown-branch", "username": 1, "annotated": true}
{"created": 1733349913.7200255, "levelname": "ERROR", "name": "annotated_logger.example.1c354f32-dc76-4a6a-8082-751106213cbd", "message": "Uncaught Exception in logged function", "exc_info": "Traceback (most recent call last):\n  File \"/home/crimsonknave/code/annotated-logger/annotated_logger/__init__.py\", line 758, in wrap_function\n  result = wrapped(*new_args, **new_kwargs)  # pyright: ignore[reportCallIssue]\n  File \"<stdin>\", line 5, in split_username\nTypeError: 'int' object is not iterable", "action": "__main__:split_username", "branch": "unknown-branch", "username": 1, "success": false, "exception_title": "'int' object is not iterable", "annotated": true}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<makefun-gen-0>", line 2, in split_username
  File "/home/crimsonknave/code/annotated-logger/annotated_logger/__init__.py", line 758, in wrap_function
    result = wrapped(*new_args, **new_kwargs)  # pyright: ignore[reportCallIssue]
  File "<stdin>", line 5, in split_username
TypeError: 'int' object is not iterable

The example demonstrates the key pieces:

• Logger configuration: you must instantiate an AnnotatedLogger class, which holds all logger settings. Here the logger name is set (note that logging config must be updated if the name does not start with annotated_logger, or no handler will process the messages), and a branch annotation is added so the value appears in every log record.
• Decorator alias: the example assigns the decorator to a shorter alias for readability, though calling @al.annotate_logs() directly works fine.
• Injected logger: the decorated method accepts an annotated_logger parameter that behaves like a standard logger but carries the extra features. The decorator inserts this argument before invoking the method and adjusts the method signature so callers never pass it—see how the method is called with just name. Decorator options exist so type checkers can correctly parse the modified signature.
• Per-method annotations: calling annotate on the injected logger adds whatever keyword arguments are supplied to the extra field for all subsequent log messages using that logger. Annotations persist across messages within the logger instance and can be overridden by re-annotating with the same key.
• Standard log calls: individual messages can still include their own fields via the same mechanism used with a regular logger.
• Automatic exception logging: in the second method call, the int passed to name and the list argument trigger an exception, which the decorator logs automatically before re-raising it. This makes it easier to determine whether a method actually finished, assuming the process was not killed.

Log message fields

For each decorated method that completes without raising, the package emits a success message that automatically includes the success, run_time, and count fields, alongside any configured annotations.

How Annotated Logger works internally

The package interacts with Python's logging module through two primary classes. AnnotatedAdapter is a subclass of logging.LoggerAdapter, and all annotated_logger arguments are instances of it. The actual annotation injection happens in AnnotatedFilter, a subclass of logging.Filter. Each AnnotatedAdapter instance owns its own AnnotatedFilter instance; the adapter's annotate method passes annotations to the filter, where they are stored until a message is logged. At that point, the filter calculates which annotations apply and updates the existing LogRecord object.

Since every method invocation creates its own AnnotatedAdapter object, it also gets its own AnnotatedFilter, preventing annotations from leaking between method calls.

Type hint support

The library is fully type hinted, including for decorated methods, though the decorator needs some extra guidance. Four optional arguments matter for typing: _typing_self, _typing_requested, _typing_class, and provided. The three arguments prefixed with _typing don't affect runtime behavior — they only override the method signature for type checkers. Setting provided to True tells the decorator that the caller will supply the annotated_logger, so the signature remains unchanged. _typing_self defaults to True; provided, _typing_class, and _typing_requested default to False.

class Example:
    @annotate_logs(_typing_requested=True)
    def foo(self, annotated_logger):
        ...

e = Example()
e.foo()

Plugin system

Plugins hook into two events: when the decorator catches an exception and when a log message is emitted. A custom plugin needs to define filter and uncaught_exception methods, or it can inherit from annotated_logger.plugins.BasePlugin, which provides noop implementations for both.

When a message is logged, the filter method of each plugin runs, in config order, against the logging.LogRecord object. Plugins can modify the record and, like any logging filter, can suppress a message by returning False.

The uncaught_exception method fires when the decorator catches an unhandled exception in the decorated method. It receives the exception and the logger (the method's annotated_logger instance). This lets a plugin annotate the forthcoming exception log message.

The example plugin below demonstrates both behaviors. It inherits from BasePlugin for convenience and stores configuration in __init__. Both methods achieve the same outcome — setting flagged=True when a word matches — but differ subtly. The filter method adds an annotation directly to the record it's processing. The uncaught_exception method has no specific record to annotate, so it sets the annotation on the logger itself. This only matters if a later plugin emits its own log message after FlagWordPlugin has run; that message would also carry flagged=True.

from annotated_logger.plugins import BasePlugin

class FlagWordPlugin(BasePlugin):
    """Plugin that flags any log message/exception that contains a word in a list."""
    def __init__(self, *wordlist):
        """Save the wordlist."""
        self.wordlist = wordlist

    def filter(self, record):
    """Add annotation if the message contains words in the wordlist."""
    for word in self.wordlist:
        if word in record.msg:
            record.flagged = True

    def uncaught_exception(self, exception, logger):
    """Add annotation if exception title contains words in the wordlist."""
    for word in self.wordlist:
        if word in str(exception)
            logger.annotate(flagged=True)

AnnotatedLogger(plugins=[FlagWordPlugin("danger", "Will Robinson")])

Plugins exist in an ordered list with BasePlugin always first. The order matters because filters modify records in sequence. One filter can break another by removing or renaming a field, or a filter may depend on a field that an earlier filter adds. Plugin order also matters on exceptions. For example, both BasePlugin and RequestsPlugin set exception_title; since BasePlugin runs first, RequestsPlugin's value wins. In other cases, plugin ordering determines whether an annotation is present when another plugin emits or forwards a message. A filter that returns False halts processing, so later plugins never see the message.

The bundled plugins handle common needs:

• GitHubActionsPlugin — emits log messages as GitHub Actions workflow commands (notice::).
• NameAdjusterPlugin — adds a pre/postfix to fields to avoid collisions in downstream log processing.
• RemoverPlugin — removes fields, such as sensitive ones like password or async-only fields like taskName.
• NestedRemoverPlugin — removes a field at any depth within a dictionary.
• RenamerPlugin — renames fields (for instance, levelname to level).
• RequestsPlugin — adds a title and status code when an exception inherits from requests.exceptions.HTTPError.
• RuntimeAnnotationsPlugin — applies dynamic annotations.

Configuration with dictConfig

To integrate with projects using other logging packages, pass a dictConfig-compliant dictionary as the config argument when initializing the Annotated Logger. Alternatively, pass config=False and reference annotated_logger.DEFAULT_LOGGING_CONFIG for the default configuration, modifying it as needed.

One special case applies: if your config contains a filter named annotated_filter, the Annotated Logger replaces it with a filter created by the new instance. This ensures annotations and settings apply to all messages using that filter. Creating your own filter from AnnotatedFilter works, but it won't inherit the rest of your configuration.

dictConfig dict merging behaves inconsistently — some parts overwrite correctly, others lose references. Build a complete logging config and call it once. When you supply config, the Annotated Logger adds its adjustments and then calls logging.config.dictConfig. The logging_config.py example walks through more detailed setups.

Testing with the pytest mock

The package includes a pytest mock for asserting logged messages. Testing logs can be overdone, but sometimes it's the most direct way to verify a loop iteration or an alert-triggering message's format. Use the annotated_logger_mock fixture to intercept, record, and forward all log messages.

def test_logs(annotated_logger_mock):
    with pytest.raises(KeyError):
        complicated_method()
    annotated_logger_mock.assert_logged(
        "ERROR",  # Log level
        "That's not the right key",  # Log message
        present={"success": False, "key": "bad-key"},  # annotations and their values that are required
        absent=["fake-annotations"],  # annotations that are forbidden
        count=1  # Number of times log messages should match
    )

The assert_logged method builds on pychoir for flexible value matching. All parameters are optional. The table below lists defaults and valid values.

The present key is what gives the mock much of its power — it lets you assert only what's relevant. Tests won't break because a method's run_time changed from 0.0 to 0.1, or because hostnames differ between machines, while those fields stay useful in production logs. This mock covers everything the caplog fixture does and more.

Beyond basic annotations

Decorating entire classes

The @annotate_logs decorator can also be applied at the class level. A decorated class receives an annotated_logger attribute after initialization (the decorator cannot hook into __init__). Each decorated method on that class then gets an annotated_logger derived from the class logger. When a method calls annotate with persist=True, those annotations are stored on the class-level Annotated Logger and applied to subsequent calls from any decorated method on that instance. The class logger also carries a class annotation indicating the source class.

Logging iteration progress

For enumerable objects, the annotated_logger.iterator method logs the start, each step, and the completion of the iteration. This is helpful when paginating through an API — it makes clear whether a long pause means the requests are hanging or there are simply many pages to fetch.

By default, iterator logs the value at each step. You can suppress those values with value=False, and you can override the default info log level.

Passing loggers between methods

Because every decorated method gets its own annotated_logger, annotations from a caller are normally lost when it invokes another method. Setting provided=True on the decorator changes that contract. The decorated method no longer gets an automatically created annotated_logger; it requires the first argument to be an existing one, which becomes the basis for its own logger. The method is also tagged with a subaction annotation equal to its name, while preserving the caller's action annotation.

Annotations still do not flow back from a provided=True method to the caller, unless the calling class is itself decorated and the called method uses persist=True, in which case the annotations land on the shared instance-level logger as usual.

This pattern suits private methods extracted during a refactor, as well as common utilities invoked from many call sites.

Handling oversized messages

Log parsing tools frequently choke on very long messages — a 500 error page's HTML, for instance, can exceed what Splunk can parse, causing the whole entry to be dropped along with its annotations. Setting max_length in the Annotated Logger configuration splits such messages into multiple log entries. Each segment is annotated with split=True, split_complete=False, message_parts=#, and message_part=#; the final segment carries split_complete=True.

Only the message text is subject to splitting. Annotations never trigger it, though a plugin could truncate overly long annotation values on its own.

Hooks around method calls

The decorator's pre_call and post_call parameters accept function references that execute right before and after the decorated method, with the same arguments. These hooks can add annotations or emit log lines of their own, assuming the decorated function requests an annotated_logger. Common uses include standardizing field-level annotations in a pre_call, or logging when a post_call finds a model in an unsaved state.

Dynamic values at emit time

Static annotations cover most needs, but some values only exist at the moment a log message is written. The RuntimeAnnotationsPlugin handles these cases. Its configuration maps annotation names to callables; each callable receives the log record when the plugin's filter runs, just before emission, and its return value becomes the annotation's value.

A frequent use is tagging requests with a correlation ID. For Django projects, django-guid provides one route to such an ID.

Practical guidance

• When the decorator is used across multiple files, centralize configuration in a module such as log.py. Every file can then import with from project.log import annotate_logs, guaranteeing a consistent setup.
• Namespacing loggers matters when a package and its consumer application both use Annotated Logger. If you configure through dictConfig, a single config should cover all Annotated Loggers involved.
• Beyond tracking the current request's correlation ID, also annotate the caller's correlation ID. That lets you trace a call from service A into the specific log lines it generates in service B.
• Plugins are flexible tools. A few possibilities:
  • Forward every exception log message to a service like Sentry.
  • Filter out messages from packages such as Django (after routing those logs through your Annotated Logger filter).
  • Attach supplemental annotations for particular exception types, as RequestsPlugin does.
  • Apply runtime annotations to selected messages rather than globally through RuntimeAnnotationsPlugin.

Feedback and contributions

Questions, comments, and feature requests are welcome via the issue tracker. Pull requests are also appreciated.