Python Logging Best Practices

What and why

Logging is how you debug code you cannot put a breakpoint into. The moment your service runs on another machine, print is no longer enough: you need timestamps, levels, correlation IDs, and a way to ship records to a backend. Python’s logging module gives you all of that, but its API is famously easy to misuse.

The goal of good logging is twofold: leave a trail an on-call engineer can follow, and stay cheap enough that you can keep it on in production.

Mental model

Logging in Python has four moving parts: loggers, handlers, formatters, and filters. A logger is the entry point you call. It passes records to one or more handlers (stderr, file, HTTP). Each handler formats records and decides what to do with them. Loggers are organized in a tree by dotted name (myapp.api.users is a child of myapp.api).

code: log = logging.getLogger("myapp.api.users")
    log.info("created user", extra={"uid": 42})

 logger 'myapp.api.users'
            |
            v   (if level passes)
 handlers attached here
            |
 not handled? propagate up:
            v
 logger 'myapp.api'  -> handlers
            |
            v
 logger 'myapp'      -> handlers
            |
            v
 logger '' (root)    -> handlers (stderr, file, JSON sink)

Hands-on example

Configure once at startup, then call getLogger(__name__) everywhere else.

# logging_setup.py
import logging
import logging.config

def setup():
    logging.config.dictConfig({
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {
            "json": {"()": "pythonjsonlogger.jsonlogger.JsonFormatter",
                     "fmt": "%(asctime)s %(name)s %(levelname)s %(message)s"},
        },
        "handlers": {
            "stderr": {"class": "logging.StreamHandler", "formatter": "json"},
        },
        "root": {"level": "INFO", "handlers": ["stderr"]},
        "loggers": {
            "myapp": {"level": "DEBUG"},
            "urllib3": {"level": "WARNING"},
        },
    })

# anywhere else
import logging
log = logging.getLogger(__name__)

def create_user(email):
    log.info("creating user", extra={"email": email})

getLogger(__name__) ensures the logger name matches the module path, so the tree mirrors your package layout. That makes it trivial to silence noisy libraries: set urllib3 to WARNING and you stop seeing every retry.

For request-scoped fields like trace IDs, use contextvars plus a filter:

import contextvars, logging
request_id: contextvars.ContextVar[str] = contextvars.ContextVar("request_id", default="-")

class ContextFilter(logging.Filter):
    def filter(self, record):
        record.request_id = request_id.get()
        return True

logging.getLogger().addFilter(ContextFilter())

Now every log record carries the current request ID without code at each call site setting it.

Common pitfalls

Calling logging.basicConfig inside library code clobbers the application’s configuration. Libraries should attach a NullHandler and leave config to the app.

# inside a library
logging.getLogger(__name__).addHandler(logging.NullHandler())

Using f-strings or % formatting at the call site forces the string to be built even when the log is filtered out. Use the lazy form so the message is only assembled if the record passes:

log.debug("processing %s items in %s", count, queue)   # lazy
log.debug(f"processing {count} items in {queue}")      # eager, wasteful

Logging exceptions with log.error(exc) loses the traceback. Use log.exception(...) inside an except block or pass exc_info=True.

Excessive INFO logs in hot paths can dominate CPU and storage. Sample (if random.random() < 0.01:) or downgrade to DEBUG. Counter metrics are cheaper than logs for high-cardinality events.

Avoid mutating the same logger multiple times. If your tests reconfigure logging without resetting handlers, you end up with duplicate output.

Production tips

Log JSON, not free text. Backends like Loki, Elastic, and Datadog parse structured records. A line like {"ts":..., "level":"INFO", "msg":"created", "uid":42, "trace_id":"abc"} is greppable and aggregatable.

Use the standard levels with discipline. DEBUG is for development noise. INFO is for the lifecycle of significant operations. WARNING is for recoverable abnormalities. ERROR is for failed operations a human should look at. CRITICAL means the process is going down.

Never log secrets. Add a redacting filter that walks the record and masks fields like password, token, authorization. Test it.

Make logs cheap to find. Always include the request ID, the user ID (when applicable), and the operation name. Three weeks from now, someone will grep for one of those.

Configure logging before importing third-party libraries. Some configure their own loggers at import time; getting in first lets you control them.

Wrap-up

Use logging.getLogger(__name__) in every module, configure once at startup with dictConfig, ship JSON, attach a contextvar-backed filter for trace IDs, and respect level discipline. Lazy-format messages, never silence the root logger globally, and make sure secrets cannot escape. With those habits, logs become the first tool you reach for in an outage instead of the last.