Logging with Python

Standards

Prefer logs to print statements
Logging objects should be named logger
Refer to boilerplate(s) below for add'l details

Basic logging setup

#%%
import logging

LEVEL = logging.INFO
FORMAT = "%(asctime)s — %(levelname)-8s — %(name)s — %(funcName)s:%(lineno)d — %(message)s"

logging.basicConfig(level=LEVEL, format=FORMAT)
logger = logging.getLogger(__name__)

# %%
# test logging outputs (DEBUG should not print because `logger` set to INFO)
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')

# %%
# set level to DEBUG to log DEBUG-level messages
logger.setLevel(logging.DEBUG)
logger.debug('debug message')
logger.info('info message')
logger.warning('warn message')
logger.error('error message')
logger.critical('critical message')

"Advanced" logging setup

Instead of using basicConfig() to configure logging, we create a Logger object and specify Formatters and Handlers

#%%
import os
import logging

LEVEL = logging.INFO
FORMAT = "%(asctime)s — %(levelname)-8s — %(name)s — %(funcName)s:%(lineno)d — %(message)s"

logger = logging.getLogger(__name__)
logger.setLevel(LEVEL)

# create console handler
console_handler = logging.StreamHandler()
console_handler.setLevel(LEVEL)
console_handler.setFormatter(logging.Formatter(FORMAT))

logger.addHandler(console_handler)

# create file handler to write logs to file
logpath = '/path/to/logname.log'
if logpath:
    # make directory if does not exist
    os.makedirs(os.path.dirname(logpath), exist_ok=True)
    file_handler = logging.handlers.RotatingFileHandler(
        filename=logpath,
        maxBytes=10485760, # 10MB
        backupCount=5,
    )
    file_handler.setLevel(LEVEL)
    file_handler.setFormatter(logging.Formatter(FORMAT))

    logger.addHandler(file_handler)

Logging setup for Libraries / Packages

Package setup

In each module (i.e., each individual python file that comprise the package), add the following after all import statements:

logger = logging.getLogger(__name__)

When the logger is called, __name__ will resolve to PackageName.module_name. This means that we can apply settings to all loggers associated with the package:

logging.getLogger('PackageName').addHandler(...)
logging.getLogger('PackageName').setLevel(...)
# etc

or, we can apply settings per specific module by specifying the entire hierarchy:

logging.getLogger('PackageName.module_name').addHandler(...)
logging.getLogger('PackageName.module_name').setLevel(...)
# etc

Script setup

In any script that imports the package where you wish to use the package loggers:

Run import statements
Configure logging (via basicConfig).
Modify package logger with .addHandler, .setFormatter, or .setLevel

The overall workflow is:

import os
from os import path, sys
import logging

# ... more imports

import PackageName

# set up logging
FORMAT = "%(asctime)s — %(levelname)-8s — %(name)s — %(funcName)s:%(lineno)d — %(message)s"
logging.basicConfig(format=FORMAT)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

# modify PackageName loggers via handlers (examples)
logging.getLogger('PackageName').setLevel(logging.DEBUG)
logging.getLogger('PackageName').addHandler(file_handler)  # see '"Advanced" setup' above

Debugging

import logging

# get list of logger objects
logging.root.manager.loggerDict

# view handlers of specific logger
logging.getLogger('loggerName').handlers

ahgraber/logging.md