Logging is a very useful tool for developers, as it can help you monitor, debug, and optimize your Python applications. Logging can also provide valuable information about the performance, behavior, and state of your applications, as well as any errors or issues that may occur.
However, logging is not as simple as using print statements to output messages to the console. Print statements have many limitations and drawbacks, such as:
- They only output to the standard output stream, which may not be suitable for persistent or structured logging.
- They are not configurable at runtime, which means you have to modify and redeploy your code every time you want to change the logging level or format.
- They do not provide any context or metadata, such as the timestamp, source file, line number, or log level of the messages.
- They are not thread-safe, which means they may cause concurrency issues or race conditions in multi-threaded or multi-process applications.
Therefore, it is recommended to use the logging module from the Python standard library instead of print statements for logging in your Python applications. The logging module is a powerful and flexible module that provides a standard way of creating and managing loggers, handlers, formatters, and filters. It also supports various features and functionalities, such as:
- Logging to different destinations, such as files, sockets, databases, or email servers.
- Logging at different levels of severity, such as DEBUG, INFO, WARNING, ERROR, or CRITICAL.
- Logging with different formats and styles, such as plain text, JSON, XML, or HTML.
- Logging with different scopes and namespaces, such as root logger, module logger, class logger, or function logger.
- Logging with different strategies and policies, such as rotating files, compressing files, or deleting old files.
In this blog post, I will show you how to enable logs in your Python application using the logging module. I will also show you some examples and best practices of using the logging module in different scenarios and situations.
Basic Logging
To enable basic logging in your Python application, you need to import the logging module and create a logger object. A logger object is an instance of the Logger class that provides methods for logging messages at different levels of severity. You can create a logger object by calling the getLogger() function and passing a name argument. The name argument can be any string that identifies the source of the log messages. For example:
import logging
# Create a logger object
logger = logging.getLogger("my_app")
By default, the logger object has a level of WARNING, which means it will only log messages that have a level of WARNING or higher. You can change the level of the logger object by calling the setLevel() method and passing a level argument. The level argument can be one of the predefined constants in the logging module that represent the levels of severity. For example:
import logging
# Create a logger object
logger = logging.getLogger("my_app")
# Set the level of the logger object to INFO
logger.setLevel(logging.INFO)
To log a message using the logger object, you can call one of its methods that correspond to the levels of severity. For example:
import logging
# Create a logger object
logger = logging.getLogger("my_app")
# Set the level of the logger object to INFO
logger.setLevel(logging.INFO)
# Log some messages at different levels of severity
logger.debug("This is a debug message")
logger.info("This is an info message")
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.critical("This is a critical message")
By default, the logger object will output the log messages to the standard error stream (stderr), which is usually the console. The output will look something like this:
INFO:my_app:This is an info message
WARNING:my_app:This is a warning message
ERROR:my_app:This is an error message
CRITICAL:my_app:This is a critical message
As you can see, each log message consists of three parts: the level name, the logger name, and the message itself. This is the default format of the log messages that is provided by the logging module. You can customize the format of the log messages by using handlers and formatters.
Handlers and Formatters
A handler object is an instance of a subclass of Handler class that defines how and where to output the log messages from a logger object. A formatter object is an instance of Formatter class that defines how to format and style the log messages before outputting them. You can create handler objects and formatter objects by instantiating their respective classes and passing some arguments. For example:
import logging
# Create a handler object for writing log messages to a file
file_handler = logging.FileHandler("app.log")
# Create a handler object for writing log messages to the console
console_handler = logging.StreamHandler()
# Create a formatter object for formatting log messages
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
In this example, I have created two handler objects: one for writing log messages to a file named app.log, and one for writing log messages to the standard output stream (stdout), which is usually the console. I have also created a formatter object that formats the log messages with four fields: the timestamp, the logger name, the level name, and the message itself.
To use the handler objects and the formatter object in your Python application, you need to add them to the logger object by calling the addHandler() method and passing a handler object as an argument. You also need to set the formatter object for each handler object by calling the setFormatter() method and passing a formatter object as an argument. For example:
import logging
# Create a logger object
logger = logging.getLogger("my_app")
# Set the level of the logger object to INFO
logger.setLevel(logging.INFO)
# Create a handler object for writing log messages to a file
file_handler = logging.FileHandler("app.log")
# Create a handler object for writing log messages to the console
console_handler = logging.StreamHandler()
# Create a formatter object for formatting log messages
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
# Set the formatter object for each handler object
file_handler.setFormatter(formatter)
console_handler.setFormatter(formatter)
# Add the handler objects to the logger object
logger.addHandler(file_handler)
logger.addHandler(console_handler)
# Log some messages at different levels of severity
logger.debug("This is a debug message")
logger.info("This is an info message")
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.critical("This is a critical message")
Now, if you run this code, you will see that the log messages are outputted to both the file and the console, and they are formatted with the specified fields. The output will look something like this:
2021-10-18 12:34:56,789 – my_app – INFO – This is an info message
2021-10-18 12:34:56,789 – my_app – WARNING – This is a warning message
2021-10-18 12:34:56,789 – my_app – ERROR – This is an error message
2021-10-18 12:34:56,789 – my_app – CRITICAL – This is a critical message
The logging module provides various classes for creating different types of handler objects and formatter objects. For example, you can use RotatingFileHandler or TimedRotatingFileHandler to create handler objects that rotate the log files based on size or time. You can also use SMTPHandler or HTTPHandler to create handler objects that send log messages via email or HTTP. You can also use JSONFormatter or HTMLFormatter to create formatter objects that format log messages in JSON or HTML format.
Complete Example:
Complete example is shown in below code, here we will use Python’s Flask framework, which is used to build web applications, though you can use logger library with core python or any of the framework.
create a requirements.txt file:
Flask
requests>=2.25.1
Create an app.py file with following code:
from flask import Flask, request, jsonify
import logging
# code downloaded from https://dataxone.com
# Create a Flask app instance
app = Flask(__name__)
# Set up a logger for the app
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
handler = logging.FileHandler("app.log")
formatter = logging.Formatter("%(asctime)s - %(name)s - %(filename)s - %(funcName)s -%(lineno)d - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info(f"This is sample Info Log")
logger.error(f"This is sample Error Log")
@app.route("/", methods=["GET"])
def hello():
logger.info(f"This is sample Info Log - in hello function")
return "Hello"
# Run the app in debug mode
if __name__ == "__main__":
app.run(debug=True)
Once you run it, file called app.log will be created, showing something like following:
References:
https://docs.python.org/3/library/logging.html#formatter-objects
https://docs.python.org/3/howto/logging.html