python Log file - colorful, daily rotating

 You need to keep track of the events or activities that occur during the execution of the program you are developing now. You create a log file for this purpose. Log files are also necessary for debugging during development. Log files that serve this purpose must contain the following entry:

• Logging time : The time when the log is written. It can be expressed in mili second units. However, if the log frequency is low, the log in seconds may be OK.
• Log Level : It represents the importance of the log and is generally divided into debug, info, notice, .. error, and so on.
• Log contents : What you want to record. It is good to be compact and to express the contents clearly.

The most used package for logging in Python is logging. I think even people who use the logging library a lot have not used many handlers other than StreamHandler and FileHandler.

However, FileHandler is insufficient for managing log files by splitting them into time units or days. A module that can manage log files more efficiently is TimedRotatingFileHandler.

As the name suggests, TimedRotatingFileHandler is a handler that manages log files that are loaded into the disk at specific time intervals by rotating them.

If your process is registered in crontab and runs only for a short time, using FileHandler alone is sufficient. However, if it runs continuously as a daemon process, it is recommended to use TimedRotatingFileHandler. to manage log files more efficiently.

How to use TimedRotatingFileHandler

General logging can be used to output to the screen or to a file, depending on the purpose. Of course, you can use both. In most cases, you will use both.

The basic usage of TimedRotatingFileHandler is as follows:

The following example uses the logging module to enable both file output and screen output simultaneously.

import sys
import logging
import logging.handlers

logger_file = './mylogging.log'

logger = logging.getLogger('mylogger')
fomatter = logging.Formatter('[%(levelname)s] %(asctime)s > %(message)s')
fileHandler = logging.handlers.TimedRotatingFileHandler(logger_file, when='midnight', backupCount=10)

streamHandler = logging.StreamHandler()
fileHandler.setFormatter(fomatter)
streamHandler.setFormatter(fomatter)
logger.addHandler(fileHandler)
logger.addHandler(streamHandler)
logger.setLevel(logging.DEBUG)   #DEBUG, INFO, WARNING, ERROR, CRITICAL

msg = 'Hello World'
logger.debug(msg)
logger.info(msg)
logger.warning(msg)
logger.error(msg)
logger.critical(msg)

When you run this program, the following is displayed on the screen and also written to the log file.

D:\study\python\logtest>python mylog1.py
[DEBUG] 2024-07-26 15:12:22,147 > Hello World
[INFO] 2024-07-26 15:12:22,147 > Hello World
[WARNING] 2024-07-26 15:12:22,147 > Hello World
[ERROR] 2024-07-26 15:12:22,148 > Hello World
[CRITICAL] 2024-07-26 15:12:22,148 > Hello World

(base) D:\study\python\logtest>dir
 Volume in drive D is D 드라이브
 Volume Serial Number is 287A-6C32

 Directory of D:\study\python\logtest

2024-07-26  오후 03:12    <DIR>          .
2024-07-26  오후 03:11    <DIR>          ..
2024-07-26  오후 03:11               690 mylog1.py
2024-07-26  오후 03:12               239 mylogging.log
               2 File(s)            929 bytes
               2 Dir(s)  1,399,240,646,656 bytes free

TimedRotatingFileHandler

The constructor for TimedRotatingFileHandler is as follows:

class logging.handlers.TimedRotatingFileHandler(filename, when='h', interval=1, backupCount=0, encoding=None, delay=False, utc=False, atTime=None, errors=None)

You can use the when to specify the type of interval. The list of possible values is below. Note that they are not case sensitive.

Value	Type of interval	If/how atTime is used
'S'	Seconds	Ignored
'M'	Minutes	Ignored
'H'	Hours	Ignored
'D'	Days	Ignored
'W0'-'W6'	Weekday (0=Monday)	Used to compute initial rollover time
'midnight'	Roll over at midnight, if atTime not specified, else at time atTime	Used to compute initial rollover time

When using weekday-based rotation, specify ‘W0’ for Monday, ‘W1’ for Tuesday, and so on up to ‘W6’ for Sunday. In this case, the value passed for interval isn’t used.

The system will save old log files by appending extensions to the filename. The extensions are date-and-time based, using the strftime format %Y-%m-%d_%H-%M-%S or a leading portion thereof, depending on the rollover interval.

When computing the next rollover time for the first time (when the handler is created), the last modification time of an existing log file, or else the current time, is used to compute when the next rotation will occur.

If the utc argument is true, times in UTC will be used; otherwise local time is used. If backupCount is nonzero, at most backupCount files will be kept, and if more would be created when rollover occurs, the oldest one is deleted. The deletion logic uses the interval to determine which files to delete, so changing the interval may leave old files lying around.

If delay is true, then file opening is deferred until the first call to emit().

If atTime is not None, it must be a datetime.time instance which specifies the time of day when rollover occurs, for the cases where rollover is set to happen “at midnight” or “on a particular weekday”. Note that in these cases, the atTime value is effectively used to compute the initial rollover, and subsequent rollovers would be calculated via the normal interval calculation.

In my example, I set the when parameter to midnight. So when the date changes, the logging files up to this point will be automatically backed up. And a new backup file will be created to continue logging.

I set the backupCount to 10. So the backup files will be kept up to 10. After 10, the oldest backup file will be deleted and a new backup will be created, so the number of backup files can always be kept at 10. If this value is 0, the number of backup files can be kept unlimited.

Colorful StreamHandler

StreamHandler provides the ability to output logging results to stdout. The logging module provides screen output using StreamHandler, but does not support colors. Therefore, if you want to specify colors according to log levels, you will need to do some work.

The following example applies screen output color according to the log level.

import sys
import logging
import logging.handlers

logger_file = './mylogging.log'

logger = logging.getLogger('mylogger')
fomatter = logging.Formatter('[%(levelname)s] %(asctime)s > %(message)s')
fileHandler = logging.handlers.TimedRotatingFileHandler(logger_file, when='midnight', backupCount=10)

streamHandler = logging.StreamHandler()
fileHandler.setFormatter(fomatter)
streamHandler.setFormatter(fomatter)
logger.addHandler(fileHandler)
logger.addHandler(streamHandler)
logger.setLevel(logging.DEBUG)   #DEBUG, INFO, WARNING, ERROR, CRITICAL


# Reset
Color_Off='\033[0m'       # Text Reset

# Regular Colors
Black='\033[0;30m'        # Black
Red='\033[0;31m'          # Red
Green='\033[0;32m'        # Green
Yellow='\033[0;33m'       # Yellow
Blue='\033[0;34m'         # Blue
Purple='\033[0;35m'       # Purple
Cyan='\033[0;36m'         # Cyan
White='\033[0;37m'        # White

# Bold
BBlack='\033[1;30m'       # Black
BRed='\033[1;31m'         # Red
BGreen='\033[1;32m'       # Green
BYellow='\033[1;33m'      # Yellow
BBlue='\033[1;34m'        # Blue
BPurple='\033[1;35m'      # Purple
BCyan='\033[1;36m'        # Cyan
BWhite='\033[1;37m'       # White

# Underline
UBlack='\033[4;30m'       # Black
URed='\033[4;31m'         # Red
UGreen='\033[4;32m'       # Green
UYellow='\033[4;33m'      # Yellow
UBlue='\033[4;34m'        # Blue
UPurple='\033[4;35m'      # Purple
UCyan='\033[4;36m'        # Cyan
UWhite='\033[4;37m'       # White

# Background
On_Black='\033[40m'       # Black
On_Red='\033[41m'         # Red
On_Green='\033[42m'       # Green
On_Yellow='\033[43m'      # Yellow
On_Blue='\033[44m'        # Blue
On_Purple='\033[45m'      # Purple
On_Cyan='\033[46m'        # Cyan
On_White='\033[47m'       # White

# High Intensity
IBlack='\033[0;90m'       # Black
IRed='\033[0;91m'         # Red
IGreen='\033[0;92m'       # Green
IYellow='\033[0;93m'      # Yellow
IBlue='\033[0;94m'        # Blue
IPurple='\033[0;95m'      # Purple
ICyan='\033[0;96m'        # Cyan
IWhite='\033[0;97m'       # White

# Bold High Intensity
BIBlack='\033[1;90m'      # Black
BIRed='\033[1;91m'        # Red
BIGreen='\033[1;92m'      # Green
BIYellow='\033[1;93m'     # Yellow
BIBlue='\033[1;94m'       # Blue
BIPurple='\033[1;95m'     # Purple
BICyan='\033[1;96m'       # Cyan
BIWhite='\033[1;97m'      # White

# High Intensity backgrounds
On_IBlack='\033[0;100m'   # Black
On_IRed='\033[0;101m'     # Red
On_IGreen='\033[0;102m'   # Green
On_IYellow='\033[0;103m'  # Yellow
On_IBlue='\033[0;104m'    # Blue
On_IPurple='\033[0;105m'  # Purple
On_ICyan='\033[0;106m'    # Cyan
On_IWhite='\033[0;107m'   # White

nc = 0

def init_log(daemon):
    global nc
    nc = daemon

def display_header(h):
    if nc == 0:
        sys.stdout.write(h)
        sys.stdout.flush() 

def display_tail():
    if nc == 0:
        sys.stdout.write(Color_Off)
        sys.stdout.flush() 

def my_log(level, msg):
    if level == 'debug':
        display_header(White)
        logger.debug(msg)
        display_tail()
    elif level == 'info':
        display_header(Yellow)
        logger.info(msg)
        display_tail()
    elif level == 'error':      
        display_header(Purple)
        logger.info(msg)
        display_tail()

    elif level == 'warning':
        display_header(Green)
        logger.info(msg)
        display_tail()
    elif level == 'critical':
        display_header(Red)
        logger.info(msg)
        display_tail()
    else:
        display_header(Blue)
        logger.info(msg)
        display_tail()


msg = 'Hello World'
my_log('debug', msg)
my_log('info', msg)
my_log('warning', msg)
my_log('error', msg)
my_log('critical', msg)

When you run this program, the following is displayed on the screen and also written to the log file.

Why I used the nc variable

In the above example, I used the nc(no console) variable to determine whether to use the sys.stdout function or not. The reason I used this method is because the process is in daemon mode, which does not use stdout. It is not recommended to use stdout for daemon processes, because stdout is redirected to the null device.

So if you want to use the above logging in a daemon process that does not need screen output, you can call init_log(1) beforehand to disable stdout.

Wrapping up

logging.handlers — The Logging handlers page provides usage of various Handlers, including TimedRotatingFileHandler and StreamHandler.

• StreamHandler
• FileHandler
• NullHandler
• WatchedFileHandler
• BaseRotatingHandler
• RotatingFileHandler
• TimedRotatingFileHandler
• SocketHandler
• DatagramHandler
• SysLogHandler
• NTEventLogHandler
• SMTPHandler
• MemoryHandler
• HTTPHandler
• QueueHandler
• QueueListener

In addition to local file logging, it can also send logging data to a remote server via udp/tcp communication, and it also has a function to send events via SMTP. If you want to make full use of the logging function, it is a good idea to read it carefully.