Allow mix extra through ini file

[lucienfregosi]

Hi thanks for your work on that log formatter.

I want to use it through an ini file but I also want to add some extra param like
root.info('test file config', extra={'jobId': jobId, 'flowId': flowId})

I didn't find a way to specify inside the ini file this specific option mix_extra=True do you have an idea if it's possible ? or do I have to declare the logger inside the code for using extra option ?

Many thanks

[MyColorfulDays]

@lucienfregosi Becasue of logging.config.fileConfig not support formatter args or kwargs, you can choose one function:

import logging
import os
from logging.config import fileConfig


def set_single_logger_mix_extra():
    fileConfig(os.path.join(os.path.dirname(__file__), 'logger_config.ini'))
    logger_name = ''
    # NOTICE: `logger_name` is `''` or `None` will return root logger, `'root'` will return a sub logger named root
    logger = logging.getLogger(logger_name)
    for handle in logger.handlers or []:
        if isinstance(handle.formatter, JsonFormatter):
            handle.formatter.mix_extra = True
    logger.info('test file config', extra={'jobId': 123, 'flowId': 456})


def set_all_logger_mix_extra():
    fileConfig(os.path.join(os.path.dirname(__file__), 'logger_config.ini'))
    all_logger = [logging.root] + list(logging.root.manager.loggerDict.values())
    for logger in all_logger:
        for handle in logger.handlers or []:
            if isinstance(handle.formatter, JsonFormatter):
                handle.formatter.mix_extra = True
    logger_name = ''
    logger = logging.getLogger(logger_name)
    logger.info('test file config', extra={'jobId': 123, 'flowId': 456})

https://github.com/MyColorfulDays/jsonformatter/blob/master/test/logger_config.ini Here is the logger_config.ini file.

[FlorianHall]

You can actually instantiate the JSON logger like this and just pass your kwargs:

  json_formatter:
      (): jsonformatter.JsonFormatter
      mix_extra: True
      fmt: '[%(asctime)s] [%(levelname)s] [%(name)s] [%(message)s]'
      answer: 42

Just remember to use the keys defined by the JsonFormatter Class. 'format' -> 'fmt'

[MyColorfulDays]

@FlorianHall
Yes, this requirment can be implemented by dict config.
The format keyword argument in dict config is already supported in next version v0.4.0(not release yet), so in v0.3.x, the fmt should be used.

import logging
import logging.config

DICT_CONFIG = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'console_jsonformatter': {
            '()': 'jsonformatter.JsonFormatter', # https://docs.python.org/3/library/logging.config.html#user-defined-objects
            'fmt': """{
                "Name": "name",
                "Levelno": "levelno",
                "Levelname": "levelname",
                "Pathname": "pathname",
                "Filename": "filename",
                "Module": "module",
                "Lineno": "lineno",
                "FuncName": "funcName",
                "Created": "created",
                "Asctime": "asctime",
                "Msecs": "msecs",
                "RelativeCreated": "relativeCreated",
                "Thread": "thread",
                "ThreadName": "threadName",
                "Process": "process",
                "Message": "message",
                "status": "status"
            }""",
            'record_custom_attrs': {
                'status': lambda **record_attrs: 'failed' if record_attrs['levelname'] in ['ERROR', 'CRITICAL'] else 'success'
            },
            'mix_extra': True
        },
    },
    'handlers': {
        'console_handler': {
            'level': 'INFO',
            'formatter': 'console_jsonformatter',
            'class': 'logging.StreamHandler',
        },
    },
    'loggers': {
        '': { # root logger
            'handlers': ['console_handler'],
            'level': 'INFO',
            'propagate': False
        },
    }
}

logging.config.dictConfig(DICT_CONFIG)
logging.info("hello, %s", 'jsonformatter')

[xiang1030]

Thank you very much for your contribution in log json

I want to use dictConfig to read the yaml/json format configuration file, and want to add record_custom_attrs, just like the demo file above uses DICT_CONFIG, but I don’t know how to use the yaml/json type configuration file while supporting record_custom_attrs.

Many thanks

[MyColorfulDays]

Hi, @xiang1030
The record_custom_attrs can't define in yaml/json cofig file.
Beacause the value of record_custom_attrs is a python function, the value of yaml/json can't define python function.
The suggestion is creating a single python file as log config file.

[xiang1030]

ok I understand.I like your project very much

[antariksh-nykaa]

Hi, thanks a lot for this really helpful formatter.

Using it with Django and having mix_extra=True, causes error in the default response/request logging that Django does. The request being a WSGIRequest type / socket.socket type, causes TypeError 'is not JSON serializable' error. Django itself sends params like self.request and server_time under 'extra' keyword param to logger in files django/core/servers/basehttp.py and django/utils/log.py.

Is there any way to avoid this error.

[antariksh-nykaa]

I have made a fix for the JSON serializable issue by using a child class , which also enables the Django user to have mix_extra enabled directly via a custom formatter class (wrapper on your library class). Let me know if I should share a PR for that.

[MyColorfulDays]

Hi, @antariksh-nykaa
Beacause of JsonFormatter support json.dumps all optional parameters, the TypeError of xx is not JSON serializeble can be solved by json.dumps optional parameter default or cls.
You can reference this Case 4. Solve cumtom LogRecord attribute is not JSON serializable

[MyColorfulDays]

@antariksh-nykaa
The document in branch v0.4.x is more clearly, and I had create FAQ for reference.
Thank you for your intention to help improve.

[xiang1030]

Hi,I try to use this command “$ pip install jsonformatter --upgrade” to upgrade,
But https://pypi.org/ only has 0.3.1,not find the latest version 0.4.X.
So I can only download the project first, and then use the command "python setup.py install" to install.
Now it can run normally in django, thank you very much.

[MyColorfulDays]

Hi @xiang1030
The version 0.4.X is not release yet, so you donwnload the version 0.4.X and use command install it is right.
Because python update the logging source code in new versions higher than python 3.6, after update project, I will release new version soon.