Python Dynamic Configuration

I am sure you are already aware of the importance of configuration files in any programming language. Managing configuration files is the most important part of any software development process.

In this blog, we will discuss one easy-to-use python trick that could be very handy for a quick program or repository you are creating for your individual project or for your organization.

In order to understand this, we will first discuss the traditional approach and then we will discuss how a small class in python can help you avoid multiple iterations in your config file.

Traditional Approach

In order to make both approaches more intelligible, we will take an example code and will implement both approaches to see the impact of the trick.

Let’s create a folder configuration and add these 2 files inside it.

settings.ini file looks something like this:

[RUN]
num_cores = 2
num_files = -1

Now to parse the configuration, we use another file. let’s say config.py

import os
import configparser

parser = configparser.ConfigParser()
parser.read_file(open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "settings.ini")))

NUM_CORES = parser.get('RUN','num_cores')

print ("Number of cores available : ", parser.get('RUN','num_cores'))

The output of the above script will look something like this:

➜  configuration python config.py
Number of cores available :  2

This is straightforward and easy, now every time, we want to add a new config, there is an easy way just to add a new variable and use that inside code. so, let’s consider now we also want the value of NUM_FILES in the code, then new code will look something like this:

import os
import configparser

parser = configparser.ConfigParser()
parser.read_file(open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "settings.ini")))

NUM_CORES = parser.get('RUN','num_cores')
NUM_FILES = parser.get('RUN','num_files')

print ("Number of cores available : ", parser.get('RUN','num_cores'))
print ("Number of cores available : ", parser.get('RUN','num_files'))

and now the NUM_FILES variable can be used across your project to get the value of this variable.

But how many times has this happened to you that you have to add these same lines every time you want to add a new variable, you would agree that this could be painful sometimes and it could halt your train of thought.

Dynamic configuration:

What if I tell you there is an easy way out and you can initialize the whole .ini file dynamically and there is no need to parse every variable explicitly.

Now, in order to do that, I have added a new file dynamic_config_parser.py and its content will look something like this:

import configparser

class DynamicConfig:
    def __init__(self, conf):
        if not isinstance(conf, dict):
            raise TypeError(f'dict expected, found {type(conf).__name__}')

        self._raw = conf
        for key, value in self._raw.items():
            setattr(self, key, value)

class DynamicConfigInit:
    """
    This class is used to dynamically load static variables from the settings.ini file. Any va
    variable declared in the settings.ini can be parsed directly.
    """
    def __init__(self, conf):
        if not isinstance(conf, configparser.ConfigParser):
            raise TypeError(f'ConfigParser expected, found {type(conf).__name__}')

        self._raw = conf
        for key, value in self._raw.items():
            setattr(self, key, DynamicConfig(dict(value.items())))

and now your config.py will look something like this:

import os
import configparser
from dynamic_config_parser import DynamicConfigInit

parser = configparser.ConfigParser()
parser.read_file(open(os.path.join(os.path.dirname(os.path.abspath(__file__)), "settings.ini")))
STATIC_CONFIG = DynamicConfigInit(parser)

print("Number of cores from dynamic config: ", STATIC_CONFIG.RUN.num_cores)

Viola! That’s it. We are done.

As you see, now there is no requirement to add every variable every time into config.py, and dynamic config initialization is handling this for you and you can save a long repetitive file with a huge configuration.