-
Notifications
You must be signed in to change notification settings - Fork 85
ConfStore Specifications
Note:This wiki is out-of-date at the moment. Please refer to this document for ConfStore Specification
ConfStore Framework provides an easy-to-use platform which can be used to load or store configurations on a node from any backing storage (be local or distributed), reliably and efficiently. It is a common framework and agnostic to the backing storage software e.g. FileStorage, PillarStorage etc.
Typical list of features provided by ConfStore Framework
- Basic Configuration Handling like loading / saving configuration from/to files.
- KV Store Abstraction over various File or DB Storage
- Copy / Backup configuration from one store to the other
- Python Interfaces
ConfStore Framework implements a python framework and can be consumed directly by python programs.
ConfStore - Abstracted Storage for storing config entries and working with it.
KVStore - Abstraction over various KV based Storages, works as Config backend.
Index - Config Domain containing a set of config keys and values loaded from a config backend.
Key - Config Key representing a given config entry. It can also be hierarchical e.g. “k1>k2>k3” or “x>y[2]>z”.
JSON - File in json format working as the storing configuration.
Depending on the configuration backend you will need to setup backend storage which could be any of the following.
- File Storage (Any config file, e.g. /etc/cortx/sspl.conf) of any format, e.g. json, yaml, xml, etc.
- Consul Storage
- Salt Pillar Storage
ConfStore uses KVStore module which offers an abstraction over various backends, represented as URLs. A ConfStore is exposed as singleton instance to enable use of configuration infrastructure. Conf entries are associated with domains called index. There can be multiple indexes and corresponding conf entries each index loaded from a defined backend. Config key represents a config entry with corresponding value. Config key can also be hierarchical format, i.e. “k1.k2.k3” which is stored in hierarchy when stored in the json format. Key can also have list in the middle, e.g. “x.y[2].z”.
Example below shows a sample program working with config entries stored in a json file.
Example:
from cortx.utils.conf_store import Conf
# Load Configuration
Conf.load(‘index1’, ‘json:///etc/cortx/cortx.conf’)
Conf.load(‘index2’, ‘consul://consul.cortx.colo.com/cortx/prefix’)
# Get Config entries
val1 = Conf.get(‘index1’, ‘key1’)
val2 = Conf.get(‘index1’, ‘keygroup2>key>subkey’)
# Modify Config entries
Conf.set(‘index1’, ‘keygroup3>key>subkey’, ‘value’)
Conf.save(‘index1’)
KVStore allows range of Configuration Backends which could be either file based or any other Key Value database based such as Consul. This is achieved merely by pointing to your configuration file or database in the form of a URL.
**Examples: **
File: “json:///etc/myconf.json”
Consul: “consul://server:port/”
ConfStore key delimiter can be configured with the user desired custom one. ConfStore allows the user to configure delimiter from one of the following below
':' (Colon)
'>'(Greater than)
'.'(Dot)
'|'(Pipe)
';' (semi-colon)
'/' (Forward slash)
Below example shows how to configure a custom delimiter
Example: Using ConfStore init configure custom delimiter
from cortx.utils.ConfStore import Conf
Conf.load(‘index1’, ‘json://etc/sspl.conf’)
# Configuring ‘.’ as delimiter
Conf.init(delim=’.’)
Conf.get(‘index1’, ‘cluster.id’)
ConfStore Keys may also include array indexes in the keys. e.g. “x.y[1].z”. Key component with index will be considered to be an array and corresponding value will be returned / acted upon. Refer to the example below for JSON config file containing arrays / lists.
Example: JSON Config File
{
"dsn" : "DBI:mysql:test",
"user" : "tester",
"colors" : [ "red", "green", "blue" ],
"stats" : [
{ "health" : 32, "vitality" : 11 },
{ "health" : 50, "vitality" : 12 }
]
}
Below is the corresponding example to show how keys with arrays/lists can be accessed.
Example: Using ConfStore to fetch and store keys with array indexes
from cortx.utils.ConfStore import Conf
Conf.load(‘index1’, ‘json://etc/sspl.conf’)
Conf.get(‘index1’, ‘stats[0]>health’’) # Get config values by indexed -> return value: 32
Conf.get(‘index1’, ‘colors[1]’’) # Get Config values by Indexed -> return value: green
Conf.set(‘index1’, ‘stats[2]>health’, ‘100’)
Conf.set(‘index1’, ‘colors[3]’, ‘CMYK’)
Conf.delete(‘index1’, ‘stats[1]>health’)
# Provisioner
Conf.load(‘index1’, ‘pillar://pillar-server/cortx’)
Conf.set()
# CSM Setup
Conf.load(‘index1’, ‘pillar://pillar-server/cortx’)
Conf.load(‘index2’, ‘json:///etc/cortx/csm.conf’) # Empty
Conf.copy(‘index1’, ‘index2’, key_list) # “mykey*”
# CSM
Conf.load(‘index2’, ‘json:///etc/cortx/csm.conf’) # Empty
Conf.get(‘index2’, ‘cluster.id’)
Users of confStore can even subscribe for the changes in the configuration backend so that if someone changes configuration files, callers are notified about it. This is done with the help of callback which is called in case of changes in the configuration values.
Example:
from cortx.utils.ConfStore import Conf
Conf.load(‘sspl’, ‘json:///etc/sspl.conf’)
# Set config values
Conf.subscribe(‘sspl’, Conf.ACT_CALLBACK, my_callback)
# Conf.subscribe(‘sspl’, Conf.ACT_RELOAD)
def my_callback(index):
# Take Lock
Conf.load(index, ‘json:///etc/sspl.conf’)
Search key API use to Search for given key and value under a parent node. Returns list of keys that matched the criteria (i.e. has given value)
Example:
from cortx.utils.conf_store import Conf
Conf.load('index', conf_file)
keys = Conf.search('index', 'parent_key', 'key_to_search', 'value_to_search')
For programs other than python, ConfStore can be used in the CLI form. Below are the example commands
# url= json:///mypath/file.json
# conf $url set “k1=v1”
# conf $url get “k1”
v1
# conf $url set “k2>k3=v3;k4=v4”
# conf $url get “k1;k2>k3”
v1;v3
# conf $url delete “k2>k3”
# conf $url set “k4[2]>k5=v6” <= Note : Indexing works only for json file format
# conf $url get “k4”
[[{}, {}, {'k5': 'v6'}]]
# conf $url get “k4[2]”
[{'k5': 'v6'}]
# conf $url get “k4[2]>k5”
[‘v6’]
# conf $url set "k6>k4[2]>k5=v3”
# conf $url get “k6”
[{'k4': [{}, {}, {'k5': 'v3'}]}]
Generic format for CLI is as below
Where:
url representing the Backend for conf store
command represents the action from: set, get, delete
arguments based on the command, next argument
Specific command format as follows
Where:
kv_spec One or more Key Value pair in “key=val” format. Multiple key value pairs are separated using “;”
Example(s): “k1=v1”, “k1=v1;k2=v2”
This command adds the keys and specified values to the conf store. If the key exists, then old value is overwritten.
Where:
key_list One or more Key(s). Multiple keys are separated using “;”
Example(s): “k1”, “k1>k2;k3”
This command retrieves the values for the keys in the argument.
Where:
key_list One or more Key(s). Multiple keys are separated using “;”
Example(s): “k1”, “k1>k2;k3”
This command deletes the keys mentioned in the arguments (and the associated values) from the conf store.
Where:
index Identifier for configuration. This could be Module ID or even any common ID.
url URL representing the Backend for conf store.
Where:
delim delimiter for configure custom delimiter of user choice. By default This could be either one of the following.
':' (Colon) or '>'(Greater than) or '.'(Dot) or '|'(Pipe) or ';' (semi-colon) or '/' (Forward slash)
Input Parameters:
index Identifier for configuration. This could be Module ID or even any common ID.
Key Config Key whose value needs to be obtained.
default_value If key is not present, default value is returned
Returns:
Value of the Config Key or Default Value
Where: index Identifier for configuration. This could be Module ID or even any common ID. key List of keys to be set val List of values
Where: source_Index Index which needs to be saved. destination_index
Where:
Index Index which needs to be saved.
Where:
Index Identifier for configuration. This could be Module ID or even any common ID.
Action Action to be taken when there is a change in the configuration backend. Possible values:
ConfStore.ACT_RELOAD – Reload Configuration Values
ConfStore.ACT_CALLBACK – Callback given function
Parameters In case of ACT_CALLBACK, parameters is the function reference which needs to be called back.