Merge pull request #141 from rhosqeauto/master

Update Stable branch with latest master

.gitignore

@@ -1,3 +1,4 @@
+!*j2
 nosetests.xml
 ansible.cfg
 *.py[co]
@@ -15,4 +16,5 @@ hosts*
 *infrared.cfg
 *.egg-info/
 tmp/
+playbooks/*.retry
 

README.rst

@@ -25,16 +25,17 @@ So, After cloning repo from GitHub::
 
   $ pip install -e .
 
-Conf
-====
-
-``infrared`` will look for ``infrared.cfg`` in the following order:
+.. note:: ``infrared`` will look for ``infrared.cfg`` in the following order:
 
 #. In working directory: ``./infrared.cfg``
 #. In user home directory: ``~/.infrared.cfg``
 #. In system settings: ``/etc/infrared/infrared.cfg``
 
-.. note:: To specify a different directory or different filename, override the
+ If the configuration file ``infrared.cfg`` doesn't exist in any of
+ the locations above, the InfraRed project's dir will be used as the default
+ location for configurations.
+
+ To specify a different directory or different filename, override the
  lookup order with ``IR_CONFIG`` environment variable::
 
     $ IR_CONFIG=/my/config/file.ini ir-provision --help
@@ -74,22 +75,74 @@ The merging priority order is:
 
 InfraRed input arguments
 ------------------------
-InfraRed accepts the next sources of the input arguments (in priority order):
+InfraRed extends the ``clg`` and ``argpars`` packages with the following types
+that need to be defined in `.spec` files:
+
+* **Value**: String values
+* **YamlFile**: Expects path to YAML files. Will search for files in the settings directory before trying to resolve
+  absolute path. For the argument name is "arg-name" and of subparser "SUBCOMMAND" of command "COMMAND", the default
+  search path would be::
+
+    settings_dir/COMMAND/SUBCOMMAND/arg/name/arg_value
+
+* **Topology**: Provisioners allow to dynamically define the provisioned
+  nodes topology. InfraRed provides several
+  'mini' YAML files to describe different roles: ``controller``, ``compute``,
+  ``undercloud``, etc...
+  These 'mini' files are then merged into one topology file according to the
+  provided ``--topology-nodes`` argument value.
+
+  The ``--topology-nodes`` argument can have the following format:
+   * ``--topology-nodes=1_controller,1_compute``
+   * ``--topology-nodes=1_controller``
+   * ``--topology-nodes=3_controller,1_compute,1_undercloud``
+
+ InfraRed will read dynamic topology by following the next steps:
+  #. Split the topology value with ','.
+  #. Split each node with '_' and get pair (number, role). For every pair
+     look for the topology folder (configured in the infrared.cfg file) for
+     the appropriate mini file (controller.yaml, compute.yaml, etc). Load the
+     role the defined number of times into the settings.
+
+ .. note:: The default search path for topology files is
+       ``settings/provivisioner/topology``. Users can add their own topology
+       roles there and reference them on runtime
+
+These arguments will accept input from sources in the following priority
+order:
+
+#. Command line arguments:
+   ``ir-provision virsh --host-address=some.host.com --host-user=root``
+#. Environment variables: ``HOST_ADRRESS=earth.example.com ir-provision virsh --host-user=root``
+#. Predefined arguments in ini file specified using ``--from-file`` option::
+
+    ir-provision virsh --host-address=some.host.com --from-file=user.ini
 
-1. Command line arguments:  ``ir-provision virsh --host=some.host.com --ssh_user=root``
-2. Predefined arguments in ini file. Use the --from-file option to specify ini file::
-  
-    ir-provision virsh --host=some.host.com --from-file=user.ini
-  
     cat user.ini
     [virsh]
-    ssh_user=root
-    ssh_key=mkey.pm
+    host-user=root
+    host-key=mkey.pm
+
+#. Defaults defined in ``.spec`` file for each argument.
+
+  .. note:: The sample `ini` file with the default values can be generated with:
+   ``ir-povision virsh --generate-conf-file=virsh.ini``. Generated file will contain
+   all the default arguments values defined in the spec file.
+
+Arguments of the above types will be automatically injected into settings
+YAML tree in a nested dict from.
+
+Example:
+The input for ``ir-COMMAND`` and argument ``--arg-name=arg-value`` maps to:
 
+  .. code-block:: yaml
 
-3. Environment variables: ``HOST=earth ir-provision virsh --ssh_user=root``
+      COMMAND:
+          arg:
+              name: "arg-value"
 
-Command line arguments have the highest priority. All the undefined variables will be replaced by that arguments from file or from environment.
+"arg-value" can be a simple string or be resolved into a more advanced
+dictionary depending on the argument type in ``.spec`` file
 
 Extra-Vars
 ----------
@@ -109,28 +162,26 @@ Add new Plugins
 
 There are two steps that should be done when adding a new plugin to InfraRed:
 
-1. Creating a specification file:
+#. Creating a specification file:
     InfraRed uses ArgParse wrapper module called 'clg' in order to create a parser that based on `spec` file
     (YAML format file) containing the plugin options.
     The spec file should be named as the new plugin name with '.spec' extension and located inside the plugin dir
     under the InfraRed 'setting' dir.
     For more details on how to use this module, please visit the 'clg' module `homepage <http://clg.readthedocs
     .org/en/latest/>`_.
 
-2. Creating a default spec file (default.ini). 
-    This file should contain the default values for the command line arguments. All the default values should go under the name section names as a new plugin. Example::
-      
-      [virsh]
-      topology=all-in-one.yml
-      network=default.yml
-      ssh-key=~/.ssh/id_rsa
-      ssh-user=root
-
-3. Creating settings files.
+#. Creating settings files.
     Settings files are files containing data which defines how the end result of the playbook execution will be
     looked like. Settings file are file in YAML format, end with ".yml" extension. Those files located under the
     plugin's dir which itself located under the 'settings' dir in the InfraRed project's dir.
     The end result of the playbook execution is based on the data created by merging of several settings files together
     with other values, all are received by the user.
     When adding a new plugin, there is a need to create those settings files containing the needed data for the
     playbook execution.
+
+
+Known issues
+============
+
+#. PROBLEM: sshpass package cannot be installed during virsh provisioning.
+   SOLUTION: install rhos-release tool. Install osp-d with rhos-release: ``rhos-release 7-director``

cli/conf.py

@@ -1,13 +1,18 @@
 import ConfigParser
 
-import clg
+import exceptions
 import os
-import yaml
-import yamlordereddictloader
-from cli import exceptions
+
 from cli import utils
+from cli import logger
 
-DEFAULT_INI = 'default.ini'
+LOG = logger.LOG
+DEFAULT_CONF_DIRS = dict(
+    settings='settings',
+    modules='library',
+    roles='roles',
+    playbooks='playbooks'
+)
 
 
 def load_config_file():
@@ -29,112 +34,23 @@ def load_config_file():
     for path in (env_path, cwd_path, utils.USER_PATH, utils.SYSTEM_PATH):
         if path is not None and os.path.exists(path):
             _config.read(path)
-            return _config
-
-    conf_file_paths = "\n".join([cwd_path, utils.USER_PATH, utils.SYSTEM_PATH])
-    raise exceptions.IRFileNotFoundException(
-        conf_file_paths,
-        "IR configuration not found. "
-        "Please set it in one of the following paths:\n")
-
-
-def IniFileType(value):
-    """
-    The custom type for clg spec
-    :param value: the argument value.
-    :return: dict based on a provided ini file.
-    """
-    _config = ConfigParser.ConfigParser()
-    _config.read(value)
-
-    d = dict(_config._sections)
-    for k in d:
-        d[k] = dict(_config._defaults, **d[k])
-        for key, value in d[k].iteritems():
-            # check if we have lists
-            value = utils.string_to_list(value, append_to_list=False)
-            d[k][key] = value
-        d[k].pop('__name__', None)
-
-    return d
+            break
+    else:
+        LOG.warning("Configuration file not found, using InfraRed project dir")
+        project_dir = os.path.dirname(os.path.dirname(__file__))
 
+        _config.add_section('defaults')
+        for option, value in DEFAULT_CONF_DIRS.iteritems():
+            _config.set('defaults', option, os.path.join(project_dir, value))
 
-class SpecManager(object):
-    """
-    Holds everything related to specs.
-    """
-
-    SPEC_EXTENSION = '.spec'
+    # Validates settings dir exists
+    settings_dir = _config.get('defaults', 'settings')
+    if not os.path.exists(settings_dir):
+        raise exceptions.IRFileNotFoundException(
+            settings_dir,
+            "Settings directory doesn't exist: ")
 
-    @classmethod
-    def parse_args(cls, module_name, config, args=None):
-        """
-        Looks for all the specs for specified module
-        and parses the commandline input arguments accordingly.
-
-        :param module_name: the module name: installer|provisioner|tester
-        """
-        cmd = clg.CommandLine(cls._get_specs(module_name, config))
-        res_args = vars(cmd.parse(args))
-
-        # always load default values for command0
-        default_file = os.path.join(
-            config.get('defaults', 'settings'),
-            module_name,
-            res_args['command0'],
-            DEFAULT_INI
-        )
-        defaults = IniFileType(default_file)[res_args['command0']]
-
-        # override defaults with env variables
-        for arg_name, arg_value in res_args.iteritems():
-            upper_arg_name = arg_name.upper()
-            if arg_value is None and upper_arg_name in os.environ:
-                defaults[arg_name] = os.getenv(upper_arg_name)
-
-        # override defaults with the ini file args
-        if 'from-file' in res_args:
-            file_args = res_args['from-file']
-            if file_args is not None and res_args['command0'] in file_args:
-                defaults = utils.dict_merge(
-                    file_args[res_args['command0']],
-                    defaults)
-
-        # replace defaults with cli
-        utils.dict_merge(res_args, defaults)
-
-        return res_args
-
-    @classmethod
-    def _get_specs(cls, module_name, config):
-        """
-        Gets specs files as a dict from settings/<module_name> folder.
-        :param module_name: the module name: installer|provisioner|tester
-        """
-        res = {}
-        for spec_file in cls._get_all_specs(config, subfolder=module_name):
-            spec = yaml.load(open(spec_file),
-                             Loader=yamlordereddictloader.Loader)
-            utils.dict_merge(res, spec)
-        return res
-
-    @classmethod
-    def _get_all_specs(cls, config, subfolder=None):
-        root_dir = utils.validate_settings_dir(
-            config.get('defaults', 'settings'))
-        if subfolder:
-            root_dir = os.path.join(root_dir, subfolder)
-
-        res = []
-        for dirpath, _, filenames in os.walk(root_dir):
-            for filename in [f for f in filenames
-                             if f.endswith(cls.SPEC_EXTENSION)]:
-                res.append(os.path.join(dirpath, filename))
-
-        return res
+    return _config
 
 
 config = load_config_file()
-
-# update clg types
-clg.TYPES.update({'IniFile': IniFileType})

cli/exceptions.py

@@ -77,3 +77,9 @@ class IRConfigurationException(IRException):
     General exception for any kind of configuration issues.
     """
     pass
+
+
+class IRInfiniteLookupException(IRException):
+    def __init__(self, value):
+        message = "Lookup circular reference detected for: {}".format(value)
+        super(IRInfiniteLookupException, self).__init__(message)