cib Design and Improvements

Metadata
cEP	17
Version	1.0
Title	cib Design and Improvements
Authors	Saksham Bansal mailto:sakshambansal1999@yahoo.com
Status	Implementation due
Type	Process

Abstract

This cEP describes changes to the design of cib which will allow it to install individual bears and their dependencies. It suggests using conda packages for bears that have different installation instructions for each distribution and enhancing of the existing Requirement classes.

The cEP discusses Ansible which is a simple IT automation software as a means to install bears on different distributions by creating a playbook with the required installation instructions for all bears.

Introduction

The current design of the cib tool allows it to perform basic operations such as running pip install on bears and then installing its dependencies. However, some bears are not even getting installed on certain distributions and cib is running into errors. Moreover, cib is not able to handle special cases such as of bears that have different installation instructions for each distribution i.e. DartLintBear and bears that have bear dependencies such as the ClangCloneDetectionBear. Additionally, cib needs to be more user friendly with a nice interface and should be able to provide verbose output, ask user for permission before proceeding.

Another alternative to cib that can be used for installing bears and their dependencies is Ansible. Ansible allows to orchestrate steps of any process for different distributions by creating "playbooks" in YAML. Each playbook contains a list of tasks which calls to any of the ansible modules. A sample playbook example is shown below:

tasks:
  - name: ensure apache is at the latest version
    yum:
      name: httpd
      state: latest
  - name: write the apache config file
    template:
      src: /srv/httpd.j2
      dest: /etc/httpd.conf
    notify:
    - restart apache
  - name: ensure apache is running (and enable it at boot)
    service:
      name: httpd
      state: started
      enabled: yes

Ansible provides numerous packaging modules that can be used to install dependencies of linter bears. Another advantage of Ansible is its ability to do a large amount of fact gathering from the user's machine and provide relevant information such as Ansible_distribution and Ansible_pkg_mgr. These gathered facts can be used to automatically decide in certain cases such as shown below.

- name: install basic package
  action: >
    {{ ansible_pkg_mgr }} name=package_name

Other than this, Ansible also allows to have root access for installing packages and printing verbose output. Ansible also allows filtering tasks based on tags. This can be used to install bears for a specific language, certain capability etc if the appropriate tags are added to the installation tasks of the bears.

Proposed Changes

Conda Packaging

Create a new repository called coala-bear packages.
This repository will have a folder called conda packages that contains conda recepies for all the bears that require different installation instructions on each distribution. Each conda package requires 3 files i.e. meta.yaml, build.sh, bld.bat.
These packages will be uploaded to Anaconda Cloud.
- On Anaconda Cloud create an organization called coala.
- The conda packages for bears and any others related to coala will be uploaded here.
- These packages can be installed by running conda config --add channels coala which will add coala to the list of channels for the user. Then running conda install bear_name will search conda packages available under coala as well and install the required bear for the user.
Move the generate_package to this repository and allow it to generate conda packages only for the bears that are in the folder conda packages.

Procedure for the installation of the bears using cib

Identify the valid bear names that need to be installed as specified by the user on the command line.
Install bear on the valid bear names. cib will check whether the bear can be installed from Anaconda Cloud by using CondaRequirement and then check whether it can be installed from PyPi using PipRequirement.
After installing the bear cib will proceed to installing its requirements. Cib should call methods from the requirement classes to do this.
- Add new Requirement classes to support different dependencies from various linters.
- The DistributionRequirement class needs to be split into different classes for each package manager and call its method to introduce more consistency.
cib then proceeds to install bear dependencies such as for ClangCloneDetectionBear by running install_bears function for each dependency bear.

Process for upgrading bears in cib

Check the bear name is installed using CondaRequirement or PipRequirement.
Iterate through Requirement instances to make sure that the required version of the linter is installed or upgrade the version if required.

Split `DistributionRequirement` for each distribution

This is the list of all distributions http://distro.readthedocs.io/en/latest/#distro.id available from distro package (excluding Darwin and Windows). Supporting all these platforms would be too broad for this project. Therefore, the following platforms listed below have been chosen to be completed as a part of this project.

Operating System	Tools
Debian/Ubuntu	apt-get
CentOS	yum
Fedora	dnf
Arch Linux	pacman
openSUSE/SUSE	zypper
Gentoo Linux	portage
Darwin	brew

Create a class PlatformRequirement representing a general requirement for a given distribution. This would be similiar to PackageRequirement but it will have a method to check whether the user is on the correct platform.
Then create classes such as AptRequirement, YumRequirement etc. that will inherit from the PlatformRequirement class.
Deprecate the DistributionRequirement class.

Ansible for downloading bear and dependencies

Write a script in python to create a playbook that will have the installation instruction for each bears and its dependencies. Additionaly, a playbook with all instructions.
- The script iterates over the REQUIREMENT instances of the bear. The cases for different tasks that will be generated in the playbook for different kinds of REQUIREMENT are shown in the code samples.
- Additionally the script would add relevant tags to each installation task. This would include the bear name, language etc.
- Other options for user interaction can also supported by Ansible and can be easily added.

Code Samples

cib improvements

Making use of conda packages in cib

from dependency_management.requirements.PipRequirement import (
    PipRequirement)
from dependency_management.requirements.CondaRequirement import (
    CondaRequirement)


def install_bears(bear_names_list):
    bear_failed_list = []
    for bear_name in bear_names_list:
        if PipRequirement(bear_name).is_installed() or \
           CondaRequirement(bear_name).is_installed():
            print(bear_name + ' is already installed.')
            continue
        if not PipRequirement(bear_name).install_package() or \
           not CondaRequirement(bear_name).install_package():
            install_requirements(bear_name)
            install_bear_dependencies(bear_name)
            print('DONE')
        else:
            print('Failed')
            bear_failed_list.append(bear_name)
    return bear_failed_list

Installing bear dependencies

import importlib


def install_bear_dependencies(bear_name):
    package_object = importlib.import_module(
        'coala' + bear_name + '.' + bear_name)
    for bear in getattr(package_object, bear_name).BEAR_DEPS:
        install_bears(bear)

UI to ask permission for dependencies to be installed

import sys
import coalib.misc.Constants as consts


def display_dependencies(bears):
    print('The bears have the following dependencies')
    for bear in bears:
        for requirement in bear.REQUIREMENTS:
            print(requirement.package, requirement.version)
    choice = raw_input().lower()
    if choice in consts.TRUE_STRINGS:
        return True
    elif choice in consts.FALSE_STRINGS:
        return False
    else:
        sys.stdout.write("Please respond with 'yes' or 'no'")

Splitting `DistributionRequirement`

import distro
from functools import lru_cache
import platform
from sarge import run, Capture
from coala_utils.decorators import generate_eq, generate_repr


@generate_eq("platform", "package", "version", "repo")
@generate_repr()
class PlatformRequirement:
    """
    This class helps keeping track of bear distribution requirements.
    """

    REQUIREMENTS = {}

    def __init__(self, platform: str, package: str, version='', repo=''):
        """
        :param platform:  A string with the name of the distribution.
                          Currently, the supported platforms are Debian,
                          Ubuntu, CentOS, Fedora, Arch Linux, openSUSE/SUSE,
                          Gentoo Linux, Darwin
        :param package: A string with the name of the package to be installed.
        :param version: Version string. Leave empty to specify latest version.
        :param repo:    The repository from which the package is to be
                        installed.
        """
        self.platform = platform
        self.package = package
        self.version = version
        self.repo = repo

    def __str__(self):
        """
        Just return package name, followed by version if given.
        """
        if self.version:
            return "{}{} on {}".format(self.package,
                                       self.version,
                                       self.platform)
        else:
            return "{} on {}".format(self.package,
                                     self.platform)

    def install_package(self):
        """
        Runs the install command for the package given in a sub-process.
        """
        run(" ".join(self.install_command()), stdout=Capture(),
            stderr=Capture())

    def install_command(self):
        """
        Returns a string with the installation command, to be used by
        "install_package()".

        :raises NotImplementedError: Method is not implemented
        """
        raise NotImplementedError

    def is_installed(self):
        """
        Check if the requirement is satisfied.

        :return: Returns True if satisfied, False if not.
        :raises NotImplementedError: Method is not implemented
        """
        raise NotImplementedError

    @classmethod
    @lru_cache()
    def get_platform(cls):
        return platform.system(), distro.id()

    def check_platform(self):
        """
        Check if the user is on the target platform

        :return: Returns True if user is on platform for this Requirement,
        otherwise false.
        """
        if self.get_platform()[0] is 'Darwin':
            return self.platform == 'Darwin'
        elif self.get_platform()[0] is 'Linux':
            return self.get_platform()[1] == self.platform

Ansible

Package Requirement

- pip:
    name: package_name
    version: 0.11

This can be used to install the pip dependencies for Bears. Ansible currently supports the following language specific packaging modules.

Distribution Requirement

Package name is same across different OS families

- name: Install foo
  package: name=foo state=latest

Ansible provides a generic and abstracted OS package module that can be used to install the required package on all distributions. For example for the ArtisticStyleBear that uses a DistributionRequirement as shown:

REQUIREMENTS = {
        DistributionRequirement(
            apt_get='astyle',
            dnf='astyle'
        )
    }

Then indent can be installed simply as follows

- name: Install astyle
  package: name=astyle state=latest

When the package name is different across OS families

# Load a variable file based on the OS type, or use default if not found.
with_first_found:
    - "../vars/{{ ansible_distribution }}.yml"
    - "../vars/{{ ansible_os_family }}.yml"
    - "../vars/default.yml"

- name: Install package_name
  package: >
    name={{ package_name }}
    state=latest

- name: Install package_name2
  package: >
    name={{ package_name2 }}
    state=latest

Then for each OS family describe a variable file that contains the name of the package on that OS family.

# vars/default.yml
package_name: name1
package_name2: name2

---
# vars/Debian.yml
package_name: name3
package_name2: name4
---
# vars/Archlinux.yml
package_name: name5
package_name2: name6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cEP-0017.md

cEP-0017.md

cib Design and Improvements

Abstract

Introduction

Proposed Changes

Conda Packaging

Procedure for the installation of the bears using cib

Process for upgrading bears in cib

Split `DistributionRequirement` for each distribution

Ansible for downloading bear and dependencies

Code Samples

cib improvements

Making use of conda packages in cib

Installing bear dependencies

UI to ask permission for dependencies to be installed

Splitting `DistributionRequirement`

Ansible

Package Requirement

Distribution Requirement

Package name is same across different OS families

When the package name is different across OS families

Files

cEP-0017.md

Latest commit

History

cEP-0017.md

File metadata and controls

cib Design and Improvements

Abstract

Introduction

Proposed Changes

Conda Packaging

Procedure for the installation of the bears using cib

Process for upgrading bears in cib

Split DistributionRequirement for each distribution

Ansible for downloading bear and dependencies

Code Samples

cib improvements

Making use of conda packages in cib

Installing bear dependencies

UI to ask permission for dependencies to be installed

Splitting DistributionRequirement

Ansible

Package Requirement

Distribution Requirement

Package name is same across different OS families

When the package name is different across OS families

Split `DistributionRequirement` for each distribution

Splitting `DistributionRequirement`