Oreum Core Tools oreum_core

---

1. Description and Scope

This is an ever-growing package of core tools for use on client projects by Oreum Industries.

• Provides an essential workflow for data curation, EDA, basic ML using the core scientific Python stack incl. numpy, scipy, matplotlib, seaborn, pandas, scikit-learn, umap-learn
• Optionally provides an advanced Bayesian modeling workflow in R&D and Production using a leading probabilistic programming stack incl. pymc, pytensor, arviz (do pip install oreum_core[pymc])
• Optionally enables a generalist black-box ML workflow in R&D using a leading Gradient Boosted Trees stack incl. catboost, xgboost, optuna, shap (do pip install oreum_core[tree])
• Also includes several utilities for text cleaning, sql scripting, file handling

This package is:

• A work in progress (v0.y.z) and liable to breaking changes and inconvenience to the user
• Solely designed for ease of use and rapid development by employees of Oreum Industries, and selected clients with guidance

This package is not:

• Intended for public usage and will not be supported for public usage
• Intended for contributions by anyone not an employee of Oreum Industries, and unsolicited contributions will not be accepted.

Notes

• Project began on 2021-01-01
• The README.md is MacOS and POSIX oriented
• See LICENCE.md for licensing and copyright details
• See pyproject.toml for various package details
• This uses a logger named 'oreum_core', feel free to incorporate or ignore
• Hosting:
  • Source code repo on GitHub
  • Source code release on GitHub
  • Package release on PyPi

2. Instructions to Create Dev Environment

For local development on MacOS

2.0 Pre-requisite installs via homebrew

1. Install Homebrew, see instructions at https://brew.sh
2. Install direnv, git, git-lfs, graphviz, zsh

$> brew update && upgrade
$> brew install direnv git git-lfs graphviz zsh

2.1 Git clone the repo

Assumes direnv, git, git-lfs and zsh installed as above

$> git clone https://github.com/oreum-industries/oreum_core
$> cd oreum_core

Then allow direnv on MacOS to autorun file .envrc upon directory open

2.2 Create virtual environment and install dev packages

Notes:

• We use conda virtual envs controlled by mamba (quicker than conda)
• We install packages using miniforge (sourced from the conda-forge repo) wherever possible and only use pip for packages that are handled better by pip and/or more up-to-date on pypi
• Packages might not be the very latest because we want stability for pymc which is usually in a state of development flux
• See cheat sheet of conda commands
• The Makefile creates a dev env and will also download and preinstall miniforge if not yet installed on your system

2.2.1 Create the dev environment

From the dir above oreum_core/ project dir:

$> make -C oreum_core/ dev

This will also create some files to help confirm / diagnose successful installation:

• dev/install_log/blas_info.txt for the BLAS MKL installation for numpy
• dev/install_log/pipdeptree[_rev].txt lists installed package deps (and reversed)
• LICENSES_THIRD_PARTY.md details the license for each package used

2.2.2 (Optional best practice) Test successful installation of dev environment

From the dir above oreum_core/ project dir:

$> make -C oreum_core/ test-dev-env

This will also add files dev/install_log/[numpy|scipy].txt which detail successful installation (or not) for numpy, scipy

2.2.3 (Useful during env install experimentation): To remove the dev environment

From the dir above oreum_core/ project dir:

$> make -C oreum_core/ uninstall-env

2.3 Code Linting & Repo Control

2.3.1 Pre-commit

We use pre-commit to run a suite of automated tests for code linting & quality control and repo control prior to commit on local development machines.

• Precommit is already installed by the make dev command (which itself calls pip install -e .[dev])
• The pre-commit script will then run on your system upon git commit
• See this project's .pre-commit-config.yaml for details

2.3.2 Github Actions

We use Github Actions aka Github Workflows to run:

1. A suite of automated tests for commits received at the origin (i.e. GitHub)
2. Publishing to PyPi upon creating a GH Release

• See Makefile for the CLI commands that are issued
• See .github/workflows/* for workflow details

2.3.3 Git LFS

We use Git LFS to store any large files alongside the repo. This can be useful to replicate exact environments during development and/or for automated tests

• This requires a local machine install (see Getting Started)
• See .gitattributes for details

2.4 Configs for Local Development

Some notes to help configure local development environment

2.4.1 Git config ~/.gitconfig

[user]
    name = <YOUR NAME>
    email = <YOUR EMAIL ADDRESS>

2.5 Install VSCode IDE

We strongly recommend using VSCode for all development on local machines, and this is a hard pre-requisite to use the .devcontainer environment (see section 3)

This repo includes relevant lightweight project control and config in:

oreum_core.code-workspace
.vscode/extensions.json
.vscode/settings.json

---

3. Code Standards

Even when writing R&D code, we strive to meet and exceed (even define) best practices for code quality, documentation and reproducibility for modern data science projects.

3.1 Code Linting & Repo Control

We use a suite of automated tools to check and enforce code quality. We indicate the relevant shields at the top of this README. See section 1.4 above for how this is enforced at precommit on developer machines and upon PR at the origin as part of our CI process, prior to master branch merge.

These include:

• ruff - extremely fast standardised linting and formatting, which replaces black, flake8, isort
• interrogate - ensure complete Python docstrings
• bandit - test for common Python security issues

We also run a suite of general tests pre-packaged in precommit.

---

Copyright 2024 Oreum OÜ t/a Oreum Industries. All rights reserved. See LICENSE.md.

Oreum OÜ t/a Oreum Industries, Sepapaja 6, Tallinn, 15551, Estonia, reg.16122291, oreum.io

---

Oreum OÜ © 2024