Skip to content

Commit

Permalink
mv to z pkg
Browse files Browse the repository at this point in the history
  • Loading branch information
@ryan-williams
ryan-williams committed Dec 30, 2024
1 parent 6a24f08 commit 56eac99
Show file tree
Hide file tree
Showing 89 changed files with 202 additions and 192 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: CI
on:
push:
branches: [ main ]
branches: [ main, z ]
tags: [ "**" ]
pull_request:
branches: [ "**" ]
Expand All @@ -25,7 +25,7 @@ jobs:
- name: Install test dependencies
run: |
pip install -e .[all]
python -c 'import utz; print("utz version: %s, file: %s" % (utz.__version__, utz.__file__))'
python -c 'import z; print("z version: %s, file: %s" % (z.__version__, z.__file__))'
- name: Test
env:
SHELL: /bin/bash # Required by test_proc.py::test_pipeline_shell_executable_warning
Expand Down
194 changes: 102 additions & 92 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,51 +1,52 @@
# utz
*("yoots")*: utilities I've missed in the Python standard library, Pytest, Pandas, Plotly, …
# z
Python utilities, augmenting the standard library, Pandas, Plotly, Pytest, etc.

[![](https://img.shields.io/pypi/v/utz?color=blue&style=flat-square)][utz]
[![](https://img.shields.io/pypi/v/z?color=blue&style=flat-square)][z]

<!-- toc -->
- [Install](#install)
- [Use](#use)
- [`utz.proc`: `subprocess` wrappers; shell out to commands, parse output](#utz.proc)
- [`utz.collections`: Collection/list helpers](#utz.collections)
- [`utz.cd`: "change directory" contextmanager](#utz.cd)
- [`utz.fn`: decorator/function utilities](#utz.fn)
- [`utz.decos`: compose decorators](#utz.decos)
- [`utz.call`: only pass expected `kwargs` to functions](#utz.call)
- [`utz.env`: `os.environ` wrapper + `contextmanager`](#utz.env)
- [`utz.gzip`: deterministic GZip helpers](#utz.gzip)
- [`utz.plot`: Plotly helpers](#utz.plots)
- [`utz.setup`: `setup.py` helper](#utz.setup)
- [`utz.test`: `dataclass` test cases, `raises` helper](#utz.test)
- [`utz.parametrize`: `pytest.mark.parametrize` wrapper, accepts `dataclass` instances](#utz.parametrize)
- [`utz.raises`: `pytest.raises` wrapper, match a regex or multiple strings](#utz.raises)
- [`utz.time`: `now`/`today` helpers](#utz.time)
- [`utz.hash_file`](#utz.hash_file)
- [`utz.docker`, `utz.tmpdir`, etc.](#misc)
- [`z.proc`: `subprocess` wrappers; shell out to commands, parse output](#z.proc)
- [`z.collections`: Collection/list helpers](#z.collections)
- [`z.cd`: "change directory" contextmanager](#z.cd)
- [`z.fn`: decorator/function utilities](#z.fn)
- [`z.decos`: compose decorators](#z.decos)
- [`z.call`: only pass expected `kwargs` to functions](#z.call)
- [`z.env`: `os.environ` wrapper + `contextmanager`](#z.env)
- [`z.gzip`: deterministic GZip helpers](#z.gzip)
- [`z.plot`: Plotly helpers](#z.plots)
- [`z.setup`: `setup.py` helper](#z.setup)
- [`z.test`: `dataclass` test cases, `raises` helper](#z.test)
- [`z.parametrize`: `pytest.mark.parametrize` wrapper, accepts `dataclass` instances](#z.parametrize)
- [`z.raises`: `pytest.raises` wrapper, match a regex or multiple strings](#z.raises)
- [`z.time`: `now`/`today` helpers](#z.time)
- [`z.hash_file`](#z.hash_file)
- [`z.docker`, `z.tmpdir`, etc.](#misc)
<!-- /toc -->

## Install <a id="install"></a>
```bash
pip install utz
pip install z
```
- `utz` has one dependency, [`stdlb`] (wild-card standard library imports).
- `z` has one dependency, [`stdlb`] (wild-card standard library imports).
- ["Extras"][extras] provide optional deps (e.g. [Pandas], [Plotly], …).

## Use <a id="use"></a>

I usually do this at the top of Jupyter notebooks:

```python
from utz import *
from z import *
```

This imports most standard library modules/functions (via [`stdlb`]), as well as the `utz` members below.
This imports most standard library modules/functions (via [`stdlb`]), as well as the `z` members below.

Below are a few modules, in rough order of how often I use them:

### [`utz.proc`]: [`subprocess`] wrappers; shell out commands, parse output <a id="utz.proc"></a>
### [`z.proc`]: [`subprocess`] wrappers; shell out commands, parse output <a id="z.proc"></a>

```python
from utz.proc import *
from z.proc import *

# Run a command
run('git', 'commit', '-m', 'message') # Commit staged changes
Expand All @@ -70,10 +71,10 @@ pipeline(['seq 10', 'head -n5']) # '1\n2\n3\n4\n5\n'

See also: [`test_proc.py`].

### [`utz.collections`]: Collection/list helpers <a id="utz.collections"></a>
### [`z.collections`]: Collection/list helpers <a id="z.collections"></a>

```python
from utz.collections import *
from z.collections import *

# Verify a collection has one element, return it
singleton(["aaa"]) # "aaa"
Expand All @@ -82,18 +83,20 @@ singleton(["aaa", "bbb"]) # error

See also: [`test_collections.py`].

### [`utz.cd`]: "change directory" contextmanager <a id="utz.cd"></a>
### [`z.cd`]: "change directory" contextmanager <a id="z.cd"></a>

```python
from utz import cd
from z import cd
with cd('..'): # change to parent dir
...
```

### [`utz.fn`]: decorator/function utilities <a id="utz.fn"></a>
### [`z.fn`]: decorator/function utilities <a id="z.fn"></a>

#### `z.decos`: compose decorators <a id="z.decos"></a>

#### `utz.decos`: compose decorators <a id="utz.decos"></a>
```python
from utz import decos
from z import decos
from click import option

common_opts = decos(
Expand All @@ -110,9 +113,10 @@ def subcmd2(n: int, v: bool):
...
```

#### `utz.call`: only pass expected `kwargs` to functions <a id="utz.call"></a>
#### `z.call`: only pass expected `kwargs` to functions <a id="z.call"></a>

```python
from utz import call, wraps
from z import call, wraps
def fn1(a, b):
...

Expand All @@ -124,9 +128,10 @@ call(fn1, **kwargs) # passes {a, b}, not {c, d}
call(fn2, **kwargs) # passes {a, b, c}, not {d}
```

### [`utz.env`]: `os.environ` wrapper + `contextmanager` <a id="utz.env"></a>
### [`z.env`]: `os.environ` wrapper + `contextmanager` <a id="z.env"></a>

```python
from utz import env, os
from z import env, os

# Temporarily set env vars
with env(FOO='bar'):
Expand All @@ -139,20 +144,22 @@ The `env()` contextmanager also supports configurable [`on_conflict`] and [`on_e

See also [`test_env.py`].

### [`utz.gzip`]: deterministic GZip helpers <a id="utz.gzip"></a>
### [`z.gzip`]: deterministic GZip helpers <a id="z.gzip"></a>

```python
from utz import deterministic_gzip_open, hash_file
from z import deterministic_gzip_open, hash_file
with deterministic_gzip_open('a.gz', 'w') as f:
f.write('\n'.join(map(str, range(10))))
hash_file('a.gz') # dfbe03625c539cbc2a2331d806cc48652dd3e1f52fe187ac2f3420dbfb320504
```

See also: [`test_gzip.py`].

### [`utz.plot`]: [Plotly] helpers <a id="utz.plots"></a>
### [`z.plot`]: [Plotly] helpers <a id="z.plots"></a>
Helpers for Plotly transformations I make frequently, e.g.:

```python
from utz import plot
from z import plot
import plotly.express as px
fig = px.bar(x=[1, 2, 3], y=[4, 5, 6])
plot(
Expand All @@ -167,10 +174,10 @@ plot(
)
```

Example usages: [hudcostreets/nj-crashes][hudcostreets/nj-crashes utz.plots], [ryan-williams/arrayloader-benchmarks][ryan-williams/arrayloader-benchmarks utz.plots].
Example usages: [hudcostreets/nj-crashes][hudcostreets/nj-crashes z.plots], [ryan-williams/arrayloader-benchmarks][ryan-williams/arrayloader-benchmarks z.plots].

### [`utz.setup`]: `setup.py` helper <a id="utz.setup"></a>
[`utz/setup.py`](src/utz/setup.py) provides defaults for various `setuptools.setup()` params:
### [`z.setup`]: `setup.py` helper <a id="z.setup"></a>
[`z/setup.py`](src/z/setup.py) provides defaults for various `setuptools.setup()` params:
- `name`: use parent directory name
- `version`: parse from git tag (otherwise from `git describe --tags`)
- `install_requires`: read `requirements.txt`
Expand All @@ -181,44 +188,45 @@ Example usages: [hudcostreets/nj-crashes][hudcostreets/nj-crashes utz.plots], [r

For an example, see [`gsmo==0.0.1`](https://github.com/runsascoded/gsmo/blob/v0.0.1/setup.py) ([and corresponding release](https://pypi.org/project/gsmo/)).

This library also "self-hosts" using `utz.setup`; see [pyproject.toml](pyproject.toml):
This library also "self-hosts" using `z.setup`; see [pyproject.toml](pyproject.toml):

```toml
[build-system]
requires = ["setuptools", "utz[setup]==0.4.2", "wheel"]
requires = ["setuptools", "z[setup]==0.4.2", "wheel"]
build-backend = "setuptools.build_meta"
```

and [setup.py](setup.py):

```python
from utz.setup import setup
from z.setup import setup

extras_require = {
# …
}

# Various fields auto-populated from git, README.md, requirements.txt, …
setup(
name="utz",
name="z",
version="0.8.0",
extras_require=extras_require,
url="https://github.com/runsascoded/utz",
url="https://github.com/runsascoded/z",
python_requires=">=3.10",
)

```

The `setup` helper can be installed via a pip "extra":
```bash
pip install utz[setup]
pip install z[setup]
```

### [`utz.test`]: `dataclass` test cases, `raises` helper <a id="utz.test"></a>
### [`z.test`]: `dataclass` test cases, `raises` helper <a id="z.test"></a>

#### `utz.parametrize`: [`pytest.mark.parametrize`] wrapper, accepts [`dataclass`] instances <a id="utz.parametrize"></a>
#### `z.parametrize`: [`pytest.mark.parametrize`] wrapper, accepts [`dataclass`] instances <a id="z.parametrize"></a>

```python
from utz import parametrize
from z import parametrize
from dataclasses import dataclass


Expand Down Expand Up @@ -251,22 +259,24 @@ def test_fn(f, fmt, expected):

[`test_parametrize.py`] contains more examples, customizing test "ID"s, adding parameter sweeps, etc.

#### `utz.raises`: `pytest.raises` wrapper, match a regex or multiple strings <a id="utz.raises"></a>
#### `z.raises`: `pytest.raises` wrapper, match a regex or multiple strings <a id="z.raises"></a>

### [`utz.time`]: `now`/`today` helpers <a id="utz.time"></a>
### [`z.time`]: `now`/`today` helpers <a id="z.time"></a>

`now` and `today` are wrappers around `datetime.datetime.now` that expose convenient functions:

```python
from utz import now, today
from z import now, today
now() # 2024-10-11T15:43:54Z
today() # 2024-10-11
now().s # 1728661583
now().ms # 1728661585952
```

Use in conjunction with [`utz.bases`] codecs for easy timestamp-nonces:
Use in conjunction with [`z.bases`] codecs for easy timestamp-nonces:

```python
from utz import b62, now
from z import b62, now
b62(now().s) # A18Q1l
b62(now().ms) # dZ3fYdS
b62(now().us) # G31Cn073v
Expand Down Expand Up @@ -319,60 +329,60 @@ Sample values for various units and codecs:

(generated by [`time-slug-grid.py`](scripts/time-slug-grid.py))

### [`utz.hash_file`](src/utz/hash.py) <a id="utz.hash_file"></a>
### [`z.hash_file`](src/z/hash.py) <a id="z.hash_file"></a>

```python
from utz import hash_file
from z import hash_file
hash_file("path/to/file") # sha256 by default
hash_file("path/to/file", 'md5')
```

### [`utz.docker`](src/utz/docker/), [`utz.tmpdir`](src/utz/tmpdir.py), etc. <a id="misc"></a>
### [`z.docker`](src/z/docker/), [`z.tmpdir`](src/z/tmpdir.py), etc. <a id="misc"></a>

Misc other modules:
- [o](src/utz/o.py): `dict` wrapper exposing keys as attrs (e.g.: `o({'a':1}).a == 1`)
- [docker](src/utz/docker/): DSL for programmatically creating Dockerfiles (and building images from them)
- [bases][`utz.bases`]: encode/decode in various bases (62, 64, 90, …)
- [tmpdir](src/utz/tmpdir.py): make temporary directories with a specific basename
- [escape](src/utz/escape.py): split/join on an arbitrary delimiter, with backslash-escaping; `utz.esc` escapes a specific character in a string.
- [ssh](src/utz/ssh.py): SSH tunnel wrapped in a context manager
- [backoff](src/utz/backoff.py): exponential-backoff utility
- [git](src/utz/git): Git helpers, wrappers around [GitPython](https://gitpython.readthedocs.io/en/stable/)
- [pnds](src/utz/pnds.py): [pandas](https://pandas.pydata.org/) imports and helpers
- [ctxs][`utz.ctxs`]: compose `contextmanager`s


[utz]: https://pypi.org/project/utz/
[extras]: https://github.com/runsascoded/utz/blob/main/setup.py#L3-L34
- [o](src/z/o.py): `dict` wrapper exposing keys as attrs (e.g.: `o({'a':1}).a == 1`)
- [docker](src/z/docker/): DSL for programmatically creating Dockerfiles (and building images from them)
- [bases][`z.bases`]: encode/decode in various bases (62, 64, 90, …)
- [tmpdir](src/z/tmpdir.py): make temporary directories with a specific basename
- [escape](src/z/escape.py): split/join on an arbitrary delimiter, with backslash-escaping; `z.esc` escapes a specific character in a string.
- [ssh](src/z/ssh.py): SSH tunnel wrapped in a context manager
- [backoff](src/z/backoff.py): exponential-backoff utility
- [git](src/z/git): Git helpers, wrappers around [GitPython](https://gitpython.readthedocs.io/en/stable/)
- [pnds](src/z/pnds.py): [pandas](https://pandas.pydata.org/) imports and helpers
- [ctxs][`z.ctxs`]: compose `contextmanager`s


[z]: https://pypi.org/project/z/
[extras]: https://github.com/runsascoded/z/blob/main/setup.py#L3-L34
[`stdlb`]: https://pypi.org/project/stdlb/
[`utz.proc`]: src/utz/proc/__init__.py
[`utz.process`]: src/utz/process/__init__.py
[`z.proc`]: src/z/proc/__init__.py
[`z.process`]: src/z/process/__init__.py
[`subprocess`]: https://docs.python.org/3/library/subprocess.html
[`test_proc.py`]: test/test_proc.py
[`utz.collections`]: src/utz/collections.py
[`z.collections`]: src/z/collections.py
[`test_collections.py`]: test/test_collections.py
[`utz.context`]: src/utz/context.py
[`utz.bases`]: src/utz/bases.py
[`utz.time`]: src/utz/time.py
[`z.context`]: src/z/context.py
[`z.bases`]: src/z/bases.py
[`z.time`]: src/z/time.py
[`test_context.py`]: test/test_context.py
[`utz.setup`]: src/utz/setup.py
[`utz.cd`]: src/utz/cd.py
[`utz.fn`]: src/utz/fn.py
[`utz.ctxs`]: src/utz/context.py
[`utz.gzip`]: src/utz/gzip.py
[`z.setup`]: src/z/setup.py
[`z.cd`]: src/z/cd.py
[`z.fn`]: src/z/fn.py
[`z.ctxs`]: src/z/context.py
[`z.gzip`]: src/z/gzip.py
[`test_gzip.py`]: test/test_gzip.py
[`utz.env`]: src/utz/environ.py
[`on_conflict`]: src/utz/environ.py#L9-13
[`on_exit`]: src/utz/environ.py#L16-19
[`z.env`]: src/z/environ.py
[`on_conflict`]: src/z/environ.py#L9-13
[`on_exit`]: src/z/environ.py#L16-19
[`test_env.py`]: test/test_env.py
[`utz.plot`]: src/utz/plots.py
[`z.plot`]: src/z/plots.py
[Plotly]: https://plotly.com/python/
[hudcostreets/nj-crashes utz.plots]: https://github.com/search?q=repo%3Ahudcostreets%2Fnj-crashes%20utz.plot&type=code
[ryan-williams/arrayloader-benchmarks utz.plots]: https://github.com/search?q=repo%3Aryan-williams%2Farrayloader-benchmarks%20utz.plot&type=code
[hudcostreets/nj-crashes z.plots]: https://github.com/search?q=repo%3Ahudcostreets%2Fnj-crashes%20z.plot&type=code
[ryan-williams/arrayloader-benchmarks z.plots]: https://github.com/search?q=repo%3Aryan-williams%2Farrayloader-benchmarks%20z.plot&type=code

[tdbs parametrize_cases]: https://github.com/single-cell-data/TileDB-SOMA/blob/1.14.2/apis/python/tests/parametrize_cases.py
[roundtrips]: https://github.com/single-cell-data/TileDB-SOMA/blob/1.14.2/apis/python/tests/test_dataframe_io_roundtrips.py
[`utz.test`]: src/utz/test.py
[`z.test`]: src/z/test.py
[`test_parametrize.py`]: test/test_parametrize.py
[`pytest.mark.parametrize`]: https://docs.pytest.org/en/stable/how-to/parametrize.html
[`dataclass`]: https://docs.python.org/3/library/dataclasses.html
Expand Down
2 changes: 1 addition & 1 deletion scripts/time-slug-grid.py
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env python
from bs4 import BeautifulSoup
from click import command
from utz import *
from z import *


@command()
Expand Down
Loading

0 comments on commit 56eac99

Please sign in to comment.