-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
3 changed files
with
48 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,47 @@ | ||
| <!DOCTYPE html> | ||
| <html xmlns="http://www.w3.org/1999/xhtml"><head><title>Zarr (zarr.Zarr)</title><meta charset="utf-8"/><link rel="stylesheet" href="../../odoc.support/odoc.css"/><meta name="generator" content="odoc 2.4.2"/><meta name="viewport" content="width=device-width,initial-scale=1.0"/><script src="../../odoc.support/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body class="odoc"><nav class="odoc-nav"><a href="../index.html">Up</a> – <a href="../index.html">zarr</a> » Zarr</nav><header class="odoc-preamble"><h1>Module <code><span>Zarr</span></code></h1></header><div class="odoc-content"><div class="odoc-spec"><div class="spec module anchored" id="module-Node"><a href="#module-Node" class="anchor"></a><code><span><span class="keyword">module</span> <a href="Node/index.html">Node</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div><div class="spec-doc"><p>This module provides functionality for manipulating Zarr nodes.</p></div></div><div class="odoc-spec"><div class="spec module anchored" id="module-Indexing"><a href="#module-Indexing" class="anchor"></a><code><span><span class="keyword">module</span> <a href="Indexing/index.html">Indexing</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div><div class="spec-doc"><p>A module housing functions for creating and manipulating indices and slices for working with Zarr arrays.</p></div></div><div class="odoc-spec"><div class="spec module anchored" id="module-ArrayMetadata"><a href="#module-ArrayMetadata" class="anchor"></a><code><span><span class="keyword">module</span> <a href="ArrayMetadata/index.html">ArrayMetadata</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div><div class="spec-doc"><p>A module which contains functionality to work with a parsed JSON Zarr array metadata document.</p></div></div><div class="odoc-spec"><div class="spec module anchored" id="module-GroupMetadata"><a href="#module-GroupMetadata" class="anchor"></a><code><span><span class="keyword">module</span> <a href="GroupMetadata/index.html">GroupMetadata</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div><div class="spec-doc"><p>A module which contains functionality to work with a parsed JSON Zarr group metadata document.</p></div></div><div class="odoc-spec"><div class="spec module anchored" id="module-Storage"><a href="#module-Storage" class="anchor"></a><code><span><span class="keyword">module</span> <a href="Storage/index.html">Storage</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div></div><div class="odoc-spec"><div class="spec module anchored" id="module-Codecs"><a href="#module-Codecs" class="anchor"></a><code><span><span class="keyword">module</span> <a href="Codecs/index.html">Codecs</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div><div class="spec-doc"><p>An array has an associated list of codecs. Each codec specifies a bidirectional transform (an encode transform and a decode transform). This module contains building blocks for creating and working with a chain of codecs.</p></div></div></div></body></html> | ||
| <html xmlns="http://www.w3.org/1999/xhtml"><head><title>Zarr (zarr.Zarr)</title><meta charset="utf-8"/><link rel="stylesheet" href="../../odoc.support/odoc.css"/><meta name="generator" content="odoc 2.4.2"/><meta name="viewport" content="width=device-width,initial-scale=1.0"/><script src="../../odoc.support/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body class="odoc"><nav class="odoc-nav"><a href="../index.html">Up</a> – <a href="../index.html">zarr</a> » Zarr</nav><header class="odoc-preamble"><h1>Module <code><span>Zarr</span></code></h1><p><code>zarr</code> Provides an Ocaml implementation of the Zarr version 3 storage format specification. It supports creation of arrays and groups as well as chunking arrays along any dimension. One can store a Zarr hierarchy in memory or on disk. Zarr also supports reading zarr hierarchies created using other implementations, as long as they are spec-compliant.</p><p>Consult the <a href="#examples" title="examples">examples</a> and <a href="#limitations" title="limitations">limitations</a> for more info.</p></header><nav class="odoc-toc"><ul><li><a href="#references">References</a></li><li><a href="#node">Node</a></li><li><a href="#metadata">Metadata</a></li><li><a href="#storage">Storage</a></li><li><a href="#codecs">Codecs</a></li><li><a href="#indexing">Indexing</a></li><li><a href="#examples">Examples</a><ul><li><a href="#create_array">Create, read & write array.</a></li><li><a href="#sharding">Using sharding codec.</a></li><li><a href="#explore">Explore a Zarr hierarchy.</a></li></ul></li><li><a href="#extensions">Extension Points</a></li><li><a href="#limitations">Limitations</a></li></ul></nav><div class="odoc-content"><h4 id="references"><a href="#references" class="anchor"></a>References</h4><ul><li><a href="https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html">The Zarr Version 3 specification.</a></li><li><a href="https://zarr.dev/">Zarr community site.</a></li></ul><h2 id="node"><a href="#node" class="anchor"></a>Node</h2><div class="odoc-spec"><div class="spec module anchored" id="module-Node"><a href="#module-Node" class="anchor"></a><code><span><span class="keyword">module</span> <a href="Node/index.html">Node</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div><div class="spec-doc"><p>This module provides functionality for manipulating Zarr nodes.</p></div></div><h2 id="metadata"><a href="#metadata" class="anchor"></a>Metadata</h2><div class="odoc-spec"><div class="spec module anchored" id="module-ArrayMetadata"><a href="#module-ArrayMetadata" class="anchor"></a><code><span><span class="keyword">module</span> <a href="ArrayMetadata/index.html">ArrayMetadata</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div><div class="spec-doc"><p>A module which contains functionality to work with a parsed JSON Zarr array metadata document.</p></div></div><div class="odoc-spec"><div class="spec module anchored" id="module-GroupMetadata"><a href="#module-GroupMetadata" class="anchor"></a><code><span><span class="keyword">module</span> <a href="GroupMetadata/index.html">GroupMetadata</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div><div class="spec-doc"><p>A module which contains functionality to work with a parsed JSON Zarr group metadata document.</p></div></div><h2 id="storage"><a href="#storage" class="anchor"></a>Storage</h2><div class="odoc-spec"><div class="spec module anchored" id="module-Storage"><a href="#module-Storage" class="anchor"></a><code><span><span class="keyword">module</span> <a href="Storage/index.html">Storage</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div></div><h2 id="codecs"><a href="#codecs" class="anchor"></a>Codecs</h2><div class="odoc-spec"><div class="spec module anchored" id="module-Codecs"><a href="#module-Codecs" class="anchor"></a><code><span><span class="keyword">module</span> <a href="Codecs/index.html">Codecs</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div><div class="spec-doc"><p>An array has an associated list of codecs. Each codec specifies a bidirectional transform (an encode transform and a decode transform). This module contains building blocks for creating and working with a chain of codecs.</p></div></div><h2 id="indexing"><a href="#indexing" class="anchor"></a>Indexing</h2><div class="odoc-spec"><div class="spec module anchored" id="module-Indexing"><a href="#module-Indexing" class="anchor"></a><code><span><span class="keyword">module</span> <a href="Indexing/index.html">Indexing</a></span><span> : <span class="keyword">sig</span> ... <span class="keyword">end</span></span></code></div><div class="spec-doc"><p>A module housing functions for creating and manipulating indices and slices for working with Zarr arrays.</p></div></div><h2 id="examples"><a href="#examples" class="anchor"></a>Examples</h2><h3 id="create_array"><a href="#create_array" class="anchor"></a>Create, read & write array.</h3><pre class="language-ocaml"><code>open Zarr | ||
| open Zarr.Node | ||
| open Zarr.Codecs | ||
| open Zarr.Storage | ||
|
|
||
| let store = | ||
| Result.get_ok @@ FilesystemStore.open_or_create "testdata.zarr" in | ||
| let group_node = Result.get_ok @@ GroupNode.of_path "/some/group" in | ||
| FilesystemStore.create_group store group_node; | ||
| let array_node = Result.get_ok @@ ArrayNode.(group_node / "name") in | ||
| FilesystemStore.create_array | ||
| ~codecs:[`Transpose [|2; 0; 1|]; `Bytes BE; `Gzip L2] | ||
| ~shape:[|100; 100; 50|] | ||
| ~chunks:[|10; 15; 20|] | ||
| Bigarray.Float32 | ||
| Float.neg_infinity | ||
| array_node | ||
| store; | ||
| let slice = Owl_types.[|R [0; 20]; I 10; R []|] in | ||
| let x = | ||
| Result.get_ok @@ | ||
| FilesystemStore.get_array store array_node slice Bigarray.Float32 in | ||
| let x' = | ||
| Owl.Dense.Ndarray.Generic.map | ||
| (fun _ -> Owl_stats_dist.uniform_rvs 0. 10.) x in | ||
| FilesystemStore.set_array store array_node slice x';</code></pre><h3 id="sharding"><a href="#sharding" class="anchor"></a>Using sharding codec.</h3><pre class="language-ocaml"><code>let config = | ||
| {chunk_shape = [|5; 3; 5|] | ||
| ;codecs = [`Transpose [|2; 0; 1|]; `Bytes LE; `Gzip L5] | ||
| ;index_codecs = [`Bytes BE; `Crc32c] | ||
| ;index_location = Start} in | ||
| let shard_node = Result.get_ok @@ ArrayNode.(group_node / "another") in | ||
| FilesystemStore.create_array | ||
| ~codecs:[`ShardingIndexed config] | ||
| ~shape:[|100; 100; 50|] | ||
| ~chunks:[|10; 15; 20|] | ||
| Bigarray.Complex32 | ||
| Complex.zero | ||
| shard_node | ||
| store;</code></pre><h3 id="explore"><a href="#explore" class="anchor"></a>Explore a Zarr hierarchy.</h3><p>Functions to query a zarr hierarchy are provided. These include listing all nodes, finding children of a group node, resizing an array, deleting nodes, obtaining metadata of a node, and more.</p><pre class="language-ocaml"><code>let a, g = FilesystemStore.find_all_nodes store in | ||
| FilesystemStore.reshape store array_node [|25; 32; 10|]; | ||
| let meta = | ||
| Result.get_ok @@ FilesystemStore.group_metadata store group_node in | ||
| GroupMetadata.show meta; | ||
| FilesystemStore.array_exists store shard_node; | ||
| let a, g = FilesystemStore.find_child_nodes store group_node in | ||
| FilesystemStore.erase_group_node store group_node;</code></pre><h2 id="extensions"><a href="#extensions" class="anchor"></a>Extension Points</h2><p>This library also provides custom extensions not defined in the version 3 specification. These are tabulated below:</p><table class="odoc-table"><tr><th><p>Extension Point</p></th><th><p>Details</p></th></tr><tr><td><p>Data Types</p></td><td><p><code>char</code>, <code>complex32</code>, <code>int</code> (63-bit integer), <code>nativeint</code></p></td></tr></table><h2 id="limitations"><a href="#limitations" class="anchor"></a>Limitations</h2><p>Although this implementation tries to be spec compliant, it does come with a few limitations:</p><ul><li>Ocaml does not have support for unsigned integers and thus this library cannot support reading values of datatypes <code>uint32</code>, <code>uint64</code> and <code>complex128</code>.</li><li>This implementation does not support reading and writing Zarr heirarchies created using the version 1 or 2 specifications.</li></ul></div></body></html> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,2 @@ | ||
| <!DOCTYPE html> | ||
| <html xmlns="http://www.w3.org/1999/xhtml"><head><title>index (zarr.index)</title><meta charset="utf-8"/><link rel="stylesheet" href="../odoc.support/odoc.css"/><meta name="generator" content="odoc 2.4.2"/><meta name="viewport" content="width=device-width,initial-scale=1.0"/><script src="../odoc.support/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body class="odoc"><nav class="odoc-nav"><a href="../index.html">Up</a> – zarr</nav><header class="odoc-preamble"><h1 id="zarr-index"><a href="#zarr-index" class="anchor"></a>zarr index</h1></header><nav class="odoc-toc"><ul><li><a href="#library-zarr">Library zarr</a></li></ul></nav><div class="odoc-content"><h2 id="library-zarr"><a href="#library-zarr" class="anchor"></a>Library zarr</h2><p>The entry point of this library is the module: <a href="Zarr/index.html"><code>Zarr</code></a>.</p></div></body></html> | ||
| <html xmlns="http://www.w3.org/1999/xhtml"><head><title>index (zarr.index)</title><meta charset="utf-8"/><link rel="stylesheet" href="../odoc.support/odoc.css"/><meta name="generator" content="odoc 2.4.2"/><meta name="viewport" content="width=device-width,initial-scale=1.0"/><script src="../odoc.support/highlight.pack.js"></script><script>hljs.initHighlightingOnLoad();</script></head><body class="odoc"><nav class="odoc-nav"><a href="../index.html">Up</a> – zarr</nav><header class="odoc-preamble"><h1 id="the-zarr-library"><a href="#the-zarr-library" class="anchor"></a>The <code>zarr</code> library</h1><p>The Zarr library provides an OCaml implementation of the Zarr version 3 storage format specification for chunked & compressed multi-dimensional arrays, designed for use in parallel computing. The storage format is used by many companies including Google, NASA, Microsoft and <a href="https://zarr.dev/adopters/">many others</a>. Zarr's goal is to provide the following features:</p><ul><li>Chunk multi-dimensional arrays along any dimension.</li><li>Store arrays in memory, on disk, inside a Zip file or any remote storage backend.</li><li>Read and write arrays concurrently from multiple threads or processes.</li><li>Organize arrays into hierarchies using groups.</li></ul><p>See <a href="https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html">Zarr V3 specification</a>.</p><p>Author: Zolisa Bleki</p></header><nav class="odoc-toc"><ul><li><a href="#entry-point">Entry Point</a></li></ul></nav><div class="odoc-content"><h2 id="entry-point"><a href="#entry-point" class="anchor"></a>Entry Point</h2><p>The entry point of this library is the module <a href="Zarr/index.html"><code>Zarr</code></a>.</p></div></body></html> |