Skip to content

Commit

Permalink
no more Ti (#479)
Browse files Browse the repository at this point in the history
* no more Ti

* go back to :time

* no Ti check

* use Dim{:time}

* adds dims

* add YAX

* adds dim Variables

* export it

* fix tests

* fix docs

* fix msc

* fixes doc strings

* no Rasters

* more docs
  • Loading branch information
lazarusA authored Dec 16, 2024
1 parent 718da93 commit 301ea46
Show file tree
Hide file tree
Showing 26 changed files with 295 additions and 211 deletions.
36 changes: 19 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,18 +107,20 @@ using YAXArrays
Let's assemble a `YAXArray` with 4 dimensions i.e. time, x,y and a variable dimension with two variables.

```julia
using YAXArrays, DimensionalData
using YAXArrays
using YAXArrays: YAXArrays as YAX

axlist = (
Dim{:time}(range(1, 20, length=20)),
X(range(1, 10, length=10)),
Y(range(1, 5, length=15)),
Dim{:Variable}(["var1", "var2"]))
YAX.time(range(1, 20, length=20)),
lon(range(1, 10, length=10)),
lat(range(1, 5, length=15)),
Variables(["var1", "var2"]))
```
```
↓ time 1.0:1.0:20.0,
X 1.0:1.0:10.0,
Y 1.0:0.2857142857142857:5.0,
Variable ["var1", "var2"]
lon 1.0:1.0:10.0,
lat 1.0:0.2857142857142857:5.0,
Variables ["var1", "var2"]
```

and the corresponding data.
Expand Down Expand Up @@ -148,8 +150,8 @@ ds = YAXArray(axlist, data, props)
│ 20×10×15×2 YAXArray{Float64,4} │
├────────────────────────────────┴─────────────────────────────────────────────── dims ┐
↓ time Sampled{Float64} 1.0:1.0:20.0 ForwardOrdered Regular Points,
X Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
Y Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
lon Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
lat Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
⬔ Variable Categorical{String} ["var1", "var2"] ForwardOrdered
├──────────────────────────────────────────────────────────────────────────── metadata ┤
Dict{String, String} with 5 entries:
Expand All @@ -168,17 +170,17 @@ ds = YAXArray(axlist, data, props)
For axis can be via `.`

```julia
ds.X
ds.lon
```
```
X Sampled{Float64} ForwardOrdered Regular Points
lon Sampled{Float64} ForwardOrdered Regular Points
wrapping: 1.0:1.0:10.0
```

or better yet via `lookup`

```julia
lookup(ds, :X)
lookup(ds, :lon)
```
```
Sampled{Float64} ForwardOrdered Regular Points
Expand All @@ -187,7 +189,7 @@ wrapping: 1.0:1.0:10.0

note that also the `.data` field can be use
```julia
lookup(ds, :X).data
lookup(ds, :lon).data
```
```
1.0:1.0:10.0
Expand All @@ -196,15 +198,15 @@ lookup(ds, :X).data
The data for one variables, i.e. `var1` can be accessed via:

```julia
ds[Variable=At("var1")]
ds[Variables=At("var1")]
```
```
╭──────────────────────────────╮
│ 20×10×15 YAXArray{Float64,3} │
├──────────────────────────────┴────────────────────────────────────────────── dims ┐
↓ time Sampled{Float64} 1.0:1.0:20.0 ForwardOrdered Regular Points,
X Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
Y Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points
lon Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
lat Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points
├───────────────────────────────────────────────────────────────────────── metadata ┤
Dict{String, String} with 5 entries:
"var1" => "one of your variables"
Expand Down
1 change: 0 additions & 1 deletion docs/Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ MultivariateStats = "6f286f6a-111f-5878-ab1e-185364afe411"
NetCDF = "30363a11-5582-574a-97bb-aa9a979735b9"
OnlineStats = "a15396b6-48d5-5d58-9928-6d29437db91e"
PlotUtils = "995b91a9-d308-5afd-9ec6-746e21dbc043"
Rasters = "a3a2b9e3-a471-40c9-b274-f788e487c689"
SkipNan = "aed68c70-c8b0-4309-8cd1-d392a74f991a"
Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
TimeSeries = "9e3dc215-6440-5c97-bce1-76c03772f85e"
Expand Down
12 changes: 7 additions & 5 deletions docs/src/UserGuide/combine.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,10 @@ The resulting array `whole_year` still has one dimension, i.e. time, but with 12

````@example cat
using YAXArrays
using YAXArrays: YAXArrays as YAX
first_half = YAXArray((Dim{:time}(1:6),), rand(6))
second_half = YAXArray((Dim{:time}(7:12),), rand(6))
first_half = YAXArray((YAX.time(1:6),), rand(6))
second_half = YAXArray((YAX.time(7:12),), rand(6))
whole_year = cat(first_half, second_half, dims = 1)
````

Expand All @@ -25,10 +26,11 @@ Note that using a `Dataset` instead is a more flexible approach in handling diff

````@example concatenatecubes
using YAXArrays
using YAXArrays: YAXArrays as YAX
temperature = YAXArray((Dim{:time}(1:6),), rand(6))
precipitation = YAXArray((Dim{:time}(1:6),), rand(6))
temperature = YAXArray((YAX.time(1:6),), rand(6))
precipitation = YAXArray((YAX.time(1:6),), rand(6))
cubes = [temperature,precipitation]
var_axis = Dim{:variable}(["temp", "prep"])
var_axis = Variables(["temp", "prep"])
combined = concatenatecubes(cubes, var_axis)
````
50 changes: 28 additions & 22 deletions docs/src/UserGuide/compute.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,13 @@ Let's start by creating an example dataset:

````@example compute
using YAXArrays
using YAXArrays: YAXArrays as YAX
using Dates
axlist = (
Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-01-30")),
Dim{:lon}(range(1, 10, length=10)),
Dim{:lat}(range(1, 5, length=15)),
YAX.time(Date("2022-01-01"):Day(1):Date("2022-01-30")),
lon(range(1, 10, length=10)),
lat(range(1, 5, length=15)),
)
data = rand(30, 10, 15)
properties = Dict(:origin => "user guide")
Expand Down Expand Up @@ -103,6 +104,7 @@ Here, we will define a simple function, that will take as input several `YAXArra

````@example mapCube
using YAXArrays, Zarr
using YAXArrays: YAXArrays as YAX
using Dates
````

Expand Down Expand Up @@ -130,23 +132,23 @@ Note the `.` after `f`, this is because we will slice across time, namely, the f
Here, we do create `YAXArrays` only with the desired dimensions as

````@ansi mapCube
lon = YAXArray(Dim{:lon}(range(1, 15)))
lat = YAXArray(Dim{:lat}(range(1, 10)))
lon_yax = YAXArray(lon(range(1, 15)))
lat_yax = YAXArray(lat(range(1, 10)))
````

And a time Cube's Axis

````@example mapCube
tspan = Date("2022-01-01"):Day(1):Date("2022-01-30")
time = YAXArray(Dim{:time}(tspan))
time_yax = YAXArray(YAX.time(tspan))
````

note that the following can be extended to arbitrary `YAXArrays` with additional data and dimensions.

Let's generate a new `cube` using `mapCube` and saving the output directly into disk.

````@ansi mapCube
gen_cube = mapCube(g, (lon, lat, time);
gen_cube = mapCube(g, (lon_yax, lat_yax, time_yax);
indims = (InDims(), InDims(), InDims("time")),
outdims = OutDims("time", overwrite=true, path="my_gen_cube.zarr", backend=:zarr,
outtype = Float32)
Expand All @@ -170,7 +172,7 @@ gen_cube.data[1, :, :]
but, we can generate a another cube with a different `output order` as follows

````@ansi mapCube
gen_cube = mapCube(g, (lon, lat, time);
gen_cube = mapCube(g, (lon_yax, lat_yax, time_yax);
indims = (InDims("lon"), InDims(), InDims()),
outdims = OutDims("lon", overwrite=true, path="my_gen_cube.zarr", backend=:zarr,
outtype = Float32)
Expand All @@ -197,15 +199,17 @@ which outputs the same as the `gen_cube.data[1, :, :]` called above.
Here, we will consider different scenarios, namely how we deal with different input cubes and how to specify the output ones. We will illustrate this with the following test example and the subsequent function definitions.

````@example outdims
using YAXArrays, Dates
using YAXArrays
using YAXArrays: YAXArrays as YAX
using Dates
using Zarr
using Random
axlist = (
Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-01-05")),
Dim{:lon}(range(1, 4, length=4)),
Dim{:lat}(range(1, 3, length=3)),
Dim{:variables}(["a", "b"])
YAX.time(Date("2022-01-01"):Day(1):Date("2022-01-05")),
lon(range(1, 4, length=4)),
lat(range(1, 3, length=3)),
Variables(["a", "b"])
)
Random.seed!(123)
Expand Down Expand Up @@ -308,7 +312,7 @@ Here, the goal is to operate at the pixel level (longitude, latitude), and then
Random.seed!(123)
data = rand(3.0:5.0, 5, 4, 3)
axlist = (Dim{:lon}(1:4), Dim{:lat}(1:3), Dim{:depth}(1:7),)
axlist = (lon(1:4), lat(1:3), Dim{:depth}(1:7),)
yax_2d = YAXArray(axlist, rand(-3.0:0.0, 4, 3, 7))
````

Expand All @@ -318,8 +322,8 @@ and
Random.seed!(123)
data = rand(3.0:5.0, 5, 4, 3)
axlist = (Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-01-05")),
Dim{:lon}(1:4), Dim{:lat}(1:3),)
axlist = (YAX.time(Date("2022-01-01"):Day(1):Date("2022-01-05")),
lon(1:4), lat(1:3),)
properties = Dict("description" => "multi dimensional test cube")
yax_test = YAXArray(axlist, data, properties)
Expand Down Expand Up @@ -359,13 +363,14 @@ First, create the raster array:

````@example compute_mapcube
using YAXArrays
using YAXArrays: YAXArrays as YAX
using DimensionalData
using Dates
axlist = (
Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-01-30")),
Dim{:lon}(range(1, 10, length=10)),
Dim{:lat}(range(1, 5, length=15)),
YAX.time(Date("2022-01-01"):Day(1):Date("2022-01-30")),
lon(range(1, 10, length=10)),
lat(range(1, 5, length=15)),
)
data = rand(30, 10, 15)
raster_arr = YAXArray(axlist, data)
Expand Down Expand Up @@ -432,13 +437,14 @@ The following code does a time mean over all grid points using multiple CPUs of

````julia
using YAXArrays
using YAXArrays: YAXArrays as YAX
using Dates
using Distributed

axlist = (
Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-01-30")),
Dim{:lon}(range(1, 10, length=10)),
Dim{:lat}(range(1, 5, length=15)),
YAX.time(Date("2022-01-01"):Day(1):Date("2022-01-30")),
lon(range(1, 10, length=10)),
lat(range(1, 5, length=15)),
)
data = rand(30, 10, 15)
properties = Dict(:origin => "user guide")
Expand Down
4 changes: 2 additions & 2 deletions docs/src/UserGuide/convert.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ m2 = collect(a.data)

A `Raster` as defined in [Rasters.jl](https://rafaqz.github.io/Rasters.jl/stable/) has a same supertype of a `YAXArray`, i.e. `AbstractDimArray`, allowing easy conversion between those types:

````@example convert
````julia
using Rasters

lon, lat = X(25:1:30), Y(25:1:30)
Expand All @@ -43,7 +43,7 @@ ras = Raster(rand(lon, lat, time))
a = YAXArray(dims(ras), ras.data)
````

````@example convert
````julia
ras2 = Raster(a)
````

Expand Down
8 changes: 5 additions & 3 deletions docs/src/UserGuide/create.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ We can create a new YAXArray by filling the values directly:

````@example create
using YAXArrays
using YAXArrays: YAXArrays as YAX
a1 = YAXArray(rand(10, 20, 5))
````

Expand All @@ -18,9 +20,9 @@ We can also specify the dimensions with custom names enabling easier access:
using Dates
axlist = (
Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-01-30")),
Dim{:lon}(range(1, 10, length=10)),
Dim{:lat}(range(1, 5, length=15)),
YAX.time(Date("2022-01-01"):Day(1):Date("2022-01-30")),
lon(range(1, 10, length=10)),
lat(range(1, 5, length=15)),
)
data2 = rand(30, 10, 15)
properties = Dict(:origin => "user guide")
Expand Down
20 changes: 12 additions & 8 deletions docs/src/UserGuide/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,12 @@ It is possible to concatenate several cubes that shared the same dimensions usin
Let's create two dummy cubes
````@example howdoi
using YAXArrays
using YAXArrays: YAXArrays as YAX
axlist = (
Dim{:time}(range(1, 20, length=20)),
Dim{:lon}(range(1, 10, length=10)),
Dim{:lat}(range(1, 5, length=15))
YAX.time(range(1, 20, length=20)),
lon(range(1, 10, length=10)),
lat(range(1, 5, length=15))
)
data1 = rand(20, 10, 15)
Expand Down Expand Up @@ -125,7 +127,7 @@ t = Date("2020-01-01"):Month(1):Date("2022-12-31")
create YAXArray axes

```@example howdoi
axes = (Dim{:Lon}(1:10), Dim{:Lat}(1:10), Dim{:Time}(t))
axes = (Lon(1:10), Lat(1:10), YAX.Time(t))
```

create the YAXArray
Expand Down Expand Up @@ -169,7 +171,7 @@ using Dates # To generate the dates of the time axis
using DimensionalData # To use the "Between" option for selecting data
t = Date("2020-01-01"):Month(1):Date("2022-12-31")
axes = (Dim{:Lon}(1:10), Dim{:Lat}(1:10), Dim{:Time}(t))
axes = (Lon(1:10), Lat(1:10), YAX.Time(t))
var1 = YAXArray(axes, reshape(1:3600, (10, 10, 36)))
var2 = YAXArray(axes, reshape((1:3600)*5, (10, 10, 36)))
Expand All @@ -195,7 +197,7 @@ using DimensionalData # To use the "Between" selector for selecting data
t = Date("2020-01-01"):Month(1):Date("2022-12-31")
common_axis = Dim{:points}(1:100)
time_axis = Dim{:Time}(t)
time_axis = YAX.Time(t)
# Note that longitudes and latitudes are not dimensions, but YAXArrays
longitudes = YAXArray((common_axis,), rand(1:369, 100)) # 100 random values taken from 1 to 359
Expand Down Expand Up @@ -324,7 +326,9 @@ You will not be able to save this dataset, first you will need to rename those `
In this section we will use `MarketData.jl` and `TimeSeries.jl` to simulate some stocks.

````@example howdoi
using YAXArrays, DimensionalData
using YAXArrays
using YAXArrays: YAXArrays as YAX
using DimensionalData
using MarketData, TimeSeries
stocks = Dict(:Stock1 => random_ohlcv(), :Stock2 => random_ohlcv(), :Stock3 => random_ohlcv())
Expand All @@ -334,7 +338,7 @@ d_keys = keys(stocks)
currently there is not direct support to obtain `dims` from a `TimeArray`, but we can code a function for it

````@example howdoi
getTArrayAxes(ta::TimeArray) = (Dim{:time}(timestamp(ta)), Dim{:variable}(colnames(ta)), );
getTArrayAxes(ta::TimeArray) = (YAX.time(timestamp(ta)), Variables(colnames(ta)), );
nothing # hide
````
then, we create the `YAXArrays` as
Expand Down
Loading

0 comments on commit 301ea46

Please sign in to comment.