api: plan for narwhals.stable.v2

[MarcoGorelli]

I think we can aim for narwhals.stable.v2 some time in 2025

The main things I'd like to achieve are:

• remove "interchange" level of support, and instead only have:
  • "full": must support Series and have a defined row order (pandas, polars.DataFrame, pyarrow.Table, modin.DataFrame, cudf.DataFrame)
  • "lazy": must support collect, adding/removing/subsetting columns, filtering rows. Initially, at least, row-order-dependent operations (such as nw.cum_sum) will be left out. This will include Polars.LazyFrame, DuckDBPyRelation, pyspark.DataFrame, ibis.Table, dask.DataFrame
• make the breaking changes listed in breaking changes

Candidate changes:

• rework from_native. It will get simpler once we remove interchange-level things, but it's still fairly complicated. Not sure how much I like it to be completely honest, but I also haven't come up with anything better

Usage of narwhals.stable.v1 should remain unaffected

Tentative date: March 2025

[MarcoGorelli]

I think I'd also like to remove row-order-dependent expressions:

- arg_max - arg_min - cum_count - cum_max - cum_min - cum_prod - cum_sum - diff - drop_nulls - ewm_mean - filter - gather_every - head - is_first_distinct - is_last_distinct - mode - rolling_mean - rolling_std - rolling_sum - rolling_var - sample - shift - sort - unique

These can all stay available for Series, so people would still be able to use them for eager execution

All of these would of course stay available in narwhals.stable.v1

---

In #1689 (comment), we find:

• Plotly is using Expr.cum_sum
• scikit-lego is using Expr.shift

So, it would take more planning to be able to do this safely

Example (maybe?) of how shift_by could be implemented for Polars:

def shift_by(expr, by):
    idx = pl.arg_sort_by(by)
    return pl.col(expr).gather(idx).shift().sort_by(idx)

[MarcoGorelli]

🤔 fill_null with strategies like forward would also become problematic

it's debatable whether to have by kwargs (which would be required for the lazy-only layer) or to have *_by variants

[MarcoGorelli]

I just tried the following:

import polars as pl
import numpy as np

rng = np.random.default_rng(1)
N = 10_000_000
a = rng.normal(size=N)
b = rng.normal(size=N)
c = rng.normal(size=N)
data = {'a': a, 'b': b, 'c': c}
df_pl = pl.DataFrame(data)
df_pd = pd.DataFrame(data)

def shift_by(expr, by, lag):
    idx = pl.arg_sort_by(by)
    return pl.col(expr).gather(idx).shift(lag).sort_by(idx)

def add_lags_polars_1(df, cols, lags):
    return df.with_columns(
        pl.col(col).shift(-lag).alias(f'{col}_{lag}')
        for col in cols
        for lag in lags
    )

def add_lags_polars_2(df, cols, lags):
    return df.with_row_index().with_columns(
        shift_by(col, 'index', -lag).alias(f'{col}_{lag}')
        for col in cols
        for lag in lags
    )

def add_lags_polars_3(df, cols, lags):
    return df.with_columns(
        df[col].shift(-lag).alias(f'{col}_{lag}')
        for col in cols
        for lag in lags
    )

Timings:

• add_lags_polars_1 (current): 1.63ms
• add_lags_polars_2 (using shift_by): ~1300ms
• add_lags_polars_3 (using Series.shift): 4.3ms

So, I'm not sure it's really warranted to remove Expr.shift