Does GX still support in memory spark sources?

[Carnegie118]

Hi

I used GX last year and it supported in memory spark processing. However, I'm struggling to find the equivalent feature in the v0.17.

Is it supported?

[Kilo59]

Yes.

https://docs.greatexpectations.io/docs/guides/connecting_to_your_data/fluent/in_memory/connect_in_memory_data

import great_expectations as gx
import pyspark.pandas as ps

context = gx.get_context()

datasource = context.sources.add_spark("my_spark_datasource")

data_asset = datasource.add_dataframe_asset(name="my_df_asset")

df = ps.DataFrame(
    {'a': [1, 2, 3, 4, 5, 6],
     'b': [100, 200, 300, 400, 500, 600],
     'c': ["one", "two", "three", "four", "five", "six"]},
    index=[10, 20, 30, 40, 50, 60])

my_batch_request = data_asset.build_batch_request(dataframe=dataframe)