Great_Expectations Conditional Expectation in Spark 3.2.1 with Pandas API in DataBricks

[fullysane]

We want to implement Great_Expectations' in DataBricks with Conditional Expectation. According to GE's documentation https://docs.greatexpectations.io/docs/reference/expectations/conditional_expectations is only available for Pandas this argument must be set to "pandas" by default, thus, demanding the appropriate syntax. Other engines might be implemented in the future.

Does anyone knows if it can be implemented in Spark 3.2.1 which integrated with Pandas APIs? if not, any suggestion for handle conditional expectation in DataBricks with Spark 3.2.1?

[austiezr]

Hey @fullysane ! Thanks for reaching out about this. We do currently have experimental support for conditional expectations on spark; you will need to pass the condition_parser as great_expectations__experimental__. This is experimental functionality so you may see unexpected behavior from time to time.

[ashitabh]

Can you provide a working JSON config example as I am getting the following error?
"exception_message": "TypeError: expect_column_values_to_not_be_null() got an unexpected keyword argument 'condition_parser'",
"exception_traceback": "Traceback (most recent call last):\n File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/great_expectations/data_asset/data_asset.py", line 249, in wrapper\n return_obj = func(self, **evaluation_args)\n File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/great_expectations/dataset/sparkdf_dataset.py", line 106, in inner_wrapper\n success_df = func(self, col_df, *args, **kwargs)\nTypeError: expect_column_values_to_not_be_null() got an unexpected keyword argument 'condition_parser'\n"