Data Types from Parquet file

[jschenz-chwy]

Hello all,

I am trying to run a data assistant against a parquet file that I have exported from Snowflake, and I am having no luck in getting this to work. I am new to GE, so there may be a simple solution that I am not aware of. The question that I am hoping to find some help with is this: Is there a workable approach to running a data assistant against a parquet file?

Python version 3.10.5
great_expectations, version 0.15.34

What I am seeing is that numeric and date columns in my dataset are all raising the same error with either TypeError: object of type \'decimal.Decimal\' has no len() or TypeError: object of type \'datetime.date\' has no len().

This feels like there is some improper type inference going on somewhere, and that great expectations and pandas have two distinct understandings of what the data look like.

Here is the code that I am trying to run:

context = ge.data_context.DataContext()

batch_request = {
    "datasource_name": "my-dataset",
    "data_connector_name": "default_inferred_data_connector_name",
    "data_asset_name": "my-dataset.snappy.parquet",
    "limit": 1000,
}

expectation_suite_name = "my-dataset.snappy.parquet.warning"

validator = context.get_validator(
    batch_request=BatchRequest(**batch_request),
    expectation_suite_name=expectation_suite_name,
)

exclude_column_names = []

result = context.assistants.onboarding.run(
    batch_request=batch_request,
    exclude_column_names=exclude_column_names,
)
validator.expectation_suite = result.get_expectation_suite(
    expectation_suite_name=expectation_suite_name
)

But when I do that, I am getting these errors from pandas:

2022-11-25T11:33:38-0600 - WARNING - Exceptions
{('column_values.length.min', '5127bd6996719565f7ee3f23eece478b', ()): {'metric_configuration': {
  "metric_name": "column_values.length.min",
  "metric_domain_kwargs": {
    "column": "INCOME_CODE",
    "batch_id": "2824d836cd30cab12e7469915579cbba"
  },
  "metric_domain_kwargs_id": "5127bd6996719565f7ee3f23eece478b",
  "metric_value_kwargs": {},
  "metric_value_kwargs_id": [],
  "id": [
    "column_values.length.min",
    "5127bd6996719565f7ee3f23eece478b",
    []
  ]
}, 'num_failures': 3, 'exception_info': {{'exception_traceback': 'Traceback (most recent call last):
  File "/.../great_expectations/execution_engine/execution_engine.py", line 407, in resolve_metrics
    resolved_metrics[metric_to_resolve.id] = metric_fn(
  File "/.../great_expectations/expectations/metrics/metric_provider.py", line 34, in inner_func
    return metric_fn(*args, **kwargs)
  File "/.../great_expectations/expectations/metrics/column_aggregate_metric_provider.py", line 84, in inner_func
    return metric_fn(
  File "/.../great_expectations/expectations/metrics/column_aggregate_metrics/column_values_length_min.py", line 21, in _pandas
    return column.map(len).min()
  File "/.../pandas/core/series.py", line 4539, in map
    new_values = self._map_values(arg, na_action=na_action)
  File "/.../pandas/core/base.py", line 890, in _map_values
    new_values = map_f(values, mapper)
  File "pandas/_libs/lib.pyx", line 2918, in pandas._libs.lib.map_infer
TypeError: object of type \'decimal.Decimal\' has no len()

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/.../great_expectations/validator/validation_graph.py", line 178, in resolve_validation_graph
    self._execution_engine.resolve_metrics(
  File "/.../great_expectations/execution_engine/execution_engine.py", line 411, in resolve_metrics
    raise ge_exceptions.MetricResolutionError(
great_expectations.exceptions.exceptions.MetricResolutionError: object of type \'decimal.Decimal\' has no len()', 
'exception_message': "object of type 'decimal.Decimal' has no len()", 'raised_exception': True}}}}
occurred while resolving metrics.
2022-11-25T11:33:38-0600 - WARNING - column_values.length.min was not found in the resolved Metrics for ParameterBuilder.