Data validation from (secondary) external data source

[stephon-barrett]

Greetings,

I come with a question around utilizing multiple data sources in a single script. Say for example, I am using GE for connecting to a warehouse in AWS that I have created some expectations for and to create some further complex expectations, I need to assert some of the fields in the warehouse against a seperate db that lives somewhere else. Is it possible to connect to a secondary data source to compare and validate some fields against the first (the warehouse)? Does GE have support for this use case? If so, what docs should I be looking at on how to do so.

Thank you big!

[rdodev]

Hey @stephon-barrett 👋

If understand your inquiry correctly, there are couple of workarounds to accomplish what you are after. The caveat here is that the part of the code is in experimental state so YMMV.

One way to accomplish this is with a query store, as described in this document. Bear in mind this document is slightly out of date and step 2 needs to be done in V3 API style, but the rest of the document should work as-is as an example.

The other way to do it would be if the Checkpoint is all done dynamically with python code with use of Evaluation Parameters. Here, for example, you could define evaluation params in the suite then in the pipeline you pull the data from the secondary db (e.g. val_1) and then pass it in when running the checkpoint: context.run_checkpoint(evaluation_parameters={"val_1": val_1}, etc...).