BigQuery Temp Table Support

[Shinnnyshinshin]

BigQuery temp tables

By default, Great Expectations will attempt to improve performance for most sql backends by creating temporary tables as the result of queries. This behavior can be modified using the boolean flag create_temp_table, which is passed in as a batch_spec_passthrough parameter.

In the case of BigQuery, which historically did not support the creation of temp tables, Great Expectations allowed for a named permanent table to be used as a "temporary" table, with the name passed in as a batch_spec_passthrough parameter.

This meant that BigQuery users were expected to either:

• Keep the resulting tables.
• Manually run a script to delete tables that were created by Great Expectations.
• Set the Database to automatically delete tables after a set period of time.

As of Great Expectations 0.15.3, users no longer have to set the bigquery_temp_table parameter, but Great Expectations will automatically set tables that are created as the result of queries to expire after 1 day.

What does this mean for me?

• CLI no longer asks for bigquery_temp_table parameter when setting up a BigQuery datasource (for V2 and V3).
• Great Expectations will now emit a DeprecationWarning if you try to pass it in a bigquery_temp_table parameter.

Additional Information

Currently BigQuery does support temp tables, but in a limited scope (either within a script or a session), neither are fully compatible with GE, as it interacts with DB back-ends through SqlAlchemy engine objects. Therefore, the decision was made to use Google Data Definition Language to create a table and set the expiration at creation time (1 day).

More information can be found in this very helpful StackOverflow thread

[camilotorresmestra]

Is there any way (from the client side) to configure the temporary table with a different expiration time than 1 day?