great_expectations: Your data validation with guarantee on high quality data.

One way to validate data in real time is to use Cloud Pub/Sub and Cloud Functions in combination with great_expectations. Cloud Pub/Sub is a messaging service that enables decoupled and asynchronous communication between components of an application, while Cloud Functions is a serverless computing service that enables code execution in response to events.

In this use case, data is published to a cloud pub/sub topic and a cloud function is triggered to perform great_expectations validations on the incoming data. If the data passes validation, it can be stored in a database or sent to downstream processes. If the data does not pass validation, an alert can be sent to notify the appropriate personnel. This use case ensures that data quality issues can be identified early in the pipeline and resolved in real time, reducing the risk of making decisions based on inaccurate data.

A data lake is a central repository where data from various sources can be stored in its raw and unstructured format. great_expectations can be used to monitor data quality in a data lake by connecting to the storage layer of the data lake (e.g. Google Cloud Storage) and defining expectations for the data. Expectations can be defined for various aspects of the data, such as data types, value ranges, and relationships between columns.

great_expectations can then be set up to perform regular validations in the data lake and display any issues in the data quality dashboard. This use case helps ensure that data quality issues are identified early in the pipeline and that the data in the data lake is of high quality and suitable for use.

BigQuery is a serverless, highly scalable and cost-effective data warehouse that enables the analysis of large data sets with SQL-like queries. great_expectations can be used to automate data validation in BigQuery by defining expectations for the data. For this, a great_expectations data source must be created for BigQuery and the data source is used to perform validations on the data in BigQuery.

The results of the validations can be displayed in the data quality dashboard and can also trigger notifications if the data does not withstand the validations. In this setup, it is ensured that the data in BigQuery is accurate, reliable and consistent as well as reducing the need for manual data validation processes.

During data preprocessing, raw data is cleaned and converted into a format that can be used by downstream processes, such as machine learning models. great_expectations can be used to validate data in data preprocessing pipelines by defining expectations for the data and performing validations on the data as it passes through the pipeline. For example, if a machine learning model requires numerical data, great_expectations can be used to ensure that the data is indeed numerical before it is passed to the model. This deployment option helps ensure that the data used by downstream processes is of high quality and reduces the risk of inaccurate results.