Azure cloud based data cleaning pipeline
- I use a cloud based platform (Azure) and SQL for data cleaning and transferring to a Data Warehouse.
- The Pipeline outputs a Power BI Dashboard for automated data reporting.
- The Data is "Refreshed" and the most recent input data is displayed in the Power BI Dashboard.
- I host, maintain, and run two Azure SQL databases.
- Database 1: Raw Data Storage.
- Database 2: Data Warehouse.
- Stored procedures clean raw data and transfer clean data to the Data Warehouse. Learn More Here
- Azure Data Factory: a cloud based platform which ingests data, performs transfer functions, and triggers SQL stored procedures to manipulate data in a SQL database.
- Learn More Here
- POWER BI
- The last step in the data pipeline is to trigger a refresh of the target Power BI dataset. Whatever condition the target data table is in at the time of a refresh will transmit the corresponding information.
- Learn More Here