Split Connector and Hubs PBI reports and switch to FOCUS exports (#530)

docs/changelog.md

@@ -17,6 +17,7 @@ Explore the latest and greatest features and enhancements from the FinOps toolki
    <summary class="fs-2 text-uppercase">On this page</summary>
 
 - [🔄️ Unreleased](#️-unreleased)
+- [🛠️ v0.2](#️-v02)
 - [🛠️ v0.1.1](#️-v011)
 - [🚚 v0.1](#-v01)
 - [🌱 v0.0.1](#-v001)
@@ -89,12 +90,17 @@ Legend:
 📊 Power BI reports
 {: .fs-5 .fw-500 .mt-4 mb-0 }
 
+> ➕ Added:
+>
+> 1. [Cost Management connector report](./power-bi/connector.md) to support the Cost Management connector.
+>
 > ✏️ Changed:
 >
 > 1. Updated [Cost summary](./power-bi/cost-summary.md) and [Commitment discounts](./power-bi/commitment-discounts.md) reports to [FOCUS 1.0 preview](./focus/README.md).
-> 2. Removed unused custom visualizations.
-> 3. Organized setup instructions in Cost summary to match other reports.
-> 4. Updated troubleshooting documentation.
+> 2. Updated [Cost summary](./power-bi/cost-summary.md) and [Commitment discounts](./power-bi/commitment-discounts.md) reports to only use [FinOps hubs](./finops-hub/README.md).
+> 3. Removed unused custom visualizations.
+> 4. Organized setup instructions in Cost summary to match other reports.
+> 5. Updated troubleshooting documentation.
 >
 > 🛠️ Fixed:
 >
@@ -105,29 +111,33 @@ Legend:
 > 4. Fixed null error when Billing Account ID is empty ([#473](https://github.com/microsoft/finops-toolkit/issues/473)).
 > 5. Added missing commitment discount refunds to the actual cost data ([#447](https://github.com/microsoft/finops-toolkit/issues/447)).
 
+🏦 FinOps hubs
+{: .fs-5 .fw-500 .mt-4 mb-0 }
+
+> 🛠️ Fixed:
+>
+> 1. Fixed error in some China regions where deployment scripts were not supported ([#259](https://github.com/microsoft/finops-toolkit/issues/259)).
+>
+> ✏️ Changed:
+>
+> 1. Switch from amortized cost exports to FOCUS cost exports.
+> 2. Updated ingestion container month folders from `yyyyMMdd-yyyyMMdd` to `yyyyMM`.
+> 3. Renamed **msexports_extract** pipeline to **msexports_ExecuteETL**.
+> 4. Renamed **msexports_transform** pipeline to **msexports_ETL_ingestion**.
+
 🖥️ PowerShell
 {: .fs-5 .fw-500 .mt-4 mb-0 }
 
-> ➕ Added:
+> 🛠️ Fixed:
 >
-> 1. New-FinOpsCostExport
-> 2. Remove-FinOpsHubScope
+> 1. Fixed typo in Deploy-FinOpsHub causing it to fail.
 >
 > 🗑️ Removed:
 >
 > 1. `ConvertTo-FinOpsSchema` and `Invoke-FinOpsSchemaTransform` are no longer being maintained and will be removed in a future update.
 >    - With native support for FOCUS 1.0 preview in Cost Management, we are deprecating both commands, which only support FOCUS 0.5.
 >    - If you would like to see the PowerShell commands updated to 1.0 preview, please let us know in discussions or via a GitHub issue.
 
-🌐 Open data
-{: .fs-5 .fw-500 .mt-4 mb-0 }
-
-> ➕ Added:
->
-> 1. [Resource types](./open-data/README.md#-resource-types) to map Azure resource types to friendly display names.
-> 2. [Get-FinOpsResourceType](./powershell/data/Get-FinOpsResourceType.md) PowerShell command to support resource type to display name mapping.
-> 3. [Sample exports](./open-data/README.md#-sample-data) for each of the datasets that can be exported from Cost Management.
-
 <br>
 
 ## 🛠️ v0.1.1

docs/finops-hub/README.md

@@ -62,15 +62,16 @@ FinOps hubs will streamline implementing the FinOps Framework, are being designe
 - Clean up duplicated data in daily Cost Management exports (and save money on storage).
 - Convert exported data to parquet for faster data access.
 - Connect Power BI to subscriptions, resource groups, and other scopes.
-- Connect Power BI to Azure Government and Microsoft Online Services Agreement accounts.
+- Connect Power BI to Azure Government and Azure China.
+- Connect Power BI to Microsoft Online Services Agreement (MOSA) subscriptions<sup>1</sup>.
 - Report on multiple subscriptions, resource groups, or billing accounts.
-- Preview cost data in the [FinOps Open Cost and Usage Specification (FOCUS)](https://focus.finops.org) schema.
+- Streamlined deployment and management with PowerShell.
+- Full alignment with the [FinOps Open Cost and Usage Specification (FOCUS)](../focus/README.md).
 - _Coming soon: Ingest data from subscriptions in multiple tenants into a single storage account._
-- _Coming soon: Normalize cost data across account types._
-- _Coming soon: Connect Power BI to Azure China accounts._
-- _Coming soon: Streamlined deployment and management with PowerShell._
 - _Coming soon: Ingest data into Azure Data Explorer._
 
+_<sup>1) MOSA (or PAYG) subscriptions are only supported in FinOps hubs 0.1.x. FinOps hubs 0.2 requires FOCUS cost data from Cost Management exports, which are not supported for MOSA subscriptions. Please contact support about transitioning to a Microsoft Customer Agreement account.</sup>_
+
 <br>
 
 ## 📦 What's included
@@ -85,7 +86,7 @@ Once deployed, you can report on the data in Power BI or by connecting to the st
 
 <img alt="Screenshot of the cost summary report" style="max-width:200px" src="https://user-images.githubusercontent.com/399533/216882658-45f026f1-c895-48ca-81e2-35765af8e29e.png">
 <img alt="Screenshot of the services cost report" style="max-width:200px" src="https://user-images.githubusercontent.com/399533/216882700-4e04b589-0580-4e49-9b40-9f5948792975.png">
-<img alt="Screenshot of the commitment-based discounts coverage report" style="max-width:200px" src="https://user-images.githubusercontent.com/399533/216882916-bb7ecfa3-d092-4ae2-88e1-7a0425c14dca.png">
+<img alt="Screenshot of the commitment discounts coverage report" style="max-width:200px" src="https://user-images.githubusercontent.com/399533/216882916-bb7ecfa3-d092-4ae2-88e1-7a0425c14dca.png">
 
 [Browse reports](../power-bi/README.md){: .btn .mt-2 .mb-4 .mb-md-0 .mr-4 }
 [See the template](./template.md){: .btn .mt-2 .mb-4 .mb-md-0 .mr-4 }
@@ -99,10 +100,18 @@ Once deployed, you can report on the data in Power BI or by connecting to the st
 
    [![Deploy To Azure](https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/deploytoazure.svg?sanitize=true)](https://aka.ms/finops/hubs/deploy) &nbsp; [![Deploy To Azure US Gov](https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/deploytoazuregov.svg?sanitize=true)](https://aka.ms/finops/hubs/deploy/gov) &nbsp; [![Deploy To Azure China](https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/1-CONTRIBUTION-GUIDE/images/deploytoazurechina.svg?sanitize=true)](https://aka.ms/finops/hubs/deploy/china)
 
-3. [Create a new cost export](https://learn.microsoft.com/azure/cost-management-billing/costs/tutorial-export-acm-data?tabs=azure-portal) using the following settings:
+3. [Create a new FOCUS cost export](https://aka.ms/exportsv2) using the following settings:
+
+   <!-- TODO: Replace the portal link with the docs link when exports v2 docs are available.
+   1. [Create a new FOCUS cost export](https://learn.microsoft.com/azure/cost-management-billing/costs/tutorial-export-acm-data?tabs=azure-portal) using the following settings:
+   -->
 
-   - **Metric** = `Amortized cost`
-   - **Export type** = `Daily export of month-to-date costs`
+   - **Type of data** = `Cost and usage details (FOCUS)`
+     <blockquote class="important" markdown="1">
+       _FinOps hubs 0.2 requires FOCUS cost data. White FOCUS is fully supported, the option to export FOCUS cost data from Cost Management is currently in preview and has not rolled out to everyone yet. In order to create and manage FOCUS exports, please use the [Exports preview link](https://aka.ms/exportsv2)._
+     </blockquote>
+   - **Dataset version** = `1.0-preview (v1)`
+   - **Frequency** = `Daily export of month-to-date costs`
      <blockquote class="tip" markdown="1">
        _Configuring a daily export starts in the current month. If you want to backfill historical data, create a one-time export and set the start/end dates to the desired date range._
      </blockquote>
@@ -138,31 +147,23 @@ After deploying a hub instance, there are several ways for you to get started:
 
    If you'd like to create your own reports or add cost data to an existing report, you can either [copy queries from a pre-built report](../power-bi/README.md#setup-a-finops-hub-report) or [connect manually](../power-bi/README.md#connect-manually) using the Azure Data Lake Storage Gen2 connector.
 
-   <!-- NOTE TO CONTRIBUTORS: Keep this info note in sync with the same one under #3 below. -->
+3. Connect to Microsoft Fabric for advanced queries.
 
-   <blockquote class="note" markdown="1">
-     _The schema may change multiple times before the 0.1 release. We will ensure Power BI reports have backwards compatibility, but if you access data directly, you may run into breaking changes with new releases. Familiarize yourself with [upcoming releases](https://aka.ms/finops/toolkit/roadmap) and review the [changelog](../changelog.md) for breaking changes before you update._
-   </blockquote>
+   If you use OneLake in Microsoft Fabric, you can create a shortcut to the `ingestion` container in your hubs storage account to run SQL or KQL queries directly against the data in hubs. [Learn more](https://learn.microsoft.com/fabric/real-time-analytics/onelake-shortcuts?tabs=adlsgen2).
 
-3. Access the cost data from custom tools.
+4. Access the cost data from custom tools.
 
    Cost data is stored in an [Azure Data Lake Storage Gen2](https://learn.microsoft.com/azure/storage/blobs/data-lake-storage-introduction) account. You can use any tool that supports Azure Data Lake Storage Gen2 to access the data. Refer to the [data dictionary](./data-dictionary.md) for details about available columns.
 
-   <!-- NOTE TO CONTRIBUTORS: Keep this info note in sync with the same one under #2 above. -->
-
-   <blockquote class="note" markdown="1">
-     _The schema may change multiple times before the 0.1 release. We will ensure Power BI reports have backwards compatibility, but if you access data directly, you may run into breaking changes with new releases. Familiarize yourself with [upcoming releases](https://aka.ms/finops/toolkit/roadmap) and review the [changelog](changelog.md) for breaking changes before you update._
-   </blockquote>
-
-4. Apply cost allocation logic, augment, or manipulate your cost data using Data Factory.
+5. Apply cost allocation logic, augment, or manipulate your cost data using Data Factory.
 
    [Data Factory](https://learn.microsoft.com/azure/data-factory/introduction) is used to ingest and transform data. We recommend using Data Factory as a cost-efficient solution to apply custom logic to your cost data. Do not modify built-in pipelines or data in the **msexports** container. If you create custom pipelines, monitor new data in the **ingestion** container and use a consistent prefix to ensure they don't overlap with new pipelines. Refer to [data processing](./data-processing.md) for details about how data is processed.
 
    <blockquote class="important" markdown="1">
      _Keep in mind this is the primary area we are planning to evolve in [upcoming FinOps toolkit releases](https://aka.ms/finops/toolkit/roadmap). Please familiarize yourself with our roadmap to avoid conflicts with future updates. Consider [contributing to the project](../CONTRIBUTING.md) to add support for new scenarios to avoid conflicts._
    </blockquote>
 
-5. Generate custom alerts using Power Automate.
+6. Generate custom alerts using Power Automate.
 
    You have many options for generating custom alerts. [Power Automate](https://powerautomate.microsoft.com/connectors/details/shared_azureblob/azure-blob-storage) is a great option for people who are new to automation but you can also use [Data Factory](https://learn.microsoft.com/azure/data-factory/introduction), [Functions](https://learn.microsoft.com/azure/azure-functions/functions-overview), or any other service that supports custom code or direct access to data in Azure Data Lake Storage Gen2.
 

docs/finops-hub/data-processing.md

@@ -17,16 +17,20 @@ FinOps hubs perform a number of data processing activities to clean up, normaliz
 
 ```mermaid
 sequenceDiagram
-    Cost Management->>msexports: ① Export amortized costs
-    msexports->>ingestion: ② msexports_Transform
+    Cost Management->>msexports: ① Export data
+    msexports->>msexports: ② msexports_ExecuteETL
+    msexports->>ingestion: ② msexports_ETL_ingestion
     Power BI-->>ingestion: ③ Read data
 ```
 
 <br>
 
 1. Cost Management exports raw cost details to the **msexports** container.
-2. The **msexports_transform** pipeline saves the raw data in parquet format to the **ingestion** container.
-3. Power BI reads cost data from the **ingestion** container.
+2. The **msexports_ExecuteETL** pipeline kicks off the extract-transform-load (ETL) process when files are added to storage.
+3. The **msexports_ETL_ingestion** pipeline transforms the data to a standard schema and saves the raw data in parquet format to the **ingestion** container.
+4. Power BI reads cost data from the **ingestion** container.
+
+<br>
 
 ---