diff --git a/docs/wiki/ADO Dashboard - Deployment Guide.md b/docs/wiki/ADO Dashboard - Deployment Guide.md index abc8ee8..c583a17 100644 --- a/docs/wiki/ADO Dashboard - Deployment Guide.md +++ b/docs/wiki/ADO Dashboard - Deployment Guide.md @@ -17,7 +17,7 @@ The CCO Insights Azure DevOps Contributions dashboard requires infrastructure be # Deployment -Here is the link to the required [bicep][BicepOverview] template that will deploy and connect the architecture presented previously. +[Here][ADODashboardBicepTemplate] is the link to the required [bicep][BicepOverview] template that will deploy and connect the architecture presented previously. ## Pre-requisites @@ -87,4 +87,5 @@ After successfully deploying the back-end go to the Azure portal and manually ru [infrastructure]: [src]: [ADODailySyncFolder]: -[WorkflowsFolder]: \ No newline at end of file +[WorkflowsFolder]: +[ADODashboardBicepTemplate]: \ No newline at end of file diff --git a/docs/wiki/ADO Dashboard.md b/docs/wiki/ADO Dashboard.md index 5c16303..25000f2 100644 --- a/docs/wiki/ADO Dashboard.md +++ b/docs/wiki/ADO Dashboard.md @@ -12,10 +12,12 @@ As part of the Continuous Cloud Optimization Insights solution, a dashboard is i - Number of Projects and branches - Pull requests - Average pull requests per day - - Comparison between number of open vs closed pull requests over the last months + - Comparison between number of open vs closed pull requests over the last months* - Comparison between number of lines added vs deleted per month - Top contributors measured by changes in their pull requests -- Branches created over the last months +- Branches created over the last months* + + > ***NOTE**: time span depends on when the active branches in scope were created. An important note is that the **the ADO Dashboard can be published in the PowerBI online service with auto refresh enabled**. This differs from other dashboards of CCO Insights because for the ADO Dashboard, dynamic queries are not being done directly from the PowerBI file, meaning that it can be published and consumed from the [PowerBI online][PublishPowerBI] service. diff --git a/docs/wiki/GitHub Dashboard - Deployment Guide.md b/docs/wiki/GitHub Dashboard - Deployment Guide.md index 995dc37..68d8b54 100644 --- a/docs/wiki/GitHub Dashboard - Deployment Guide.md +++ b/docs/wiki/GitHub Dashboard - Deployment Guide.md @@ -19,7 +19,7 @@ This dashboard requires infrastructure being deployed in Azure. The infrastructu # Setup -Here is the link to the required [bicep][BicepOverview] template that will deploy and connect the architecture presented previously. +[Here][GitHubDashboardBicepTemplate] is the link to the required [bicep][BicepOverview] template that will deploy and connect the architecture presented previously. ## Prerequisites @@ -64,9 +64,19 @@ Finally, navigate to the root folder of the repository where you will find the [ Now you are ready to deploy your backend in your environment: ![deploy-backend][DeployBackend] -After successfully deploying the backend go to the Azure portal and manually run the `InitializeTables` endpoint. Make sure you see the tables in your Storage Account before moving forward. +After successfully deploying the backend go to the Azure portal and manually run the `InitializeTables` endpoint. -![storage-tables][StorageTables] +On the Azure portal, go to the Storage account and under the Tables section in the Storage browser, verify that the following tables are present: +- Repository +- Forks +- Clones +- OpenPullRequests +- ClosedPullRequests +- Stargazers +- Contributors +- Traffic +- Issues +- Releases
@@ -88,4 +98,5 @@ After successfully deploying the backend go to the Azure portal and manually run [infrastructure]: [src]: [GitHubDailySyncfolder]: -[WorkflowsFolder]: \ No newline at end of file +[WorkflowsFolder]: +[GitHubDashboardBicepTemplate]: \ No newline at end of file diff --git a/docs/wiki/GitHub Dashboard.md b/docs/wiki/GitHub Dashboard.md index f437c31..3e4f194 100644 --- a/docs/wiki/GitHub Dashboard.md +++ b/docs/wiki/GitHub Dashboard.md @@ -15,10 +15,12 @@ As part of the Continuous Cloud Optimization Insights solution, a dashboard is i - Number of open pull requests - Average pull requests per day - Pull requests' lifecycle (in days) - - Comparison between number of open vs closed pull requests over the last months + - Comparison between number of open vs closed pull requests over the last months* - Comparison between number of lines added vs deleted per month - Top contributors measured by changes in their pull requests + > ***NOTE**: time span depends on when the active branches in scope were created. + An important note is that the **the Github Dashboard can be published in the PowerBI online service with auto refresh enabled**. This differs from other dashboards of CCO Insights because for the Github Dashboard, dynamic queries are not being done directly from the PowerBI file, meaning that it can be published and consumed from the [PowerBI online][PublishPowerBI] service.
diff --git a/docs/wiki/Governance Dashboard - Deployment Guide.md b/docs/wiki/Governance Dashboard - Deployment Guide.md index efd0b8d..b2acfb8 100644 --- a/docs/wiki/Governance Dashboard - Deployment Guide.md +++ b/docs/wiki/Governance Dashboard - Deployment Guide.md @@ -42,7 +42,7 @@ Registering this Resource Provider has no cost or performance penalty on the sub # Installing the custom connector -The CCO Azure Governance Dashboard requires to install the Power BI Custom Connector located in the same folder as the CCO Governance Dashboard: ([CCoDashboardAzureConnector.mez][CCoDashboardAzureConnector]). This Custom Connector allows us to leverage information from Azure Management REST APIs that requires POST methods and errors control +The CCO Azure Governance Dashboard requires to install the Power BI Custom Connector located in the same folder as the CCO Governance Dashboard: ([CCoDashboardAzureConnector.mez][CCoDashboardAzureConnector]). This Custom Connector allows us to leverage information from Azure Management REST APIs that require POST methods and error handling. To install the custom connector you must copy the file [CCoDashboardAzureConnector.mez][CCoDashboardAzureConnector] from the **ccodashboard/dashboards/CCODashboard-Governance/** folder to the folder that Power BI creates by default in the Documents folder in your PC. If this folder doesn't exist, you can create a new one with this name. diff --git a/docs/wiki/Governance Dashboard - Reports.md b/docs/wiki/Governance Dashboard - Reports.md index 0f783b1..5a2084a 100644 --- a/docs/wiki/Governance Dashboard - Reports.md +++ b/docs/wiki/Governance Dashboard - Reports.md @@ -4,7 +4,7 @@ - [Management Groups and Subscriptions Hierarchy Overview page](#management-groups-and-subscriptions-hierarchy-overview-page) - [Tags and naming standards page](#tags-and-naming-standards-page) - [Azure Regulatory Standards Forecast](#azure-regulatory-standards-forecast) - - [Azure Resources Security & Compliance page](#azure-resources-security--compliance-page) + - [Azure Resources Security \& Compliance page](#azure-resources-security--compliance-page) - [Azure Policies page](#azure-policies-page) - [Azure Blueprints page](#azure-blueprints-page) @@ -80,7 +80,7 @@ You can filter the information by: - Subscription with assigned blueprints - Blueprint Definition - +![GovernanceSubsBlueprints][GovernanceSubsBlueprints] diff --git a/docs/wiki/Governance Dashboard.md b/docs/wiki/Governance Dashboard.md index fc967d0..ebdb99d 100644 --- a/docs/wiki/Governance Dashboard.md +++ b/docs/wiki/Governance Dashboard.md @@ -9,7 +9,16 @@ # Overview -The CCO Azure Governance Dashboard is aligned with the Microsoft Cloud Adoption Framework governance principles and will allow you to get quick insights around Management Groups, Subscriptions, Blueprints, Polices, Naming Standards, Tagging and Regulatory Standards compliance. The information captured on this Power BI Dashboard can help Cloud Teams, Operations Teams or business decision makers to have a snapshot of the current Azure configuration in just few minutes. +The CCO Azure Governance Dashboard is aligned with the Microsoft Cloud Adoption Framework governance principles and will allow you to get quick insights around important cloud assets and configuration, such as: +- Management Groups, +- Subscriptions, +- Blueprints, +- Polices, +- Naming Standards, +- Tagging +- and Regulatory Standards compliance. + +The information captured on this Power BI Dashboard can help Cloud Teams, Operations Teams or business decision makers to have a snapshot of the current Azure configuration in just few minutes.
diff --git a/docs/wiki/Home.md b/docs/wiki/Home.md index 886819f..52d1ba5 100644 --- a/docs/wiki/Home.md +++ b/docs/wiki/Home.md @@ -21,7 +21,7 @@ The Continuous Cloud Optimization Insights (CCO Insights) project is a set of Power BI Desktop Reports that enables monitoring, operation and infrastructure teams to quickly gain insights about their existing Azure Platform footprint, resources and code contribution characteristics on Azure DevOps and GitHub. CCO Insights is developed using Power Query M language and DAX that pulls information directly from different Azure REST API. -![OverviewImage](./media/OverviewImage.png) +![OverviewImage][OverviewImage] CCO Insights currently includes 4 different dashboards to discover information about your Azure, Azure DevOps and GitHub cloud platforms: @@ -38,13 +38,13 @@ The [**Troubleshooting Guide**][TroubleshootingGuide] chapter contains guidance # List of assets -1. **queries folder**: Includes the M queries used in the Dashboard to pull data from Azure and Graph REST APIs. This content is for reference purposes to facilitate the Data Model comprehension and to enable contributors to expand the Dashboard capabilities. -2. **docs/assets/pictures folder**: Contains all the images that the Dashboard will use when loading data from Azure. The content of this folder is dynamic and updated regularly. Make sure the computer running the Dashboard also has access to [this URL][GraphicalElements] via the internet. -3. **dashboards folder**: This folder contains sub folders with different versions of the CCO Insights dashboards. - - ***CCODashboard-Infra folder*** has a more generic version of the Dashboard that includes information from Azure Advisor, Azure Defender , Azure Networking REST APIs, Azure Compute and more REST and Graph APIs. This dashboard requires the installation of a [custom connector][CustomConnector]. - - ***CCODashboard-Governance folder*** has a dashboard aligned with the Microsoft Cloud Adoption Framework governance principles and will allow to get quick insights around Management Groups, Subscriptions, Blueprints, Polices, Naming Standards, Tagging and Regulatory Standards compliance. This dashboard requires the installation of a [custom connector][CustomConnector]. - - [***GitHub Contributions Dashboard folder***][GitHubContributionsDashboard]: has a dashboard to get insights about the contributions to your GitHub project. - - [***Azure DevOps Contributions Dashboard folder***][AdoContributionsDashboard]: has a dashboard to get insights about the contributions to your Azure DevOps (ADO) project. +1. **[queries folder][QueriesFolder]**: Includes the M queries used in the Dashboard to pull data from Azure and Graph REST APIs. This content is for reference purposes to facilitate the Data Model comprehension and to enable contributors to expand the Dashboard capabilities. +2. **[docs/assets/pictures folder][GraphicalElementsFolder]**: Contains all the images that the Dashboard will use when loading data from Azure. The content of this folder is dynamic and updated regularly. Make sure the computer running the Dashboard also has access to [this URL][GraphicalElementsFolder] via the internet. +3. **[dashboards folder][DashboardsFolder]**: This folder contains sub folders with different versions of the CCO Insights dashboards. + - ***[CCODashboard-Infra folder][InfraDashboardFolder]*** has a more generic version of the Dashboard that includes information from Azure Advisor, Azure Defender , Azure Networking REST APIs, Azure Compute and more REST and Graph APIs. This dashboard requires the installation of a [custom connector][CustomConnector]. + - ***[CCODashboard-Governance folder][GovDashboardFolder]*** has a dashboard aligned with the Microsoft Cloud Adoption Framework governance principles and will allow to get quick insights around Management Groups, Subscriptions, Blueprints, Polices, Naming Standards, Tagging and Regulatory Standards compliance. This dashboard requires the installation of a [custom connector][CustomConnector]. + - [***GitHub Contributions Dashboard folder***][GitHubDashboardFolder]: has a dashboard to get insights about the contributions to your GitHub project. + - [***Azure DevOps Contributions Dashboard folder***][ADODashboardFolder]: has a dashboard to get insights about the contributions to your Azure DevOps (ADO) project.
@@ -115,6 +115,7 @@ Any use of third-party trademarks or logos are subject to those third-party's po [MicrosoftsTrademarkAndBrandGuidelines]: +[OverviewImage]: <./media/OverviewImage.png> [YouTubeVideos]: [Video0]: [Video1]: @@ -130,8 +131,14 @@ Any use of third-party trademarks or logos are subject to those third-party's po [AzureInfrastructureDashboard]: <./Infrastructure-Dashboard> [AzureGovernanceDashboard]: <./Governance Dashboard> -[GitHubContributionsDashboard]: <./GitHub Dashboard> +[GitHubContributionsDashboard]: [AdoContributionsDashboard]: <./ADO Dashboard> [TroubleshootingGuide]: <./Troubleshooting%20Guide> [CustomConnector]: <./Governance Dashboard%20-%20Deployment Guide#installing-the-custom-connector> -[GraphicalElements]: +[GraphicalElementsFolder]: +[QueriesFolder]: +[DashboardsFolder]: +[InfraDashboardFolder]: +[GovDashboardFolder]: +[GitHubDashboardFolder]: +[ADODashboardFolder]: \ No newline at end of file diff --git a/docs/wiki/Infrastructure-Dashboard-Deployment Guide.md b/docs/wiki/Infrastructure-Dashboard-Deployment Guide.md index 5e9a3f2..e5f4c04 100644 --- a/docs/wiki/Infrastructure-Dashboard-Deployment Guide.md +++ b/docs/wiki/Infrastructure-Dashboard-Deployment Guide.md @@ -5,7 +5,7 @@ - [Installing the custom connector](#installing-the-custom-connector) - [Azure Advisor Recommendations](#azure-advisor-recommendations) - [Generating Azure Advisor recommendations manually](#generating-azure-advisor-recommendations-manually) -- [Azure Defender Recommendations](#azure-defender-recommendations) +- [Recommendations from Microsoft Defender for Azure](#recommendations-from-microsoft-defender-for-azure) - [Setting up the Azure Infrastructure Dashboard](#setting-up-the-azure-infrastructure-dashboard) - [Template download](#template-download) - [Environment selection](#environment-selection) @@ -15,7 +15,7 @@ - [Refresh the dashboard](#refresh-the-dashboard) - [Credentials for management.azure.com REST API request:](#credentials-for-managementazurecom-rest-api-request) - [Credentials for graph.windows.net API](#credentials-for-graphwindowsnet-api) - - [Enter Access Web content credentials](#enter-access-web-content-credentials) + - [Enter Organizational credentials](#enter-organizational-credentials) --- @@ -58,7 +58,7 @@ Registering these 2 Resource Providers has no cost or performance penalty on the # Installing the custom connector -The CCO Azure Infrastructure Dashboard requires you to install the Power BI Custom Connector located in the same folder as the CCO Infrastructure Dashboard: ([CCoDashboardAzureConnector.mez][CCoDashboardAzureConnector]). This Custom Connector allows us to leverage information from Azure Management REST APIs that requires POST methods and errors control. +The CCO Azure Infrastructure Dashboard requires you to install the Power BI Custom Connector located in the same folder as the CCO Infrastructure Dashboard: ([CCoDashboardAzureConnector.mez][CCoDashboardAzureConnector]). This Custom Connector allows us to leverage information from Azure Management REST APIs that require POST methods and error handling. To install the custom connector you must copy the file [CCoDashboardAzureConnector.mez][CCoDashboardAzureConnector] from the **ccodashboard/dashboards/CCODashboard-Infrastructure/** folder to the folder that Power BI creates by default in the Documents folder in your PC. If this folder doesn't exist, you can create a new one with this name. @@ -92,13 +92,13 @@ Open the Azure Portal with your Azure Account https://portal.azure.com
-# Azure Defender Recommendations +# Recommendations from Microsoft Defender for Azure -Microsoft Defender for Cloud is a Cloud Security Posture Management (CSPM) and Cloud Workload Protection Platform (CWPP) for all of your Azure, on-premises, and multicloud (Amazon AWS and Google GCP) resources. Defender for Cloud fills three vital needs as you manage the security of your resources and workloads in the cloud and on-premises. +Microsoft Defender for Cloud is a Cloud Security Posture Management (CSPM) and Cloud Workload Protection Platform (CWPP) for all of your Azure, on-premises, and multi-cloud (Amazon AWS and Google GCP) resources. Defender for Cloud fills the vital needs as you manage the security of your resources and workloads in the cloud and on-premises. -You can find more information at the official Azure Defender site [here][SecurityCenterIntro]. +You can find more information at the official Microsoft Defender for Azure [site][SecurityCenterIntro]. -The subscriptions will need to use the **paid** tier if you want to detect and see the alerts in the Azure Defender Alerts page of the dashboard. +The subscriptions will need to use the **paid** tier if you want to detect and see the alerts in the Microsoft Defender for Azure Alerts page of the dashboard.
@@ -166,7 +166,7 @@ If the permissions and credentials are properly flushed it should ask you for cr ![credentials5][Credentials5] -### Enter Access Web content credentials +### Enter Organizational credentials - Make sure that you select **Organization account** type. - Click on **Sign in**. @@ -196,8 +196,11 @@ If the permissions and credentials are properly flushed it should ask you for cr [Credentials5]: <./media/Credentials5.png> [LogAnalytics]: <./media/loganalyticsAPI.PNG> [Credentials7]: <./media/Credentials7.png> +[CustomConnectorFolder]: <./media/customconnectorfolder.PNG> +[CustomConnectorSecurity]: <./media/customconnectorsecurity.PNG> [GenerateAllSubscriptionsAdvisorRecommendations.ps1]: [CCODashboardInfra]: -[PowerBIDownload]: \ No newline at end of file +[PowerBIDownload]: +[CCoDashboardAzureConnector]: \ No newline at end of file diff --git a/docs/wiki/Infrastructure-Dashboard-Reports.md b/docs/wiki/Infrastructure-Dashboard-Reports.md index 6b455e7..33a63a8 100644 --- a/docs/wiki/Infrastructure-Dashboard-Reports.md +++ b/docs/wiki/Infrastructure-Dashboard-Reports.md @@ -24,7 +24,7 @@ ## CCO Azure Infrastructure Dashboard overview page In this page, you will be able to identify the top 10 recommendations that Azure Advisor has identified, the top 10 most attacked resources and the number of subscription owners. You can also locate all the deployed resources in a map. -It’s important to mention that this tab just gives you a quick view. All the recommendations will be available with more details in the following tabs +It’s important to mention that this tab just gives you a quick view. All the recommendations will be available with more details in the following tabs. You can filter the information by: @@ -67,7 +67,17 @@ You can filter the information by: ## Azure Compute Overview page -In this tab, you will be able to identify the number of VMs, the Operating System, the SKU, the Availability Set name, the location, the VM Size, the VNET and subnet each VM is connected, the private IP address and if the VM has any extension installed. +In this tab, you will be able to identify the +- number of VMs, +- the Operating System, +- the SKU, +- the Availability Set name, +- the location, +- the VM Size, +- the VNET +- and subnet each VM is connected, +- the private IP address +- and if the VM has any extension installed. You can filter the information by: @@ -194,7 +204,16 @@ You can filter the information by: ## Azure Kubernetes Service page -In this page, you will be able to identify the number of AKS Clusters, Nodes, Pods, Containers, Container images, Service principals and Azure Container Instances . All the information related to these resources will be shown (IPs, pods in use, status, network, image repositories, RBAC roles …). +In this page, you will be able to identify the +- number of AKS Clusters, +- Nodes, +- Pods, +- Containers, +- Container images, +- Service principals +- and Azure Container Instances . + +All the information related to these resources will be shown (IPs, pods in use, status, network, image repositories, RBAC roles …). You can filter the information by: diff --git a/docs/wiki/OLD - Deployment Guide - ADO Dashboard.md b/docs/wiki/OLD - Deployment Guide - ADO Dashboard.md deleted file mode 100644 index 0cd7279..0000000 --- a/docs/wiki/OLD - Deployment Guide - ADO Dashboard.md +++ /dev/null @@ -1,122 +0,0 @@ -### _Navigation_ - -- [Overview](#overview) -- [Infrastructure requirements](#infrastructure-requirements) - - [Deployment](#deployment) - - [Pre-requisites](#pre-requisites) - - [Back-end Deployment](#back-end-deployment) -- [Dashboard](#dashboard) - ---- - -
- -# Overview - -As part of the Continuous Cloud Optimization Insights solution, a dashboard is included to track the contributions made to an Azure DevOps repository. The objective is to monitor not only the cloud environment, but also all the resources used for its design, deployment and maintenance. This dashboard allows you to monitor different metrics such as: -- Number of Projects and branches -- Pull requests - - Average pull requests per day - - Comparison between number of open vs closed pull requests over the last months -- Comparison between number of lines added vs deleted per month -- Top contributors measured by changes in their pull requests -- Branches created over the last months - -An important note about this dashboard is that **this dashboard can be published in the PowerBI online service with auto refresh enabled**. The difference with the current versions of the other dashboards of CCO Insights is that, for this one, no dynamic queries are being done directly from the PowerBI file, meaning that it can be published and consumed directly from the [PowerBI online][PublishPowerBI] service. - -
- -# Infrastructure requirements - -The CCO Insights Azure DevOps Contributions dashboard requires a infrastructure being deployed in Azure. The infrastructure consists of a PowerShell Function App,Application Insights for monitoring and a Storage Account where results from the Azure DevOps REST API calls will be stored in different tables. The following diagram represents the infrastructure to be deployed. - -![ADO Dashboard Architecture][ADODashboardArchitecture] - -## Deployment - -Here is the link to the required [bicep][BicepOverview] template that will deploy and connect the architecture presented previously. - -### Pre-requisites - -In order to successfully use the deploy.bicep and workflow provided, you will need to have: -- This repository forked in your own environment. -- An Azure subscription. If you don't have one you can create one for free using this [link][GetAzure]. If you already have an Azure tenant but you want to create a new subscription you can follow the instructions [here][CreateSubscription]. -- A [resource group][ResourceGroup] already created. -- A service principal with Owner permissions in your subscription. You will need owner permissions because as part of the architecture you will be creating a Managed Identity that will require a role assignment to save the retrieved data in the Storage Account. You can create your service principal with Contributor rights by running the following commands: - ```sh - az ad sp create-for-rbac --name "<>" --role "Contributor" --scopes /subscriptions/<> --output "json" - ``` -- A secret in your GitHub repository with the name `AZURE_CREDENTIALS`. You can use the output from the previous command to generate this secret. The format of the secret should be: - ```json - { - "clientId": "", - "ClientSecret": "", - "SubscriptionId": "", - "TenantId": "" - } - ``` -- Another secret in your Azure DevOps repository with the name `ADOPAT`. This will store the value of a PAT token you will need to generate with the following permissions: - | Scope | Permission | - |-------| ---------- | - | Code | Read | - | Graph | Read | - | Identity | Read | - | Project and Team | Read | - -- In the [local.settings.json][local.settings.json] file, update the values for the `organization`, `resourceGroup` and `storageAccount` with the names you want to configure in your environment. Also, make sure that these names match the values in the [deploy.bicep][deploy.bicep] file for the same resources. - - > Note: The **organization** corresponds to the ADO organization from where the information needs to be retrieved. - -### Back-end Deployment - -In the [infrastructure][infrastructure] folder you will find a `deploy.bicep` file which is the template that will be used to deploy the infrastructure. Update the first two parameters (`name` and `staname`) with your unique values. **Name** will be used to compose the name of all resources except for the storage account, which will leverage the **staname** parameter. - -In the [src][src] folder you can find the source code that will be deployed in the Function App once the infrastructure is ready. You will deploy two endpoints: -- **InitializeTables**: you will need to run this endpoint once manually to initialize the Storage Account with the required tables and collect all the data history available in the Azure DevOps API. -- **ADODailySync**: this endpoint will be automatically run on a daily basis and will add more data to the already created storage account tables. If you don't want a daily cadence, you can update the cron expression in the `function.json` file under the [ADO DailySync folder][ADODailySyncFolder]. - -Finally, navigate to the root folder of the repository where you will find the [workflows folder][WorkflowsFolder] under the `.github` folder. There you can locate the workflow that you will have to run to deploy the back-end of the dashboard. The only parameter you will need to setup manually while triggering the workflow is the `resourceGroupName` that you created earlier. - -Now you are ready to deploy the back-end solution in your environment: -![deploy-back-end][DeployBackend] - -After successfully deploying the back-end go to the Azure portal and manually run the `InitializeTables` endpoint. Verify that you can see the tables in your Storage Account below before moving forward. - -![storage-tables][StorageTables] - -
- -# Dashboard - -With the previous back-end deployed, you can now download the [ADOContributions.pbit][AdoContributionsDashboard] and run it locally. You will be asked to enter: -- The name of the Storage Account you deployed. -![Storage Account Name][StorageAccountName] -- The Storage account access key. - -After that you will be able to monitor your contributions! - -![Ado Contributions][AdoContributions] - - - -[PublishPowerBI]: -[BicepOverview]: -[GetAzure]: -[CreateSubscription]: -[ResourceGroup]: - - -[ADODashboardArchitecture]: <./media/github-dashboard-architecture.png> -[DeployBackend]: <./media/ado-run-workflow.png> -[StorageTables]: <./media/ado-storage-tables.png> -[StorageAccountName]: <./media/ado-storage-account.png> -[AdoContributions]: <./media/Ado-contributions-dashboard.png> - - -[local.settings.json]: -[deploy.bicep]: -[infrastructure]: -[src]: -[ADODailySyncFolder]: -[WorkflowsFolder]: -[AdoContributionsDashboard]: diff --git a/docs/wiki/OLD - Deployment Guide - GitHub Dashboard.md b/docs/wiki/OLD - Deployment Guide - GitHub Dashboard.md deleted file mode 100644 index b6e82bc..0000000 --- a/docs/wiki/OLD - Deployment Guide - GitHub Dashboard.md +++ /dev/null @@ -1,122 +0,0 @@ -### _Navigation_ - -- [Overview](#overview) -- [Infrastructure requirements](#infrastructure-requirements) - - [Deployment](#deployment) - - [Prerequisites](#prerequisites) - - [Backend Deployment](#backend-deployment) -- [Dashboard](#dashboard) - ---- - -
- -# Overview - -As part of the Continuous Cloud Optimization Insights solution, a dashboard is included to track the contributions made to a GitHub repository. The objective is to monitor not only the cloud environment, but also all the resources used for its design, deployment and maintenance. This dashboard allows you to monitor different metrics such as: -- Number of contributors -- Number of clones, forks, watchers, stars -- Pull requests - - Number of open pull requests - - Average pull requests per day - - Pull requests' lifecycle (in days) - - Comparison between number of open vs closed pull requests over the last months -- Comparison between number of lines added vs deleted per month -- Top contributors measured by changes in their pull requests. - -An important note about this dashboard is that **this dashboard can be published in the PowerBI online service with auto refresh enabled**. The difference with the current versions of the other dashboards of CCO Insights is that, for this one, no dynamic queries are being done directly from the PowerBI file, meaning that it can be published and consumed directly from the [PowerBI online][PublishPowerBI] service. - -
- -# Infrastructure requirements - -This dashboard requires an infrastructure being deployed in Azure. The infrastructure consists of a Powershell Function App, an Application Insights for monitoring and a Storage Account where results from the GitHub REST API calls will be stored in different tables. The following diagram represents the infrastructure to be deployed. - -![ADO Dashboard Architecture][GHDashboardArchitecture] - -## Deployment - -As part of this solution we offer you already the required [bicep][BicepOverview] template that will deploy and connect the architecture presented previously. - -### Prerequisites - -In order to successfully user the deploy.bicep and workflow provided, you will need to have: -- This repository forked in your own environment. -- An Azure subscription. If you don't have one you can create one for free using this [link][GetAzure]. If you already have an Azure tenant but you want to create a new subscription you can follow the instructions [here][CreateSubscription]. -- A [resource group][ResourceGroup] already created. -- A service principal with Owner permissions in your subscription. You will need owner permissions because as part of the architecture you will be creating a Managed Identity that will require a role assignment to save the retrieve data in the Storage Account. You can create your service principal with Contributor permissions by running the following commands: - ```sh - az ad sp create-for-rbac --name "<>" --role "Contributor" --output "json" - ``` -- A secret in your GitHub repository with the name `AZURE_CREDENTIALS`. You can user the output from the previous command to generate this secret. The format of the secret should be: - ```json - { - "clientId": "", - "ClientSecret": "", - "SubscriptionId": "", - "TenantId": "" - } - ``` -- Another secret in your GitHub repository with the name `PAT`. This will be store the value of a PAT token you will need to generate with the following permissions: - | Scope | Permission | - |-------| ---------- | - | repo | Full control of private repositories | - | user | Update ALL user data | - | admin:repo_hook | Full control of repository hooks | - | admin:org | Full control of orgs and teams, read and write org projects | -- In the [local.settings.json][local.settings.json] file, update the values for the `owner`, `repository`, `resourceGroup` and `storageAccount` with the names you want to configure in your environment. Also, make sure that these names match the values in the [deploy.bicep][deploy.bicep] file for the same resources. - - > Note: The **owner** and **repository** names correspond to the GitHub organization and repository name from where the information needs to be retrieved. - -### Backend Deployment - -In the [infrastructure][infrastructure] folder you will find a `deploy.bicep` file which is the template that will be used to deploy the infrastructure. Please, go ahead and update the first two parameters (`name` and `staname`) with your unique values. **Name** will be used to compose the name of all resources except for the storage account, which will leverage the **staname**. - -In the [src][src] folder you can find the source code that will be deployed in the Function App once the infrastructure is ready. Basically you will deploy two endpoints: -- **InitializeTables**: you will need to run this endpoint once manually to initialize the Storage Account with the required tables and collect all the data history available in the GitHub API. -- **GitHubDailySync**: this endpoint will be automatically run in a daily basis and will add more data to the already created storage account tables. If you don't want a daily cadence you can update the cron expression in the `function.json` file under the [GitHub DailySync folder][GitHubDailySyncfolder]. - -Finally, if you go to the root folder of the repository, you will find the [workflows folder][WorkflowsFolder] under the `.github` folder. There you can locate the workflow that you will have to run to deploy the backend of the dashboard. The only parameter you will need to setup manually while triggering the workflow in the `resourceGroupName` that you created earlier. - -Now you are ready to deploy your backend in your environment: -![deploy-backend][DeployBackend] - -After successfully deploying the backend go to the Azure portal and manually run the `InitializeTables` endpoint. Make sure you see the tables in your Storage Account before moving forward. - -![storage-tables][StorageTables] - -
- -# Dashboard - -With the previous backend deployed, you can now download the [GitHubContributions.pbit][GitHubContributionsDashboard] file and open it locally. You will be asked to enter: -- The Storage Account name of the Storage Account you deployed. -![Storage Account Name][StorageAccountName] -- The Storage account access key. - -After that you will be able to monitor your contributions! - -![GitHub Contributions][GitHubContributions] - - -[PublishPowerBI]: -[BicepOverview]: -[GetAzure]: -[CreateSubscription]: -[ResourceGroup]: - - -[GHDashboardArchitecture]: <./media/github-dashboard-architecture.png> -[DeployBackend]: <./media/run-workflow.jpg> -[StorageTables]: <./media/storage-tables.jpg> -[StorageAccountName]: <./media/github-storage-account.jpg> -[GitHubContributions]: <./media/Github-contributions-dashboard.jpg> - - -[local.settings.json]: -[deploy.bicep]: -[infrastructure]: -[src]: -[GitHubDailySyncfolder]: -[WorkflowsFolder]: -[GitHubContributionsDashboard]: \ No newline at end of file diff --git a/docs/wiki/OLD - Deployment Guide - Governance Dashboard.md b/docs/wiki/OLD - Deployment Guide - Governance Dashboard.md deleted file mode 100644 index 70dbcc8..0000000 --- a/docs/wiki/OLD - Deployment Guide - Governance Dashboard.md +++ /dev/null @@ -1,266 +0,0 @@ -### _Navigation_ - -- [Overview](#overview) - - [Requirements](#requirements) -- [APIs in use](#apis-in-use) -- [Resource Providers requirements](#resource-providers-requirements) -- [Installing the custom connector](#installing-the-custom-connector) -- [Setting up the CCO Azure Governance Dashboard Governance](#setting-up-the-cco-azure-governance-dashboard-governance) - - [Template download](#template-download) - - [Environment selection](#environment-selection) - - [Modify Privacy settings](#modify-privacy-settings) - - [Credentials](#credentials) - - [Clean Credentials on the Data Source](#clean-credentials-on-the-data-source) - - [Refresh the dashboard](#refresh-the-dashboard) - - [Credentials for management.azure.com REST API request](#credentials-for-managementazurecom-rest-api-request) - - [Credentials for Custom Connector](#credentials-for-custom-connector) -- [Report Pages](#report-pages) - - [Management Groups and Subscriptions Hierarchy Overview page](#management-groups-and-subscriptions-hierarchy-overview-page) - - [Tags and naming standards page](#tags-and-naming-standards-page) - - [Azure Regulatory Standards Forecast](#azure-regulatory-standards-forecast) - - [Azure Resources Security & Compliance page](#azure-resources-security--compliance-page) - - [Azure Policies page](#azure-policies-page) - - [Azure Blueprints page](#azure-blueprints-page) - ---- - -
- -# Overview - -The CCO Azure Governance Dashboard is aligned with the Microsoft Cloud Adoption Framework governance principles and will allow to get quick insights around Management Groups, Subscriptions, Blueprints, Polices, Naming Standards, Tagging and Regulatory Standards compliance. The information captured on this Power BI Dashboard can help Cloud Teams, Operations Teams or business decision makers to have a snapshot of the current Azure configuration in just few minutes. - -## Requirements - -- The CCO Azure Governance Dashboard is a Power BI Template that requires to download and install the Microsoft Power BI Desktop Edition from the Microsoft Store. Below you can find the minimum requirements to run the Dashboard - - Windows 10 version **14393.0** or **higher**. - - Internet access from the computer running Microsoft Power BI desktop. - - An Azure account on the desired tenant space with permissions on the subscriptions to read from the Azure Services described above. - - Install the custom connector and allow the use of any extension to load data without validation or warning. - -
- -# APIs in use - -The CCO Azure Governance Dashboard Governance pulls the information from several APIs. You can read the public documentation if you need further information about the calls and methods available: -

- -| API Name| Dashboard API Version | Last API version | Using latest version| -| --- | :---: | :---: |:---: | -| [Resource Groups][ResourceGroups] |2019-05-01 |2019-05-01|:heavy_check_mark:| -| [Azure Resources][AzureResources] |2019-05-01 |2019-05-01|:heavy_check_mark:| -| [Azure Subscriptions][AzureSubscriptions] |2020-01-01 |2020-01-01|:heavy_check_mark:| -| [Azure Locations][AzureLocations] |2019-05-01 |2019-05-01|:heavy_check_mark:| -| [Azure Blueprints][AzureBlueprints] |2018-11-01-preview |2018-11-01-preview|:heavy_check_mark:| -| [Azure Policies][AzurePolicies] |2019-09-01 |2019-09-01|:heavy_check_mark:| -| [Azure Regulatory Compliances][AzureRegulatoryCompliances] |2019-01-01-preview |2019-01-01-preview|:heavy_check_mark:| -| [Azure Assessments][AzureAssessments] |2020-01-01 |2020-01-01|:heavy_check_mark:| -| [Azure Secure Scores][AzureSecureScores] |2020-01-01 |2020-01-01|:heavy_check_mark:| -| [Azure Secure Scores Controls][AzureSecureScoresControls] |2020-01-01-preview |2020-01-01-preview|:heavy_check_mark:| - -API URLs by environment: - -| API Name| API URL | Environment| -|--- |--- |--- | -| Management |https://management.azure.com/|Global| -| Management |https://management.usgovcloudapi.net/|US Government| -| Management |https://management.chinacloudapi.cn/|China| - -
- -# Resource Providers requirements - -Although some of the Resource Providers might be enabled by default, you need to make sure that at least the **Microsoft.Security** resource provider is registered across all the subscriptions that you plan analyze using the Dashboard. - -Registering this Resource Provider has no cost or performance penalty on the subscription: - -1. Click on **Subscriptions**. -2. Click on the Subscription name you want to configure. -3. Click on **Resource Providers**. -4. Click on **Microsoft.Security** and **Register**. - -
- -# Installing the custom connector - -The CCO Azure Governance Dashboard requires to install the Power BI Custom Connector located in the same folder as the CCO Governance Dashboard ([CCoDashboardAzureConnector.mez][CCoDashboardAzureConnector]). This Custom Connector allows us to leverage information from Azure Management REST APIs that requires POST methods and errors control - -To install the custom connector you must copy the file [CCoDashboardAzureConnector.mez][CCoDashboardAzureConnector] from the **ccodashboard/dashboards/CCODashboard-Governance/** folder to the folder that Power BI creates by default in the Documents folder in your PC. If this folder doesn't exist, you can create a new one with this name. - -The path should be **C:\Users\\%username%\Documents\Power BI Desktop\Custom Connectors** or if you are using OneDrive to backup the documents folder this path would not work for you and you should manually go to your documents folder and create the folder structure there. - -![CustomConnectorFolder][CustomConnectorFolder] - -Then go to Power BI Options and under Global category in the Security section, select **(Not Recommended) Allow any extension to load without validation or warning** and click **OK**. - -![CustomConnectorSecurity][CustomConnectorSecurity] - -
- -# Setting up the CCO Azure Governance Dashboard Governance - -## Template download -Download and open the `.pbit` file from [CCODashboard-Governance][CCODashboardGovernance] folder. - -## Environment selection - -Before start loading data you need to select which type of environment you're using: - -- Select "Global" for Microsoft Azure commercial environments. This is the default selection. -- Select [US-Government][UsGovernment] for Azure Us government services. Azure Government is a separate instance of the Microsoft Azure service. It addresses the security and compliance needs of United States federal agencies, state and local governments, and their solution providers. - -![selector][SelectorGov] - -## Modify Privacy settings - -- Go to File -> Options -> Privacy and set to Always ignore privacy level settings. - -![Privacy][Privacy] - -## Credentials - -By default, the template doesn't have any Azure Account credentials preloaded. Hence, the first step to start loading subscriptions data is to sign-in with the right user credentials. - -**IMPORTANT NOTE**: Power BI Desktop caches the credentials after the first logon. It is important to clear the credentials from Power BI desktop if you plan to switch between Azure Global and any other region like US Government or China. The same concept applies if you plan to switch between tenants. Otherwise, the staged credentials will be used again for the different Azure environments and the authentication or data load process will fail. - -### Clean Credentials on the Data Source - -In some cases, old credentials are cached by previous logins using Power BI Desktop and the dashboard might show errors or blank fields. - -- Click on Data sources in **Current file/Global permissions** -- Click on **Clear Permissions**. -- Click on **Clear All Permissions**. - -![credentials1][Credentials1] ![credentials2][Credentials2] - -### Refresh the dashboard - -If the permissions and credentials are properly flushed it should ask you for credentials for each REST API and you will have to set the Privacy Levels for each of them. - -- Click on **Refresh**. - -![refreshgovernance][RefreshGovernance] - -### Credentials for management.azure.com REST API request - -- Click on **Organizational Account**. -- Click on **Sign in**. -- Click on **Connect**. - -![credentials4][Credentials4] - -### Credentials for Custom Connector - -- Click on **Organizational Account**. -- Click on **Sign in**. -- Click on **Connect**. - -![CustomConnector][CustomConnector] - -
- -# Report Pages - -## Management Groups and Subscriptions Hierarchy Overview page - -In this page, you will be able to identify easily the hierarchy within your environment with the view of the Management Groups and Subscriptions. -It's important to mention that this page just gives you a quick view. - -![overview][Overview] - -## Tags and naming standards page - -In this page you will be able to sort and filter all your Resources and Resource groups based on Tags. It will help you identify any missing Tag and if your naming standards and Tags classifications adheres to your organization guidelines or policies. - -You can filter the information by: - -- Management Group with subscriptions -- Subscription - -![TagsOverview][TagsOverview] - -## Azure Regulatory Standards Forecast - -In this page you can compare your current Azure resources compliance against selected Regulatory Standards, to understand how far from a given Regulatory Standard your current Azure footprint is today. For more information check the published [Regulatory Standards][RegulatoryStandards]. - -You can filter the information by: - -- Subscription -- Regulatory Compliance -- Assessment Category - -![regulatorycompliance][RegulatoryCompliance] - -## Azure Resources Security & Compliance page - -In this page you can check the compliance status of your Azure resources based on the Azure Security Center Secure Score Controls and the corresponding Policy Set or Regulatory Standard. - -You can filter the information by: - -- Subscription -- Policy Set -- Regulatory Standard Name -- Secure Controls -- Policy Category - -![regulatory compliance resources][RegulatoryComplianceResources] - -## Azure Policies page - -In this page of the report, you will be able to identify the total amount of policies that are you applying in your environment. It will also give a high-level overview of which policies has less compliance level and which resources require more attention. - -You can filter the information by: - -- Management Group with subscriptions -- Subscription -- Policy scope - -If you navigate to a impacted resource you will see a quick description of the applied policies. - -![policies][policies] - -## Azure Blueprints page - -In this page of the report, you will be able to identify the total amount of blueprints that are you applying in your environment. It will also show which are the artifacts within the blueprints. - -You can filter the information by: - -- Subscription with assigned blueprints -- Blueprint Definition - -![governanceSubsBlueprints][GovernanceSubsBlueprints] - - -[ResourceGroups]: -[AzureResources]: -[AzureSubscriptions]: -[AzureLocations]: -[AzureBlueprints]: -[AzurePolicies]: -[AzureRegulatoryCompliances]: -[AzureAssessments]: -[AzureSecureScores]: -[AzureSecureScoresControls]: -[UsGovernment]: -[RegulatoryStandards]: - - -[CustomConnectorFolder]: <./media/customconnectorfolder.PNG> -[CustomConnectorSecurity]: <./media/customconnectorsecurity.PNG> -[SelectorGov]: <./media/selectorGov.PNG> -[Privacy]: <./media/governancePrivacy.png> -[Credentials1]: <./media/Credentials1.png> -[Credentials2]: <./media/Credentials2.png> -[RefreshGovernance]: <./media/refreshgovernance1.png> -[Credentials4]: <./media/Credentials4.png> -[CustomConnector]: <./media/customconnector.PNG> -[Overview]: <./media/GovernanceOverview.png> -[TagsOverview]: <./media/TagsOverview.png> -[RegulatoryCompliance]: <./media/regulatorycompliance.png> -[RegulatoryComplianceResources]: <./media/regulatorycomplianceresources.png> -[policies]: <./media/governancePolicies.png> -[GovernanceSubsBlueprints]: <./media/governanceSubsBlueprints.png> - - -[CCODashboardGovernance]: -[CCoDashboardAzureConnector]: diff --git a/docs/wiki/OLD - Deployment Guide - Infrastructure-Dashboard.md b/docs/wiki/OLD - Deployment Guide - Infrastructure-Dashboard.md deleted file mode 100644 index dfd519d..0000000 --- a/docs/wiki/OLD - Deployment Guide - Infrastructure-Dashboard.md +++ /dev/null @@ -1,467 +0,0 @@ -### _Navigation_ - -- [Overview](#overview) - - [Requirements](#requirements) -- [APIs in use](#apis-in-use) -- [Resource Providers requirements](#resource-providers-requirements) -- [Azure Advisor Recommendations](#azure-advisor-recommendations) - - [Generating Azure Advisor recommendations manually](#generating-azure-advisor-recommendations-manually) -- [Azure Security Center Recommendations](#azure-security-center-recommendations) -- [Setting up the Continuous Cloud Optimization Azure Infrastructure Power BI Dashboard](#setting-up-the-continuous-cloud-optimization-azure-infrastructure-power-bi-dashboard) - - [Template download](#template-download) - - [Environment selection](#environment-selection) - - [Modify Privacy settings](#modify-privacy-settings) - - [Credentials](#credentials) - - [Clean Credentials on the Data Source](#clean-credentials-on-the-data-source) - - [Refresh the dashboard](#refresh-the-dashboard) - - [Credentials for management.azure.com REST API request:](#credentials-for-managementazurecom-rest-api-request) - - [Credentials for graph.windows.net API](#credentials-for-graphwindowsnet-api) - - [Credentials for api.loganalytics.io API](#credentials-for-apiloganalyticsio-api) - - [Enter Access Web content credentials](#enter-access-web-content-credentials) -- [Report Pages](#report-pages) - - [CCO Azure Infrastructure Dashboard overview page](#cco-azure-infrastructure-dashboard-overview-page) - - [Azure Advisor Recommendations page](#azure-advisor-recommendations-page) - - [Azure Security Center Alerts page](#azure-security-center-alerts-page) - - [Azure Compute Overview page](#azure-compute-overview-page) - - [Azure VNETs and Subnets Recommendations page](#azure-vnets-and-subnets-recommendations-page) - - [Azure Network Security Groups page](#azure-network-security-groups-page) - - [Role Based Access Control page](#role-based-access-control-page) - - [Service Principal Role Based Access Control page](#service-principal-role-based-access-control-page) - - [IaaS Usage and Limits page](#iaas-usage-and-limits-page) - - [IaaS Idle Resources Dashboard page](#iaas-idle-resources-dashboard-page) - - [Azure Kubernetes Service Dashboard Overview page](#azure-kubernetes-service-dashboard-overview-page) - - [Azure Kubernetes Service page](#azure-kubernetes-service-page) - ---- - -
- -# Overview - -The Continuous Cloud Optimization Azure Infrastructure Power BI Dashboard is a report that aims to aggregate and consolidate the information generated by several Azure services to gain quick insights on your subscriptions to enable data driven business and technical optimization decisions. The main data sources for this Azure Infrastructure Dashboard are the **Azure Advisor REST API**, **Azure Security Center REST API**, **Azure Graph REST API**, **Log Analytics API** and several **Azure IaaS REST APIs**. - -## Requirements - -- The CCO Azure Infrastructure Dashboard is a Power BI Template that requires to download and install the Microsoft Power BI Desktop Edition from the Microsoft Store. Below you can find the minimum requirements to run the Dashboard - - Windows 10 version **14393.0** or **higher**. - - Internet access from the computer running Microsoft Power BI desktop. - - An Azure account on the desired tenant space with permissions on the subscriptions to read from the Azure Services described above. - - The subscriptions will need to use the Azure Security Center **Standard** plan if you want to detect and see the alerts in the Azure Security Center Alerts page of the CCO Azure Infrastructure Dashboard. - -Below you can find the list of providers and the actions that you will need to permit to allow to run the CCO Power BI Dashboard: - - -| Resource Provider Name| Permissions | -| --- | --- | -|Azure Advisor| Microsoft.Advisor/generateRecommendations/action
-|*|*/Read| - -**IMPORTANT**: You must follow [this procedure][OnboardToLighthouse] to implement Azure delegated resource management to get data from subscriptions in other tenants. - -
- -# APIs in use - -The CCO Azure Infrastructure Dashboard pulls the information from several APIs. You can read the public documentation if you need further information about the calls and methods available: -

- - -| API Name| Dashboard API Version | Last API version | Using latest version| -| --- | :---: | :---: |:---: | -| [Azure Advisor][AzureAdvisor] | 2020-01-01|2020-01-01|:heavy_check_mark:| -| [Azure Security Center Alerts][AzureSecurityCenterAlerts] |2019-01-01 |2019-01-01|:heavy_check_mark:| -| [Azure Kubernetes Service][AzureKubernetesService] | 2019-08-01|2019-08-01|:heavy_check_mark:| -| [Azure Compute][AzureCompute] | 2019-03-01|2019-03-01|:heavy_check_mark:| -| [Azure Disks][AzureDisks] | 2019-03-01|2019-03-01|:heavy_check_mark:| -| [Azure Virtual Networks][AzureVirtualNetworks] | 2019-04-01|2019-04-01|:heavy_check_mark:| -| [Azure Network Interfaces][AzureNetworkInterfaces] |2019-04-01 |2019-04-01|:heavy_check_mark:| -| [Resource Groups][ResourceGroups] |2019-05-01 |2019-05-01|:heavy_check_mark:| -| [Azure Resources][AzureResources] |2019-10-01 |2019-10-01|:heavy_check_mark:| -| [Azure Subscriptions][AzureSubscriptions] |2020-01-01 |2020-01-01|:heavy_check_mark:| -| [Azure Locations][AzureLocations] |2019-05-01 |2019-05-01|:heavy_check_mark:| -| [Azure Role Assignments][AzureRoleAssignments] |2015-07-01 |2015-07-01|:heavy_check_mark:| -| [Azure Role Definitions][AzureRoleDefinitions] |2015-07-01 |2015-07-01|:heavy_check_mark:| -| [Azure Container Registry][AzureContainerRegistry] | 2017-10-01|2017-10-01|:heavy_check_mark:| -| Log Analytics Rest API ([1][LogAnalyticsRestAPI1], [2][LogAnalyticsRestAPI2]) |v1 |v1|:heavy_check_mark:| -| [Azure Active Directory Graph API][AzureActiveDirectoryGraphAPI] | 1.6|1.6 |:heavy_check_mark:| - -API URLs by environment: - -| API Name| API URL | Environment| -| --- | :---: | :---: | -| Management |https://management.azure.com/|Global| -| Azure AD Graph |https://graph.windows.net/|Global| -| Management |https://management.usgovcloudapi.net/|US Government| -| Azure AD Graph |https://graph.microsoft.us/|US Government| -| Management |https://management.chinacloudapi.cn/|China| -| Azure AD Graph |https://graph.chinacloudapi.cn/|China| - -
- -# Resource Providers requirements - -Although some of the Resource Providers might be enabled by default, you need to make sure that at least the **Microsoft.Advisor** and the **Microsoft.Security** resource providers are registered across all the subscriptions that you plan analyze using the Dashboard. - -Registering these 2 Resource Providers has no cost or performance penalty on the subscription: - -1. Click on **Subscriptions**. -2. Click on the Subscription name you want to configure. -3. Click on **Resource Providers**. -4. Click on **Microsoft.Advisor** and **Register**. -5. Click on **Microsoft.Security** and **Register**. - -![resource providers][ResourceProviders] - -
- -# Azure Advisor Recommendations -Azure Advisor is a personalized cloud consultant that helps you follow best practices to optimize your Azure deployments. It analyzes your resource configuration and usage telemetry. It then recommends solutions to help improve the performance, security, and high availability of your resources while looking for opportunities to reduce your overall Azure spend. - -The Continuous Optimization Power BI Dashboard will directly pull data from Azure Advisor REST APIs to aggregate all the information across the Azure account subscriptions. This requires generating the recommendations before the first time we load the template else the Dashboard will be empty or will fail because it was unable to download any data. - -To do so, you need to generate the recommendations for the first time manually from the Azure Portal, or programmatically using the script [GenerateAllSubscriptionsAdvisorRecommendations.ps1][GenerateAllSubscriptionsAdvisorRecommendations.ps1] - -## Generating Azure Advisor recommendations manually - -Open the Azure Portal with your Azure Account https://portal.azure.com - -1. Click on **Advisor**. -2. Expand the subscriptions drop-down menu. -3. Select the subscription you want to update or generate the recommendations for the first time. -4. Wait until the recommendations for the selected subscriptions has been loaded. -5. Repeat these steps for each subscription you want to generate Azure Advisor recommendations. - -![AdvisorRecommendations][AdvisorRecommendations] - -
- -# Azure Security Center Recommendations - -Azure Security Center provides unified security management and advanced threat protection for workloads running in Azure, on-premises, and in other clouds. It delivers visibility and control over hybrid cloud workloads, active defense that reduces your exposure to threats, and intelligent detection to help you keep pace with rapidly evolving cyber-attacks. - -You can find more information at the official Azure Security Center site [here][SecurityCenterIntro]. - -Azure Security Center is offered in two tiers: - -- Free -- Standard - -The Standard tier is offered [free for the first 60 days][FreeForTheFirst60Days]. - -The subscriptions will need to use the **Standard** tier if you want to detect and see the alerts in the Azure Security Center Alerts page of the dashboard. - -The following picture shows the steps to configure Azure Security Center plan for Azure Subscriptions - -1. Click on **Security Center**. -2. Click on **Click on top to learn more**. -3. Click on **Select the subscription you want to configure**. -4. Click on **Free** or **Standard** plan and the click **Save**. - -![SecurityCenterStandardRecommendations][SecurityCenterStandardRecommendations] - -
- -# Setting up the Continuous Cloud Optimization Azure Infrastructure Power BI Dashboard - -## Template download - -Download and open the `.pbit` file from [CCODashboard-Infra][CCODashboardInfra] folder. - -## Environment selection - -Before start loading data you need to select which type of environment you're using: - -- Select "Global" for Microsoft Azure commercial environments. This is the default selection. -- Select [US-Government][UsGovernment] for Azure Us government services. Azure Government is a separate instance of the Microsoft Azure service. It addresses the security and compliance needs of United States federal agencies, state and local governments, and their solution providers. -- **Preview feature:** Select [China][China] to load data from cloud applications in Microsoft Azure operated by 21Vianet (Azure China). - -![selector][Selector] - -## Modify Privacy settings - -- Go to File -> Options -> Privacy and set to Always ignore privacy level settings. - -![Privacy][Privacy] - -## Credentials - -By default, the template doesn’t have any Azure Account credentials preloaded. Hence, the first step to start showing subscriptions data is to sign-in with the right user credentials. - -**IMPORTANT NOTE**: Power BI Desktop caches the credentials after the first logon. It is important to clear the credentials from Power BI desktop if you plan to switch between Azure GLobal and any other region like US Government or China. The same concept applies if you plan to switch between tenants. Otherwise, the staged credentials will be used again for the different Azure environments and the authentication or data load process will fail. - -### Clean Credentials on the Data Source - -In some cases, old credentials are cached by previous logins using Power BI Desktop and the dashboard might show errors or blank fields. - -- Click on Data sources in **Current file/Global permissions**. -- Click on **Clear Permissions**. -- Click on **Clear All Permissions**. - -![credentials1][Credentials1] ![credentials2][Credentials2] - -### Refresh the dashboard - -If the permissions and credentials are properly flushed it should ask you for credentials for each REST API and you will have to set the Privacy Levels for each of them. - -- Click on **Refresh**. - -![credentials3][Credentials3] - -### Credentials for management.azure.com REST API request: - -- Click on **Organizational Account**. -- Click on **Sign in**. -- Click on **Connect**. - - -![credentials4][Credentials4] - -### Credentials for graph.windows.net API - -- Click on **Organizational Account**. -- Click on **Sign in**. -- Click on **Connect**. - -![credentials5][Credentials5] - -### Credentials for api.loganalytics.io API - -- Click on **Organizational Account**. -- Click on **Sign in**. -- Click on **Connect**. - -![loganalytics][LogAnalytics] - -### Enter Access Web content credentials - -- Make sure that you select **Organization account** type. -- Click on **Sign in**. - -![credentials7][Credentials7] - -
- -# Report Pages - -## CCO Azure Infrastructure Dashboard overview page - -In this page, you will be able to identify the top 5 of recommendations that Azure Advisor has identified, the top 10 most attacked resources and the number of subscription owners. You can also locate all the deployed resources in a map. -It’s important to mention that this tab just gives you a quick view. All the recommendations will be available with more details in the following tabs - -You can filter the information by: - -- Tenant -- Subscription -- Resource Tags - -![overview][Overview] - -## Azure Advisor Recommendations page - -In this page of the report, you will be able to identify the total amount of recommendations that Azure Advisor has identified, to what resources each recommendations apply and to what subscription as well. - -You can filter the information by: - -- Tenant -- Subscription -- Resource type - -It will also give a high-level overview of what subscriptions require more attention and has more recommendations to snooze or implement. - -If you navigate to a impacted resource you will see a quick description, potential solution and in some cases a link to a website where you can find all the steps to solve the problem. - -![advisor][Advisor] - -## Azure Security Center Alerts page - -The third tab is used to show the Azure Security Center Advanced Threat Analytics Alerts from all the subscriptions a given Azure account has access to. Is important to remark that subscriptions will need to use the Standard plan if you want to detect and see the alerts in the Power BI Dashboard. - -You can filter the information by: - -- Tenant -- Subscription -- Attack type -- Data range - -![security Center alerts][SecurityCenterAlerts] - -## Azure Compute Overview page - -In this tab, you will be able to identify the number of VMs, the Operating System, the SKU, the Availability Set name, the location, the VM Size, the VNET and subnet each VM is connected, the private IP address and if the VM has any extension installed. - -You can filter the information by: - -- Tenant -- Subscription -- Resource Group -- Vm extension - -![azure compute][IMG_AzureCompute] - -## Azure VNETs and Subnets Recommendations page - -In this tab, you will be able to identify VNETs with only one subnet, if there are any VNET peering and if some of the subnets is exhausting its IP Pool. - -You can filter the information by: - -- Tenant -- Subscription -- Resource Group -- VNET -- Subnet -- Networking Interface - -![azure networking][AzureNetworking] - -**IMPORTANT**: It is important to mention that although a VNET with only one subnet might not be an issue, it might be a good lead to investigate if that is the best network segmentation for the applications running on it. - -## Azure Network Security Groups page - -In this tab, you will be able to identify all the NSGs assigned to a VM or Subnet. On each one, you can check all the rules that are being applied - -You can filter the information by: - -- Tenant -- Subscription -- VM -- VNET -- Subnet -- NSG assignment - -![azure NSGs][AzureNSGs] - -## Role Based Access Control page - -This tab is used to show the Azure RBAC permissions from all the subscriptions a given Azure account has access to. You will be able to identify the roles applied to all Azure resources and if the subscriptions have custom roles. - -You can filter the information by: - -- Tenant -- Subscription -- Object type -- User - -![azure rbac][AzureRbac] - -## Service Principal Role Based Access Control page - -This tab is used to show Azure Services Principals RBAC permissions from all the subscriptions a given Azure account has access to. You will be able to identify the roles applied to all Azure resources and if the subscriptions have custom roles. - -You can filter the information by: - -- Tenant -- Subscription -- Object type -- User - -![azure rbacSP][AzureRbacSP] - -## IaaS Usage and Limits page - -This tab allows to identify the usage of any Compute, Storage and Networking Azure resource and validate the limits for each region and subscription. - -You can filter the information by: - -- Tenant -- Subscription -- Azure Region - -![usage and limits][UsageAndLimits] - -## IaaS Idle Resources Dashboard page - -This tab is lists all the Public IPs, Network Interfaces and Disks that are disconnected, idle or unattached. - -You can filter the information by: - -- Tenant -- Subscription - -![azure Idle][IdleResources] - -## Azure Kubernetes Service Dashboard Overview page - -In this page, you will be able to identify the number of AKS Clusters, Nodes, Pods, Containers, Service Principals and Azure Security Center recommendations. It’s important to mention that this tab just gives you a quick view. All the detailed information will be available in the following tab. - -You can filter the information by: - -- Subscription -- AKS Cluster - -![aks][Aks1] - -**IMPORTANT**: to receive all the information related to the Pods, Containers and Container Images a log analytics workspace configured **is required**. - - -## Azure Kubernetes Service page - -In this page, you will be able to identify the number of AKS Clusters, Nodes, Pods, Containers, Container images, Service principals and Azure Container Instances . All the information related to these resources will be shown (IPs, pods in use, status, network, image repositories, RBAC roles …). - -You can filter the information by: - -- Subscription -- AKS Cluster -- Namespace -- Cluster Node - -**IMPORTANT**: to receive all the information related to the Pods, Containers and Container Images a log analytics workspace configured **is required**. - - -![aks][Aks2] - -
- - -[OnboardToLighthouse]: -[AzureAdvisor]: -[AzureSecurityCenterAlerts]: -[AzureKubernetesService]: -[AzureCompute]: -[AzureDisks]: -[AzureVirtualNetworks]: -[AzureNetworkInterfaces]: -[ResourceGroups]: -[AzureResources]: -[AzureSubscriptions]: -[AzureLocations]: -[AzureRoleAssignments]: -[AzureRoleDefinitions]: -[AzureContainerRegistry]: -[LogAnalyticsRestAPI1]: -[LogAnalyticsRestAPI2]: -[AzureActiveDirectoryGraphAPI]: -[SecurityCenterIntro]: -[FreeForTheFirst60Days]: -[UsGovernment]: -[China]: - - -[ResourceProviders]: <./media/resourceproviders.png> -[AdvisorRecommendations]: <./media/AdvisorRecommendations.png> -[SecurityCenterStandardRecommendations]: <./media/EnableSecurityCenterStandard.png> -[Selector]: <./media/selector.png> -[Privacy]: <./media/governancePrivacy.png> -[Credentials1]: <./media/Credentials1.png> -[Credentials2]: <./media/Credentials2.png> -[Credentials3]: <./media/Credentials3.png> -[Credentials4]: <./media/Credentials4.png> -[Credentials5]: <./media/Credentials5.png> -[LogAnalytics]: <./media/loganalyticsAPI.PNG> -[Credentials7]: <./media/Credentials7.png> -[Overview]: <./media/OverviewImage.png> -[Advisor]: <./media/Advisor.png> -[SecurityCenterAlerts]: <./media/SecurityCenterAlerts.png> -[IMG_AzureCompute]: <./media/AzureCompute.png> -[AzureNetworking]: <./media/AzureNetworking.png> -[AzureNSGs]: <./media/NSGs.png> -[AzureRbac]: <./media/RBAC.png> -[AzureRbacSP]: <./media/RBACServicePrincipals.png> -[UsageAndLimits]: <./media/UsageAndLimits.png> -[IdleResources]: <./media/IdleResources.png> -[Aks1]: <./media/aks.PNG> -[Aks2]: <./media/aks2.png> - - - -[GenerateAllSubscriptionsAdvisorRecommendations.ps1]: -[CCODashboardInfra]: \ No newline at end of file diff --git a/docs/wiki/Troubleshooting Guide.md b/docs/wiki/Troubleshooting Guide.md index aea62d7..0aa21c9 100644 --- a/docs/wiki/Troubleshooting Guide.md +++ b/docs/wiki/Troubleshooting Guide.md @@ -11,6 +11,7 @@ - [Log Analytics REST API timeout (CCO AKS dashboard add-on only)](#log-analytics-rest-api-timeout-cco-aks-dashboard-add-on-only) - [Data Model Relationships missing](#data-model-relationships-missing) - [Errors regarding missing `column1` on refresh](#errors-regarding-missing-column1-on-refresh) + - [Custom connector not recognized](#custom-connector-not-recognized) --- @@ -103,6 +104,19 @@ Infrastructure: Try using a native user account to the AAD tenant you are connecting to instead of a guest user account. +## Custom connector not recognized + +To make sure the custom connector is configured correctly, please check the following security settings in Power BI Desktop. + +Under the File menu, click on the "Options and Settings" button, then chose Options. In the dialog, click on Security and select "Allow any extension to load without validation of warning". + +![PowerBISecurityConfiguration][PowerBISecurityConfiguration] + +Under the Home menu, click on "Get Data", then chose the "More..." option. Select Azure from the list on the left to filter the options. + +![PowerBIDataConnectors][PowerBIDataConnectors] + +Once this configuration is in place, you can use the solution to get data. [PowerBIDesktop]: @@ -119,5 +133,7 @@ Try using a native user account to the AAD tenant you are connecting to instead [WrongPrivacyLevelConfig]: <./media/WrongPrivacyLevelConfig.png> [Credentials5]: <./media/Credentials5.png> [LocaleOptionsPowerBI]: <./media/locale_options_powerBI.PNG> +[PowerBISecurityConfiguration]: <./media/PowerBI-SecurityConfiguration.png> +[PowerBIDataConnectors]: <./media/PowerBI-DataConnectors.png> diff --git a/docs/wiki/media/PowerBI-DataConnectors.png b/docs/wiki/media/PowerBI-DataConnectors.png new file mode 100644 index 0000000..98b8bcf Binary files /dev/null and b/docs/wiki/media/PowerBI-DataConnectors.png differ diff --git a/docs/wiki/media/PowerBI-SecurityConfiguration.png b/docs/wiki/media/PowerBI-SecurityConfiguration.png new file mode 100644 index 0000000..ba30301 Binary files /dev/null and b/docs/wiki/media/PowerBI-SecurityConfiguration.png differ