New accounting check tool

README.md

@@ -1,17 +1,12 @@
-# fedcloud-vm-monitoring
+# fedcloud-monitoring-tools
 
-This repository contains a Python tool to monitor usage of EGI FedCloud
-providers and remove long-running instances. The clients work with OpenStack
-cloud providers supporting the OIDC protocol.
+This repository contains a set of Python tools to monitor usage of EGI FedCloud
+providers. The clients work with OpenStack cloud providers supporting the OIDC
+protocol.
 
 ## Requirements
 
 - Python v3.9+
-- A Check-in account member of the VOs to be monitored
-- For getting the EGI user identity, cloud providers have to enable the
-  `"identity:get_user"` API call for the user (see
-  [VO auditing](https://docs.egi.eu/providers/cloud-compute/openstack/aai/#vo-auditing)
-  for more information)
 
 ## Installation
 
@@ -25,7 +20,20 @@ Some sites use certificates issued by certificate authorities that are not
 included in the default OS distribution, if you find SSL errors, please
 [install the EGI Core Trust Anchors certificates](https://fedcloudclient.fedcloud.eu/install.html#installing-egi-core-trust-anchor-certificates)
 
-## Running the monitor
+## fedcloud-vo-monitor
+
+`fedcloud-vo-monitor` checks the usage of a VO (e.g. running VMs, floating IPs
+allocated, security groups) and identifies potential issues in the running VMs.
+
+### Requirements
+
+- A Check-in account member of the VOs to be monitored
+- For getting the EGI user identity, cloud providers have to enable the
+  `"identity:get_user"` API call for the user (see
+  [VO auditing](https://docs.egi.eu/providers/cloud-compute/openstack/aai/#vo-auditing)
+  for more information)
+
+### Running the monitor
 
 For running the tool, you just need a
 [valid Check-in token](https://docs.egi.eu/users/aai/check-in/obtaining-tokens/),
@@ -47,8 +55,8 @@ You can tune the behavior with the following parameters:
 - `--show-quotas BOOLEAN`: whether to show quotas for the VO or not (default:
   `True`)
 - `--check-ssh BOOLEAN`: Check SSH version on target VMs (default: `False`)
-- `--check-cups BOOLEAN`: Check whether TCP/UDP port 631 is accessible
-  (default: `False`)
+- `--check-cups BOOLEAN`: Check whether TCP/UDP port 631 is accessible (default:
+  `False`)
 
 If you have access to
 [Check-in LDAP](https://docs.egi.eu/users/aai/check-in/vos/#ldap) for VO
@@ -60,7 +68,7 @@ membership, you can specify the settings with the following options:
 The `ldap-server`, `ldap-base-dn` and `ldap-search-filter`, can further tune the
 usage of LDAP, but should work for most cases without changes.
 
-### Sample output
+#### Sample output
 
 ```shell
 $ fedcloud-vo-monitor --vo cloud.egi.eu
@@ -193,7 +201,33 @@ Getting VMs information  [####################################]  100%
 [-] WARNING: Less than 3 security groups per instance
 ```
 
+## fedcloud-sla-monitor
+
+`fedcloud-sla-monitor` checks the configuration of sites supporting SLAs. It
+compares the reported usage in the accoutnting portal and the information
+retrieved from the cloud-info-provider and reports any deviations.
+
+### Requirements
+
+- An IGTF certificate to query GOCDB SLA lists
+
+### Running the monitor
+
+
+```shell
+$ fedcloud-sla-monitor --help
+Usage: fedcloud-sla-monitor [OPTIONS]
+
+Options:
+  --site TEXT         Site to check
+  --user-cert TEXT    User certificate (for GOCDB queries)  [required]
+  --vo-map-file TEXT  SLA-VO mapping file
+  --help              Show this message and exit.
+```
+
 ## Useful links
 
 - [OpenStack API](https://docs.openstack.org/api-ref/)
 - [OpenStack API examples](https://docs.openstack.org/keystone/pike/api_curl_examples.html)
+
+

fedcloud_vm_monitoring/accounting.py

@@ -0,0 +1,57 @@
+"""Class for interaction with the accounting portal"""
+
+import datetime
+import numbers
+
+import httpx
+
+ACCOUNTING_URL = "https://accounting.egi.eu/"
+SITE_VO_ACCOUNTING = (
+    "cloud/sum_elap_processors/SITE/VO/"
+    "{start_year}/{start_month}/{end_year}/{end_month}"
+    "/all/onlyinfrajobs/JSON/"
+)
+
+
+class Accounting:
+    def __init__(self):
+        self._data = {}
+
+    def _get_accounting_data(self):
+        """Gets accounting data for sites / vos over the last 90 days"""
+        today = datetime.date.today()
+        start = today - datetime.timedelta(days=90)
+        url = ACCOUNTING_URL + SITE_VO_ACCOUNTING.format(
+            start_year=start.year,
+            start_month=start.month,
+            end_year=today.year,
+            end_month=today.month,
+        )
+        # accounting generates a redirect here
+        r = httpx.get(url, follow_redirects=True)
+        self._data = r.json()
+        return self._data
+
+    def site_vos(self, site):
+        if not self._data:
+            self._get_accounting_data()
+        for col in self._data:
+            if col["id"] == site:
+                return set(
+                    [
+                        vo[0]
+                        for vo in col.items()
+                        if isinstance(vo[1], numbers.Number)
+                        and vo[1] != 0
+                        and vo[0] not in ["Total", "Percent"]
+                    ]
+                )
+        return set([])
+
+    def all_sites(self):
+        if not self._data:
+            self._get_accounting_data()
+        for col in self._data:
+            if col["id"] == "xlegend":
+                return [site[1] for site in col.items() if site[0] != "id"]
+        return []

fedcloud_vm_monitoring/appdb.py

@@ -2,7 +2,7 @@
 
 import requests
 
-site_query = """
+sites_supporting_vo_query = """
 {
   sites(filter: {cloudComputingShares: {VO: {eq: "%s"}}}) {
     items {
@@ -11,24 +11,49 @@
   }
 }"""
 
+vos_in_site_query = """
+{
+  sites(filter: {name: {eq: "%s"}}) {
+    items {
+      cloudComputingShares {
+        items {
+          VO
+        }
+      }
+    }
+  }
+}"""
+
 
 class AppDB:
     graphql_url = "https://is.appdb.egi.eu/graphql"
 
-    def __init__(self, vo):
-        self.vo = vo
+    def __init__(self):
         self.sites = {}
 
-    def get_sites_for_vo(self):
-        params = {"query": site_query % self.vo}
+    def get_sites_for_vo(self, vo):
+        params = {"query": sites_supporting_vo_query % vo}
         r = requests.get(
             self.graphql_url, params=params, headers={"accept": "application/json"}
         )
         r.raise_for_status()
         data = r.json()["data"]["sites"]["items"]
         return [i["name"] for i in data]
 
-    def vo_check(self, site):
+    def vo_check(self, site, vo):
         if not self.sites:
-            self.sites = self.get_sites_for_vo()
+            self.sites = self.get_sites_for_vo(vo)
         return site in self.sites
+
+    def get_vo_for_site(self, site):
+        params = {"query": vos_in_site_query % site}
+        r = requests.get(
+            self.graphql_url, params=params, headers={"accept": "application/json"}
+        )
+        r.raise_for_status()
+        sites_items = r.json()["data"]["sites"]["items"]
+        if sites_items:
+            data = sites_items.pop()["cloudComputingShares"]["items"]
+        else:
+            return []
+        return [i["VO"] for i in data]

fedcloud_vm_monitoring/data/vos.yaml

@@ -0,0 +1,146 @@
+D4SCIENCE:
+  - d4science.org
+WENMR:
+  - enmr.eu
+OBSEA:
+  - vo.obsea.es
+NBISBILS:
+  - vo.nbis.se
+MSO4SC:
+  - imath.cesga.es
+BIOISI:
+  - bioisi
+TERRADUE:
+  - geohazards.terradue.com
+GEODAB:
+  - vo.geoss.eu
+CLARIN:
+  - vo.clarin.eu
+DEIMOS:
+  - vo.nextgeoss.eu
+EMSOERIC:
+  - vo.emso-eric.eu
+TrainingInfrastructure:
+  - training.egi.eu
+Notebooks:
+  - vo.notebooks.egi.eu
+AoD:
+  - vo.access.egi.eu
+EXTraS:
+  - extras-fp7.eu
+Fusion:
+  - fusion
+LSGC:
+  - biomed
+MRILab:
+Peachnote:
+  - peachnote.com
+OPENBIOMAP:
+  - vo.openbiomaps.org
+IIASA:
+EMPHASIS:
+  - vo.emphasisproject.eu
+EOSCSYNERGY:
+  - eosc-synergy.eu
+  - worsica.vo.incd.pt
+  - cryoem.instruct-eric.eu
+  - lagoproject.net
+  - umsa.certi-sc-cz
+  - mswss.ui.savbka.sk
+  - o3as.data.kit.edu
+VESPA:
+  - vo.europlanet-vespa.eu
+ECRIN:
+  - vo.crmdr.org
+STARS4ALL:
+  - vo.stars4all.eu
+GOSAFE:
+  - gosafe.eng.it
+DIGITBRAIN:
+  - vo.digitbrain.eu
+BELLE2:
+  - belle
+OPERAS:
+  - vo.operas-eu.org
+DEEP:
+  - deep-hybrid-datacloud.eu
+POLICYCLOUD:
+  - vo.ai4publicpolicy.eu
+CSCALE:
+  - aquamonitor.c-scale.eu
+  - eval.c-scale.eu
+  - terrascope.c-scale.eu
+  - waterwatch.c-scale.eu
+  # these below are actually removed
+  - HighResLandSurf.c-scale.eu
+  - return.c-scale.eu
+  - hisea.c-scale.eu
+  - coastmonitor.c-scale.eu
+  - in-sar-cubes.c-scale.eu
+  - plankton.c-scale.eu
+  - lost-salvage.c-scale.eu
+  - gltfca.c-scale.eu
+  - geohazards.c-scale.eu
+  - pangeo.c-scale.eu
+EISCAT3D:
+  - eiscat.se
+BINARE:
+  - vo.binare-oy.eu
+COS4CLOUD:
+  - cos4cloud-eosc.eu
+PEROVSKITE:
+  - perla-pv.ro
+ENES:
+  - vo.enes.org
+MINKE:
+  - minka-sdg.org
+BD4NRG:
+  - vo.bd4nrg.eu
+PITHIANRF:
+  - vo.esc.pithia.eu
+  - vo.pithia.eu
+CEITEC: []
+PLOCAN:
+  - vo.plocan.eu
+ENVRIFAIR:
+  - vo.envri-fair.eu
+ICECUBE:
+  - icecube
+MATRYCS:
+  - vo.matrycs.eu
+OPENCOASTS:
+  - opencoast.eosc-hub.eu
+AIIDALAB:
+  - vo.max-centre.eu
+PANGEO:
+  - vo.pangeo.eu
+LETHE:
+  - vo.lethe-project.eu
+LATITUDO40:
+  - vo.latitudo40.com.eu
+EUROSCIENCEGATEWAY:
+  - vo.usegalaxy.eu
+CESSDA:
+  - vo.cessda.eduteams.org
+SEADATANET:
+  - vo.seadatanet.org
+ERIES:
+  - vo.eries.eu
+EUROSEA:
+  - vo.eurosea.marine.ie
+ANERIS:
+  - vo.aneris.eu
+OPENRISKNET:
+  - openrisknet.org
+OIPUB:
+  - vo.oipub.com
+DECIDO:
+  - vo.decido-project.eu
+AI4PUBLICPOLICY:
+  - vo.ai4publicpolicy.eu
+PSMA:
+  - vo.radiotracers4psma.eu
+EUREKA3D:
+  - culturalheritage.vo.egi.eu
+NEURODESK:
+  - vo.neurodesk.eu