Analytics improvement #12176

BLasan · 2023-10-24T10:52:31Z

Purpose

fixes: Is Api using mediation policies api-manager#2214

Goals

Load API and Operation Policies of the APIs/API Products to the gateway and send the corresponding policies in the analytics event.

Approach

Load API Policies and Operation Policies to the gateway on the deployment.
Fetch Operation and API Policies related to the the resource and the method on the API invocation.
To Enable/Disable Policies in the event, a separate configuration enablePolicy is introduced under analytics configuration in deployment.toml file. If this is enabled then operation and API policies will get loaded to the analytics event

[apim.analytics]
enable = true
auth_token = ""
enablePolicy = true

tharikaGitHub · 2023-11-02T05:08:03Z

...cs/src/main/java/org/wso2/carbon/apimgt/common/analytics/publishers/dto/OperationPolicy.java

@@ -0,0 +1,70 @@
+/*
+ * Copyright (c) 2020, WSO2 Inc. (http://www.wso2.org) All Rights Reserved.


Let's change the license header

tharikaGitHub · 2023-11-02T05:08:32Z

...lytics/src/main/java/org/wso2/carbon/apimgt/common/analytics/publishers/dto/URITemplate.java

@@ -0,0 +1,65 @@
+package org.wso2.carbon.apimgt.common.analytics.publishers.dto;


Let's include the license header for this file.

tharikaGitHub · 2023-11-02T05:10:29Z

...ain/java/org/wso2/carbon/apimgt/gateway/handlers/analytics/SynapseAnalyticsDataProvider.java

@@ -151,6 +158,41 @@ public API getApi() throws DataNotFoundException {
 api.setApiVersion(apiObj.getApiVersion());
 api.setApiCreator(apiObj.getApiProvider());
 api.setApiCreatorTenantDomain(MultitenantUtils.getTenantDomain(api.getApiCreator()));
+ List<URITemplate> uriTemplates = new ArrayList<>();
+ for (URLMapping uriTemplate : apiObj.getUrlMappings()) {
+ org.wso2.carbon.apimgt.common.analytics.publishers.dto.URITemplate uriTemplateObj = new org.wso2.carbon.apimgt.common.analytics.publishers.dto.URITemplate();


Formatting has gone beyond 120 limit.

tharikaGitHub · 2023-11-02T05:11:09Z

...ain/java/org/wso2/carbon/apimgt/gateway/handlers/analytics/SynapseAnalyticsDataProvider.java

+ uriTemplateObj.setAuthScheme(uriTemplate.getAuthScheme());
+ List<org.wso2.carbon.apimgt.common.analytics.publishers.dto.OperationPolicy> operationPolicies = new ArrayList<>();
+ for (OperationPolicy operationPolicy : uriTemplate.getOperationPolicies()) {
+ org.wso2.carbon.apimgt.common.analytics.publishers.dto.OperationPolicy operationPolicyObj = new org.wso2.carbon.apimgt.common.analytics.publishers.dto.OperationPolicy();


Let's format the code.

tharikaGitHub · 2023-11-02T05:11:15Z

...ain/java/org/wso2/carbon/apimgt/gateway/handlers/analytics/SynapseAnalyticsDataProvider.java

+ }
+ List<org.wso2.carbon.apimgt.common.analytics.publishers.dto.OperationPolicy> apiPolicyList = new ArrayList<>();
+ for(OperationPolicy apiPolicy: apiObj.getApiPolicies()){
+ org.wso2.carbon.apimgt.common.analytics.publishers.dto.OperationPolicy operationPolicyObj = new org.wso2.carbon.apimgt.common.analytics.publishers.dto.OperationPolicy();


Let's format the code.

tharikaGitHub · 2023-11-02T05:12:10Z

...bon.apimgt.impl/src/main/java/org/wso2/carbon/apimgt/impl/dao/SubscriptionValidationDAO.java

+
+ }
+
+ private void attachPolicies(Connection connection, String revisionId, API api) throws SQLException {


Let's fix the rest of the formatting issues in this file.

tharikaGitHub · 2023-11-02T05:12:47Z

.../main/java/org/wso2/carbon/apimgt/impl/dao/constants/SubscriptionValidationSQLConstants.java

@@ -570,7 +570,26 @@ public class SubscriptionValidationSQLConstants {
 "AM_API_RESOURCE_SCOPE_MAPPING.SCOPE_NAME FROM AM_API_URL_MAPPING LEFT JOIN AM_API_RESOURCE_SCOPE_MAPPING" +
 " ON AM_API_URL_MAPPING.URL_MAPPING_ID=AM_API_RESOURCE_SCOPE_MAPPING.URL_MAPPING_ID WHERE " +
 "AM_API_URL_MAPPING.API_ID = ? AND AM_API_URL_MAPPING.REVISION_UUID = ?";
-
+// public static final String GET_OPERATION_POLICIES_PER_URI_BY_API_SQL = "SELECT AUM.HTTP_METHOD, AUM.URL_PATTERN, "


Let's remove the commented out code if not required.

tharikaGitHub · 2023-11-02T05:13:10Z

...nal.service/src/gen/java/org/wso2/carbon/apimgt/internal/service/dto/OperationPolicyDTO.java

@@ -0,0 +1,157 @@
+package org.wso2.carbon.apimgt.internal.service.dto;


Let's add the license header and remove wildcard imports.

This is a swagger generated file. Should we modify this class as well?

Yes that's right, no need to add the header here as it is an auto generated file.

codecov · 2023-11-15T10:44:54Z

Codecov Report

Attention: 191 lines in your changes are missing coverage. Please review.

Comparison is base (0457739) 39.36% compared to head (2a61dfa) 20.83%.
Report is 152 commits behind head on master.

Files	Patch %	Lines
...bon/apimgt/impl/dao/SubscriptionValidationDAO.java	0.00%	90 Missing ⚠️
.../service/utils/SubscriptionValidationDataUtil.java	0.00%	48 Missing ⚠️
...ndlers/analytics/SynapseAnalyticsDataProvider.java	0.00%	36 Missing ⚠️
.../wso2/carbon/apimgt/api/model/OperationPolicy.java	0.00%	4 Missing ⚠️
...wso2/carbon/apimgt/api/model/subscription/API.java	0.00%	4 Missing ⚠️
...rbon/apimgt/api/model/subscription/URLMapping.java	0.00%	4 Missing ⚠️
...rg/wso2/carbon/apimgt/keymgt/model/entity/API.java	25.00%	3 Missing ⚠️
...o2/carbon/apimgt/impl/APIManagerConfiguration.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #12176       +/-   ##
===========================================
- Coverage   39.36%   20.83%   -18.53%     
===========================================
  Files        1789     1193      -596     
  Lines      131570   101689    -29881     
  Branches    19008    14280     -4728     
===========================================
- Hits        51797    21191    -30606     
- Misses      73195    77710     +4515     
+ Partials     6578     2788     -3790

Flag	Coverage Δ
integration_tests	`?`
unit_tests	`20.83% <0.52%> (-0.05%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tharikaGitHub · 2023-12-21T05:43:44Z

...ain/java/org/wso2/carbon/apimgt/gateway/handlers/analytics/SynapseAnalyticsDataProvider.java

+ org.wso2.carbon.apimgt.common.analytics.publishers.dto.URITemplate uriTemplateObj
+ = new org.wso2.carbon.apimgt.common.analytics.publishers.dto.URITemplate();
+ if (uriTemplate.getHttpMethod() != null && uriTemplate.getHttpMethod()
+ .equals(messageContext.getProperty("api.ut.HTTP_METHOD")) && uriTemplate.getUrlPattern() != null


You can use the constant[1] here.

[1] https://github.com/wso2/carbon-apimgt/blob/master/components/apimgt/org.wso2.carbon.apimgt.gateway/src/main/java/org/wso2/carbon/apimgt/gateway/APIMgtGatewayConstants.java#L33

tharikaGitHub · 2023-12-21T05:44:09Z

.../main/java/org/wso2/carbon/apimgt/internal/service/utils/SubscriptionValidationDataUtil.java

@@ -100,8 +102,32 @@ private static APIDTO fromAPItoDTO(API model) {
 urlMappingDTO.setThrottlingPolicy(urlMapping.getThrottlingPolicy());
 urlMappingDTO.setUrlPattern(urlMapping.getUrlPattern());
 urlMappingDTO.setScopes(urlMapping.getScopes());
+ List<OperationPolicyDTO> operationPolicyDTOList = new ArrayList<>();
+ for(OperationPolicy operationPolicy: urlMapping.getOperationPolicies()) {


tharikaGitHub · 2023-12-21T05:44:13Z

.../main/java/org/wso2/carbon/apimgt/internal/service/utils/SubscriptionValidationDataUtil.java

 urlMappingsDTO.add(urlMappingDTO);
 }
+ List<OperationPolicyDTO> apiPolicies = new ArrayList<>();
+ for(OperationPolicy apiPolicy: model.getApiPolicies()) {


tharikaGitHub · 2023-12-21T05:44:17Z

.../main/java/org/wso2/carbon/apimgt/internal/service/utils/SubscriptionValidationDataUtil.java

@@ -132,8 +158,32 @@ public static APIListDTO fromAPIToAPIListDTO(API model) {
 urlMappingDTO.setThrottlingPolicy(urlMapping.getThrottlingPolicy());
 urlMappingDTO.setUrlPattern(urlMapping.getUrlPattern());
 urlMappingDTO.setScopes(urlMapping.getScopes());
+ List<OperationPolicyDTO> operationPolicyDTOList = new ArrayList<>();
+ for(OperationPolicy operationPolicy: urlMapping.getOperationPolicies()) {


tharikaGitHub · 2023-12-21T05:44:20Z

.../main/java/org/wso2/carbon/apimgt/internal/service/utils/SubscriptionValidationDataUtil.java

 urlMappingsDTO.add(urlMappingDTO);
 }
+ List<OperationPolicyDTO> apiPolicies = new ArrayList<>();
+ for(OperationPolicy apiPolicy: model.getApiPolicies()) {


Formatting issue. You can use https://tharika.medium.com/how-to-setup-code-formatting-in-my-ide-86374ec8fbd2 to fix the formatting using a template instead of requiring to manually do it.

tharikaGitHub · 2024-01-04T05:55:06Z

...bon.apimgt.impl/src/main/java/org/wso2/carbon/apimgt/impl/dao/SubscriptionValidationDAO.java

@@ -1170,10 +1177,16 @@ private void attachURlMappingDetailsOfApiProduct(Connection connection, API api)
 }
 }
 }
+
+ if(configs.containsKey(POLICY_ENABLED_FOR_ANALYTICS)) {


tharikaGitHub · 2024-01-04T05:55:10Z

...bon.apimgt.impl/src/main/java/org/wso2/carbon/apimgt/impl/dao/SubscriptionValidationDAO.java

@@ -1260,6 +1273,112 @@ private void attachURLMappingDetails(Connection connection, String revisionId, A
 }
 }
 }
+
+ if(configs.containsKey(POLICY_ENABLED_FOR_ANALYTICS)) {


BLasan · 2024-01-19T14:52:31Z

Replication of #12207

BLasan force-pushed the analytics-improvement branch 5 times, most recently from 587c32b to 7524f90 Compare October 27, 2023 14:58

tharikaGitHub reviewed Nov 2, 2023

View reviewed changes

BLasan mentioned this pull request Nov 20, 2023

[Doc]: Enabling/Disabling Operation Policies in Analytics Event wso2/docs-apim#7416

Open

BLasan force-pushed the analytics-improvement branch from 3c9071a to e5e052f Compare December 21, 2023 05:22

tharikaGitHub reviewed Dec 21, 2023

View reviewed changes

tharikaGitHub previously approved these changes Dec 21, 2023

View reviewed changes

BLasan dismissed tharikaGitHub’s stale review via cf9ace4 January 2, 2024 15:35

BLasan added 7 commits January 2, 2024 21:06

feat: send operation policies in analytics event

041951c

add: API Policies to the API object in gateway

445119e

remove: commented codes

cbb16d7

Feat: Adding Operation Policies to API Product

1da7188

Add: Parameters to the OperationPolicyDTO

028c809

Add: Populate operation policy parameters to the object

241507b

Improved the code quality

b371fd7

BLasan added 4 commits January 2, 2024 21:12

Add: System Property to enable operations policies in analytics event

fdafcd5

Fix: Code Format Issues

00cdb8a

Remove: Replicate Static Import

bf1e7f6

Add: Configuration to enable/disable Policy Loading

2a61dfa

BLasan force-pushed the analytics-improvement branch from cf9ace4 to 2a61dfa Compare January 2, 2024 15:52

tharikaGitHub reviewed Jan 4, 2024

View reviewed changes

BLasan closed this Jan 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Analytics improvement #12176

Analytics improvement #12176

BLasan commented Oct 24, 2023 •

edited

Loading

tharikaGitHub Nov 2, 2023

tharikaGitHub Nov 2, 2023

tharikaGitHub Nov 2, 2023

tharikaGitHub Nov 2, 2023

tharikaGitHub Nov 2, 2023

tharikaGitHub Nov 2, 2023

tharikaGitHub Nov 2, 2023

tharikaGitHub Nov 2, 2023

BLasan Dec 21, 2023

tharikaGitHub Dec 21, 2023

codecov bot commented Nov 15, 2023 •

edited

Loading

tharikaGitHub Dec 21, 2023

tharikaGitHub Dec 21, 2023

tharikaGitHub Dec 21, 2023

tharikaGitHub Dec 21, 2023

tharikaGitHub Dec 21, 2023

tharikaGitHub Jan 4, 2024

tharikaGitHub Jan 4, 2024

BLasan commented Jan 19, 2024

		@@ -0,0 +1,70 @@
		/*
		* Copyright (c) 2020, WSO2 Inc. (http://www.wso2.org) All Rights Reserved.

		@@ -0,0 +1,65 @@
		package org.wso2.carbon.apimgt.common.analytics.publishers.dto;


		}

		private void attachPolicies(Connection connection, String revisionId, API api) throws SQLException {

		@@ -0,0 +1,157 @@
		package org.wso2.carbon.apimgt.internal.service.dto;

Analytics improvement #12176

Analytics improvement #12176

Conversation

BLasan commented Oct 24, 2023 • edited Loading

Purpose

Goals

Approach

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 15, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BLasan commented Jan 19, 2024

BLasan commented Oct 24, 2023 •

edited

Loading

codecov bot commented Nov 15, 2023 •

edited

Loading