Need to support per API Log Filtering based on API Resource #1564
Comments
Update 2023-03-13Understood the codebase and identified what methods need to be changed |
Update 2023-03-14Identified that previously log info per API was added to the AM_API table. |
Update (2023-03-17)Added a new LOG_LEVEL column to AM_API_URL_MAPPING table to store the log status of the each resource and removed the LOG_LEVEL column from AM_API table. The problem with this approach is that
On 17 th I had a call with Arshash, Tharika and Dushani, We were decided to schedule a design review to talk about this issue. |
Update 2023-03-21Had the design review today. At this meeting we finalyzed the DB design. Attendees: Arshardh Ifthikar, Chamila Adhikarinayake, Isuru Udana, Nandika Jayawardana, Tharika Madurapperuma, Thilini Shanika Meeting Notes
|
Update 2023-03-23Changed the methods related to
According to the suggestions given by the design review.
|
Update 2023-03-24
|
Update 2023-03-28
|
Update 2023-03-30
|
Update 2023-04-04
|
Update 2023-04-07
|
Update 2023-04-12
|
Update 2023-04-20Had the code review on 18th, At this meeting I've showed a small demo of the implementation, and explained the codebase. Needs to be changed, Since if the context contains forward slashes, then whole splitting process will get wronged. Like this. And also Tharindu mentioned that it is not good to match resources using regex again in this method. Since regex matching is a very high CPU intensive task. Tharindu suggested to discuss this again in another meeting to find a solution to this. We had another discussion for this issue on 20th with Isuru Udana. Isuru mentioned that if we put a matching resource pattern property in a messageContext, then we can resolve this issue easily and it would be beneficial for future fixes as well. |
|
Date: 23rd & 24th May 2023 Tasks:
Update Currently working on building the branch from the public repository. It seems there is an issue with the dependencies in the local machines with has caused a compilations error.
|
|
Date: 25th of May 2023 Completed watching both the available recordings (code review & discussion on implementation with synapse) for this feature. The pack was built (along with the benchmark test failure) and the current fix was debugged for the current version of the fix using the branch https://github.com/shnrndk/carbon-apimgt/tree/ResourceLevelLogfilteringCopy. |
|
Update
In the current implementation, resource-level log details are identified in the logs handler which is executed at first before creating the API object from the API class. When logs are enabled this is called in 4 instances; handleRequestInFlow, handleRequestOutFlow, handleResponseInFlow and handleResponseOutFlow. Here, in the implementation, the identification of the resource is happening as follows.
In synapse, there is a comparison of resources happening using the findResource() method. Unlike to the approach above, this happens within process() method in the API class. By the time this happens, the message context is populated with the respective API object. Following is the flow here;
The difference between the two approaches are, considering where we are trying now to implement this solution, we do not have an API object when we execute the log handler. But in synapse, when it is doing the matching for resources, the API is already process and the object is there. The class execution happens in the following order;
Hence, in order to get a similar implementation here, we need to figure out a way to create an api object in the logs handler and then using that try to create similar methods to do the comparison. |
|
Update
The API collection can be obtained from the message context using the synapse configuration. Using that, we can iterate through the api set to identify the API. In the below code, as for testing, the method identifyAPI() has been taken as it is from synapse. Once an API is identified, the resources can be obtained from the selected api and then proceed with step 3 & 4 as in above.
This approach cannot be directly implemented here since in identifyAPI(), the API is processed which is done in synapses not in the global handle. Hence for identification of the API a different approach is needed to be taken into consideration.
Currently, API resource path is identified by splitting the api context received from getTransportInURL().
Here, we make an assumption that the api resource path is as follows; ex: api/version/resource/identifier. This is directly compared with the resource path in log properties. However, we need to have a better approach to find the resource first from an API and that resource should be compared with the resource path in log properties.
When it comes to the resources in log properties, these log-enabled resources may belong to multiple APIs. Since now the API object is involved in this scenario, we need to identify the API as well in this mechanism without processing the API. |
|
Update Next Steps Resource
APIs
|
|
Update
Using this approach allows to replace the IdentifyAPI() method which processes the API. However, dropping this method causes a java.lang.NullPointerException when the resources are being searched for REST methods that were not invoked. ex: OPTIONS method run first when GET is called. And when resources are searched for OPTIONS, the following error happens. |
|
Update Current implementation to identify the resource |
used the implementation of URITemplateBasedDispatcher for resource matching issue: wso2/api-manager#1564
|
Update The following fixes were done
|
|
Update
|
|
Note: API Resource Log Filtering happens at the global handler through the getAPILogLevel() method in the LogsHandler class. When this method is called, we are yet to process both the API and the Resouces, which is done by the process() method in the API class in the synapse. Hence, many parameters in the message context are not set/configured at this moment. In order to do log filtering for resources, first we need to identify the resource based on the information available in the message context. In order to get that specific information needed to identify the resource, we need to find the specific API that has been invoked from the collection of the APIs. To do that, we can use the identifyAPI() method from AbstractApiHandler class in wso2-synapse. Once both the API and the resource has been identified, it will be matched with the map of resources in logProperties to log the resource based on both resource name and resource method. This will be done for handleRequestInFlow, handleRequestOutFlow, handleResponseInFlow and handleResponseOutFlow. To find the resource, we use findResouce methods from wso2-synapse. When running this method, it was modified to set the selected object to the message context to reduce repetitive executions of the methods in instances such as CORSRequestHandler, etc. The resource path is matched using the dispatcher helper logic. -> helper.getString().equals(key.get("resourcePath") To find the API, we can use multiple options. One option is the match the API context path by matching the URL that can be obtained from axis2MsgContext.getProperty("TransportInURL") and the inbuild getContext() method of the API. Another approach is to match the API context using the inbuild getConext() method of the API and the matchApiPath() method from ApiUtils in synaspe. Yet, we have a specific method in synapse to find the API, called identifyAPI() method. The current approach was developed by getting the necessary parts from that method and dispatchToAPI () method to come up with new methods called getSelectedAPI() and identifySelectedAPI (). However, all these happen before the API is being processed at the global handler. Hence, when the API is processed the same code segment is called in the identifyAPI() method. Yet, since process() method is written inside this method, we cannot call this method in the global handler. Hence we have to study dispatchToAPI(), identifyAPI(), canProcess() and process() methods and refactor the methods in a way where we can isolate identifying an API and processing an API. |
Update 05/07/2023I have started to work on this feature on 29/07/2023. Had a call with Chamika to get an idea on how to refactor the identifyAPI by isolating the logic on identifying API. Setting up the RESTConstants.REST_FULL_REQUEST_PATH property is happening inside the ApiUtils.getFullRequestPath(synCtx) which is located inside the getIdentifiedApi method.;. As per the suggestions I have modified the if condition and the I have reduced the canProcess to the original method only. |
Issue: wso2/api-manager#1564 (cherry picked from commit f0010ed)
used the implementation of URITemplateBasedDispatcher for resource matching issue: wso2/api-manager#1564 (cherry picked from commit 5b3818c)
Update 18/07/2023
|
Update 24/07/2023
|
Update 31/07/2023
|
|
Closing as the task is completed. |
Description
Currently we do not have the capability to enable per API Logging based on API Resource as well. We need to add an improvement to have this capability in the product.
As an additional requirement, the changes to this log configuration should be logged as audit logs. These audit logs should contain the timestamp, type of the event, subject identified (if applicable) and the outcome (Success / Failure).
Affected Component
APIM
Version
4.2.0
Related Issues
No response
Suggested Labels
No response
The text was updated successfully, but these errors were encountered: