Merge pull request #7771 from RakhithaRR/master-ws-perf

Update streaming perf results for 4.3.0
wso2 · Mar 18, 2024 · 0ce62de · 0ce62de
2 parents 0cff709 + 9bfa7f1
commit 0ce62de
Show file tree

Hide file tree

Showing 10 changed files with 24 additions and 57 deletions.
diff --git a/...d-install/performance-test-results/websocket-cpu-mem-graphs/1000connections.png b/...d-install/performance-test-results/websocket-cpu-mem-graphs/1000connections.png
diff --git a/...nd-install/performance-test-results/websocket-cpu-mem-graphs/100connections.png b/...nd-install/performance-test-results/websocket-cpu-mem-graphs/100connections.png
diff --git a/...nd-install/performance-test-results/websocket-cpu-mem-graphs/200connections.png b/...nd-install/performance-test-results/websocket-cpu-mem-graphs/200connections.png
diff --git a/...nd-install/performance-test-results/websocket-cpu-mem-graphs/300connections.png b/...nd-install/performance-test-results/websocket-cpu-mem-graphs/300connections.png
diff --git a/...nd-install/performance-test-results/websocket-cpu-mem-graphs/400connections.png b/...nd-install/performance-test-results/websocket-cpu-mem-graphs/400connections.png
diff --git a/...nd-install/performance-test-results/websocket-cpu-mem-graphs/600connections.png b/...nd-install/performance-test-results/websocket-cpu-mem-graphs/600connections.png
diff --git a/...nd-install/performance-test-results/websocket-cpu-mem-graphs/800connections.png b/...nd-install/performance-test-results/websocket-cpu-mem-graphs/800connections.png
diff --git a/...and-install/performance-test-results/websocket-cpu-mem-graphs/cpu_usage_430.png b/...and-install/performance-test-results/websocket-cpu-mem-graphs/cpu_usage_430.png
diff --git a/...and-install/performance-test-results/websocket-cpu-mem-graphs/mem_usage_430.png b/...and-install/performance-test-results/websocket-cpu-mem-graphs/mem_usage_430.png
diff --git a/...ll-and-setup/setup/deployment-best-practices/performance-tests-results-async.md b/...ll-and-setup/setup/deployment-best-practices/performance-tests-results-async.md
@@ -25,7 +25,7 @@ The performance tests were conducted for streaming APIs of WSO2 API Manager. Tes
     <td>Apache JMeter Client</td>
     <td>c5.large</td>
     <td>2</td>
-    <td>2</td>
+    <td>4</td>
   </tr>
   <tr>
     <td>Apache JMeter Server 01</td>
@@ -55,43 +55,25 @@ The performance tests were conducted for streaming APIs of WSO2 API Manager. Tes
     <td>WSO2 API Manager</td>
     <td>c5.large</td>
     <td>2</td>
-    <td>2</td>
+    <td>4</td>
   </tr>
 </tbody>
 </table>
 
 - Token type: JWT
 - The operating system is Ubuntu 18.04
-- MySQL version in RDS instance is 5.7
-- Java version is JDK 11
+- MySQL version in RDS instance is 8.0
+- Java version is Temurin JDK 17
 
 ### Performance test scripts
 
 All scripts used to run the performance tests and analyze results are in the following repositories.
 
-- [https://github.com/wso2/performance-common](https://github.com/wso2/performance-common)
-
 - [https://github.com/wso2/performance-apim](https://github.com/wso2/performance-apim)
 
 ## Results
 
-## Server Sent Events
-
-Test for SSE API was done by using an SSE back end which emitted events in different frequencies. The error rate with 
-concurrent users and the CPU and memory percentage consumed was recorded at that time.
-
-The following graph depicts the Error %, CPU % and Memory % variation in the server during the time in which the 
-request was processed.
-
-[![Server Sent Events Test Results]({{base_path}}/assets/img/setup-and-install/performance-test-results/sse_perf_test_results.png)]({{base_path}}/assets/img/setup-and-install/performance-test-results/sse_perf_test_results.png)
-
-**Key observations:**
-
-- Five thousand (5000) concurrent connections can be handled with zero errors, with an event rate of 10 Event/sec (Average size of an Event used was 82B)
-
-- CPU spike is expected during the processing time, however it will settle once the requests are served.
-
-## WebSub 
+### WebSub 
 
 Test for WebSub API was done for varying number of subscribers and measuring the associated error, CPU and Memory % 
 consumed.
@@ -112,59 +94,44 @@ this will settle when the events are delivered.
     10,000 concurrent users mean a lot, and it is not very common. It is recommended to scale horizontally or 
     vertically to support more concurrent users with acceptable response times. When scaling horizontally, two or more Gateway nodes need to be used with a load balancer. Another load test must be carried out to measure the performance after scaling.
 
-## WebSocket
-
-Test for WebSocket API was done using a WebSocket server, which sends back the reversed message received from the client. The ramp up period, error rate, concurrent connections, and CPU and memory usages were recorded during that time.
-
-The following graph depicts the ramp up period with error rate.
+### WebSocket
 
-<img src="{{base_path}}/assets/img/setup-and-install/performance-test-results/ws-streaming-api-ramp-up-vs-error-rate.png" alt="websocket_ramp_up_period_vs_error_rate" width="1000"/>
+Test for WebSocket API was done using a WebSocket server, which sends back the reversed message received from the client. The CPU and memory usages for concurrent connections were recorded during that time. The tests were run for 30 seconds.
 
-The following table shows the number of concurrent connections with error rate. A 500 bytes string payload was used as the message. Size of the payload was reduced when non-zero error rates were observed without any failed connections.
+The following table shows the number of concurrent connections with error rate. A 500 bytes string payload was used as the message.
 
 | Events/second  | Message Size | Attempted Concurrent Connections | Failed Connections | Error Rate |
 |----------------|--------------|----------------------------------|--------------------|------------|
 | 1              | 500 bytes    | 100                              | 0                  | 0%         |
 |                |              | 200                              | 0                  | 0%         |
 |                |              | 300                              | 0                  | 0%         |
 |                |              | 400                              | 0                  | 0%         |
-|                |              | 600                              | 48                 | 0%         |
-|                |              | 800                              | 78                 | 0%         |
-|                |              | 1000                             | 74                 | 0%         |
+|                |              | 600                              | 0                  | 0%         |
+|                |              | 800                              | 0                  | 0%         |
+|                |              | 1000                             | 0                  | 0%         |
 | 5              | 500 bytes    | 100                              | 0                  | 0%         |
 |                |              | 200                              | 0                  | 0%         |
 |                |              | 300                              | 0                  | 0%         |
 |                |              | 400                              | 0                  | 0%         |
-|                |              | 600                              | 42                 | 32.90%     |
-|                |              | 800                              | 64                 | 65.25%     |
-|                |              | 1000                             | 62                 | 70.14%     |
+|                |              | 600                              | 0                  | 3.24%      |
+|                |              | 800                              | 0                  | 44.54%     |
+|                |              | 1000                             | 0                  | 45.59%     |
 | 10             | 500 bytes    | 100                              | 0                  | 0%         |
 |                |              | 200                              | 0                  | 0%         |
-|                |              | 300                              | 0                  | 46.10%     |
-|                | 250 bytes    | 300                              | 0                  | 14.29%     |
-|                | 125 bytes    | 300                              | 0                  | 12.70%     |
-|                | 60 bytes     | 300                              | 0                  | 29.12%     |
-|                |              | 400                              | 4                  | 62.17%     |
-|                |              | 600                              | 56                 | 85.14%     |
+|                |              | 300                              | 0                  | 0%         |
+|                |              | 400                              | 0                  | 16.72%     |
+|                |              | 600                              | 0                  | 57.60%     |
+
+!!! note
+      The backend used here is a Tornado backend. The error rates observed in the above graph are due to limitations in the Tornado backend. API Manager
+      was able to succesfully handle upto 1000 concurrent connections even in the 10 events per second scenario.
 
 **Key observations:**
 
-- 400 concurrent connections can be made with zero error rate and without any failed connections, at five events per second.
-- 200 concurrent connections can be made with zero error rate and without any failed connections, at ten events per second.
+- 1000 concurrent connections can be made without any failed connections, at ten events per second.
 
 The following graphs depict the variation of CPU and Memory % with concurrent connections.
 
-<img src="{{base_path}}/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/100connections.png" alt="100_connections" width="500"/>
-
-<img src="{{base_path}}/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/200connections.png" alt="200_connections" width="500"/>
-
-<img src="{{base_path}}/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/300connections.png" alt="300_connections" width="500"/>
-
-<img src="{{base_path}}/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/400connections.png" alt="400_connections" width="500"/>
-
-<img src="{{base_path}}/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/600connections.png" alt="600_connections" width="500"/>
-
-<img src="{{base_path}}/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/800connections.png" alt="800_connections" width="500"/>
-
-<img src="{{base_path}}/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/1000connections.png" alt="1000_connections" width="500"/>
+<img src="{{base_path}}/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/cpu_usage_430.png" alt="CPU Usage" width="900"/>
 
+<img src="{{base_path}}/assets/img/setup-and-install/performance-test-results/websocket-cpu-mem-graphs/mem_usage_430.png" alt="Memory Usage" width="900"/>