asgardeo · UdeshAthukorala · Aug 6, 2026
diff --git a/benchmarks/v1.0.0-alpha/kubernetes/readme.md b/benchmarks/v1.0.0-alpha/kubernetes/readme.md
@@ -0,0 +1,126 @@
+Version: v1.0.0-alpha
+
+Deployment Pattern: kubernetes (Azure AKS)
+
+Thunder Image: ghcr.io/thunder-id/thunderid:1.0.0-alpha
+
+Database Type: Postgres (DB) with In-Memory caching
+
+Performance Repo: https://github.com/asgardeo/thunder-performance
+
+Test Configuration: USE_DELAYS=true, Warm-up: 2m, Test Duration: 15m per scenario
+
+> **Note on the 10000 concurrent user run:** that run is not a valid benchmark. Error rates were
+> 93-100% across every scenario. The single JMeter client VM (Standard_F8s_v2, 8 vCore) was
+> saturated (load average ~100-126) and had exhausted its ephemeral ports (9963 sockets in
+> TIME_WAIT at the end of the Client Credentials scenario). It is recorded here for traceability
+> only, and its numbers must not be quoted as Thunder throughput. See run 2 below.
+
+
+## Summary
+
+Only the 1000 concurrent user run produced usable numbers.
+
+| Scenario Name | Heap Size | Concurrent Users | Label | # Samples | Error % | Throughput (Requests/sec) | Average Response Time (ms) | 95th Percentile of Response Time (ms) |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| Client Credentials Grant Type | N/A | 1000 | 1 Get access token | 2527085 | 0.30 | 3228.94 | 308.20 | 1367.00 |
+| Authorization Code Grant Type | N/A | 1000 | 1 Send request to authorize endpoint | 127987 | 0.00 | 164.11 | 20.57 | 36.00 |
+| Authorization Code Grant Type | N/A | 1000 | 2 Start Authentication Flow | 127985 | 0.01 | 164.10 | 12.36 | 32.00 |
+| Authorization Code Grant Type | N/A | 1000 | 3 Perform authentication | 127993 | 0.08 | 164.11 | 26.16 | 39.00 |
+| Authorization Code Grant Type | N/A | 1000 | 4 Obtain authorization code | 127995 | 0.04 | 164.12 | 15.43 | 37.00 |
+| Authorization Code Grant Type | N/A | 1000 | 5 Obtain access token | 127989 | 0.09 | 164.11 | 12.97 | 30.00 |
+| User Authentication with Credentials | N/A | 1000 | 1 Perform user authentication | 3468259 | 0.00 | 4443.35 | 224.35 | 1151.00 |
+
+The Authorization Code scenario carries a Gaussian think-time delay of 6000ms (range 2000ms)
+between iterations because USE_DELAYS was true, so its throughput is delay-bound rather than
+server-bound. The Client Credentials and User Authentication scenarios have no such timer and
+run at maximum throughput.
+
+
+## Test Runs
+
+### 1. All Scenarios — 1000 Concurrent Users
+
+Test Duration: 15m per scenario
+
+Warm-up Time: 2m
+
+USE_DELAYS: true
+
+Total Samples: 6635293
+
+Data: [results-1000conc.csv](results-1000conc.csv)
+
+**K8s Spec**
+
+- AKS Version: 1.34
+- Node Pool: Standard_F8s_v2, count: 2, autoscale min: 2, max: 5, max-pods-per-node: 30
+- Thunder Pods: requests 1 vCore + 256Mi, limits 1.5 vCore + 512Mi, replicas: 2, HPA max-pods: 10 (CPU 65%, Memory 75%)
+- Nginx Ingress Pods: requests 0.5 vCore + 500Mi, limits 1 vCore + 1000Mi, min-pods: 2, max-pods: 5 (CPU 80%, Memory 80%)
+- JMeter Client VM: Standard_F8s_v2, Heap Size: 4g
+- Database: Postgres 17
+- Caching: In-Memory
+- DB Specs:
+  - Config: GP_Standard_D2s_v3
+  - Runtime: GP_Standard_D2s_v3
+  - User: GP_Standard_D2s_v3
+
+| Scenario Name | Concurrent Users | Label | # Samples | Error Count | Error % | Throughput (Requests/sec) | Average Response Time (ms) | 95th Percentile of Response Time (ms) |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| Client Credentials Grant Type | 1000 | 1 Get access token | 2527085 | 7558 | 0.30 | 3228.94 | 308.20 | 1367.00 |
+| Authorization Code Grant Type | 1000 | 1 Send request to authorize endpoint | 127987 | 4 | 0.00 | 164.11 | 20.57 | 36.00 |
+| Authorization Code Grant Type | 1000 | 2 Start Authentication Flow | 127985 | 8 | 0.01 | 164.10 | 12.36 | 32.00 |
+| Authorization Code Grant Type | 1000 | 3 Perform authentication | 127993 | 106 | 0.08 | 164.11 | 26.16 | 39.00 |
+| Authorization Code Grant Type | 1000 | 4 Obtain authorization code | 127995 | 50 | 0.04 | 164.12 | 15.43 | 37.00 |
+| Authorization Code Grant Type | 1000 | 5 Obtain access token | 127989 | 109 | 0.09 | 164.11 | 12.97 | 30.00 |
+| User Authentication with Credentials | 1000 | 1 Perform user authentication | 3468259 | 136 | 0.00 | 4443.35 | 224.35 | 1151.00 |
+
+### 2. All Scenarios — 10000 Concurrent Users (INVALID — load generator saturated)
+
+Test Duration: 15m per scenario
+
+Warm-up Time: 2m
+
+USE_DELAYS: true
+
+Total Samples: 12763069
+
+Data: [results-10000conc.csv](results-10000conc.csv)
+
+**Why this run is not usable**
+
+- Error rates of 93.39% (Client Credentials), 98.23-100.00% (Authorization Code steps) and 99.85%
+  (User Authentication).
+- JMeter client load average reached 125.91 / 91.62 / 51.33 on an 8 vCore VM.
+- 9963 of 9968 client TCP sockets were in TIME_WAIT at the end of the Client Credentials scenario,
+  indicating ephemeral port exhaustion on the load generator.
+- The AKS side is also undersized for this level: 2 x Standard_F8s_v2 nodes with Thunder capped at
+  10 pods of 1.5 vCore cannot absorb 10000 concurrent users.
+
+To produce a valid 10000 concurrent user run, the load generator needs to be scaled out (multiple
+JMeter clients or a larger VM SKU with tuned `net.ipv4.ip_local_port_range` and
+`tcp_tw_reuse`), and the AKS node pool and Thunder HPA ceiling need to be raised.
+
+**K8s Spec**
+
+- AKS Version: 1.34
+- Node Pool: Standard_F8s_v2, count: 2, autoscale min: 2, max: 5, max-pods-per-node: 30
+- Thunder Pods: requests 1 vCore + 256Mi, limits 1.5 vCore + 512Mi, replicas: 2, HPA max-pods: 10 (CPU 65%, Memory 75%)
+- Nginx Ingress Pods: requests 0.5 vCore + 500Mi, limits 1 vCore + 1000Mi, min-pods: 2, max-pods: 5 (CPU 80%, Memory 80%)
+- JMeter Client VM: Standard_F8s_v2, Heap Size: 4g
+- Database: Postgres 17
+- Caching: In-Memory
+- DB Specs:
+  - Config: GP_Standard_D2s_v3
+  - Runtime: GP_Standard_D2s_v3
+  - User: GP_Standard_D2s_v3
+
+| Scenario Name | Concurrent Users | Label | # Samples | Error Count | Error % | Throughput (Requests/sec) | Average Response Time (ms) | 95th Percentile of Response Time (ms) |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| Client Credentials Grant Type | 10000 | 1 Get access token | 2267130 | 2117166 | 93.39 | 2733.27 | 1982.92 | 10815.00 |
+| Authorization Code Grant Type | 10000 | 1 Send request to authorize endpoint | 707047 | 704340 | 99.62 | 906.60 | 409.81 | 182.00 |
+| Authorization Code Grant Type | 10000 | 2 Start Authentication Flow | 707470 | 706684 | 99.89 | 907.14 | 262.82 | 72.00 |
+| Authorization Code Grant Type | 10000 | 3 Perform authentication | 701902 | 701780 | 99.98 | 900.21 | 1391.38 | 10047.00 |
+| Authorization Code Grant Type | 10000 | 4 Obtain authorization code | 705076 | 692562 | 98.23 | 904.07 | 619.27 | 3039.00 |
+| Authorization Code Grant Type | 10000 | 5 Obtain access token | 707342 | 707336 | 100.00 | 906.98 | 313.39 | 387.00 |
+| User Authentication with Credentials | 10000 | 1 Perform user authentication | 6967102 | 6956696 | 99.85 | 8660.87 | 451.74 | 467.00 |
diff --git a/benchmarks/v1.0.0-alpha/kubernetes/results-10000conc.csv b/benchmarks/v1.0.0-alpha/kubernetes/results-10000conc.csv
@@ -0,0 +1,27 @@
+Scenario Name,Heap Size,Concurrent Users,Label,# Samples,Error Count,Error %,Throughput (Requests/sec),Average Response Time (ms),Standard Deviation of Response Time (ms),Minimum Response Time (ms),Maximum Response Time (ms),75th Percentile of Response Time (ms),90th Percentile of Response Time (ms),95th Percentile of Response Time (ms),98th Percentile of Response Time (ms),99th Percentile of Response Time (ms),99.9th Percentile of Response Time (ms),Received (KB/sec),Sent (KB/sec),WSO2 Thunder 1 GC Throughput (%),WSO2 Thunder 1 Memory Footprint (M),Average WSO2 Thunder 1 Memory Footprint After Full GC (M),Standard Deviation of WSO2 Thunder 1 Memory Footprint After Full GC (M),WSO2 Thunder 2 GC Throughput (%),WSO2 Thunder 2 Memory Footprint (M),Average WSO2 Thunder 2 Memory Footprint After Full GC (M),Standard Deviation of WSO2 Thunder 2 Memory Footprint After Full GC (M),JMeter Client GC Throughput (%),JMeter Client Memory Footprint (M),Average JMeter Client Memory Footprint After Full GC (M),Standard Deviation of JMeter Client Memory Footprint After Full GC (M),WSO2 Thunder 1 Load Average - Last 1 minute,WSO2 Thunder 1 Load Average - Last 5 minutes,WSO2 Thunder 1 Load Average - Last 15 minutes,WSO2 Thunder 2 Load Average - Last 1 minute,WSO2 Thunder 2 Load Average - Last 5 minutes,WSO2 Thunder 2 Load Average - Last 15 minutes,JMeter Client Load Average - Last 1 minute,JMeter Client Load Average - Last 5 minutes,JMeter Client Load Average - Last 15 minutes
+Client Credentials Grant Type,N/A,10000,1 Get access token,2267130,2117166,93.39,2733.27,1982.92,4576.28,0,47343,151.00,9983.00,10815.00,16127.00,19327.00,35327.00,1304.56,903.71,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,125.91,91.62,51.33
+Authorization Code Grant Type,N/A,10000,1 Send request to authorize endpoint,707047,704340,99.62,906.60,409.81,2190.28,0,30300,5.00,33.00,182.00,6015.00,13695.00,22015.00,366.24,240.62,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,41.63,50.70,48.26
+Authorization Code Grant Type,N/A,10000,2 Start Authentication Flow,707470,706684,99.89,907.14,262.82,1861.97,0,25592,6.00,31.00,72.00,3439.00,11903.00,24575.00,365.36,235.13,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,41.63,50.70,48.26
+Authorization Code Grant Type,N/A,10000,3 Perform authentication,701902,701780,99.98,900.21,1391.38,3468.90,0,25450,7.00,7935.00,10047.00,12223.00,14143.00,17535.00,368.48,334.40,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,41.63,50.70,48.26
+Authorization Code Grant Type,N/A,10000,4 Obtain authorization code,705076,692562,98.23,904.07,619.27,2762.62,0,25531,23.00,277.00,3039.00,11391.00,15295.00,25087.00,356.97,251.81,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,41.63,50.70,48.26
+Authorization Code Grant Type,N/A,10000,5 Obtain access token,707342,707336,100.00,906.98,313.39,1885.20,0,25526,7.00,57.00,387.00,7967.00,11327.00,18431.00,359.10,362.50,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,41.63,50.70,48.26
+User Authentication with Credentials,N/A,10000,1 Perform user authentication,6967102,6956696,99.85,8660.87,451.74,2148.91,0,64756,3.00,19.00,467.00,10239.00,10815.00,18687.00,3500.82,3458.37,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,58.49,76.52,66.76
+
+
+NOTE: This run is NOT a valid benchmark. Error rates were 93-100% across all scenarios.
+The single JMeter client VM (Standard_F8s_v2, 8 vCore) was saturated (load average ~100-126)
+and exhausted ephemeral ports (9963 sockets in TIME_WAIT). Numbers below reflect load
+generator and cluster saturation, not Thunder throughput.
+
+K8s Spec:
+AKS Version: 1.34
+Node Pool: Standard_F8s_v2, count:2, autoscale min:2 max:5, max-pods-per-node:30
+Thunder Pods: requests 1 vCore + 256Mi, limits 1.5 vCore + 512Mi, replicas:2, HPA max-pods:10 (CPU 65%, Memory 75%)
+Nginx Ingress Pods: requests 0.5 vCore + 500Mi, limits 1 vCore + 1000Mi, min-pods:2, max-pods:5 (CPU 80%, Memory 80%)
+JMeter Client VM: Standard_F8s_v2, Heap Size:4g
+Caching: In-Memory
+
+DB Specs:
+Config: GP_Standard_D2s_v3 (Postgres 17)
+Runtime: GP_Standard_D2s_v3 (Postgres 17)
+User: GP_Standard_D2s_v3 (Postgres 17)
diff --git a/benchmarks/v1.0.0-alpha/kubernetes/results-1000conc.csv b/benchmarks/v1.0.0-alpha/kubernetes/results-1000conc.csv
@@ -0,0 +1,22 @@
+Scenario Name,Heap Size,Concurrent Users,Label,# Samples,Error Count,Error %,Throughput (Requests/sec),Average Response Time (ms),Standard Deviation of Response Time (ms),Minimum Response Time (ms),Maximum Response Time (ms),75th Percentile of Response Time (ms),90th Percentile of Response Time (ms),95th Percentile of Response Time (ms),98th Percentile of Response Time (ms),99th Percentile of Response Time (ms),99.9th Percentile of Response Time (ms),Received (KB/sec),Sent (KB/sec),WSO2 Thunder 1 GC Throughput (%),WSO2 Thunder 1 Memory Footprint (M),Average WSO2 Thunder 1 Memory Footprint After Full GC (M),Standard Deviation of WSO2 Thunder 1 Memory Footprint After Full GC (M),WSO2 Thunder 2 GC Throughput (%),WSO2 Thunder 2 Memory Footprint (M),Average WSO2 Thunder 2 Memory Footprint After Full GC (M),Standard Deviation of WSO2 Thunder 2 Memory Footprint After Full GC (M),JMeter Client GC Throughput (%),JMeter Client Memory Footprint (M),Average JMeter Client Memory Footprint After Full GC (M),Standard Deviation of JMeter Client Memory Footprint After Full GC (M),WSO2 Thunder 1 Load Average - Last 1 minute,WSO2 Thunder 1 Load Average - Last 5 minutes,WSO2 Thunder 1 Load Average - Last 15 minutes,WSO2 Thunder 2 Load Average - Last 1 minute,WSO2 Thunder 2 Load Average - Last 5 minutes,WSO2 Thunder 2 Load Average - Last 15 minutes,JMeter Client Load Average - Last 1 minute,JMeter Client Load Average - Last 5 minutes,JMeter Client Load Average - Last 15 minutes
+Client Credentials Grant Type,N/A,1000,1 Get access token,2527085,7558,0.30,3228.94,308.20,900.04,2,30777,273.00,855.00,1367.00,2127.00,2623.00,15039.00,4363.61,1067.59,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,6.70,4.92,3.06
+Authorization Code Grant Type,N/A,1000,1 Send request to authorize endpoint,127987,4,0.00,164.11,20.57,166.18,2,5089,13.00,22.00,36.00,66.00,97.00,5023.00,137.48,43.56,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,0.49,1.17,2.01
+Authorization Code Grant Type,N/A,1000,2 Start Authentication Flow,127985,8,0.01,164.10,12.36,17.46,0,1093,11.00,19.00,32.00,58.00,83.00,213.00,439.40,58.45,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,0.49,1.17,2.01
+Authorization Code Grant Type,N/A,1000,3 Perform authentication,127993,106,0.08,164.11,26.16,212.16,0,5047,15.00,24.00,39.00,72.00,109.00,5023.00,219.54,86.12,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,0.49,1.17,2.01
+Authorization Code Grant Type,N/A,1000,4 Obtain authorization code,127995,50,0.04,164.12,15.43,29.93,0,5016,13.00,22.00,37.00,72.00,107.00,297.00,58.18,221.95,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,0.49,1.17,2.01
+Authorization Code Grant Type,N/A,1000,5 Obtain access token,127989,109,0.09,164.11,12.97,20.84,0,5005,12.00,19.00,30.00,49.00,67.00,150.00,185.10,78.25,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,0.49,1.17,2.01
+User Authentication with Credentials,N/A,1000,1 Perform user authentication,3468259,136,0.00,4443.35,224.35,564.41,2,18137,129.00,571.00,1151.00,2111.00,2895.00,5759.00,6389.59,1774.27,N/A,N/A,N/A,N/A,N/A,N/A,N/A,N/A,,,,,N/A,N/A,N/A,N/A,N/A,N/A,1.88,1.67,1.62
+
+
+K8s Spec:
+AKS Version: 1.34
+Node Pool: Standard_F8s_v2, count:2, autoscale min:2 max:5, max-pods-per-node:30
+Thunder Pods: requests 1 vCore + 256Mi, limits 1.5 vCore + 512Mi, replicas:2, HPA max-pods:10 (CPU 65%, Memory 75%)
+Nginx Ingress Pods: requests 0.5 vCore + 500Mi, limits 1 vCore + 1000Mi, min-pods:2, max-pods:5 (CPU 80%, Memory 80%)
+JMeter Client VM: Standard_F8s_v2, Heap Size:4g
+Caching: In-Memory
+
+DB Specs:
+Config: GP_Standard_D2s_v3 (Postgres 17)
+Runtime: GP_Standard_D2s_v3 (Postgres 17)
+User: GP_Standard_D2s_v3 (Postgres 17)