Performance Test Report
Introduction
This document provides details on the performance measurement approach of prioritized scenarios of MOSIP modules and provides detailed reports on the results.
Scope
APIs in all modules are prioritized based on the high load expectation and usage frequency by other modules.
There are 3 types of performance scenarios covered in this report
Prioritized single API performance (For most modules)
End to end UI API sequence performance (For pre-registration module)
End to end message delivery performance (For web-sub module)
All individual test scenarios from the modules are listed below.
Module Name
API Scenario
API Endpoint
Kernel
Add Audits
v1/auditmanager/audits
Kernel
Client Id - Secret Key
v1/authmanager/authenticate/clientidsecretkey
Kernel
User Id Pwd
v1/authmanager/authenticate/internal/useridPwd
Kernel
Validate Token
v1/authmanager/authorize/admin/validateToken
Kernel
Generate OTP
v1/otpmanager/otp/generate
Kernel
Validate OTP
v1/otpmanager/otp/validate?key={keyValidate}&otp={otpValidate}
Kernel
Send SMS
v1/notifier/sms/send
Kernel
Send EMAIL
v1/notifier/email/send
Kernel
Generate PRID
v1/pridgenerator/prid
Kernel
Generate UIN
v1/idgenerator/uin
Kernel
Generate VID
v1/idgenerator/vid
Kernel
Generate RID
v1/ridgenerator/generate/rid/{centerID}/{machineID}
IDA
Auth with OTP
idauthentication/v1/auth/{mispLicenseKey}/{rpPartnerId}/{rpApiKey}
IDA
KYC with OTP
idauthentication/v1/kyc/{mispLicenseKey}/{ekycPartnerId}/{ekycApiKey}
IDA
Auth with bio
idauthentication/v1/auth/{mispLicenseKey}/{rpPartnerId}/{rpApiKey}
IDA
KYC with bio
idauthentication/v1/kyc/{mispLicenseKey}/{ekycPartnerId}/{ekycApiKey}
IDA
Send OTP
idauthentication/v1/otp/{mispLicenseKey}/{rpPartnerId}/{rpApiKey}
Pre-Registration
Prereg UI end to end flow
Prereg UI end to end flow
Syncdata
Public Key Verify
v1/syncdata/tpm/publickey/verify
Syncdata
Get Certificate
v1/syncdata/getCertificate?applicationId={appID}&referenceId={refID}
Syncdata
Get User Details
v1/syncdata/userdetails?keyindex={keyIndex}
Syncdata
Get Client Settings
v1/syncdata/v2/clientsettings?keyindex={keyIndex}
Syncdata
Get Configs
v1/syncdata/configs/{name}
Syncdata
Get LatestID Schema
v1/syncdata/latestidschema?schemaVersion={schemaVersion}
Syncdata
Get CAcertificates
v1/syncdata/getcacertificates
Regproc
Sync Registration Packet Details
registrationprocessor/v1/registrationstatus/sync
Regproc
Get Packet status
registrationprocessor/v1/registrationstatus/search
Regproc
Upload Registration Packet
registrationprocessor/v1/packetreceiver/registrationpackets
Resident
Request/Send OTP
resident/v1/req/otp
Resident
RID Check Status
resident/v1/rid/check-status
Resident
Auth Lock
resident/v1/req/auth-lock
Resident
Auth Unlock
resident/v1/req/auth-unlock
Resident
Credential Request
resident/v1/req/credential
Resident
Auth History
resident/v1/req/auth-history
Resident
Generate VID
resident/v1/vid
Datashare
Create Data Share URL
v1/datashare/create/{policyId}/{subscriberId}
Datashare
Get Data Share File
v1/datashare/get/{policyId}/{subscriberId}/{randomShareKey}
IDRepo
Retrieve Identity using UIN
idrepository/v1/identity/idvid/{uin}
IDRepo
Retrieve Identity using VID
idrepository/v1/identity/idvid/{vid}
IDRepo
Add Identity
idrepository/v1/identity/
IDRepo
Update Identity
idrepository/v1/identity/
IDRepo
Auth Type Status
idrepository/v1/identity/authtypes/status
IDRepo
Create VID
idrepository/v1/vid
IDRepo
Update VID
idrepository/v1/vid/{vidGenerated}
Masterdata
Get Registration Centers
v1/masterdata/registrationcenters
Masterdata
Get Templates
v1/masterdata/templates
Masterdata
Get Templates By Lang & Temp Code
v1/masterdata/templates/{lang&TempTypeCode}
Masterdata
Get Latest ID Schema
v1/masterdata/idschema/latest?schemaVersion={schemaVersion}
Masterdata
Get Dynamic Fields
v1/masterdata/dynamicfields?pageNumber=0&pageSize=10&sortBy=name&orderBy=desc
Masterdata
Get Users
v1/masterdata/users/{userID}/{time(yyyy-MM-dd'T'HH:mm:ss.sss'Z',)}
Keymanager
Encrypt Data
v1/keymanager/encrypt
Keymanager
Encrypt Data With Pin
v1/keymanager/encryptWithPin
Keymanager
Generate JWT Signature
v1/keymanager/jwtSign
Keymanager
Decrypt Data
v1/keymanager/decrypt
Keymanager
Decrypt Data With Pin API
v1/keymanager/decryptWithPin
Keymanager
Verify JWT Signature
v1/keymanager/jwtVerify
Testing Tools & Techniques
Tools
Purpose
Jmeter
To verify the Load and Performance of all the applications.
Approach
The approach to performance testing differs based on the testing scenarios and are detailed as per the categorization as below.
Prioritized Single API Performance
Run a single pod of the application hosting the required API
Run four pods each of the dependent applications to cater the load implied from the test application
Simulate varied load profilers like 10, 30, 50 etc.. concurrent users from Jmeter for that single API for a period of one hour
Record the average and 90th percentile response time metrics specific to each load profiles
Verify that JVMs show stable memory and CPU usage over the increasing load profiles.
End to End UI API Sequence Performance
Run a single pod for each application hosting the required APIs called directly from Jmeter
Run four pods each of the dependent applications to cater the load implied from the test application
Simulate varied load profilers like 50, 80, 100 etc.. concurrent users from Jmeter for end to end sequence of APIs called from UI for a period of one hour
Record the average and 90th percentile response time metrics for entire end to end API sequence for each load profiles
Verify that JVMs show stable memory and CPU usage over the increasing load profiles.
End to End Message Delivery Performance
Run a single pod for each application hosting the required APIs called directly from Jmeter
Run four pods each of the dependent applications to cater the load implied from the test application
Simulate varied load profilers like 10, 30, 50 etc.. concurrent users to publish continues messages to the test application for a period of one hour
Record the average and 90th percentile response time metrics for publish API for each load profiles
Record the average and 90th percentile delivery time metrics for entire end to end message delivery for each load profiles
Build Details
Title
Description
Build Tag Version
1.2.0-rc2
Performance script source location
https://github.com/mosip/mosip-performance-tests-mt/tree/1.2.0
Environment Setup
Hardware / VM details
CPU (cores)
Memory (GB)
HDD (GB)
OS
Count
Purpose
8
30
768
Cent OS
1
Kubernetes console machine
8
Kubernetes node machines
4
16
500
Ubuntu
1
Database machine
4
16
835
Windows
3
Jmeter test machines
Pod Resource allocation
Pod/Application Names
Resource Allocation (Limits)
Memory (M)
CPU (m)
Heap (M)
kernel-auditmanager-service
2500
1000
1750
kernel-auth-service
2250
500
1500
kernel-otpmanager-service
1750
500
1000
kernel-notification-service
1500
500
1000
kernel-pridgenerator-service
1750
300
1000
kernel-idgenerator-service
3500
500
2000
kernel-ridgenerator-service
1750
500
1000
ida-auth-service
4000
4500
2000
ida-otp-service
3000
3000
2000
prereg-application-service
2500
1000
2000
kernel-syncdata-service
5000
500
4000
regproc-registration-status-service
4000
500
2000
regproc-stage-group-1
5000
1000
4000
resident-service
3000
500
2000
datashare-service
3000
500
2000
idrepo-identity-service
4000
1000
2000
idrepo-vid-service
3000
500
1000
kernel-masterdata-service
2250
500
1500
kernel-keymanager-service
5000
1000
4000
Test Results
Prioritized single API performance
90th Percentile Response Times
Detailed Metrics for all 14 modules is available below:
Audit Manager
Run 1: 50 concurrency
Jmeter Aggregate Report
Add Audit API Execution
231236
777
395
2206
2700
3402
3
8420
0.00%
64.22209
38.88
186.99
Response Time Percentile Graph using Jmeter
CPU and Memory Utilization Graphs using Grafana
Application Response Time Graph from Kibana
Dependent Application Response Time Graph from Kibana
Dependency app graph for auth manager service was empty so not attached.
Run 2: 30 concurrency
Jmeter Aggregate Report
Add Audit API Execution
266879
403
105
1199
1893
2409
2
3870
0.00%
74.12657
44.88
215.82
Response Time Percentile Graph using Jmeter
CPU and Memory Utilization Graphs using Grafana
Application Response Time Graph from Kibana
Dependent Application Response Time Graph from Kibana
Application dependency graph was empty so not attached.
Auth Manager
Run 1: 30 concurrency
Aggregate Report from Jmeter
Client Id - Secret Key API Execution
241902
449
202
483
597
3352
9
60051
0.16%
66.1073
290.08
28.49
User Id Pwd API Execution
220878
488
488
680
711
872
70
6616
0.00%
61.34721
215.9
30
Validate Token API Execution
451378
239
201
400
496
680
6
3506
0.00%
125.3778
582.61
235.51
Response Time Percentile Graph using Jmeter
We only have the graph for Validate token as the previous ones were cleared
CPU and Memory Utilization Graphs using Grafana
Application Response Time Graph from Kibana
Dependent Application Response Time from Kibana
Dependency app graph was empty so so not attached.
Notes:
Keycloak was having issue by end of client id secret key API run, so the keycloak was restarted before the start of user id password API
Run 2: 50 concurrency
Aggregate Report from Jmeter
Client Id - Secret Key API Execution
243069
740
499
1694
2100
2833
9
7511
0.00%
67.51412
296.36
29.09
User Id Pwd API Execution
27121
5519
3290
11676
22351
43894
40
60047
0.39%
9.05834
31.82
4.42
Validate Token API Execution
181953
989
810
2107
2496
3206
6
8111
0.00%
50.53694
234.74
94.93
Response Time Percentile Graph using Jmeter
CPU and Memory Utilization Graphs using Grafana
Application Response Time Graph from Kibana
Dependent Application Response Time Graph from Kibana
Dependency app graph was empty so so not attached.
Notes:
Keycloak was having issue by end of user id password API run, so the keycloak was restarted before the start of validate token API
Kernel services
Run 1: 30 concurrency
Aggregate Report from Jmeter
Generate OTP API Execution
87777
1229
1104
2160
2496
3207
5
7418
0.00%
24.37841
15.57
48.04
Validate OTP API Execution
150000
1275
1128
2192
2404
3196
0
6268
0.02%
23.4176
14.89
42.63
Notification Manager SMS API Request
433004
246
99
692
808
1119
2
2625
0.00%
120.2695
80.92
259.67
Notification Manager EMAIL API Request
69727
1550
1396
2992
3602
5101
4
13607
0.04%
14.87585
9.48
41.37
PRID Generator API Execution
102606
1052
1022
1893
2073
2432
5
5104
0.00%
28.49246
11.74
51
Generate UIN API Execution
39971
2698
2672
2813
2833
3536
127
8776
0.00%
11.09746
26.27
23.71
Generate VID API Execution
205377
154
139
266
321
423
5
12380
6.39%
193.1971
458.89
412.75
RID Generator API Execution
368802
291
193
743
930
1324
5
5761
0.00%
102.4346
63.42
221.07
Note: idgenerator- generate vid api threw error as “VID not available for allocation error”, so it ran only for around 20 mins rest all other api’s ran for the complete 1 hour run.
Response Time Percentile Graph using Jmeter
CPU and Memory Utilization Graphs using Grafana
Application Response Time Graph from Kibana
Note: idgenerator & pridgenerator services are vertx apps so there are no kibana graphs for the same
Dependent Application Response Time Graph from Kibana
Dependency app graph was empty so above are the graphs for the application services from kibana
Notes: Some of the kernel service api’s failed with oomkilled & authentication failed issues, so restarted the same and then continued the load test from the next set of api’s that’s why we have multiple graphs from jmeter & grafana.
Run 2: 10 & 50 concurrency
Aggregate Report from Jmeter
Generate OTP API Execution
768397
45
9
92
95
100
5
299
0.00%
213.4406
135.9
420.63
Generate OTP API Request
90000
42
8
90
91
94
4
184
0.00%
225.8061
143.77
445
Validate OTP API Execution
90000
48
14
92
95
100
6
1209
0.00%
201.6066
127.78
366.99
Notification Manager EMAIL API Request
135462
264
195
606
893
1198
3
6299
0.00%
37.62565
23.92
104.68