Performance Test Report

Introduction
This document provides details on the performance measurement approach of prioritized scenarios of MOSIP modules and provides detailed reports on the results.
Scope
APIs in all modules are prioritized based on the high load expectation and usage frequency by other modules.
There are 3 types of performance scenarios covered in this report
Prioritized single API performance (For most modules)
End to end UI API sequence performance (For pre-registration module)
End to end message delivery performance (For web-sub module)
All individual test scenarios from the modules are listed below.
Module Name
API Scenario
API Endpoint
Kernel
Add Audits
v1/auditmanager/audits
Kernel
Client Id - Secret Key
v1/authmanager/authenticate/clientidsecretkey
Kernel
User Id Pwd
v1/authmanager/authenticate/internal/useridPwd
Kernel
Validate Token
v1/authmanager/authorize/admin/validateToken
Kernel
Generate OTP
v1/otpmanager/otp/generate
Kernel
Validate OTP
v1/otpmanager/otp/validate?key={keyValidate}&otp={otpValidate}
Kernel
Send SMS
v1/notifier/sms/send
Kernel
Send EMAIL
v1/notifier/email/send
Kernel
Generate PRID
v1/pridgenerator/prid
Kernel
Generate UIN
v1/idgenerator/uin
Kernel
Generate VID
v1/idgenerator/vid
Kernel
Generate RID
v1/ridgenerator/generate/rid/{centerID}/{machineID}
IDA
Auth with OTP
idauthentication/v1/auth/{mispLicenseKey}/{rpPartnerId}/{rpApiKey}
IDA
KYC with OTP
idauthentication/v1/kyc/{mispLicenseKey}/{ekycPartnerId}/{ekycApiKey}
IDA
Auth with bio
idauthentication/v1/auth/{mispLicenseKey}/{rpPartnerId}/{rpApiKey}
IDA
KYC with bio
idauthentication/v1/kyc/{mispLicenseKey}/{ekycPartnerId}/{ekycApiKey}
IDA
Send OTP
idauthentication/v1/otp/{mispLicenseKey}/{rpPartnerId}/{rpApiKey}
Pre-Registration
Prereg UI end to end flow
Prereg UI end to end flow
Syncdata
Public Key Verify
v1/syncdata/tpm/publickey/verify
Syncdata
Get Certificate
v1/syncdata/getCertificate?applicationId={appID}&referenceId={refID}
Syncdata
Get User Details
v1/syncdata/userdetails?keyindex={keyIndex}
Syncdata
Get Client Settings
v1/syncdata/v2/clientsettings?keyindex={keyIndex}
Syncdata
Get Configs
v1/syncdata/configs/{name}
Syncdata
Get LatestID Schema
v1/syncdata/latestidschema?schemaVersion={schemaVersion}
Syncdata
Get CAcertificates
v1/syncdata/getcacertificates
Regproc
Sync Registration Packet Details
registrationprocessor/v1/registrationstatus/sync
Regproc
Get Packet status
registrationprocessor/v1/registrationstatus/search
Regproc
Upload Registration Packet
registrationprocessor/v1/packetreceiver/registrationpackets
Resident
Request/Send OTP
resident/v1/req/otp
Resident
RID Check Status
resident/v1/rid/check-status
Resident
Auth Lock
resident/v1/req/auth-lock
Resident
Auth Unlock
resident/v1/req/auth-unlock
Resident
Credential Request
resident/v1/req/credential
Resident
Auth History
resident/v1/req/auth-history
Resident
Generate VID
resident/v1/vid
Datashare
Create Data Share URL
v1/datashare/create/{policyId}/{subscriberId}
Datashare
Get Data Share File
v1/datashare/get/{policyId}/{subscriberId}/{randomShareKey}
IDRepo
Retrieve Identity using UIN
idrepository/v1/identity/idvid/{uin}
IDRepo
Retrieve Identity using VID
idrepository/v1/identity/idvid/{vid}
IDRepo
Add Identity
idrepository/v1/identity/
IDRepo
Update Identity
idrepository/v1/identity/
IDRepo
Auth Type Status
idrepository/v1/identity/authtypes/status
IDRepo
Create VID
idrepository/v1/vid
IDRepo
Update VID
idrepository/v1/vid/{vidGenerated}
Masterdata
Get Registration Centers
v1/masterdata/registrationcenters
Masterdata
Get Templates
v1/masterdata/templates
Masterdata
Get Templates By Lang & Temp Code
v1/masterdata/templates/{lang&TempTypeCode}
Masterdata
Get Latest ID Schema
v1/masterdata/idschema/latest?schemaVersion={schemaVersion}
Masterdata
Get Dynamic Fields
v1/masterdata/dynamicfields?pageNumber=0&pageSize=10&sortBy=name&orderBy=desc
Masterdata
Get Users
v1/masterdata/users/{userID}/{time(yyyy-MM-dd'T'HH:mm:ss.sss'Z',)}
Keymanager
Encrypt Data
v1/keymanager/encrypt
Keymanager
Encrypt Data With Pin
v1/keymanager/encryptWithPin
Keymanager
Generate JWT Signature
v1/keymanager/jwtSign
Keymanager
Decrypt Data
v1/keymanager/decrypt
Keymanager
Decrypt Data With Pin API
v1/keymanager/decryptWithPin
Keymanager
Verify JWT Signature
v1/keymanager/jwtVerify
Testing Tools & Techniques
Tools
Purpose
Jmeter
To verify the Load and Performance of all the applications.
Approach
The approach to performance testing differs based on the testing scenarios and are detailed as per the categorization as below.
Prioritized Single API Performance
Run a single pod of the application hosting the required API
Run four pods each of the dependent applications to cater the load implied from the test application
Simulate varied load profilers like 10, 30, 50 etc.. concurrent users from Jmeter for that single API for a period of one hour
Record the average and 90th percentile response time metrics specific to each load profiles
Verify that JVMs show stable memory and CPU usage over the increasing load profiles.
End to End UI API Sequence Performance
Run a single pod for each application hosting the required APIs called directly from Jmeter
Run four pods each of the dependent applications to cater the load implied from the test application
Simulate varied load profilers like 50, 80, 100 etc.. concurrent users from Jmeter for end to end sequence of APIs called from UI for a period of one hour
Record the average and 90th percentile response time metrics for entire end to end API sequence for each load profiles
Verify that JVMs show stable memory and CPU usage over the increasing load profiles.
End to End Message Delivery Performance
Run a single pod for each application hosting the required APIs called directly from Jmeter
Run four pods each of the dependent applications to cater the load implied from the test application
Simulate varied load profilers like 10, 30, 50 etc.. concurrent users to publish continues messages to the test application for a period of one hour
Record the average and 90th percentile response time metrics for publish API for each load profiles
Record the average and 90th percentile delivery time metrics for entire end to end message delivery for each load profiles
Build Details
Title
Description
Build Tag Version
1.2.0-rc2
Performance script source location
https://github.com/mosip/mosip-performance-tests-mt/tree/1.2.0
Environment Setup
Hardware / VM details
CPU (cores)
Memory (GB)
HDD (GB)
OS
Count
Purpose
8
30
768
Cent OS
1
Kubernetes console machine
8
​
​
Kubernetes node machines
​
​
4
16
500
Ubuntu
1
Database machine
4
16
835
Windows
3
Jmeter test machines
Pod Resource allocation
Pod/Application Names
Resource Allocation (Limits)
​
​
​
Memory (M)
CPU (m)
Heap (M)
kernel-auditmanager-service
2500
1000
1750
kernel-auth-service
2250
500
1500
kernel-otpmanager-service
1750
500
1000
kernel-notification-service
1500
500
1000
kernel-pridgenerator-service
1750
300
1000
kernel-idgenerator-service
3500
500
2000
kernel-ridgenerator-service
1750
500
1000
ida-auth-service
4000
4500
2000
ida-otp-service
3000
3000
2000
prereg-application-service
2500
1000
2000
kernel-syncdata-service
5000
500
4000
regproc-registration-status-service
4000
500
2000
regproc-stage-group-1
5000
1000
4000
resident-service
3000
500
2000
datashare-service
3000
500
2000
idrepo-identity-service
4000
1000
2000
idrepo-vid-service
3000
500
1000
kernel-masterdata-service
2250
500
1500
kernel-keymanager-service
5000
1000
4000
Test Results
Prioritized single API performance
90th Percentile Response Times
​Detailed response time details​
Detailed Metrics for all 14 modules is available below:
Audit Manager
Run 1: 50 concurrency
Jmeter Aggregate Report
Label
# Samples
Average
Median
90% Line
95% Line
99% Line
Min
Max
Error %
Throughput
Received KB/sec
Sent KB/sec
Add Audit API Execution
231236
777
395
2206
2700
3402
3
8420
0.00%
64.22209
38.88
186.99
Response Time Percentile Graph using Jmeter
CPU and Memory Utilization Graphs using Grafana
Application Response Time Graph from Kibana
Dependent Application Response Time Graph from Kibana
Dependency app graph for auth manager service was empty so not attached.
Run 2: 30 concurrency
Jmeter Aggregate Report
Label
# Samples
Average
Median
90% Line
95% Line
99% Line
Min
Max
Error %
Throughput
Received KB/sec
Sent KB/sec
Add Audit API Execution
266879
403
105
1199
1893
2409
2
3870
0.00%
74.12657
44.88
215.82
Response Time Percentile Graph using Jmeter
CPU and Memory Utilization Graphs using Grafana
Application Response Time Graph from Kibana
Dependent Application Response Time Graph from Kibana
Application dependency graph was empty so not attached.
Auth Manager
Run 1: 30 concurrency
Aggregate Report from Jmeter
Label
# Samples
Average
Median
90% Line
95% Line
99% Line
Min
Max
Error %
Throughput
Received KB/sec
Sent KB/sec
Client Id - Secret Key API Execution
241902
449
202
483
597
3352
9
60051
0.16%
66.1073
290.08
28.49
User Id Pwd API Execution
220878
488
488
680
711
872
70
6616
0.00%
61.34721
215.9
30
Validate Token API Execution
451378
239
201
400
496
680
6
3506
0.00%
125.3778
582.61
235.51
Response Time Percentile Graph using Jmeter
We only have the graph for Validate token as the previous ones were cleared
CPU and Memory Utilization Graphs using Grafana
Application Response Time Graph from Kibana
Dependent Application Response Time from Kibana
Dependency app graph was empty so so not attached.
Notes:
Keycloak was having issue by end of client id secret key API run, so the keycloak was restarted before the start of user id password API
Run 2: 50 concurrency
Aggregate Report from Jmeter
Label
# Samples
Average
Median
90% Line
95% Line
99% Line
Min
Max
Error %
Throughput
Received KB/sec
Sent KB/sec
Client Id - Secret Key API Execution
243069
740
499
1694
2100
2833
9
7511
0.00%
67.51412
296.36
29.09
User Id Pwd API Execution
27121
5519
3290
11676
22351
43894
40
60047
0.39%
9.05834
31.82
4.42
Validate Token API Execution
181953
989
810
2107
2496
3206
6
8111
0.00%
50.53694
234.74
94.93
Response Time Percentile Graph using Jmeter
CPU and Memory Utilization Graphs using Grafana
Application Response Time Graph from Kibana
Dependent Application Response Time Graph from Kibana
Dependency app graph was empty so so not attached.
Notes:
Keycloak was having issue by end of user id password API run, so the keycloak was restarted before the start of validate token API
Kernel services
Run 1: 30 concurrency
Aggregate Report from Jmeter
Label
# Samples
Average
Median
90% Line
95% Line
99% Line
Min
Max
Error %
Throughput
Received KB/sec
Sent KB/sec
Generate OTP API Execution
87777
1229
1104
2160
2496
3207
5
7418
0.00%
24.37841
15.57
48.04
Validate OTP API Execution
150000
1275
1128
2192
2404
3196
0
6268
0.02%
23.4176
14.89
42.63
Notification Manager SMS API Request
433004
246
99
692
808
1119
2
2625
0.00%
120.2695
80.92
259.67
Notification Manager EMAIL API Request
69727
1550
1396
2992
3602
5101
4
13607
0.04%
14.87585
9.48
41.37
PRID Generator API Execution
102606
1052
1022
1893
2073
2432
5
5104
0.00%
28.49246
11.74
51
Generate UIN API Execution
39971
2698
2672
2813
2833
3536
127
8776
0.00%
11.09746
26.27
23.71
Generate VID API Execution
205377
154
139
266
321
423
5
12380
6.39%
193.1971
458.89
412.75
RID Generator API Execution
368802
291
193
743
930
1324
5
5761
0.00%
102.4346
63.42
221.07
Note: idgenerator- generate vid api threw error as “VID not available for allocation error”, so it ran only for around 20 mins rest all other api’s ran for the complete 1 hour run.
Response Time Percentile Graph using Jmeter
CPU and Memory Utilization Graphs using Grafana
Application Response Time Graph from Kibana
Note: idgenerator & pridgenerator services are vertx apps so there are no kibana graphs for the same
Dependent Application Response Time Graph from Kibana
Dependency app graph was empty so above are the graphs for the application services from kibana
Notes: Some of the kernel service api’s failed with oomkilled & authentication failed issues, so restarted the same and then continued the load test from the next set of api’s that’s why we have multiple graphs from jmeter & grafana.
Run 2: 10 & 50 concurrency
Aggregate Report from Jmeter
Label
# Samples
Average
Median
90% Line
95% Line
99% Line
Min
Max
Error %
Throughput
Received KB/sec
Sent KB/sec
Generate OTP API Execution
768397
45
9
92
95
100
5
299
0.00%
213.4406
135.9
420.63
Generate OTP API Request
90000
42
8
90
91
94
4
<

Tools	Purpose
Jmeter	To verify the Load and Performance of all the applications.

Title	Description
Build Tag Version	1.2.0-rc2
Performance script source location	https://github.com/mosip/mosip-performance-tests-mt/tree/1.2.0

Module Name	API Scenario	API Endpoint
Kernel	Add Audits	v1/auditmanager/audits
Kernel	Client Id - Secret Key	v1/authmanager/authenticate/clientidsecretkey
Kernel	User Id Pwd	v1/authmanager/authenticate/internal/useridPwd
Kernel	Validate Token	v1/authmanager/authorize/admin/validateToken
Kernel	Generate OTP	v1/otpmanager/otp/generate
Kernel	Validate OTP	v1/otpmanager/otp/validate?key={keyValidate}&otp={otpValidate}
Kernel	Send SMS	v1/notifier/sms/send
Kernel	Send EMAIL	v1/notifier/email/send
Kernel	Generate PRID	v1/pridgenerator/prid
Kernel	Generate UIN	v1/idgenerator/uin
Kernel	Generate VID	v1/idgenerator/vid
Kernel	Generate RID	v1/ridgenerator/generate/rid/{centerID}/{machineID}
IDA	Auth with OTP	idauthentication/v1/auth/{mispLicenseKey}/{rpPartnerId}/{rpApiKey}
IDA	KYC with OTP	idauthentication/v1/kyc/{mispLicenseKey}/{ekycPartnerId}/{ekycApiKey}
IDA	Auth with bio	idauthentication/v1/auth/{mispLicenseKey}/{rpPartnerId}/{rpApiKey}
IDA	KYC with bio	idauthentication/v1/kyc/{mispLicenseKey}/{ekycPartnerId}/{ekycApiKey}
IDA	Send OTP	idauthentication/v1/otp/{mispLicenseKey}/{rpPartnerId}/{rpApiKey}
Pre-Registration	Prereg UI end to end flow	Prereg UI end to end flow
Syncdata	Public Key Verify	v1/syncdata/tpm/publickey/verify
Syncdata	Get Certificate	v1/syncdata/getCertificate?applicationId={appID}&referenceId={refID}
Syncdata	Get User Details	v1/syncdata/userdetails?keyindex={keyIndex}
Syncdata	Get Client Settings	v1/syncdata/v2/clientsettings?keyindex={keyIndex}
Syncdata	Get Configs	v1/syncdata/configs/{name}
Syncdata	Get LatestID Schema	v1/syncdata/latestidschema?schemaVersion={schemaVersion}
Syncdata	Get CAcertificates	v1/syncdata/getcacertificates
Regproc	Sync Registration Packet Details	registrationprocessor/v1/registrationstatus/sync
Regproc	Get Packet status	registrationprocessor/v1/registrationstatus/search
Regproc	Upload Registration Packet	registrationprocessor/v1/packetreceiver/registrationpackets
Resident	Request/Send OTP	resident/v1/req/otp
Resident	RID Check Status	resident/v1/rid/check-status
Resident	Auth Lock	resident/v1/req/auth-lock
Resident	Auth Unlock	resident/v1/req/auth-unlock
Resident	Credential Request	resident/v1/req/credential
Resident	Auth History	resident/v1/req/auth-history
Resident	Generate VID	resident/v1/vid
Datashare	Create Data Share URL	v1/datashare/create/{policyId}/{subscriberId}
Datashare	Get Data Share File	v1/datashare/get/{policyId}/{subscriberId}/{randomShareKey}
IDRepo	Retrieve Identity using UIN	idrepository/v1/identity/idvid/{uin}
IDRepo	Retrieve Identity using VID	idrepository/v1/identity/idvid/{vid}
IDRepo	Add Identity	idrepository/v1/identity/
IDRepo	Update Identity	idrepository/v1/identity/
IDRepo	Auth Type Status	idrepository/v1/identity/authtypes/status
IDRepo	Create VID	idrepository/v1/vid
IDRepo	Update VID	idrepository/v1/vid/{vidGenerated}
Masterdata	Get Registration Centers	v1/masterdata/registrationcenters
Masterdata	Get Templates	v1/masterdata/templates
Masterdata	Get Templates By Lang & Temp Code	v1/masterdata/templates/{lang&TempTypeCode}
Masterdata	Get Latest ID Schema	v1/masterdata/idschema/latest?schemaVersion={schemaVersion}
Masterdata	Get Dynamic Fields	v1/masterdata/dynamicfields?pageNumber=0&pageSize=10&sortBy=name&orderBy=desc
Masterdata	Get Users	v1/masterdata/users/{userID}/{time(yyyy-MM-dd'T'HH:mm:ss.sss'Z',)}
Keymanager	Encrypt Data	v1/keymanager/encrypt
Keymanager	Encrypt Data With Pin	v1/keymanager/encryptWithPin
Keymanager	Generate JWT Signature	v1/keymanager/jwtSign
Keymanager	Decrypt Data	v1/keymanager/decrypt
Keymanager	Decrypt Data With Pin API	v1/keymanager/decryptWithPin
Keymanager	Verify JWT Signature	v1/keymanager/jwtVerify

CPU (cores)	Memory (GB)	HDD (GB)	OS	Count	Purpose
8	30	768	Cent OS	1	Kubernetes console machine
8			Kubernetes node machines
4	16	500	Ubuntu	1	Database machine
4	16	835	Windows	3	Jmeter test machines

Pod/Application Names	Resource Allocation (Limits)
	Memory (M)	CPU (m)	Heap (M)
kernel-auditmanager-service	2500	1000	1750
kernel-auth-service	2250	500	1500
kernel-otpmanager-service	1750	500	1000
kernel-notification-service	1500	500	1000
kernel-pridgenerator-service	1750	300	1000
kernel-idgenerator-service	3500	500	2000
kernel-ridgenerator-service	1750	500	1000
ida-auth-service	4000	4500	2000
ida-otp-service	3000	3000	2000
prereg-application-service	2500	1000	2000
kernel-syncdata-service	5000	500	4000
regproc-registration-status-service	4000	500	2000
regproc-stage-group-1	5000	1000	4000
resident-service	3000	500	2000
datashare-service	3000	500	2000
idrepo-identity-service	4000	1000	2000
idrepo-vid-service	3000	500	1000
kernel-masterdata-service	2250	500	1500
kernel-keymanager-service	5000	1000	4000

Label	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
Client Id - Secret Key API Execution	241902	449	202	483	597	3352	9	60051	0.16%	66.1073	290.08	28.49
User Id Pwd API Execution	220878	488	488	680	711	872	70	6616	0.00%	61.34721	215.9	30
Validate Token API Execution	451378	239	201	400	496	680	6	3506	0.00%	125.3778	582.61	235.51

Label	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
Client Id - Secret Key API Execution	243069	740	499	1694	2100	2833	9	7511	0.00%	67.51412	296.36	29.09
User Id Pwd API Execution	27121	5519	3290	11676	22351	43894	40	60047	0.39%	9.05834	31.82	4.42
Validate Token API Execution	181953	989	810	2107	2496	3206	6	8111	0.00%	50.53694	234.74	94.93

Label	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
Generate OTP API Execution	87777	1229	1104	2160	2496	3207	5	7418	0.00%	24.37841	15.57	48.04
Validate OTP API Execution	150000	1275	1128	2192	2404	3196	0	6268	0.02%	23.4176	14.89	42.63
Notification Manager SMS API Request	433004	246	99	692	808	1119	2	2625	0.00%	120.2695	80.92	259.67
Notification Manager EMAIL API Request	69727	1550	1396	2992	3602	5101	4	13607	0.04%	14.87585	9.48	41.37
PRID Generator API Execution	102606	1052	1022	1893	2073	2432	5	5104	0.00%	28.49246	11.74	51
Generate UIN API Execution	39971	2698	2672	2813	2833	3536	127	8776	0.00%	11.09746	26.27	23.71
Generate VID API Execution	205377	154	139	266	321	423	5	12380	6.39%	193.1971	458.89	412.75
RID Generator API Execution	368802	291	193	743	930	1324	5	5761	0.00%	102.4346	63.42	221.07

Label	# Samples	Average	Median	90% Line	95% Line	99% Line	Min	Max	Error %	Throughput	Received KB/sec	Sent KB/sec
Generate OTP API Execution	768397	45	9	92	95	100	5	299	0.00%	213.4406	135.9	420.63
Generate OTP API Request	90000	42	8	90	91	94	4 <