On Premises Deployment
On-Premises Deployment
1. Wireguard
Wireguard bastion server provides a secure private channel to access the MOSIP cluster.
The bastion server restricts public access and enables access only to clients whose public keys are listed in the Wireguard server.
The bastion server listens on UDP port 51820.
If you already have a VPN configured to access nodes privately, skip Wireguard installation and continue using the same VPN.
Setup Wireguard VM and Wireguard Bastion Server
Create a Wireguard server VM as per the Hardware and Network Requirements.
Open the required ports on the bastion server VM:
cd $K8_ROOT/wireguard/
cp hosts.ini.sample hosts.iniNote:
Remove the
[Cluster]section entirely from the copiedhosts.inifile.Add the following details:
ansible_host— public IP of the Wireguard bastion server. e.g.100.10.20.56
ansible_user— user for installation. In this reference implementation we use the Ubuntu user.
ansible_ssh_private_key_file— path to the PEM key for SSH to the Wireguard server. e.g.~/.ssh/wireguard-ssh.pem
Execute
ports.yamlto open ports at the VM level using ufw:
Note:
PEM file permissions for SSH access must be set to 400:
sudo chmod 400 ~/.ssh/privkey.pemThese ports only need to be opened for UDP packet sharing.
Ensure the Wireguard server is reachable on port
51820/udpat the firewall level.
Install Docker on the Wireguard VM:
Set up the Wireguard server. SSH into the Wireguard VM:
Create a directory for storing Wireguard config files:
Install and start the Wireguard server using Docker:
Note:
Increase the number of peers if more than 30 Wireguard client configs are needed (
-e PEERS=30).Change the mounted directory as needed. All Wireguard configs will be generated in the mounted directory (
-v /home/ubuntu/wireguard/config:/config).
Setup Wireguard Client on your PC
Install the Wireguard client using the steps at wireguard.com/install.
Assign a
wireguard.conf. SSH into the Wireguard server VM and navigate to the config directory:
Assign one peer to yourself and use it from your PC to connect to the server.
Create an
assigned.txtfile to track allocated peer files, and update it each time a peer is assigned:
Use
lsto see the list of available peers.Enter your selected peer directory and update
peer.conf:
Make the following changes:
Delete the DNS IP line.
Update
AllowedIPsto the subnets CIDR range. e.g.10.10.20.0/23
Note:
The CIDR range will be provided by the infrastructure provider.
Ensure all nodes are covered in the CIDR range (nginx server, Observation K8s cluster nodes, and MOSIP K8s cluster nodes).
Share the updated
peer.confwith the respective peer to allow them to connect from their PC.Add
peer.confto your PC's/etc/wireguard/directory aswg0.conf.Start the Wireguard client and check the status:
Once connected to Wireguard, you can log in to cluster nodes using their private IPs.
2. Observation K8s Cluster Setup and Configuration
Note: Cluster creation uses RKE2 (RKE1 is EOL). The recommended approach is the Ansible-based automated setup from
$K8_ROOT/k8-cluster/on-prem/rke2/ansible. See the RKE2 documentation for full reference.
Install all the required tools listed in the Personal Computer Setup section.
Ensure the following are installed on all cluster VMs and the client machine:
ufw,wget,curl,kubectl,istioctl,helm,jq,ansible(version > 2.12.4).Set up Observation Cluster node VMs per the hardware and network requirements.
Set up passwordless SSH to the cluster nodes via PEM keys (skip if VMs are already accessible via PEM):
Generate keys on your PC:
ssh-keygen -t rsaCopy the keys to remote observation node VMs:
ssh-copy-id <remote-user>@<remote-ip>Verify SSH access:
ssh -i ~/.ssh/<your-private-key> <remote-user>@<remote-ip>
Note: Ensure the PEM file permission is set to 400:
sudo chmod 400 ~/.ssh/privkey.pem
Step 1 — Prepare Inventory File
Navigate to the ansible directory and create a copy of the sample inventory:
Update hosts.ini with the details of all cluster VMs:
Note:
ansible_host— internal IP of each node. e.g.100.10.20.56,100.10.20.57
ansible_user— user for installation. In this reference implementation we use the Ubuntu user.
ansible_ssh_private_key_file— path to the PEM key. e.g.~/.ssh/nodes-ssh.pemSandbox/Development: comment out the
[etcd]section inhosts.ini.Production: ensure all sections including
[etcd]are fully populated.
Step 2 — Enable Required Ports
Open the required ports on all nodes using ufw. For the full list of RKE2 networking requirements see docs.rke2.io:
Step 3 — Disable Swap
Disable swap on all cluster nodes (skip if already disabled):
Caution: Verify swap status with
swapon --showbefore running this playbook.
Note: RKE2 uses containerd and does not require Docker. There is no
docker.yamlstep for RKE2 clusters.
Step 4 — Run the Main Playbook
Execute the main Ansible playbook to install and configure the RKE2 cluster across all nodes:
This provisions the RKE2 cluster — installing the binary, distributing config to each node type (primary server, secondary servers, etcd, agents), and starting all services automatically.
Step 5 — Securely Store the Kubeconfig File
After the cluster is successfully created, the kubeconfig file is saved in the ansible/playbook/ directory as {{ cluster_domain }}-{{ inventory_hostname }}.yaml. Copy it to your .kube directory:
Step 6 — Access the Cluster
To use this kubeconfig, choose one of the following:
Alternatively, set the KUBECONFIG environment variable:
Step 7 — Verify Cluster Access
The output should list all nodes of the Observation cluster.
Important: Store the kubeconfig file and the
hosts.iniused for this deployment in a secure location. They are required for cluster management, adding nodes, and recovery.
3. Observation K8s Cluster Ingress and Storage Class Setup
Once the cluster is ready, ingress and a storage class must be configured before other applications can be installed.
3.a. Nginx Ingress Controller
Nginx Ingress Controller is used for ingress in the Rancher cluster.
Note:
This installs ingress into the
ingress-nginxnamespace of the Observation cluster.Verify the installation:
kubectl get all -n ingress-nginxThe output should list all pods and deployments in the
ingress-nginxnamespace.
3.b. Storage Classes
Multiple storage class options are available for on-premises K8s clusters. This reference deployment uses NFS as the storage class.
Move to the NFS directory on your personal computer:
Create a copy of
hosts.ini.sampleashosts.ini:
Update the NFS machine details in
hosts.ini:
Note:
ansible_host— internal IP of the NFS server. e.g.10.12.23.21
ansible_user— user for installation. In this reference implementation we use the Ubuntu user.
ansible_ssh_private_key_file— path to the PEM key. e.g.~/.ssh/nfs-ssh.pem
Confirm the kubeconfig is pointing to the correct Observation cluster:
Note: The output should show the expected cluster name. If not, update your kubeconfig to point to the correct cluster.
Open the required firewall ports on the NFS node:
SSH into the NFS node:
Clone the
k8s-infrarepo on the NFS VM:
Move to the NFS directory and run the install script:
Note: The script will prompt for an environment name:
where
envNameis the environment name e.g.dev,qa,uat. The NFS share will be exported at/srv/nfs/mosip/<envName>.
Switch back to your personal computer and run the NFS client provisioner:
Note: The script will prompt for:
NFS Server: IP of the NFS server.
NFS Path: Path for persisted data. e.g.
/srv/nfs/mosip/
Post-installation checks:
Check the status of the NFS Client Provisioner:
Check the storage class is registered:
Expected output:
Set
nfs-clientas the default storage class:
4. Setting up Nginx Server for Observation K8s Cluster
4.a. SSL Certificate Setup for TLS Termination
An SSL certificate is required for the Nginx server. If a valid certificate is not available, generate one using Let's Encrypt:
SSH into the nginx server.
Install the prerequisites:
Generate a wildcard SSL certificate for your domain:
Note: Replace
org.netwith your actual domain.
The DNS challenge requires you to create a TXT record in your DNS service:
Host:
_acme-challenge.org.netValue: the string prompted by the script.
Wait a few minutes for the DNS entry to propagate. Verify with:
Press Enter in the
certbotprompt to proceed.Certificates are created in
/etc/letsencrypt/and are valid for 3 months.For certificate renewal, see the wildcard SSL renewal guide.
4.b. Install Nginx
Move to the nginx directory:
Create a
hosts.inifile for the nginx server:
Add the following with the nginx server details and save:
Open the required ports:
SSH into the nginx server node:
From the nginx directory, run the install script:
Provide the following inputs when prompted:
Rancher nginx IP: internal IP of the nginx server VM.
SSL cert path: path to the SSL certificate for TLS termination.
SSL key path: path to the SSL key for TLS termination.
Cluster node IPs: IPs of the Rancher cluster nodes.
Post-installation checks:
sudo systemctl status nginxTo uninstall nginx if required:
sudo apt purge nginx nginx-commonDNS mapping: Once nginx is installed successfully, create DNS records for the Rancher cluster domains as described in the DNS requirements section (e.g.
rancher.org.net,keycloak.org.net).
5. Observation K8s Cluster Apps Installation
5.a. Rancher UI
Rancher provides full management capability for creating and administering Kubernetes clusters.
Update
hostnameinrancher-values.yaml, then install Rancher using Helm:
Connect to Wireguard (on Windows via WSL, connect to the Wireguard server from Windows, not from WSL).
Open the Rancher page in a browser.
Retrieve the bootstrap password:
Important: Assign a strong password and store it securely. It will be needed by the admin.
5.b. Keycloak
Keycloak is an OAuth 2.0-compliant Identity and Access Management (IAM) system used to manage access to Rancher for cluster administration.
After installation, access Keycloak at iam.mosip.net and retrieve credentials as per the post-installation steps.
5.c. Keycloak — Rancher UI Integration
Log in as the
adminuser in Keycloak and ensure theemailandfirstNamefields are populated for the admin user. These fields are required for Rancher authentication.In Keycloak (in the
masterrealm), create a new client with the following values:Client ID:https://<your-rancher-host>/v1-saml/keycloak/saml/metadataClient Protocol:samlRoot URL: (leave empty)
After saving, configure the client with the following settings:
Name
rancher
Enabled
ON
Login Theme
keycloak
Sign Documents
ON
Sign Assertions
ON
Encrypt Assertions
OFF
Client Signature Required
OFF
Force POST Binding
OFF
Front Channel Logout
OFF
Force Name ID Format
OFF
Name ID Format
username
Valid Redirect URIs
https://<your-rancher-host>/v1-saml/keycloak/saml/acs
IDP Initiated SSO URL Name
IdPSSOName
In the same client, go to the
Mapperstab and create the following mappers:Mapper 1 — username:
FieldValueProtocol
samlName
usernameMapper Type
User PropertyProperty
usernameFriendly Name
usernameSAML Attribute Name
usernameSAML Attribute NameFormat
BasicMapper 2 — groups:
FieldValueProtocol
samlName
groupsMapper Type
Group ListGroup Attribute Name
memberFriendly Name
(leave empty)
SAML Attribute NameFormat
BasicSingle Group Attribute
ONFull Group Path
OFFClick
Add Builtin→ select all →Add Selected.Download the Keycloak SAML descriptor XML from:
Generate a self-signed SSL certificate and key (if not already available):
In Rancher UI, navigate to:
Users & Authentication→Auth Providers→Keycloak (SAML)Configure the fields as follows:
Display Name Field
givenName
User Name Field
email or uid
UID Field
username
Groups Field
member
Entity ID Field
(leave empty)
Rancher API Host
https://<your-rancher-host>
Upload the following files:
Private Key:myservice.keyCertificate:myservice.certSAML Metadata XML: content from the Keycloak descriptor URL above
Click Enable to activate Keycloak authentication. After successful integration, Rancher users can log in using their Keycloak credentials.
5.d. RBAC for Rancher using Keycloak
Assign cluster and project roles in Rancher for Keycloak users. Add all namespaces under the
defaultproject. Non-admin users can be granted the Read-Only role at the project level.To create custom roles, follow the steps here.
To add a member to a cluster or project in Rancher:
Navigate to RBAC cluster members.
Add the member name exactly as the
usernamein Keycloak.Assign an appropriate role (e.g. Cluster Owner, Cluster Viewer).
To add a group to a cluster or project in Rancher:
Navigate to RBAC cluster members.
Click
Addand select a group from the dropdown.Assign an appropriate role.
Note: a user must be a member of the group to add it.
To create a Keycloak group:
Go to the
Groupssection in Keycloak and create a group with default roles.In the
Userssection, select a user, go to theGroupstab, and add the user to the required group.
6. MOSIP K8s Cluster Setup
Note: Cluster creation uses RKE2 (RKE1 is EOL). The recommended approach is the Ansible-based automated setup from
$K8_ROOT/k8-cluster/on-prem/rke2/ansible. See the RKE2 documentation for full reference.
Install the following tools on your PC and all cluster VMs:
kubectl,helm,ansible(version > 2.12.4),ufw,wget,curl,istioctl,jq.Add the required Helm repositories:
Set up MOSIP K8s Cluster node VMs per the Hardware and Network Requirements.
Set up passwordless SSH to the cluster nodes via PEM keys (skip if VMs are already accessible via PEM):
Generate keys on your PC:
ssh-keygen -t rsaCopy keys to remote cluster node VMs:
ssh-copy-id <remote-user>@<remote-ip>Verify SSH access:
ssh -i ~/.ssh/<your-private-key> <remote-user>@<remote-ip>
Step 1 — Prepare Inventory File
Navigate to the ansible directory and create a copy of the sample inventory:
Update hosts.ini with the details of all MOSIP cluster VMs:
Note:
ansible_host— internal IP of each node. e.g.100.10.20.56,100.10.20.57
ansible_user— user for installation. In this reference implementation we use the Ubuntu user.
ansible_ssh_private_key_file— path to the PEM key. e.g.~/.ssh/nodes-ssh.pemSandbox/Development: comment out the
[etcd]section inhosts.ini.Production: ensure all sections including
[etcd]are fully populated.
Step 2 — Enable Required Ports
Open the required ports on all nodes. For the full list of RKE2 networking requirements see docs.rke2.io:
Step 3 — Disable Swap
Disable swap on all cluster nodes (skip if already disabled):
Caution: Verify swap status with
swapon --showbefore running this playbook.
Note: RKE2 uses containerd and does not require Docker. There is no
docker.yamlstep for RKE2 clusters.
Step 4 — Run the Main Playbook
Execute the main Ansible playbook to install and configure the RKE2 cluster across all nodes:
This provisions the RKE2 cluster — installing the binary, distributing config to each node type (primary server, secondary servers, etcd, agents), and starting all services automatically.
Step 5 — Securely Store the Kubeconfig File
After the cluster is successfully created, the kubeconfig file is saved in the ansible/playbook/ directory as {{ cluster_domain }}-{{ inventory_hostname }}.yaml. Copy it to your .kube directory:
Step 6 — Access the Cluster
To use this kubeconfig, choose one of the following:
Alternatively, set the KUBECONFIG environment variable:
Step 7 — Verify Cluster Access
The output should list all nodes of the MOSIP cluster.
Important: Store the kubeconfig file and the
hosts.iniused for this deployment in a secure location. They are required for cluster management, adding nodes, and recovery.
7. MOSIP K8s Cluster Global ConfigMap, Ingress, and Storage Class Setup
7.a. Global ConfigMap
The global ConfigMap contains common configuration values shared across all namespaces in the cluster.
Update the domain names in global_configmap.yaml, then apply:
7.b. Istio Ingress Setup
Istio is a service mesh for the MOSIP K8s cluster. It provides transparent layers over existing microservices, enabling a uniform way to secure, connect, and monitor services.
This will bring up all Istio components and the Ingress Gateways. Verify the Ingress Gateway services:
Note: The response should include the following services:
istio-ingressgateway— external-facing Istio service.
istio-ingressgateway-internal— internal-facing Istio service.
istiod— Istio daemon that replicates changes to all Envoy filters.
7.c. Storage Classes
Multiple storage class options are available for on-premises K8s clusters. This reference deployment uses NFS as the storage class.
Move to the NFS directory on your personal computer:
Create a copy of
hosts.ini.sampleashosts.ini:
Update the NFS machine details in
hosts.ini:
Note:
ansible_host— internal IP of the NFS server. e.g.10.12.23.21
ansible_user— user for installation. In this reference implementation we use the Ubuntu user.
ansible_ssh_private_key_file— path to the PEM key. e.g.~/.ssh/nfs-ssh.pem
Confirm the kubeconfig is pointing to the correct MOSIP cluster:
Note: The output should show the expected cluster name. If not, update your kubeconfig to point to the correct cluster.
Open the required firewall ports on the NFS node:
SSH into the NFS node:
Clone the
k8s-infrarepo on the NFS VM:
Move to the NFS directory and run the install script:
Note: The script will prompt for an environment name:
where
envNameis the environment name e.g.dev,qa,uat. The NFS share will be exported at/srv/nfs/mosip/<envName>.
Switch back to your personal computer and run the NFS client provisioner:
Note: The script will prompt for:
NFS Server: IP of the NFS server.
NFS Path: Path for persisted data. e.g.
/srv/nfs/mosip/
Post-installation checks:
Check the status of the NFS Client Provisioner:
Check the storage class is registered:
8. Import MOSIP Cluster into Rancher UI
Log in as admin in the Rancher console.
Select
Import Existingfor cluster addition.Select
Genericas the cluster type.Fill in the
Cluster Namefield with a unique name and clickCreate.Copy the generated
kubectlcommand and execute it from your PC (ensure your kubeconfig is set to the MOSIP cluster):
Wait a few seconds for the cluster to be verified. The cluster will then appear in the Rancher management console.
9. MOSIP K8s Cluster Nginx Server Setup
9.a. SSL Certificate Creation
An SSL certificate is required for the Nginx server. If a valid certificate is not available, generate one using Let's Encrypt:
SSH into the nginx server.
Install the prerequisites:
Generate a wildcard SSL certificate for your domain:
Note: Replace
sandbox.mosip.netwith your actual domain.
The DNS challenge requires you to create a TXT record in your DNS service:
Host:
_acme-challenge.sandbox.mosip.netValue: the string prompted by the script.
Wait a few minutes for the DNS entry to propagate. Verify with:
Press Enter in the
certbotprompt to proceed.Certificates are created in
/etc/letsencrypt/and are valid for 3 months.For certificate renewal, see the wildcard SSL renewal guide.
9.b. Nginx Server Setup for MOSIP K8s Cluster
Move to the nginx directory:
Create a
hosts.inifile for the nginx server:
Add the following with nginx server details and save:
Open the required ports:
SSH into the nginx server node:
From the nginx directory, run the install script:
Provide the following inputs when prompted:
MOSIP nginx server internal IP.
MOSIP nginx server public IP.
Publicly accessible domains (comma-separated, no whitespace).
SSL cert path.
SSL key path.
Cluster node IPs (comma-separated, no whitespace).
Post-installation checks:
sudo systemctl status nginxTo uninstall nginx if required:
sudo apt purge nginx nginx-commonDNS mapping: Once nginx is installed successfully, create DNS records for the MOSIP cluster domains as described in the DNS requirements section.
9.c. Verify Nginx and Istio Wiring
Install httpbin — a utility that echoes HTTP headers received inside the cluster, useful for debugging ingress and header routing:
Test external and internal ingress (replace with your actual domain):
10. Monitoring Module Deployment
Note:
Monitoring is optional in sandbox environments.
For production environments, alternative monitoring tools may be used.
These steps can be skipped in development environments if monitoring is not needed.
If skipping, run the following commands to install the monitoring CRD, which is required by MOSIP services:
Prometheus, Grafana, and Alertmanager are used for cluster monitoring.
In Rancher console, go to
Apps & Marketplacesand select theMonitoringapp.In the Helm options, open the YAML file and disable the Nginx Ingress:
Click
Install.
11. Alerting Setup
Note:
Alerting is optional in sandbox environments.
For production environments, alternative alerting tools may be used.
These steps can be skipped in development environments if alerting is not needed.
Alerting is part of cluster monitoring. Alert notifications are sent to a configured email address or Slack channel.
Monitoring (Prometheus, Grafana, Alertmanager) must be deployed before setting up alerting.
Create a Slack incoming webhook.
Update
slack_api_urland the Slack channel name inalertmanager.yml:
Update the following values:
Update the cluster name in
patch-cluster-name.yaml:
Update:
Install the default and custom alerts:
12. Logging Module Setup and Installation
Note:
Logging is optional in sandbox environments.
For production environments, alternative logging tools may be used.
These steps can be skipped in development environments if logging is not needed.
MOSIP uses Rancher Logging and Elasticsearch to collect logs from all services and display them in a Kibana dashboard.
Install the Rancher FluentD system (required to scrape logs from all MOSIP microservices):
In Rancher UI, go to
Apps & Marketplacesand install the Logging app.Select chart version
100.1.3+up3.17.7.
Configure Rancher FluentD. Create the
clusteroutput:
Create the
clusterflow:
Install Elasticsearch, Kibana, and Istio addons:
Set the
min_agevalue inelasticsearch-ilm-script.sh(the minimum number of days to retain indices in Elasticsearch), then execute it:
MOSIP provides the following Kibana dashboards:
Logstash index pattern required by all other dashboards
Shows only error-level logs (MOSIP Error Logs dashboard)
Shows all logs for a specific service (MOSIP Service Logs dashboard)
Shows MOSIP process insights such as UINs generated and packets uploaded (MOSIP Insight dashboard)
Shows API response time trends across MOSIP services (Response Time dashboard)
Import dashboards:
View dashboards: open Kibana at
https://kibana.sandbox.xyz.net, then navigate to Menu → Dashboard → select a dashboard.
13. MOSIP External Dependencies Setup
External dependencies include all external components required by MOSIP's core services: database (PostgreSQL), object store (MinIO), HSM, and others.
See the detailed installation instructions for all external components.
Note: After installation, connect to the
mosip_pmsPostgreSQL database and extend thevalid_to_dateformpolicy-default-mobile:
14. MOSIP Modules Deployment
With the Kubernetes cluster and all external dependencies in place, proceed with MOSIP service deployment.
Note:
If
install-all.shfails at any point, follow the MOSIP Modules Deployment guide from the point of failure.The config-server and admin service may experience startup delays in this version. Apply the following fixes if needed:
For config-server — increase
failureThresholdforstartupProbeto 60:For admin-service — increase
failureThresholdforstartupProbeto 60:Once admin-service is running, re-execute
install.shafter commenting out the following lines:
15. API Testrig
MOSIP's successful deployment can be verified by running the API testrig and comparing results against the testrig benchmark.
Navigate to the Infra Root Directory:
Clone the Functional Tests repository:
Move to the apitestrig deployment directory:
Make the script executable:
Run the installer:
Note: The script prompts for the following inputs:
Cronjob hour (0–23): e.g.
6for 6 AM.Do you have a public domain and valid SSL certificate? (Y/n)
Retention days for old reports (default: 3).
Slack Webhook URL for server issue notifications.
Is the eSignet service deployed? (yes/no) — if
no, eSignet-related test cases are skipped.Is
values.yamlfor the apitestrig chart configured as part of prerequisites? (Y/n)Do you have S3 details for storing API-Testrig reports? (Y/n)
16. DSL Rig
Install Packetcreator
Navigate to the packetcreator directory:
Run the installation script:
Provide the following inputs when prompted:
NFS Host:
NFS PEM File:
User for SSH Login:
ubuntuIngress Controller Type: select
2for Istio.
Install DSL Rig
Navigate to the dslrig directory:
Run the installation script:
Note: Before running
install.sh, ensure the following flag is included in the Helm install command:The full Helm install command should look like:
When prompted, provide the following inputs:
NFS Host:
NFS PEM:
User for SSH Login:
ubuntuCronjob hour (0–23).
Do you have a public domain and valid SSL certificate? (Y/n)
Packet Utility Base URL:
https://packetcreator.packetcreator:80/v1/packetcreatorRetention days for old reports (default: 3).
Last updated
Was this helpful?