This version of the guide is currently under revision and enhancement. For any support, please reach out to us on community.mosip.io.
Version: 1.2.0.1 (Latest stable release)
Release date: 19th March 2024
Version: 1.2.0.1-B4
Release date: 12th January 2024
Version: 1.2.0.1-B3
Release date: 14th April 2023
Version: 1.2.0.1-B2
Release date: 8th Jan 2023
Version: 1.2.0.1-B1
Release date: 14th Oct 2022
Overview and Architecture
Overview
MOSIP modules are deployed as microservices in a Kubernetes cluster.
Wireguard is used as a trust network extension to access the admin, control, and observation panes.
In case of on-field registration, WireGuard is extended to the registration client machines as one of the sources of trust.
MOSIP uses server for:
SSL termination
Reverse Proxy
CDN/Cache management
Kubernetes cluster is administered using the and tools.
We have two Kubernetes clusters:
Observation cluster - This cluster is part of the observation plane and assists with administrative tasks. By design, this is kept independent from the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or its services should be internal and should never be exposed to the external world.
is used for managing the MOSIP cluster.
in this cluster is used to manage user access and rights for the observation plane.
It is recommended to configure log monitoring and network monitoring in this cluster.
MOSIP cluster - This cluster runs all the MOSIP components and certain third-party components like the kafka, keycloak etc.
Architecture diagram
Deployment repos
These are the repos that we would use for installing and configuring the MOSIP deployment. For detailed steps, please follow the guide. This section is for reference.
: contains the scripts to install and configure a Kubernetes cluster with required monitoring, logging, and alerting tools.
: contains the deployment scripts to run charts in a defined sequence.
: contains all the configuration files required by the MOSIP modules.
This version of the guide is currently under revision and enhancement. For any support, please reach out to us on community.mosip.io.
External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, and HSM etc.
List of external dependencies:
: Relational Database system used for storing data in MOSIP.
IAM: IAM tool is for authentication and authorization. Reference implementation here uses for the same purpose.
: Hardware Security Module (HSM) stores the cryptographic keys used in MOSIP. Reference implementation is provided as SoftHSM here.
: MOSIP uses S3 API compliant object store for storing biometric and other data. Reference implementation here uses .
Anti-virus: Used for document scanning and packets scanning throughout MOSIP modules. Reference implementation uses dockerised version of .
Queuing tool: Tool used for queuing messages to external MOSIP components. Reference implementation used .
Event Publisher/ streamer: MOSIP uses Kafka for publishing events to it's internal as well as external partners modules.
BioSDK: Biometric SDK for quality check and authentication purpose using biometrics.
: Performs the de-duplication of a resident's biometric data.
Message Gateway: This is for notifying residents about different OTPs and other information.
Installation
Postgres
Install Postgres
Initialize Postgres DB
Opt for yes and enter Y.
Keycloak
Install Keycloak
Initialize Keycloak
Setup SoftHSM
Setup Object store
Opt 1 for MinIO
Opt 2 for S3 (incase you are not going with MinIO installation and want s3 to be installed)
Enter the prompted details.
MinIO installation
S3 Credentials setup
ClamAV setup
ActiveMQ setup
Kafka setup
BioSDK Server setup
Reference implementation of Biometric SDK server will be installed separately in MOSIP service installation section as the same is dependent on artifactory which is a MOSIP component.
ABIS
ABIS is needed to be up and running outside the MOSIP cluster and should be able to connect to the activeMQ. For testing purpose, MOSIP has provided a mock stimulator for the same named as mock-abis which will be deployed as part of the MOSIP services installation.
MSG Gateway
MOSIP provides mock smtp server which will be installed as part of default installation, opt for Y.
Docker Secrets
Incase the images are getting pulled from private repositories.
Captcha
To setup the captcha for pre-reg and resident domains.
Landing page setup
Testrig
Authdemo Service
The Authdemo service is utilized to execute the IDA APIs that are employed by API-testrig and DSLrig.
The purpose of the Authdemo Service is to showcase the functionality of authentication.
It can be considered as a simplified iteration of an authentication service, serving as a mock or prototype for the purpose of testing.
When prompted, input the NFS host, it's pem-key, ssh login user of NFS server.
Install script will create the NFS directories i.e., /srv/nfs/mosip/packetcreator-authdemo-authcerts to store the certificates generated by Authdemo service.
These certificates will be used by API-testrig, orchestrator and packetcreator.
API-testrig
API-testRig tests the working of APIs of the MOSIP modules.
MOSIP’s successful deployment can be verified by comparing the results of API-testrig with the testrig benchmark.
When prompted, input the hour of the day to execute the API-testrig.
Daily API-testrig cron job will be executed at the very opted hour of the day.
The reports will move to the object store ( i.e., s3/minio) under automationtests bucket.
Packetcreator
Packetcreator will create packets for DSL orchestrator.
Note: It is recommended to deploy the packetcreator on a separate server/ cluster from where the other DSL orchestrators can access this service.
When prompted, input the NFS host, it's pem-key, ssh login user of NFS server.
Install script will create two NFS directories i.e., /srv/nfs/mosip/packetcreator_data, /srv/nfs/mosip/packetcreator-authdemo-authcerts.
Packetcreator_data
DSLrig/ DSLOrchestrator
DSLrig will test end-to-end functional flows involving multiple MOSIP modules.
The Orchestrator utilizes the Packet Creator to generate packets according to the defined specifications. It then communicates with the Authdemo Service making REST API calls to perform authentication-related actions or retrieve the necessary information.
When prompted, input the NFS host, it's pem-key, ssh login user of NFS server.
Install script will create NFS directories, i.e., /srv/nfs/mosip/dsl-scenarios/sandbox.xyz.net to store the DSL scenario sheet.
The Default template for DSL scenario sheet can be accessible from .
1.2.0.3
Overview
MOSIP Rapid Deployment Infrastructure is a unified, cloud-native deployment platform that transforms complex, time-consuming MOSIP deployments into automated, 3-step processes that completes fast and takes much lesser time than earlier deployments. It replaces fragmented, manual infrastructure setup with a single repository containing everything needed for production-ready MOSIP deployment.
With MOSIP RDI (Rapid Deployment Infrastructure), you can deploy complete MOSIP identity platforms with enterprise-grade security, monitoring, and automation through a streamlined CI/CD approach that reduces deployment complexity by 90%.
With the current release of 0.2.0 Infra (Rapid Deployment Infrastructure), eSignet 1.4.2 now integrates as a dedicated deployment, unlocking execution of all the features' that requires eSignet (OTP, biometric auth, KYC exchange) and which were getting skipped earlier. Countries adopting eSignet can now validate complete end-to-end workflows before production. Countries using alternative authentication methods can skip the esignet-dsf.yaml phase entirely — core MOSIP services remain unaffected.
How does the rapid deployment model improve upon previous approaches in terms of DevOps modernization?
MOSIP RDI introduces several key improvements over the earlier approach:
Infrastructure as Code (IaC): MOSIP RDI (Rapid Deployment Infrastructure) leverages tools like Terraform for declarative, version-controlled infrastructure provisioning. This ensures repeatable, automated deployments and minimizes manual intervention.
Streamlined CI/CD Pipelines: Integrated GitHub Actions automate testing, validation, and deployment workflows, enabling faster and more reliable releases with minimal manual steps.
Automated Configuration and Deployment: Ansible and Cloud-Init are used for server configuration and initialization, ensuring consistent environments across deployments.
Where and how do I start?
To begin with MOSIP RDI:
Deployment Architecture
Key Resources
You can refer to the following key resources to get started with MOSIP Rapid Deployment. Currently the links take you to repo READMEs as this is the first beta release of MOSIP RDI (Rapid Deployment Infrastructure) and we are working to bring all the relevant content to this published documentation site as well.
Comprehensive Documentation: Access the official for step-by-step guides, architecture details, and deployment instructions.
Deployment Repository: Clone the deployment codebase from the . This repository contains all necessary scripts, modules, and configuration files for a production-ready setup.
eSignet Deployment Guide: Refer to the for configuration details and required secrets.
For additional guidance, refer to the section at the end of this document for links to checklists, setup guides, and troubleshooting resources.
How it works (High-level Overview)
MOSIP RDI originally follows a 3-step deployment model. Starting with this release, the eSignet authentication stack is added as a dedicated fourth deployment phase that runs after MOSIP core services are up.
cd $INFRA_ROOT/deployment/v3/external/postgres
./install.sh
cd $INFRA_ROOT/deployment/v3/external/postgres
./init_db.sh
cd $INFRA_ROOT/deployment/v3/external/iam
./install.sh
cd $INFRA_ROOT/deployment/v3/external/iam
./keycloak_init.sh
cd $INFRA_ROOT/deployment/v3/external/hsm/softhsm
./install.sh
cd $INFRA_ROOT/deployment/v3/external/object-store/minio
./install.sh
cd $INFRA_ROOT/deployment/v3/external/object-store/
./cred.sh
cd $INFRA_ROOT/deployment/v3/external/antivirus/clamav
./install.sh
cd $INFRA_ROOT/deployment/v3/external/activemq
./install.sh
cd $INFRA_ROOT/deployment/v3/external/kafka
./install.sh
cd $INFRA_ROOT/deployment/v3/external/msg-gateway
./install.sh
cd $INFRA_ROOT/deployment/v3/external/docker-secrets
./install.sh
cd $INFRA_ROOT/deployment/v3/external/captcha
./install.sh
cd $INFRA_ROOT/deployment/v3/external/landing-page
./install.sh
contains biometric data which is used to create packets.
The default packetcreator_data can be accessible from .
Copy the packetcreator_data from the link mentioned above to the NFS directory /srv/nfs/mosip/packetcreator_data.
Ensure to use the same NFS host and path i.e., /srv/nfs/mosip/packetcreator-authdemo-authcerts for Authdemo and packetcreator service.
When prompted, input the kubernetes ingress type (i.e., Ingress/Istio) and DNS as required if you are using the Ingress-nginx.
Copy the scenario csv from the above link to the NFS directory /srv/nfs/mosip/dsl-scenarios/sandbox.xyz.net. Make sure to rename the csv files by replacing env with your domain ex: sandbox.xyz.net.
To run the dslorchestrator for sanity only, update the dslorchestrator configmap TESTLEVEL key to sanity.
The reports will move to object store (i.e., s3/minio) under dslreports bucket.
cd $INFRA_ROOT/deployment/v3/testrig/authdemo
./install.sh
cd $INFRA_ROOT/deployment/v3/testrig/apitestrig
./install.sh
cd $INFRA_ROOT/deployment/v3/testrig/packetcreator
./install.sh
cd $INFRA_ROOT/deployment/v3/testrig/dslrig
./install.sh
Unified Repository and Collaboration: All infrastructure, configuration, and deployment scripts are maintained in a single repository, promoting collaboration and reducing silos between Dev, Ops, and Security teams.
Security and Compliance: Automated vulnerability scanning, compliance checks, and secure networking (e.g., WireGuard VPN, security groups) are built into the deployment process.
Scalability and Flexibility: The cloud-native, modular architecture—built on Kubernetes and managed via Helm/Helmsman—enables seamless scaling and adaptation to changing requirements.
eSignet Integration: The eSignet authentication stack (Redis, SoftHSM, Keycloak, Mock Identity System, OIDC UI, and Partner Onboarder) is now deployable via a dedicated esignet-dsf.yaml Helmsman Desired State File and a dedicated helmsman_esignet.yml GitHub Actions workflow, integrated with MOSIP core services deployment.
cd $INFRA_ROOT/deployment/v3/mosip/conf-secrets
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/config-server
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/artifactory
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/keymanager
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/websub
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/mock-smtp
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/kernel
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/masterdata-loader
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/biosdk
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/packetmanager
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/datashare
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/prereg
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/idrepo
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/pms
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/mock-abis
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/mock-mv
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/regproc
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/admin
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/ida
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/print
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/partner-onboarder
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/mosip-file-server
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/resident
./install.sh
cd $INFRA_ROOT/deployment/v3/mosip/regclient
sudo apt-get update
sudo apt-get install jq
./install.sh
Pre-Requisites
Hardware requirements
VM’s required can be with any OS as per convenience.
Here, we are referting to Ubuntu OS (22.0.4) throughout this installation guide.
Ensure to create Ubuntu user with its directory /home/ubuntu as the same user is referenced throughout the document. Else, change the user related commands as and when needed throughout the installation.
Sl no.
Purpose
vCPU's
RAM
Storage (HDD)
no. ofVM's
HA
Network requirements
All the VM's should be able to communicate with each other.
Need stable Intra network connectivity between these VM's.
All the VM's should have stable internet connectivity for docker image download (in case of local setup ensure to have a locally accessible docker registry).
Sl no.
Purpose
Network Interfaces
DNS requirements
Incase not using public DNS system please ignore this as the same mapping needs to be performed at node level and is mentioned in the deployment document.
Domain Name
Mapping details
Purpose
Certificate requirements
As only secured https connections are allowed via nginx server will need below mentioned valid ssl certificates:
One valid wildcard ssl certificate related to domain used for accessing Observation cluster, this needs to be stored inside the nginx server VM for Observation cluster. In above e.g.: *.org.net is the similar example domain.
One valid wildcard ssl certificate related to domain used for accessing Mosip cluster, this needs to be stored inside the nginx server VM for mosip cluster. In above e.g.: *.sandbox.xyz.net is the similar example domain.
Personal computer requirements
Note:
In case unfamiliar with below mentioned tools and trying deployment from personel computer having Windows OS, suggestion is to use WSL for easy proceedings.
Please ensure the following tools with the specified versions are installed on your personal computer.
- version 2.25.1 or higher.
- version 2.12.4 or higher
- any client version above 3.0.0 and add below repos as well:
: version: 1.15.0
: version:
- version > 2.12.4
1
(ensure to setup active-passive)
2.
Observation Cluster nodes
2
8 GB
32 GB
2
2
3.
Observation Nginx server (use Loadbalancer if required)
2
4 GB
16 GB
1
Nginx+
4.
MOSIP Cluster nodes
12
32 GB
128 GB
6
6
5.
MOSIP Nginx server ( use Loadbalancer if required)
2
4 GB
16 GB
1
Nginx+
Server Interface requirement as mentioned in below table:
Observation Nginx server
One internal interface: with internet access and that is on the same network as all the rest of nodes (e.g.: inside local NAT Network).
4.
Mosip Nginx server
One internal interface : that is on the same network as all the rest of nodes (e.g.: inside local NAT Network).
One public interface : Either has a direct public IP, or a firewall NAT (global address) rule that forwards traffic on 443/tcp port to this interface IP.
Administrative IAM tool (keycloak). This is for the kubernetes administration.
3.
sandbox.xyx.net
Private IP of Nginx server for MOSIP cluster
Index page for links to different dashboards of MOSIP env. (This is just for reference, please do not expose this page in a real production or UAT environment)
4.
api-internal.sandbox.xyz.net
Private IP of Nginx server for MOSIP cluster
Internal API’s are exposed through this domain. They are accessible privately over wireguard channel
5.
api.sandbox.xyx.net
Public IP of Nginx server for MOSIP cluster
All the API’s that are publically usable are exposed using this domain.
6.
prereg.sandbox.xyz.net
Public IP of Nginx server for MOSIP cluster
Domain name for MOSIP's pre-registration portal. The portal is accessible publicly.
7.
activemq.sandbox.xyx.net
Private IP of Nginx server for MOSIP cluster
Provides direct access to activemq dashboard. It is limited and can be used only over wireguard.
8.
kibana.sandbox.xyx.net
Private IP of Nginx server for MOSIP cluster
Optional installation. Used to access kibana dashboard over wireguard.
9.
regclient.sandbox.xyz.net
Private IP of Nginx server for MOSIP cluster
Registration Client can be downloaded from this domain. It should be used over wireguard.
10.
admin.sandbox.xyz.net
Private IP of Nginx server for MOSIP cluster
MOSIP's admin portal is exposed using this domain. This is an internal domain and is restricted to access over wireguard
11.
object-store.sandbox.xyx.net
Private IP of Nginx server for MOSIP cluster
Optional- This domain is used to access the object server. Based on the object server that you choose map this domain accordingly. In our reference implementation, MinIO is used and this domain let's you access MinIO’s Console over wireguard
12.
kafka.sandbox.xyz.net
Private IP of Nginx server for MOSIP cluster
Kafka UI is installed as part of the MOSIP’s default installation. We can access kafka UI over wireguard. Mostly used for administrative needs.
13.
iam.sandbox.xyz.net
Private IP of Nginx server for MOSIP cluster
MOSIP uses an OpenID Connect server to limit and manage access across all the services. The default installation comes with Keycloak. This domain is used to access the keycloak server over wireguard
14.
postgres.sandbox.xyz.net
Private IP of Nginx server for MOSIP cluster
This domain points to the postgres server. You can connect to postgres via port forwarding over wireguard
15.
pmp.sandbox.xyz.net
Private IP of Nginx server for MOSIP cluster
MOSIP’s partner management portal is used to manage partners accessing partner management portal over wireguard
16.
onboarder.sandbox.xyz.net
Private IP of Nginx server for MOSIP cluster
Accessing reports of MOSIP partner onboarding over wireguard
17.
resident.sandbox.xyz.net
Public IP of Nginx server for MOSIP cluster
Accessing resident portal publically
18.
idp.sandbox.xyz.net
Public IP of Nginx server for MOSIP cluster
Accessing IDP over public
19.
smtp.sandbox.xyz.net
Private IP of Nginx server for MOSIP cluster
Accessing mock-smtp UI over wireguard
Openssl Need openssl version 1.1.1 specifically for regclient installation.
Check the current OpenSSL version:
openssl version
If it's not OpenSSL 1.1.1f, remove the existing OpenSSL
sudo apt remove openssl
Manually install OpenSSL 1.1.1f by following this guide:
Note :
While following the guide:
Replace all instances of openssl-1.1.1s with openssl-1.1.1f.
Skip the commands that create backups of binaries:
After completing the installation, verify the OpenSSL version again:
Once the above tools are installed, create a directory named mosip on your PC and follow the steps below:
clone k8’s infra repo with tag : 1.2.0.2 (whichever is the latest version) inside mosip directory. git clone https://github.com/mosip/k8s-infra -b v1.2.0.2
clone mosip-infra with tag : 1.2.0.2 (whichever is the latest version) inside mosip directory. git clone https://github.com/mosip/mosip-infra -b v1.2.0.2
Set below mentioned variables in bashrc
source ~/.bashrc
Note: Above mentioned environment variables will be used throughout the installation to move between one directory to other to run install scripts.
1.
Wireguard Bastion Host
2
4 GB
1.
Wireguard Bastion Host
One Private interface : that is on the same network as all the rest of nodes (e.g.: inside local NAT Network).
One public interface : Either has a direct public IP, or a firewall NAT (global address) rule that forwards traffic on 51820/udp port to this interface IP.
2.
K8 Cluster nodes
One internal interface: with internet access and that is on the same network as all the rest of nodes (e.g.: inside local NAT Network )
1.
rancher.xyz.net
Private IP of Nginx server or load balancer for Observation cluster
Rancher dashboard to monitor and manage the kubernetes cluster.
Note, the target group name of listner 80. Set target group of 443 to target group of 80. Basically, we want TLS termination at the LB and it must forward HTTP traffic (not HTTPS) to port 80 of ingress controller. So
Input of LB: HTTPS
Output of LB: HTTP --> port 80 of ingress nginx controller
Enable "Proxy Protocol v2" in the target group settings
Make sure all subnets are selected in LB -->Description-->Edit subnets.
Check health check of target groups.
Remove listner 80 from LB as we will receive traffic only on 443.
3.b. Storage class setup:
Default storage class on EKS is gp2 . GP2 by default is in Delete mode which means if PVC is deleted, the underlying storage PV is also deleted.
Enable volume expansion for the existing gp2 storage class, modify the YAML configuration by adding allowVolumeExpansion: true to the gp2 storage class configuration.
kubectl edit sc gp2 : to edit the yaml configuration.
Create storage class gp2-retain by running sc.yaml for PV in Retain mode. Set the storage class as gp2-retain in case you want to retain PV.
EBS driver installation for our storage class to work, follow the steps to setup EBS driver.
Point the above to internal ip address of the NLB.
Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.
5.b.Keycloak : Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.
cd $K8_ROOT/rancher/keycloak
./install.sh <iam.host.name>
keycloak_client.json: Used to create SAML client on Keycloak for Rancher integration.
5.c. Keycloak - Rancher Integration
Login as admin user in Keycloak and make sure an email id, and first name field is populated for admin user. This is important for Rancher authentication as given below.
Enable authentication with Keycloak using the steps given here.
In Keycloak add another Mapper for the rancher client (in Master realm) with following fields:
Protocol: saml
Name: username
Mapper Type: User Property
Specify the following mappings in Rancher's Authentication Keycloak form:
Display Name Field: givenName
User Name Field: email
UID Field: username
RBAC :
For users in Keycloak assign roles in Rancher - cluster and project roles. Under default project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects).
If you want to create custom roles, you can follow the steps given here.
Add a member to cluster/project in Rancher:
Give member name exactly as username in Keycloak
Assign appropriate role like Cluster Owner, Cluster Viewer etc.
You may create new role with fine grained acccess control.
Certificates expiry
In case you see certificate expiry message while adding users, on local cluster run these commands:
Review and update the below mentioned parameters of cluster.config.sample carefully.
name
region
version: “1.24“
instance related details
instanceName
instanceType
desiredcapacity
update the details of the subnets to be used from vpc
Install
eksctl create cluster -f mosip.cluster.config
Wait for the cluster creation to complete, generally it takes around 30 minutes to create or update cluster.
Once EKS K8 cluster is ready below mentioned output will be displayed in the console screen.
EKS cluster "my-cluster" in "region-code" region is ready
The config file for the new cluster will be created on ~/.kube/config
Make sure to backup and store the ~/.kube/config with new name. e.g. ~/.kube/mosip-cluster.config.
Change file permission using below command:
chmod 400 ~/.kube/mosip-cluster.config
Set the KUBECONFIG properly so that you can access the cluster.
export KUBECONFIG=~/.kube/mosip-cluster.config
Test cluster access:
kubect get nodes
Command will result in details of the nodes of the MOSIP cluster.
Fill the Cluster Name field with unique cluster name and select Create.
You will get the kubectl commands to be executed in the kubernetes cluster. Copy the command and execute from your PC. (make sure your kube-config file is correctly set to Mosip cluster)
Update the domain names in global_configmap.yaml and run.
kubectl apply -f global_configmap.yaml
8.b. Storage class setup:
Default storage class on EKS is gp2 . GP2 by default is in Delete mode which means if PVC is deleted, the underlying storage PV is also deleted.
To enable volume expansion for the existing gp2 storage class, modify the YAML configuration by adding allowVolumeExpansion: true to the gp2 storage class configuration.
kubectl edit sc gp2 : to edit the yaml configuration.
Create storage class gp2-retain by running sc.yaml for PV in Retain mode. Set the storage class as gp2-retain in case you want to retain PV.
We need the EBS driver for our storage class to work, follow the steps to setup EBS driver.
Also we need EFS CSI driver for the regproc services, because EBS driver only supports RWO but we need RWX, follow these to setup EFS CSI driver.
8.c. Ingress and load balancer (LB) :
Ingress is not installed by default on EKS. We use Istio ingress gateway controller to allow traffic in the cluster. Two channels are created - public and internal. See architecture.
You may view them on AWS console in Loadbalancer section.
TLS termination is supposed to be on LB. So all our traffic coming to ingress controller shall be HTTP.
Obtain AWS TLS certificate as given
Add the certificates and 443 access to the LB listener.
Update listener TCP->443 to TLS->443 and point to the certificate of domain name that belongs to your cluster.
Forward TLS->443 listner traffic to target group that corresponds to listener on port 80 of respective Loadbalancers. This is because after TLS termination the protocol is HTTP so we must point LB to HTTP port of ingress controller.
Update health check ports of LB target groups to node port corresponding to port 15021. You can see the node ports with
Enable Proxy Protocol v2 on target groups.
Make sure all subnets are included in Availability Zones for the LB. Description --> Availability Zones --> Edit Subnets
Make sure to delete the listeners for port 80 and 15021 from each of the loadbalancers as we restrict unsecured port 80 access over http.
8.e. DNS mapping
Initially all the services will be accesible only over the internal channel.
Point all your domain names to internal LoadBalancers DNS/IP intially till testing is done.
On AWS this may be done on Route 53 console.
After Go live decision enable .
8.f. Check Overall if nginx and istio wiring is set correctly
Install httpbin: This utility docker returns http headers received inside the cluster. You may use it for general debugging - to check ingress, headers etc.
cd $K8_ROOT/utils/httpbin
./install.sh
To see what's reaching httpbin (example, replace with your domain name):
curl https://api-internal.sandbox.xyz.net/httpbin/get?show_env=true
Once public access is enabled also check this:
curl https://api.sandbox.xyz.net/httpbin/get?show_env=true
In Helm options, open the YAML file and disable Nginx Ingress.
After setting slack incoming webhook update slack_api_url and slack_channel_name in alertmanager.yml.
cd $K8_ROOT/monitoring/alerting/
nano alertmanager.yml
Update
Update Cluster_name in patch-cluster-name.yaml.
cd $K8_ROOT/monitoring/alerting/
nano patch-cluster-name.yaml
Update
Install Default alerts along some of the defined custom alerts:
cd $K8_ROOT/monitoring/alerting/
./install.sh
Alerting is Installed.
Install Logging from Apps and marketplace within the Rancher UI.
Select Chart Version 100.1.3+up3.17.7 from Rancher console -> Apps & Marketplaces.
Configure Rancher FluentD
Create clusteroutput :
kubectl apply -f clusteroutput-elasticsearch.yaml
start clusterFlow
kubectl apply -f clusterflow-elasticsearch.yaml
Install elasticsearch, kibana and Istio addons
cd $K8_ROOT/logging
./install.sh
set min_age in elasticsearch-ilm-script.sh and execute the same. min_age : is the min no. of days for which indices will be stored in elasticsearch.
cd $K8_ROOT/logging
./elasticsearch-ilm-script.sh
Mosip provides set of Kibana Dashboards for checking logs and throughputs .
Brief description of these dashboards are as follows:
01-logstash.ndjson contains the logstashIndex Pattern required by the rest of the dashboards.
contains a Search dashboard which shows only the error logs of the services, called MOSIP Error Logs dashboard.
contains a Search dashboard which show all logs of a particular service, called MOSIP Service Logs dashboard.
contains dashboards which show insights into MOSIP processes, like the number of UINs generated (total and per hr), the number of Biometric deduplications processed, number of packets uploaded etc, called MOSIP Insight dashboard.
contains dashboards which show how quickly different MOSIP Services are responding to different APIs, over time, called Response Time dashboard.
This version of the guide is currently under revision and enhancement. For any support, please reach out to us on community.mosip.io.
Overview
MOSIP modules are deployed in the form of microservices in a Kubernetes cluster.
is used as a trust network extension to access the admin, control, and observation pane
It is also used for on-the-field registrations.
Deployment Repos
: contains scripts to install and configure Kubernetes cluster with required monitoring, logging and alerting tools.
: contains deployment scripts to run charts in defined sequence.
: contains all the configuration files required by the MOSIP modules.
Pre-requisites:
Hardware Requirements
VM’s required have any Operating System and can be selected as per convenience.
In this installation guide, we are referring to Ubuntu OS throughout.
Network Requirements
All the VM's should be able to communicate with each other.
Need stable Intra network connectivity between these VM's.
All the VM's should have stable internet connectivity for docker image download (in case of local setup ensure to have a locally accesible docker registry).
DNS Requirements
Note:
Only proceed to DNS mapping after the ingressgateways are installed and the load balancer is already configured.
The above table is just a placeholder for hostnames, the actual name itself varies from organisation to organisation.
Certificate requirements
As only secured https connections are allowed via nginx server, you will need the below mentioned valid ssl certificates:
One valid wildcard ssl certificate related to domain used for accesing Observation cluster which will be created using ACM (Amazon certificate manager). In above e.g. *. is the similiar example domain.
One valid wildcard ssl certificate related to domain used for accessing MOSIP cluster which will be created using ACM (Amazon certificate manager). In above e.g. *. is the similiar example domain.
Prerequisite for complete deployment in Personal Computer
client version 1.23.6
client version 3.8.2 and add below repos as well :
Installation
A Wireguard bastion host (Wireguard server) provides secure private channel to access MOSIP cluster. The host restricts public access, and enables access to only those clients who have their public key listed in Wireguard server. Wireguard listens on UDP port 51820.
Architecture diagram
Setup Wirguard VM and wireguard bastion server:
Create a Wireguard server VM in aws console with above mentioned Hardware and Network requirements.
Edit the security group and add the following inbound rules in aws console
type ‘custom TCP', port range ‘51820’ and source '0.0.0.0/0’
Setup Wireguard Client in your PC
Install in your PC.
Assign wireguard.conf:
SSH to the wireguard server VM.
Observation K8s Cluster setup and configuration
Observation K8s Cluster setup
Setup rancher cluster,
cd $K8_ROOT/rancher/aws
Copy rancher.cluster.config.sample to rancher.cluster.config
Observation K8s Cluster’s Ingress and Storage class setup
Once the rancher cluster is ready we need ingress and storage class to be set for other applications to be installed.
: used for ingress in rancher cluster.
The above will automatically spawn an .
Check the following on AWS console:
Domain name
Create the following domain names:
Rancher: rancher.xyz.net
Keycloak: keycloak.xyz.net
Point the above to internal ip address of the NLB. This assumes that you have a has been installed. On AWS this is done on Route 53 console.
Rancher K8s Cluster Apps Installation
Rancher UI : Rancher provides full CRUD capability of creating and managing kubernetes cluster.
Install rancher using Helm, update hostname in rancher-values.yaml and run the following command to install.
MOSIP K8s Cluster setup
Setup mosip cluster
cd $K8_ROOT/mosip/aws
Copy cluster.config.sample to mosip.cluster.config
Import Mosip Cluster into Rancher UI
Login as admin in Rancher console
Select Import Existing for cluster addition.
Select the Generic as cluster type to add.
Wait for few seconds after executing the command for the cluster to get verified.
Your cluster is now added to the rancher management server.
MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup
Global configmap: Global configmap contains list of necesary details to be used throughout the namespaces of the cluster for common details.
cd $K8_ROOT/mosip
Copy global_configmap.yaml.sample
Monitoring Module deployment
Prometheus and Grafana and Alertmanager tools are used for cluster monitoring.
Select 'Monitoring' App from Rancher console -> Apps & Marketplaces.
In Helm options, open the YAML file and disable Nginx Ingress.
Alerting Setup
Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel.
Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager.
Create .
After setting slack incoming webhook update slack_api_url and slack_channel_name in alertmanager.yml
Logging Module Setup and Installation
Mosip uses Rancher Fluentd and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard.
Install Rancher FluentD system : for screpping logs outs of all the microservices from Mosip k8 cluster.
Install Logging from Apps and marketplace within the Rancher UI.
Select Chart Version 100.1.3+up3.17.7 from Rancher console -> Apps & Marketplaces.
Mosip External Dependencies setup
External Dependencies: are set of external requirements needed for funtioning of MOSIP’s core services like DB, object store, hsm etc.
Check detailed installation instruction of all the .
MOSIP Modules Deployment
Now that all the Kubernetes cluster and external dependencies are already installed, you can continue with MOSIP service deployment.
Check the detailed MOSIP Modules Deployment steps.
API Testrig
MOSIP’s successfull deployment can be verified by comparing the results of api testrig with testrig benchmark.
When prompted input the hour of the day to execute the api-testrig.
On-Premises Deployment
1. Wireguard
Wireguard bastian server provides secure private channel to access MOSIP cluster.
Bastian server restricts public access, and enables access to only those clients who have their public key listed in Wireguard server.
MOSIP uses AWS load balancers for:
SSL termination
Reverse Proxy
CDN/Cache management
Loadbalancing
Kubernetes cluster is administered using the Rancher and EKS
In V3, we have two Kubernetes clusters:
Observation Cluster - This cluster is a part of the observation plane and it helps in administrative tasks. By design, this is kept independent of the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or its services should be internal and should never be exposed to the external world.
in this cluster is used for cluster user access management.
It is recommended to configure log monitoring and network monitoring in this cluster.
In case you have a internal container registry, then it should run here.
MOSIP Cluster - This cluster runs all the MOSIP components and certain third party components to secure the cluster, API’s and Data.
mosip-helm : contains packaged helm charts for all the MOSIP modules.
8 GB
1
(ensure to setup active-passive)
2
Rancher Cluster nodes (EKS managed)
2
8 GB
32 GB
2
2
3
Mosip Cluster nodes (EKS managed)
8
32 GB
64 GB
6
6
During the process, we will be creating two loadbalancers as mentioned in the first table below:
Server Interface requirement as mentioned in the second table:
Load balancer of Observation cluster
Administrative IAM tool (keycloak). This is for the kubernetes administration.
3
Private Load balancer of MOSIP cluster
Index page for links to different dashboards of Mosip env. (This is just for reference, Please do not expose this page in a real production or uat environment)
4
Private Load balancer of MOSIP cluster
Internal API’s are exposed through this domain. They are accessible privately over wireguard channel
5
Public Load balancer of MOSIP cluster
All the API’s that are publically usable are exposed using this domain.
6
Public Load balancer of MOSIP cluster
Domain name for Mosip’s pre-registration portal. The portal is accessible publicly.
7
Private Load balancer of MOSIP cluster
Provides direct access to activemq dashboard. Its limited and can be used only over wireguard
8
Private Load balancer of MOSIP cluster
Optional instalation. Used to access kibana dashboard over wireguard
9
Private Load balancer of MOSIP cluster
Regclient can be downloaded from this domain. It should be used over wireguard.
10
Private Load balancer of MOSIP cluster
Mosip’s admin portal is exposed using this domain. This is an internal domain and is restricted to access over wireguard
11
Private Load balancer of MOSIP cluster
Optional- This domain is used to access the object server. Based on the object server that you choose map this domain accordingly. In our reference implementation Minio is used and this domain lets you access Minio’s Console over wireguard
12
Private Load balancer of MOSIP cluster
Kafka UI is installed as part of the Mosip’s default installation. We can access kafka ui over wireguard. Mostly used for administrative needs.
13
Private Load balancer of MOSIP cluster
Mosip uses an Openid connect server to limit and manage access across all the services. The default installation comes with Keycloak. This domain is used to access the keycloak server over wireguard
14
Private Load balancer of MOSIP cluster
This domain points to the postgres server. You can connect to postgres via port forwarding over wireguard
15
Public Load balancer of MOSIP cluster
Mosip’s partner management portal is used to manage partners accessing partner management portal over wireguard
Save ~/.kube/config file with another name. (IMPORTANT. As in this process your existing~/.kube/config file will be overridden).
Save .pem file from AWS console and store it in ~/.ssh/ folder. (Generate a new one if you do not have this key file).
Create a directory as mosip in your PC and
clone k8’s infra repo with tag : 1.2.0.1-B2 inside mosip directory.
git clone https://github.com/mosip/k8s-infra -b v1.2.0.1-B2
clone mosip-infra with tag : 1.2.0.1-B2 inside mosip directory
git clone https://github.com/mosip/mosip-infra -b v1.2.0.1-B2
Set below mentioned variables in bashrc
source .bashrcNote:
Above mentioned environment variables will be used throughout the installation to move between one directory to other to run install scripts.
type ‘custom UDP', port range ‘51820’ and source '0.0.0.0/0’
Install docker in the Wireguard machine as given here.
Setup Wireguard server
SSH to wireguard VM
Create directory for storing wireguard config files.
mkdir -p wireguard/config
Install and start wireguard server using docker as given below:
Note:
* Increase the no of peers above in case needed more than 30 wireguard client confs. (-e PEERS=30)
* Change the directory to be mounted to wireguard docker in case needed.
All your wireguard confs will be generated in the mounted directory. (-v /home/ubuntu/wireguard/config:/config)
cd /home/ubuntu/wireguard/config
assign one of the PR for yourself and use the same from the PC to connect to the server.
create assigned.txt file to assign the keep track of peer files allocated and update everytime some peer is allocated to someone.
peer1 : peername
peer2 : xyz
Use ls cmd to see the list of peers.
get inside your selected peer directory, and add mentioned changes in peer.conf:
cd peer1
nano peer1.conf
add peer.conf in your PC’s /etc/wireguard directory as wg0.conf.
start the wireguard client and check the status:
sudo systemctl start wg-quick@wg0
sudo systemctl status wg-quick@wg0
Once Connected to wireguard you should be now able to login using private ip’s.
.
Review and update the below mentioned parameters of rancher.cluster.config carefully.
name
region
version: “1.24“
instance related details
instanceName
instanceType
update the details of the subnets to be used from vpc
Install
eksctl create cluster -f rancher.cluster.config
Wait for the cluster creation to complete, generally it takes around 30 minutes to create or update cluster.
Once EKS K8 cluster is ready below mentioned output will be displayed in the console screen.
EKS cluster "my-cluster" in "region-code" region is ready
The config file for the new cluster will be created on ~/.kube/config
Make sure to backup and store the ~/.kube/config with new name. e.g. ~/.kube/obs-cluster.config.
Change file permission using below command:
chmod 400 ~/.kube/obs-cluster.config
Set the KUBECONFIG properly so that you can access the cluster.
export KUBECONFIG=~/.kube/obs-cluster.config
Test cluster access:
kubect get nodes
Command will result in details of the nodes of the rancher cluster.
An NLB has been created. You may also see the DNS of NLB with
Note, the target group name of listner 80. Set target group of 443 to target group of 80. Basically, we want TLS termination at the LB and it must forward HTTP traffic (not HTTPS) to port 80 of ingress controller. So
Input of LB: HTTPS
Output of LB: HTTP --> port 80 of ingress nginx controller
Enable "Proxy Protocol v2" in the target group settings
Make sure all subnets are selected in LB -->Description-->Edit subnets.
Check health check of target groups.
Remove listner 80 from LB as we will receive traffic only on 443.
Storage class setup:
Default storage class on EKS is gp2 . GP2 by default is in Delete mode which means if PVC is deleted, the underlying storage PV is also deleted.
To enable volume expansion for the existing gp2 storage class, modify the YAML configuration by adding allowVolumeExpansion: true to the gp2 storage class configuration.
kubectl edit sc gp2 : to edit the yaml configuration.
Create storage class gp2-retain by running sc.yaml for PV in Retain mode. Set the storage class as gp2-retain in case you want to retain PV.
we need the EBS driver for our storage class to work, follow the steps to setup EBS driver.
Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.
Keycloak : Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.
cd $K8_ROOT/rancher/keycloak
./install.sh <iam.host.name>
keycloak_client.json: Used to create SAML client on Keycloak for Rancher integration.
Keycloak - Rancher Integration
Login as admin user in Keycloak and make sure an email id, and first name field is populated for admin user. This is important for Rancher authentication as given below.
Enable authentication with Keycloak using the steps given .
In Keycloak add another Mapper for the rancher client (in Master realm) with following fields:
Protocol: saml
Name: username
Specify the following mappings in Rancher's Authentication Keycloak form:
Display Name Field: givenName
User Name Field: email
RBAC :
For users in Keycloak assign roles in Rancher - cluster and project roles. Under default project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects).
If you want to create custom roles, you can follow the steps given here.
Add a member to cluster/project in Rancher:
Give member name exactly as username in Keycloak
Assign appropriate role like Cluster Owner, Cluster Viewer etc.
Certificates expiry
In case you see certificate expiry message while adding users, on local cluster run these commands:
Review and update the below mentioned parameters of cluster.config.sample carefully.
name
region
version: “1.24“
instance related details
instanceName
instanceType
update the details of the subnets to be used from vpc
Install
eksctl create cluster -f mosip.cluster.config
Wait for the cluster creation to complete, generally it takes around 30 minutes to create or update cluster.
Once EKS K8 cluster is ready below mentioned output will be displayed in the console screen.
EKS cluster "my-cluster" in "region-code" region is ready
The config file for the new cluster will be created on ~/.kube/config
Make sure to backup and store the ~/.kube/config with new name. e.g. ~/.kube/mosip-cluster.config.
Change file permission using below command:
chmod 400 ~/.kube/mosip-cluster.config
Set the KUBECONFIG properly so that you can access the cluster.
export KUBECONFIG=~/.kube/mosip-cluster.config
Test cluster access:
kubect get nodes
Command will result in details of the nodes of the MOSIP cluster.
Fill the Cluster Name field with unique cluster name and select Create.
You will get the kubectl commands to be executed in the kubernetes cluster. Copy the command and execute from your PC. (make sure your kube-config file is correctly set to Mosip cluster)
to
global_configmap.yaml.
Update the domain names in global_configmap.yaml and run.
kubectl apply -f global_configmap.yaml
Storage class setup:
Default storage class on EKS is gp2 . GP2 by default is in Delete mode which means if PVC is deleted, the underlying storage PV is also deleted.
To enable volume expansion for the existing gp2 storage class, modify the YAML configuration by adding allowVolumeExpansion: true to the gp2 storage class configuration.
kubectl edit sc gp2 : to edit the yaml configuration.
Create storage class gp2-retain by running sc.yaml for PV in Retain mode. Set the storage class as gp2-retain in case you want to retain PV.
we need the EBS driver for our storage class to work, follow the steps to setup EBS driver.
also we need EFS CSI driver for the regproc services, because EBS driver only supports RWO but we need RWX, follow these to setup EFS CSI driver.
Ingress and load balancer (LB) :
Ingress is not installed by default on EKS. We use Istio ingress gateway controller to allow traffic in the cluster. Two channels are created - public and internal. See architecture.
You may view them on AWS console in Loadbalancer section.
TLS termination is supposed to be on LB. So all our traffic coming to ingress controller shall be HTTP.
Obtain AWS TLS certificate as given
Add the certificates and 443 access to the LB listener.
Update listener TCP->443 to TLS->443 and point to the certificate of domain name that belongs to your cluster.
Forward TLS->443 listner traffic to target group that corresponds to listener on port 80 of respective Loadbalancers. This is because after TLS termination the protocol is HTTP so we must point LB to HTTP port of ingress controller.
Update health check ports of LB target groups to node port corresponding to port 15021. You can see the node ports with
kubectl -n istio-system get svc
Enable Proxy Protocol v2 on target groups.
Make sure all subnets are included in Availability Zones for the LB. Description --> Availability Zones --> Edit Subnets
Make sure to delete the listeners for port 80 and 15021 from each of the loadbalancers as we restrict unsecured port 80 access over http.
DNS mapping:
Initially all the services will be accesible only over the internal channel.
Point all your domain names to internal LoadBalancers DNS/IP intially till testing is done.
On AWS this may be done on Route 53 console.
After Go live decision enable .
Check Overall if nginx and istio wiring is set correctly
Install httpbin: This utility docker returns http headers received inside the cluster. You may use it for general debugging - to check ingress, headers etc.
cd $K8_ROOT/utils/httpbin
./install.sh
To see what's reaching httpbin (example, replace with your domain name):
Install Default alerts along some of the defined custom alerts:
cd $K8_ROOT/monitoring/alerting/
./install.sh
Alerting is Installed.
Configure Rancher FluentD
Create clusteroutput :
kubectl apply -f clusteroutput-elasticsearch.yaml
start clusterFlow
kubectl apply -f clusterflow-elasticsearch.yaml
Install elasticsearch, kibana and Istio addons
cd $K8_ROOT/logging
./intall.sh
set min_age in elasticsearch-ilm-script.sh and execute the same. min_age : is the min no. of days for which indices will be stored in elasticsearch.
cd $K8_ROOT/logging
./elasticsearch-ilm-script.sh
Mosip provides set of Kibana Dashboards for checking logs and throughputs .
Brief description of these dashboards are as follows:
01-logstash.ndjson contains the logstashIndex Pattern required by the rest of the dashboards.
contains a Search dashboard which shows only the error logs of the services, called MOSIP Error Logs dashboard.
contains a Search dashboard which show all logs of a particular service, called MOSIP Service Logs dashboard.
contains dashboards which show insights into MOSIP processes, like the number of UINs generated (total and per hr), the number of Biometric deduplications processed, number of packets uploaded etc, called MOSIP Insight dashboard.
contains dashboards which show how quickly different MOSIP Services are responding to different APIs, over time, called Response Time dashboard.
Open kibana dashboard from: https://kibana.sandbox.xyz.net.
Kibana --> Menu (on top left) --> Dashboard --> Select the dashboard.
Daily api testrig cron jon will be executed at the very opted hour of the day.
Purpose
vCPU’s
RAM
Storage (HDD)
no. of VM’s
HA
1
Wireguard Bastion Host
2
Loadbalancer
Purpose
Private loadbalancer Observation cluster
This will be used to access Rancher dashboard and keycloak of observation cluster.
Note: access to this will be restricted only with wireguard key holders.
Public loadbalancer MOSIP cluster
This will be used to access below mentioned services:
Pre-registration
Esignet
IDA
Partner management service api’s
Mimoto
Mosip file server
Resident
Private loadbalancer MOSIP cluster
This will be used to access all the services deployed as part of the setup inclusing external as well as all the MOSIP services.
Note: access to this will be restricted only with wireguard key holders.
Purpose VM
Network Interfaces
1
Wireguard Bastion Host
One Private interface: that is on the same network as all the rest of nodes. (Eg: inside local NAT Network )
One public interface: Either has a direct public IP, or a firewall NAT (global address) rule that forwards traffic on 51820/udp port to this interface ip.
Create copy of hosts.ini.sample as hosts.ini and update the required details for wireguard VM
cp hosts.ini.sample hosts.ini
Note :
Remove [Cluster] complete section from copied hosts.ini file.
Add below mentioned details:
Execute ports.yml to enable ports on VM level using ufw:ansible-playbook -i hosts.ini ports.yaml
Note:
Permission of the pem files to access nodes should have 400 permission. sudo chmod 400 ~/.ssh/privkey.pem
These ports are only needed to be opened for sharing packets over UDP.
Take necessary measure on firewall level so that the Wireguard server can be reachable on 51820/udp.
Install docker
execute docker.yml to install docker and add user to docker group:
ansible-playbook -i hosts.ini docker.yaml
Setup Wireguard server
SSH to wireguard VM
ssh -i <path to .pem> ubuntu@<Wireguard server public ip>
Create directory for storing wireguard config files.mkdir -p wireguard/config
Install and start wireguard server using docker as given below:
Note:
Increase the no. of peers above in case more than 30 wireguard client confs (-e PEERS=30) are needed.
Change the directory to be mounted to wireguard docker as per need. All your wireguard confs will be generated in the mounted directory (-v /home/ubuntu/wireguard/config:/config).
cd $K8_ROOT/rancher/on-prem/nginx
sudo ./install.sh
Provide below mentioned inputs as and when promted
Rancher nginx ip : internal ip of the nginx server VM.
SSL cert path : path of the ssl certificate to be used for ssl termination.
SSL key path : path of the ssl key to be used for ssl termination.
Cluster node ip's : ip’s of the rancher cluster node
Post installation check:
sudo systemctl status nginx
Steps to Uninstall nginx (in case required)sudo apt purge nginx nginx-common
DNS mapping: Once nginx server is installed sucessfully, create DNS mapping for rancher cluster related domains as mentioned in DNS requirement section. (rancher.org.net, keycloak.org.net)
5. Observation K8's Cluster Apps Installation
5.a. Rancher UI
Rancher provides full CRUD capability of creating and managing kubernetes cluster.
Install rancher using Helm, update hostname in rancher-values.yaml and run the following command to install.
Connect to the Wireguard. (in case using Windows via WSL, make sute to connect to wireguard server from windows instead of WSL).
Open Rancher page.
Get Bootstrap password using
Note: Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.
5.b. Keycloak
Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.
cd $K8_ROOT/rancher/keycloak
./install.sh <iam.host.name>
Post installation access the keycloak using iam.mosip.net and get the credentials as per the post installation steps defined .
5.c. Keycloak - Rancher UI Integration
Login as admin user in Keycloak and make sure email and firstName fields are populated for the admin user. These are required for Rancher authentication to work properly.
In Keycloak (in the master realm), create a new client with the following values:
For users in Keycloak assign roles in Rancher - cluster and project roles. Under default project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects).
If you want to create custom roles, you can follow the steps given here.
Add a member to cluster/project in Rancher:
Navigate to RBAC cluster members
Add member name exactly as username in Keycloak
Assign appropriate role like Cluster Owner, Cluster Viewer etc.
You may create new role with fine grained acccess control.
Add group to to cluster/project in Rancher:
Navigate to RBAC cluster members
Click on Add and select a group from the displayed drop-down.
Creating a Keycloak group involves the following steps:
Go to the "Groups" section in Keycloak and create groups with default roles.
Navigate to the "Users" section in Keycloak, select a user, and then go to the "Groups" tab. From the list of groups, add the user to the required group.
6. MOSIP K8s Cluster setup
Pre-requisites:
Install all the required tools mentioned in Pre-requisites for PC.
kubectl
helm
ansible
rke (version 1.3.10)
Setup MOSIP K8 Cluster node VM’s as per '.
Run env-check-setup.yaml to check if cluster nodes are fine and dont have known issues in it.
cd $K8_ROOT/rancher/on-prem
Create copy of hosts.ini.sample as hosts.ini
Setup passwordless ssh into the cluster nodes via pem keys. (Ignore if VM’s are accessible via pem’s).
Generate keys on your PC
ssh-keygen -t rsa
Open ports and Install docker on MOSIP K8 Cluster node VM’s.
cd $K8_ROOT/mosip/on-prem
create copy of hosts.ini.sample as hosts.ini and update the required details for wireguard VM.
Creating RKE Cluster Configuration file
rke config
Command will prompt for nodal details related to cluster, provide inputs w.r.t below mentioned points:
As result of rke config command cluster.ymlfile will be generated inside same directory, update the below mentioned fields:
nano cluster.yml
Remove the default Ingress install
Setup up the cluster:
Once cluster.yml is ready, you can bring up the kubernetes cluster using simple command.
This command assumes the cluster.yml file is in the same directory as where you are running the command.
Test cluster access:
kubectl get nodes
Command will result in details of the nodes of the rancher cluster.
Save Your files
Save a copy of the following files in a secure location, they are needed to maintain, troubleshoot and upgrade your cluster.:
cluster.yml: The RKE cluster configuration file.
7. MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup
7.a. Global configmap:
Global configmap contains the list of neccesary details to be used throughout the namespaces of the cluster for common details.
cd $K8_ROOT/mosip
Copy global_configmap.yaml.sample to global_configmap.yaml.
It is a service mesh for the MOSIP K8 cluster which provides transparent layers on top of existing microservices along with powerful features enabling a uniform and more efficient way to secure, connect, and monitor services.
cd $K8_ROOT/mosip/on-prem/istio
./install.sh
This will bring up all the Istio components and the Ingress Gateways.
Check Ingress Gateway services:
kubectl get svc -n istio-system
Note: Response should contain service names as mentioned below.
7.c. Storage classes
Multiple storage classes options are available for onprem K8's cluster. In this reference deployment will continue to use NFS as a storage class.
Move to nfs directory in your personel computer.
cd $K8_ROOT/mosip/nfs
Create a copy of hosts.ini.sample as hosts.ini.
cp hosts.ini.sample hosts.ini
Update the NFS machine details in hosts.ini file.
Note :
Add below mentioned details:
ansible_host : internal IP of NFS server. eg. 10.12.23.21
ansible_user : user to be used for installation, in this ref-impl we use Ubuntu user.
ansible_ssh_private_key_file : path to pem key for ssh to wireguard server. eg. ~/.ssh/wireguard-ssh.pem
Make sure Kubeconfig file is set correctly to point to required mosip cluster.
Note:
Output should show the cluster name to confirm you are pointing to right kubernetes cluster.
If not pinting to right K8 cluster change the kubeconfig to connect to right K8 cluster.
Enable firewall with required ports:
SSH to the nfs node:
Clone k8s-infra repo in nginx VM:
Move to the nfs directory:
Execute script to install nfs server:
Note:
Script prompts for below mentioned user inputs:
envName: env name eg. dev/qa/uat...
Switch to your personel computer and excute below mentioned commands:
Note:
Script prompts for:
NFS Server: NFS server ip for persistence.
Post installation check:
Check status of NFS Client Provisioner.
Check status of nfs-client storage class.
8. Import MOSIP Cluster into Rancher UI
Login as admin in Rancher console
Select Import Existing for cluster addition.
Select Generic as cluster type to add.
Fill the Cluster Name field with unique cluster name and select Create.
You will get the kubectl commands to be executed in the kubernetes cluster. Copy the command and execute from your PC (make sure your kube-config file is correctly set to MOSIP cluster).
Wait for few seconds after executing the command for the cluster to get verified.
Your cluster is now added to the rancher management server.
9. MOSIP K8 cluster Nginx server setup
9.a. SSL certificates creation
For Nginx server setup, we need ssl certificate, add the same into Nginx server.
Incase valid ssl certificate is not there generate one using letsencrypt:
SSH into the nginx server
Install Pre-requisites:
Generate wildcard SSL certificates for your domain name.
Open kibana dashboard from https://kibana.sandbox.xyz.net.
Kibana --> Menu (on top left) --> Dashboard --> Select the dashboard.
13. MOSIP External Dependencies setup
External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, HSM etc.
Click here to check the detailed installation instructions of all the external components.
Note:
Connect to mosip_pms DB in postgres and execute the query to change valid_to_date for mpolicy-default-mobile in pms.auth_policy table.
Open the terminal.
Use the psql command to connect to the PostgreSQL server. The general syntax is:
: The server address (e.g., localhost or an IP address).
Assuming other details remain same like port and user.
14. MOSIP Modules Deployment
Now that all the Kubernetes cluster and external dependencies are already installed, will continue with MOSIP service deployment.
Note:
In case of failure on execution of install-all.sh follow MOSIP Modules Deployment installation steps from the failure point.
Installation for config-server and admin service is facing delay in this version hence need to execute below mentioned commands respectively for config-server and admin-service once the script fails.
For config server:
Update failureThreshold for startupProbe to 60.
For admin service:
Update failureThreshold for startupProbe to 60.
Once admin service is up and running re-execute install.sh
15. API Testrig
MOSIP’s successfull deployment can be verified by comparing the results of api testrig with testrig benchmark.
curl https://api-internal.sandbox.xyz.net/httpbin/get?show_env=true
Once public access is enabled also check this:
curl https://api.sandbox.xyz.net/httpbin/get?show_env=true
ansible_host : public IP of Wireguard Bastion server. eg. 100.10.20.56
ansible_user : user to be used for installation. In this ref-impl we use Ubuntu user.
ansible_ssh_private_key_file : path to pem key for ssh to wireguard server. eg. ~/.ssh/wireguard-ssh.pem
get inside your selected peer directory, and add mentioned changes in peer.conf:
cd peer1
nano peer1.conf
Delete the DNS IP.
Update the allowed IP's to subnets CIDR ip . e.g. 10.10.20.0/23
Note:
CIDR Range will be shared by the Infra provider.
Make sure all the nodes are covered in the provided CIDR range. (nginx server, K8 cluster nodes for observation as well as mosip).
Share the updated peer.conf with respective peer to connect to wireguard server from Personel PC.
ansible_ssh_private_key_file : path to pem key for ssh to wireguard server. eg. ~/.ssh/nodes-ssh.pem
Make sure all the nodes are covered in the provided CIDR range. (nginx server, K8 cluster nodes for observation as well as mosip).
Execute ports.yml to enable ports on VM level using ufw:ansible-playbook -i hosts.ini ports.yaml
Disable swap in cluster nodes. (Ignore if swap is already disabled)
ansible-playbook -i hosts.ini swap.yaml
Caution: Always verify swap status with swapon --show before running the playbook to avoid unnecessary operations.
execute docker.yml to install docker and add user to docker group:ansible-playbook -i hosts.ini docker.yaml
SSH Private Key Path :
Number of Hosts:
SSH Address of host :
SSH User of host :
Make all the nodes Worker host by default.
To create an HA cluster, specify more than one host with role Control Plane and etcd host.
Network Plugin Type : Continue with canal as default network plugin.
For rest of other configurations, opt the required or default value.
Remove the default Ingress install
Update the name of the kubernetes cluster in cluster.yml.
cluster_name: observation-cluster
rke up
The last line should read Finished building Kubernetes cluster successfully to indicate that your cluster is ready to use.
Note:
Incase rke up command is unsucessfull due to any underline error then we need to fix the same by checking the logs.
Once the issue is fixed we need to remove the cluster using rke remove.
Once rke remove is executed sucessfully need to delete cluster related incomplete configuration using :
As part of the Kubernetes creation process, a kubeconfig file has been created and written at kube_config_cluster.yml, which can be used to start interacting with your Kubernetes cluster.
Copy the kubeconfig files
To access the cluster using kubeconfig file use any one of the below method:
