Version: 1.2.0.1-B3 (Latest stable release)

• Release date: 14th April 2023

• Release notes

• On-Prem Installation Guidelines

Version: 1.2.0.1-B2

• Release date: 8-Jan 2023

Version: 1.2.0.1-B1

• Release date: 14-Oct 2022

External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, and HSM etc.

List of external dependencies:

• Postgres: Relational Database system used for storing data in MOSIP.

• IAM: IAM tool is for authentication and authorization. Reference implementation here uses Keycloak for the same purpose.

• : Hardware Security Module (HSM) stores the cryptographic keys used in MOSIP. Reference implementation is provided as SoftHSM here.

• : MOSIP uses S3 API compliant object store for storing biometric and other data. Reference implementation here uses .

• Anti-virus: Used for document scanning and packets scanning throughout MOSIP modules. Reference implementation uses dockerised version of .

• Queuing tool: Tool used for queuing messages to external MOSIP components. Reference implementation used .

• Event Publisher/ streamer: MOSIP uses Kafka for publishing events to it's internal as well as external partners modules.

• BioSDK: Biometric SDK for quality check and authentication purpose using biometrics.

• : Performs the de-duplication of a resident's biometric data.

• Message Gateway: This is for notifying residents about different OTPs and other information.

Installation

Postgres

• Install Postgres

• Initialize Postgres DB

Opt for yes and enter Y.

Keycloak

• Install Keycloak

• Initialize Keycloak

Setup SoftHSM

Setup Object store

MinIO installation

S3 Credentials setup

• Opt 1 for MinIO

• Opt 2 for S3 (incase you are not going with MinIO installation and want s3 to be installed)

  • Enter the prompted details.

ClamAV setup

ActiveMQ setup

Kafka setup

BioSDK Server setup

Reference implementation of Biometric SDK server will be installed separately in MOSIP service installation section as the same is dependent on artifactory which is a MOSIP component.

ABIS

ABIS is needed to be up and running outside the MOSIP cluster and should be able to connect to the activeMQ. For testing purpose, MOSIP has provided a mock stimulator for the same named as mock-abis which will be deployed as part of the MOSIP services installation.

MSG Gateway

• MOSIP provides mock smtp server which will be installed as part of default installation, opt for Y.

Docker Secrets

Incase the images are getting pulled from private repositories.

Captcha

To setup the captcha for pre-reg and resident domains.

Landing page setup

Authdemo Service

• The Authdemo service is utilized to execute the IDA APIs that are employed by API-testrig and DSLrig.

• The purpose of the Authdemo Service is to showcase the functionality of authentication.

• It can be considered as a simplified iteration of an authentication service, serving as a mock or prototype for the purpose of testing.

• When prompted, input the NFS host, it's pem-key, ssh login user of NFS server.

• Install script will create the NFS directories i.e., /srv/nfs/mosip/packetcreator-authdemo-authcerts to store the certificates generated by Authdemo service.

These certificates will be used by API-testrig, orchestrator and packetcreator.

API-testrig

API-testRig tests the working of APIs of the MOSIP modules.

MOSIP’s successful deployment can be verified by comparing the results of API-testrig with the testrig benchmark.

• When prompted, input the hour of the day to execute the API-testrig.

• Daily API-testrig cron job will be executed at the very opted hour of the day.

• The reports will move to the object store ( i.e., s3/minio) under automationtests bucket.

Packetcreator

Packetcreator will create packets for DSL orchestrator.

Note: It is recommended to deploy the packetcreator on a separate server/ cluster from where the other DSL orchestrators can access this service.

• When prompted, input the NFS host, it's pem-key, ssh login user of NFS server.

• Install script will create two NFS directories i.e., /srv/nfs/mosip/packetcreator_data, /srv/nfs/mosip/packetcreator-authdemo-authcerts.

• Packetcreator_data contains biometric data which is used to create packets.

DSLrig/ DSLOrchestrator

• DSLrig will test end-to-end functional flows involving multiple MOSIP modules.

• The Orchestrator utilizes the Packet Creator to generate packets according to the defined specifications. It then communicates with the Authdemo Service making REST API calls to perform authentication-related actions or retrieve the necessary information.

• When prompted, input the NFS host, it's pem-key, ssh login user of NFS server.

• Install script will create NFS directories, i.e., /srv/nfs/mosip/dsl-scenarios/sandbox.xyz.net to store the DSL scenario sheet.

• The Default template for DSL scenario sheet can be accessible from .

Below is the sequence of installation of MOSIP modules and the sequence must be followed to resolve all interdependencies.

1. Config Server Secrets

2. Config Server

3. Artifactory

Installation

• Conf secrets

• Config Server

• Artifactory

• Keymanager

• WebSub

• Mock-SMTP

• Kernel

• Masterdata-loader

• Mock-biosdk

• Packetmanager

• Datashare

• Pre-reg

• Idrepo

• Partner Management Services

• Mock ABIS

• Mock-mv

• Registration Processor

• Admin

• ID Authentication

• Print

• Partner Onboarder

• MOSIP File Server

• Resident services

• Registration Client

Overview

• MOSIP modules are deployed in the form of microservices in kubernetes cluster.

• Wireguard is used as a trust network extension to access the admin, control, and observation pane.

• It is also used for the on-the-field registrations.

• MOSIP uses server for:

  • SSL termination

  • Reverse Proxy

  • CDN/Cache management
• Kubernetes cluster is administered using the and tools.

• In V3, we have two Kubernetes clusters:

Observation cluster - This cluster is a part of the observation plane and it helps in administrative tasks. By design, this is kept independent of the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or it's services should be internal and should never be exposed to the external world.

• is used for managing the MOSIP cluster.

• in this cluster is used for cluster user access management.

• It is recommended to configure log monitoring and network monitoring in this cluster.

• In case you have an internal container registry, then it should run here.

MOSIP cluster - This cluster runs all the MOSIP components and certain third party components to secure the cluster, API’s and data.

Architecture

Deployment repos

• : contains the scripts to install and configure Kubernetes cluster with required monitoring, logging and alerting tools.

• : contains the deployment scripts to run charts in defined sequence.

• : contains all the configuration files required by the MOSIP modules.

Pre-requisites

Hardware requirements

• VM’s required can be with any OS as per convenience.

• Here, we are referting to Ubuntu OS throughout this installation guide.

Network requirements

• All the VM's should be able to communicate with each other.

• Need stable Intra network connectivity between these VM's.

• All the VM's should have stable internet connectivity for docker image download (in case of local setup ensure to have a locally accesible docker registry).

• Server Interface requirement as mentioned in below table:

DNS requirements

Certificate requirements

As only secured https connections are allowed via nginx server will need below mentioned valid ssl certificates:

• One valid wildcard ssl certificate related to domain used for accessing Observation cluster, this needs to be stored inside the nginx server VM for Observation cluster. In above e.g.: *.org.net is the similiar example domain.

• One valid wildcard ssl certificate related to domain used for accesing Mosip cluster, this needs to be stored inside the nginx server VM for mosip cluster. In above e.g.: *.sandbox.xyz.net is the similiar example domain.

Tools to be installed in Personel Computers for complete deployment

• - any client version above 1.19

• - any client version above 3.0.0 and add below repos as well:

• : version: 1.15.0

• : version:

• [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html: version > 2.12.4

Installation

Wireguard

A Wireguard bastion host (Wireguard server) provides secure private channel to access MOSIP cluster. The host restricts public access, and enables access to only those clients who have their public key listed in Wireguard server. Wireguard listens on UDP port51820.

Setup Wireguard VM and wireguard bastion server

• Create a Wireguard server VM with above mentioned Hardware and Network requirements.

• Open ports and Install docker on Wireguard VM.

  • cd $K8_ROOT/wireguard/

Note:

• Permission of the pem files to access nodes should have 400 permission. sudo chmod 400 ~/.ssh/privkey.pem

• These ports are only needed to be opened for sharing packets over UDP.

• Take necessary measure on firewall level so that the Wireguard server can be reachable on 51820/udp.

• Setup Wireguard server

  • SSH to wireguard VM

  • Create directory for storing wireguard config files. mkdir -p wireguard/config

Note:

• Increase the no. of peers above in case more than 30 wireguard client confs (-e PEERS=30) are needed.

• Change the directory to be mounted to wireguard docker as per need. All your wireguard confs will be generated in the mounted directory (-v /home/ubuntu/wireguard/config:/config).

Setup Wireguard Client in your PC

• Install Wireguard client in your PC.

• Assign wireguard.conf:

  • SSH to the wireguard server VM.

• Once connected to wireguard, you should be now able to login using private IP’s.

Observation K8s Cluster setup and configuration

Observation K8s Cluster setup

• Install all the required tools mentioned in pre-requisites for PC.

  • kubectl

  • helm

  • ansible

Note:

• Make sure the permission for privkey.pem for ssh is set to 400.

• Run env-check.yaml to check if cluster nodes are fine and do not have known issues in it.

  • cd $K8_ROOT/rancher/on-prem

  • create copy of hosts.ini.sample as hosts.ini and update the required details for Observation k8 cluster nodes.

Observation K8s Cluster Ingress and Storage class setup

Once the rancher cluster is ready, we need ingress and storage class to be set for other applications to be installed.

• : used for ingress in rancher cluster.

this will install ingress in ingress-nginx namespace of rancher cluster.

Storage classes

The following storage classes can be used:

• : If you are already using VMware virtual machines, you can proceed with the vSphere storage class.

• .

• [ceph-csi](TODO Implementation in progress)

Pre-requisites:

Install Longhorn via helm

• ./install.sh

• Note: Values of below mentioned parametrs are set as by default Longhorn installation script:

  • PV replica count is set to 1. Set the replicas for the storage class appropriately.

Setting up nginx server for Observation K8s Cluster

• For Nginx server setup we need ssl certificate, add the same into Nginx server.

• Incase valid ssl certificate is not there generate one using letsencrypt:

  • SSH into the nginx server

Observation K8's Cluster Apps Installation

Rancher UI

• Rancher provides full CRUD capability of creating and managing kubernetes cluster.

• Install rancher using Helm, update hostname in rancher-values.yaml and run the following command to install.

• Login:

  • Open page.

  • Get Bootstrap password using

  Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.

Keycloak

• is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.

keycloak_client.json: Used to create SAML client on Keycloak for Rancher integration.

Keycloak - Rancher UI Integration

• Login as admin user in Keycloak and make sure an email id, and first name field is populated for admin user. This is important for Rancher authentication as given below.

• Enable authentication with Keycloak using the steps given .

• In Keycloak add another Mapper for the rancher client (in Master realm) with following fields:

RBAC for Rancher using Keycloak

• For users in Keycloak assign roles in Rancher - cluster and project roles. Under default project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects).

• If you want to create custom roles, you can follow the steps given .

• Add a member to cluster/project in Rancher:

Certificates expiry

In case you see certificate expiry message while adding users, on local cluster run these commands:

MOSIP K8s Cluster setup

• Pre-requisites:

• Install all the required tools mentioned in Pre-requisites for PC.

  • kubectl

  • helm

• ansible

• rke (version 1.3.10)

• Setup MOSIP K8 Cluster node VM’s as per the hardware and network requirements as mentioned above.

• Run env-check.yaml

Alternatively

• Test cluster access:

  • kubect get nodes

  • Command will result in details of the nodes of the rancher cluster.

MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup

Global configmap: Global configmap contains the list of neccesary details to be used throughout the namespaces of the cluster for common details.

• cd $K8_ROOT/mosip

• Copy global_configmap.yaml.sample to global_configmap.yaml.

• Update the domain names in global_configmap.yaml and run.

Storage classes

The following storage classes can be used:

• : If you are already using VMware virtual machines, you can proceed with the vSphere storage class.

• .

• [ceph-csi](TODO Implementation in progress)

For time being, we are considering Longhorn as a storage class.

• Storage class setup: Longhorn creates a storage class in the cluster for creating pv (persistence volume) and pvc (persistence volume claim).

  • Pre-requisites:

  • Install Longhorn via helm

Import MOSIP Cluster into Rancher UI

• Login as admin in Rancher console

• Select Import Existing for cluster addition.

• Select Generic as cluster type to add.

• Wait for few seconds after executing the command for the cluster to get verified.

• Your cluster is now added to the rancher management server.

MOSIP K8 cluster Nginx server setup

• For Nginx server setup, we need ssl certificate, add the same into Nginx server.

• Incase valid ssl certificate is not there generate one using letsencrypt:

  • SSH into the nginx server

Monitoring module deployment

• Prometheus and Grafana and Alertmanager tools are used for cluster monitoring.

• Select 'Monitoring' App from Rancher console -> Apps & Marketplaces.

• In Helm options, open the YAML file and disable Nginx Ingress.

Alerting setup

Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel.

• Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager.

• Create .

• After setting slack incoming webhook update slack_api_url and slack_channel_name in alertmanager.yml.

• Install Default alerts along some of the defined custom alerts:

• Alerting is installed.

Logging module setup and installation

MOSIP uses and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard.

• Install Rancher FluentD system : for screpping logs outs of all the microservices from MOSIP k8 cluster.

  • Install Logging from Apps and marketplace within the Rancher UI.

  • Select Chart Version 100.1.3+up3.17.7 from Rancher console -> Apps & Marketplaces.

Open kibana dashboard from https://kibana.sandbox.xyz.net.

Kibana --> Menu (on top left) --> Dashboard --> Select the dashboard.

MOSIP External Dependencies setup

External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, HSM etc.

Click to check the detailed installation instructions of all the external components.

MOSIP Modules Deployment

Now that all the Kubernetes cluster and external dependencies are already installed, will continue with MOSIP service deployment.

Check detailed installation steps.

Overview

• MOSIP modules are deployed in the form of microservices in kubernetes cluster.

• Wireguard is used as a trust network extension to access the admin, control, and observation pane.

• It is also used for the on-the-field registrations.

• MOSIP uses server for:

  • SSL termination

  • Reverse Proxy

  • CDN/Cache management
• Kubernetes cluster is administered using the and tools.

• In V3, we have two Kubernetes clusters:

Observation cluster - This cluster is a part of the observation plane and it helps in administrative tasks. By design, this is kept independent of the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or it's services should be internal and should never be exposed to the external world.

• is used for managing the MOSIP cluster.

• in this cluster is used for cluster user access management.

• It is recommended to configure log monitoring and network monitoring in this cluster.

• In case you have an internal container registry, then it should run here.

MOSIP cluster - This cluster runs all the MOSIP components and certain third party components to secure the cluster, API’s and data.

Architecture diagram

Deployment repos

• : contains the scripts to install and configure Kubernetes cluster with required monitoring, logging and alerting tools.

• : contains the deployment scripts to run charts in defined sequence.

• : contains all the configuration files required by the MOSIP modules.

Pre-requisites

Hardware requirements

• VM’s required can be with any OS as per convenience.

• Here, we are referring to Ubuntu OS throughout this installation guide.

Network requirements

• All the VM's should be able to communicate with each other.

• Need stable Intra network connectivity between these VM's.

• All the VM's should have stable internet connectivity for docker image download (in case of local setup ensure to have a locally accessible docker registry).

• Server Interface requirement as mentioned in below table:

DNS requirements

Certificate requirements

As only secured https connections are allowed via nginx server will need below mentioned valid ssl certificates:

• One valid wildcard ssl certificate related to domain used for accessing Observation cluster, this needs to be stored inside the nginx server VM for Observation cluster. In above e.g.: *.org.net is the similar example domain.

• One valid wildcard ssl certificate related to domain used for accesing Mosip cluster, this needs to be stored inside the nginx server VM for mosip cluster. In above e.g.: *.sandbox.xyz.net is the similar example domain.

Tools to be installed in Personel Computers for complete deployment

• - any client version above 1.19

• - any client version above 3.0.0 and add below repos as well:

• : version: 1.15.0

• : version:

• [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html: version > 2.12.4

Installation

Wireguard

A Wireguard bastion host (Wireguard server) provides secure private channel to access MOSIP cluster. The host restricts public access, and enables access to only those clients who have their public key listed in Wireguard server. Wireguard listens on UDP port51820.

Setup Wireguard VM and wireguard bastion server

• Create a Wireguard server VM with above mentioned Hardware and Network requirements.

• Open ports and Install docker on Wireguard VM.

  • cd $K8_ROOT/wireguard/

Note:

• Permission of the pem files to access nodes should have 400 permission. sudo chmod 400 ~/.ssh/privkey.pem

• These ports are only needed to be opened for sharing packets over UDP.

• Take necessary measure on firewall level so that the Wireguard server can be reachable on 51820/udp.

• Setup Wireguard server

  • SSH to wireguard VM

  • Create directory for storing wireguard config files. mkdir -p wireguard/config

Note:

• Increase the no. of peers above in case more than 30 wireguard client confs (-e PEERS=30) are needed.

• Change the directory to be mounted to wireguard docker as per need. All your wireguard confs will be generated in the mounted directory (-v /home/ubuntu/wireguard/config:/config).

Setup Wireguard Client in your PC

• Install Wireguard client in your PC.

• Assign wireguard.conf:

  • SSH to the wireguard server VM.

• Once connected to wireguard, you should be now able to login using private IP’s.

Observation K8s Cluster setup and configuration

Observation K8s Cluster setup

• Install all the required tools mentioned in pre-requisites for PC.

  • kubectl

  • helm

  • ansible

Note:

• Make sure the permission for privkey.pem for ssh is set to 400.

• Run env-check.yaml to check if cluster nodes are fine and do not have known issues in it.

  • cd $K8_ROOT/rancher/on-prem

  • create copy of hosts.ini.sample as hosts.ini and update the required details for Observation k8 cluster nodes.

Observation K8s Cluster Ingress and Storage class setup

Once the rancher cluster is ready, we need ingress and storage class to be set for other applications to be installed.

• : used for ingress in rancher cluster.

this will install ingress in ingress-nginx namespace of rancher cluster.

• Storage class setup: creates a storage class in the cluster for creating pv (persistence volume) and pvc (persistence volume claim).

Pre-requisites:

Install Longhorn via helm

• ./install.sh

• Note: Values of below mentioned parameters are set as by default Longhorn installation script:

  • PV replica count is set to 1. Set the replicas for the storage class appropriately.

Setting up nginx server for Observation K8s Cluster

• For Nginx server setup we need ssl certificate, add the same into Nginx server.

• SSL certificates can be generated in multiple ways. Either via lets encrypt if you have public DNS or via openssl certs when you don't have Public DNS.

  • Letsencrypt: Generate wildcard ssl certificate having 3 months validity when you have public DNS system using below steps.

• Post installation check:

  • sudo systemctl status nginx

• Steps to Uninstall nginx (in case required) sudo apt purge nginx nginx-common

Observation K8's Cluster Apps Installation

Rancher UI: Rancher provides full CRUD capability of creating and managing kubernetes cluster.

• Install rancher using Helm, update hostname, & add privateCA to true in rancher-values.yaml, and run the following command to install.

• Login:

  • Open page.

  • Get Bootstrap password using

  Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.

keycloak_client.json: Used to create SAML client on Keycloak for Rancher integration.

Keycloak - Rancher UI Integration

• Login as admin user in Keycloak and make sure an email id, and first name field is populated for admin user. This is important for Rancher authentication as given below.

• Enable authentication with Keycloak using the steps given .

• In Keycloak add another Mapper for the rancher client (in Master realm) with following fields:

RBAC :

• For users in Keycloak assign roles in Rancher - cluster and project roles. Under default project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects).

• If you want to create custom roles, you can follow the steps given .

• Add a member to cluster/project in Rancher:

Certificates expiry

In case you see certificate expiry message while adding users, on local cluster run these commands:

MOSIP K8s Cluster setup

• Pre-requisites:

• Install all the required tools mentioned in Pre-requisites for PC.

  • kubectl

  • helm

• ansible

• rke (version 1.3.10)

• Setup MOSIP K8 Cluster node VM’s as per the hardware and network requirements as mentioned above.

• Run env-check.yaml

Alternatively

• Test cluster access:

  • kubect get nodes

  • Command will result in details of the nodes of the rancher cluster.

MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup

Global configmap: Global configmap contains the list of neccesary details to be used throughout the namespaces of the cluster for common details.

• cd $K8_ROOT/mosip

• Copy global_configmap.yaml.sample to global_configmap.yaml.

• Update the domain names in global_configmap.yaml and run.

Import MOSIP Cluster into Rancher UI

• Login as admin in Rancher console

• Select Import Existing for cluster addition.

• Select Generic as cluster type to add.

• Wait for few seconds after executing the command for the cluster to get verified.

• Your cluster is now added to the rancher management server.

MOSIP K8 cluster Nginx server setup

• For Nginx server setup, we need ssl certificate, add the same into Nginx server.

• SSL certificates can be generated in multiple ways. Either via lets encrypt if you have public DNS or via openssl certs when you don't have Public DNS.

  • Letsencrypt: Generate wildcard ssl certificate having 3 months validity when you have public DNS system using below steps.

Monitoring module deployment

• Prometheus and Grafana and Alertmanager tools are used for cluster monitoring.

• Select 'Monitoring' App from Rancher console -> Apps & Marketplaces.

• In Helm options, open the YAML file and disable Nginx Ingress.

Alerting setup

Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel.

• Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager.

• Create .

• After setting slack incoming webhook update slack_api_url and slack_channel_name in alertmanager.yml.

• Install Default alerts along some of the defined custom alerts:

• Alerting is installed.

Logging module setup and installation

MOSIP uses and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard.

• Install Rancher FluentD system : for scraping logs outs of all the microservices from MOSIP k8 cluster.

  • Install Logging from Apps and marketplace within the Rancher UI.

  • Select Chart Version 100.1.3+up3.17.7 from Rancher console -> Apps & Marketplaces.

Open kibana dashboard from https://kibana.sandbox.xyz.net.

Kibana --> Menu (on top left) --> Dashboard --> Select the dashboard.

MOSIP External Dependencies setup

External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, HSM etc.

Click to check the detailed installation instructions of all the external components.

Configurational change in case using Openssl wildcard ssl certificate. (Only advised in development env, not recommended for Production setup)

• Add/ Update the below property in application-default.properties and comment on the below property in the *-default.properties file in the config repo.

• Add/ Update the below property in the esignet-default.properties file in the config repo.

MOSIP Modules Deployment

• Now that all the Kubernetes cluster and external dependencies are already installed, will continue with MOSIP service deployment.

• While installing a few modules, installation script prompts to check if you have public domain and valid SSL certificates on the server. Opt option n as we are using self-signed certificates. For example:

• Start installing mosip modules:

Check detailed installation steps.