1 of 14

V3 installation

Note:

This version of the guide is currently under revision and enhancement. For any support, please reach out to us on .

Version: 1.2.0.1 (Latest stable release)

Release date: 19th March 2024
Release notes
On-Prem Installation Guidelines

Version: 1.2.0.1-B4

Release date: 12th January 2024

Version: 1.2.0.1-B3

Release date: 14th April 2023

Version: 1.2.0.1-B2

Release date: 8th Jan 2023

Version: 1.2.0.1-B1

Release date: 14th Oct 2022

1.2.0.3

Overview

MOSIP Rapid Deployment Infrastructure is a unified, cloud-native deployment platform that transforms complex, time-consuming MOSIP deployments into automated, 3-step processes that completes fast and takes much lesser time than earlier deployments. It replaces fragmented, manual infrastructure setup with a single repository containing everything needed for production-ready MOSIP deployment.

With MOSIP RDI (Rapid Deployment Infrastructure), you can deploy complete MOSIP identity platforms with enterprise-grade security, monitoring, and automation through a streamlined CI/CD approach that reduces deployment complexity by 90%.

With the current release of 0.2.0 Infra (Rapid Deployment Infrastructure), eSignet 1.4.2 now integrates as a dedicated deployment, unlocking execution of all the features' that requires eSignet (OTP, biometric auth, KYC exchange) and which were getting skipped earlier. Countries adopting eSignet can now validate complete end-to-end workflows before production. Countries using alternative authentication methods can skip the esignet-dsf.yaml phase entirely — core MOSIP services remain unaffected.

How does the rapid deployment model improve upon previous approaches in terms of DevOps modernization?

MOSIP RDI introduces several key improvements over the earlier approach:

Infrastructure as Code (IaC): MOSIP RDI (Rapid Deployment Infrastructure) leverages tools like Terraform for declarative, version-controlled infrastructure provisioning. This ensures repeatable, automated deployments and minimizes manual intervention.
Streamlined CI/CD Pipelines: Integrated GitHub Actions automate testing, validation, and deployment workflows, enabling faster and more reliable releases with minimal manual steps.
Automated Configuration and Deployment: Ansible and Cloud-Init are used for server configuration and initialization, ensuring consistent environments across deployments.

Where and how do I start?

To begin with MOSIP RDI:

Deployment Architecture

Key Resources

You can refer to the following key resources to get started with MOSIP Rapid Deployment. Currently the links take you to repo READMEs as this is the first beta release of MOSIP RDI (Rapid Deployment Infrastructure) and we are working to bring all the relevant content to this published documentation site as well.

Comprehensive Documentation: Access the official for step-by-step guides, architecture details, and deployment instructions.
Deployment Repository: Clone the deployment codebase from the . This repository contains all necessary scripts, modules, and configuration files for a production-ready setup.
eSignet Deployment Guide: Refer to the for configuration details and required secrets.

For additional guidance, refer to the section at the end of this document for links to checklists, setup guides, and troubleshooting resources.

How it works (High-level Overview)

MOSIP RDI originally follows a 3-step deployment model. Starting with this release, the eSignet authentication stack is added as a dedicated fourth deployment phase that runs after MOSIP core services are up.

Complete Deployment Flow

Documentation

1.2.0.2 Overview and Architecture

Overview

MOSIP modules are deployed as microservices in a Kubernetes cluster.
Wireguard is used as a trust network extension to access the admin, control, and observation panes.
In case of on-field registration, WireGuard is extended to the registration client machines as one of the sources of trust.
MOSIP uses server for:
- SSL termination
- Reverse Proxy
- CDN/Cache management
Kubernetes cluster is administered using the and tools.
We have two Kubernetes clusters:

Observation cluster - This cluster is part of the observation plane and assists with administrative tasks. By design, this is kept independent from the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or its services should be internal and should never be exposed to the external world.

is used for managing the MOSIP cluster.
in this cluster is used to manage user access and rights for the observation plane.
It is recommended to configure log monitoring and network monitoring in this cluster.

MOSIP cluster - This cluster runs all the MOSIP components and certain third-party components like the kafka, keycloak etc.

Architecture diagram

Deployment repos

These are the repos that we would use for installing and configuring the MOSIP deployment. For detailed steps, please follow the guide. This section is for reference.

: contains the scripts to install and configure a Kubernetes cluster with required monitoring, logging, and alerting tools.
: contains the deployment scripts to run charts in a defined sequence.
: contains all the configuration files required by the MOSIP modules.

Pre-Requisites

Hardware requirements

VM’s required can be with any OS as per convenience.
Here, we are referting to Ubuntu OS (22.0.4) throughout this installation guide.
Ensure to create Ubuntu user with its directory /home/ubuntu as the same user is referenced throughout the document. Else, change the user related commands as and when needed throughout the installation.

Sl no.

Purpose

vCPU's

RAM

Storage (HDD)

no. ofVM's

Network requirements

All the VM's should be able to communicate with each other.
Need stable Intra network connectivity between these VM's.
All the VM's should have stable internet connectivity for docker image download (in case of local setup ensure to have a locally accessible docker registry).

Sl no.

Purpose

Network Interfaces

DNS requirements

Incase not using public DNS system please ignore this as the same mapping needs to be performed at node level and is mentioned in the deployment document.

Domain Name

Mapping details

Purpose

Certificate requirements

As only secured https connections are allowed via nginx server will need below mentioned valid ssl certificates:

One valid wildcard ssl certificate related to domain used for accessing Observation cluster, this needs to be stored inside the nginx server VM for Observation cluster. In above e.g.: *.org.net is the similar example domain.
One valid wildcard ssl certificate related to domain used for accessing Mosip cluster, this needs to be stored inside the nginx server VM for mosip cluster. In above e.g.: *.sandbox.xyz.net is the similar example domain.

Personal computer requirements

Note:
In case unfamiliar with below mentioned tools and trying deployment from personel computer having Windows OS, suggestion is to use WSL for easy proceedings.
Please ensure the following tools with the specified versions are installed on your personal computer.

- version 2.25.1 or higher.
- version 2.12.4 or higher
- any client version above 3.0.0 and add below repos as well:

: version: 1.15.0
: version:
- version > 2.12.4

Testrig

Authdemo Service

The Authdemo service is utilized to execute the IDA APIs that are employed by API-testrig and DSLrig.

MOSIP External Dependencies

External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, and HSM etc.

List of external dependencies:

: Relational Database system used for storing data in MOSIP.
IAM: IAM tool is for authentication and authorization. Reference implementation here uses for the same purpose.

MOSIP Modules Deployment

Below is the sequence of installation of MOSIP modules and the sequence must be followed to resolve all interdependencies.

Check the current OpenSSL version:
```
openssl version
```
If it's not OpenSSL 1.1.1f, remove the existing OpenSSL
```
sudo apt remove openssl 
```
Manually install OpenSSL 1.1.1f by following this guide:
Note :
- While following the guide:
  Replace all instances of openssl-1.1.1s with openssl-1.1.1f.
  Skip the commands that create backups of binaries:
  After completing the installation, verify the OpenSSL version again:

AWS Deployment

1. Wireguard

Wireguard Bastion server provides secure private channel to access MOSIP cluster.
Bastian server restricts public access, and enables access to only those clients who have their public key listed with it.
Bastian server listens on UDP port 51820.

Setup Wirguard VM

Create a Wireguard server VM from aws console with mentioned ''.
Edit the security group and add the following inbound rules in aws console
- type ‘custom TCP', port range ‘51820’ and source '0.0.0.0/0’

Setup Wireguard Bastian server

SSH to wireguard VM
Create directory for storing wireguard config files.mkdir -p wireguard/config
Install and start wireguard server using docker as given below:
Note: * Increase the no of peers above in case needed more than 30 wireguard client confs. (-e PEERS=30

Setup Wireguard Client in your PC

Install in your PC.
Assign wireguard.conf:
- SSH to the wireguard server VM.

2. Observation K8s Cluster setup

Setup Observation cluster:
- cd $K8_ROOT/rancher/aws
- Copy rancher.cluster.config.sample to rancher.cluster.config.

3. Observation K8s Cluster’s Ingress and Storage class setup

Once the rancher cluster is ready we need ingress and storage class to be set for other applications to be installed.

3.a. : used for ingress in rancher cluster.

The above will automatically spawn an .
Check the following on AWS console:
- An NLB has been created. You may also see the DNS of NLB with

4. Domain name mapping

Create the following domain names:
- Rancher: rancher.xyz.net
- Keycloak: keycloak.xyz.net

5. Rancher K8s Cluster Apps Installation

5.a. Rancher UI : Rancher provides full CRUD capability of creating and managing kubernetes cluster.
- Install rancher using Helm, update hostname in rancher-values.yaml and run the following command to install.

6. MOSIP K8s Cluster setup

Setup mosip cluster
- cd $K8_ROOT/mosip/aws
- Copy cluster.config.sample to mosip.cluster.config.

7. Import Mosip Cluster into Rancher UI

Login as admin in Rancher console
Select Import Existing for cluster addition.
Select the Generic as cluster type to add.

Wait for few seconds after executing the command for the cluster to get verified.
Your cluster is now added to the rancher management server.

8. MOSIP K8 Cluster Global configmap, Storage Class, Ingress and Loadbalancer setup

8.a. Global configmap: Global configmap contains list of necesary details to be used throughout the namespaces of the cluster for common details.
- cd $K8_ROOT/mosip
- Copy global_configmap.yaml.sample to global_configmap.yaml.

9. Monitoring Module deployment

Prometheus and Grafana and Alertmanager tools are used for cluster monitoring.
Note : This is optional for sandbox to be deployed in case monitoring is necesary and for production they can always go with alternate tools option.
- Select 'Monitoring' App from Rancher console -> Apps & Marketplaces.

10. Alerting Setup

Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel.
Note : This is optional for sandbox to be deployed in case alerting is necesary and for production they can always go with alternate tools option.
- Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager.

11. Logging Module Setup and Installation

Mosip uses Rancher Fluentd and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard.
Note : This is optional for sandbox to be deployed in case logging is necesary and for production they can always go with alternate tools option.
Install Rancher FluentD system : for screpping logs outs of all the microservices from Mosip k8 cluster.

12. Mosip External Dependencies setup

External Dependencies: are set of external requirements needed for funtioning of MOSIP’s core services like DB, object store, hsm etc.
Check detailed installation instruction of all the .

13. MOSIP Modules Deployment

Now that all the Kubernetes cluster and external dependencies are already installed, you can continue with MOSIP service deployment.
Check the detailed MOSIP Modules Deployment steps.

14. API Testrig

MOSIP’s successfull deployment can be verified by comparing the results of api testrig with testrig benchmark.
- - When prompted input the hour of the day to execute the api-testrig.
  - Daily api testrig cron jon will be executed at the very opted hour of the day.

-v /home/ubuntu/wireguard/config:/config

cd /home/ubuntu/wireguard/config

assign one of the PR for yourself and use the same from the PC to connect to the server.

create assigned.txt file to assign the keep track of peer files allocated and update everytime some peer is allocated to someone.
- peer1 : peername peer2 : xyz
Use ls cmd to see the list of peers.
get inside your selected peer directory, and add mentioned changes in peer.conf:
- cd peer1
- nano peer1.conf

Open Rancher page https://rancher.org.net.

Get Bootstrap password using

kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}'

Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.

5.b. Keycloak : Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.

cd $K8_ROOT/rancher/keycloak
./install.sh <iam.host.name>

keycloak_client.json: Used to create SAML client on Keycloak for Rancher integration.

8.d. Load Balancers setup for istio-ingress

The above istio installation will automatically spawn an Internal AWS Network Load Balancer (L4).
These may be also seen with

kubectl -n istio-system get svc

You may view them on AWS console in Loadbalancer section.
TLS termination is supposed to be on LB. So all our traffic coming to ingress controller shall be HTTP.
Obtain AWS TLS certificate as given
Add the certificates and 443 access to the LB listener.
Update listener TCP->443 to TLS->443 and point to the certificate of domain name that belongs to your cluster.
Forward TLS->443 listner traffic to target group that corresponds to listener on port 80 of respective Loadbalancers. This is because after TLS termination the protocol is HTTP so we must point LB to HTTP port of ingress controller.
Update health check ports of LB target groups to node port corresponding to port 15021. You can see the node ports with
Enable Proxy Protocol v2 on target groups.
Make sure all subnets are included in Availability Zones for the LB. Description --> Availability Zones --> Edit Subnets
Make sure to delete the listeners for port 80 and 15021 from each of the loadbalancers as we restrict unsecured port 80 access over http.

8.f. Check Overall if nginx and istio wiring is set correctly

Install httpbin: This utility docker returns http headers received inside the cluster. You may use it for general debugging - to check ingress, headers etc.
- cd $K8_ROOT/utils/httpbin ./install.sh
- To see what's reaching httpbin (example, replace with your domain name):
  curl https://api-internal.sandbox.xyz.net/httpbin/get?show_env=true Once public access is enabled also check this: curl https://api.sandbox.xyz.net/httpbin/get?show_env=true

sudo docker run -d \
   --name=wireguard \
   --cap-add=NET_ADMIN \
   --cap-add=SYS_MODULE \
   -e PUID=1000 \
   -e PGID=1000 \
   -e TZ=Asia/Calcutta\
   -e PEERS=30 \
   -p 51820:51820/udp \
   -v /home/ubuntu/wireguard/config:/config \
   -v /lib/modules:/lib/modules \
   --sysctl="net.ipv4.conf.all.src_valid_mark=1" \
   --restart unless-stopped \
   ghcr.io/linuxserver/wireguard

cd $K8_ROOT/rancher/aws
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install \                               
  ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace  \
-f nginx.values.yaml

cd $K8_ROOT/rancher/rancher-ui
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update
helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --create-namespace \
  -f rancher-values.yaml

On-Premises Deployment

1. Wireguard

Wireguard bastian server provides secure private channel to access MOSIP cluster.
Bastian server restricts public access, and enables access to only those clients who have their public key listed in Wireguard server.
Bastion server listens on UDP port 51820.
In case you already have VPN configured to access nodes privately please skip Wireguard installation and continue to use the same VPN.

Setup Wireguard VM and wireguard bastion server

Create a Wireguard server VM with mentioned '.
Open required ports in the Bastian server VM.
- cd $K8_ROOT/wireguard/

Note:
Permission of the pem files to access nodes should have 400 permission. sudo chmod 400 ~/.ssh/privkey.pem
These ports are only needed to be opened for sharing packets over UDP.
Take necessary measure on firewall level so that the Wireguard server can be reachable on 51820/udp.

Install docker
- execute docker.yml to install docker and add user to docker group:
Setup Wireguard server

Note:
Increase the no. of peers above in case more than 30 wireguard client confs (-e PEERS=30) are needed.
Change the directory to be mounted to wireguard docker as per need. All your wireguard confs will be generated in the mounted directory (-v /home/ubuntu/wireguard/config:/config).

Setup Wireguard Client in your PC

Install Wireguard client in your PC using .
Assign wireguard.conf:
- SSH to the wireguard server VM.

2. Observation K8s Cluster setup and configuration

Install all the required tools mentioned in '' section.
Setup Observation Cluster node VM’s hardware and network configuration as per ().
Setup passwordless SSH into the cluster nodes via pem keys. (Ignore if VM’s are accessible via pem’s).

Note:
Make sure the permission for privkey.pem for ssh is set to 400.

Open ports and install docker on Observation K8 Cluster node VM’s.
- cd $K8_ROOT/rancher/on-prem
- Copy hosts.ini.sample to hosts.ini and update required details.

3. Observation K8s Cluster Ingress, Storageclass setup

Once the rancher cluster is ready, we need ingress and storage class to be set for other applications to be installed.

3.a.: used for ingress in rancher cluster.

Note:
This will install ingress in namespace ingress-nginx of Observation cluster.
Crosscheck using below mentioned command:
kubectl get all -n ingress-nginx

3.b. Storage classes

Multiple storage classes options are available for onprem K8's cluster. In this reference deployment will continue to use NFS as a storage class.

Move to nfs directory in your personel computer.
Create a copy of hosts.ini.sample as hosts.ini.
Update the NFS machine details in hosts.ini file.
Note :

4. Setting up nginx server for Observation K8s Cluster

4.a. SSL Certificate setup for TLS termination

For Nginx server setup we need ssl certificate, add the same into Nginx server.
Incase valid ssl certificate is not there generate one using letsencrypt:
- SSH into the nginx server

4.b. Install Nginx :

Move to nginx directory in your local:
cd $K8_ROOT/mosip/on-prem/nginx/
Open required ports :

Clone
Provide below mentioned inputs as and when promted
- Rancher nginx ip : internal ip of the nginx server VM.

5. Observation K8's Cluster Apps Installation

5.a. Rancher UI

Rancher provides full CRUD capability of creating and managing kubernetes cluster.

Install rancher using Helm, update hostname in rancher-values.yaml and run the following command to install.
Login:
- Connect to the Wireguard. (in case using Windows via WSL, make sute to connect to wireguard server from windows instead of WSL).

Note: Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.

5.b. Keycloak

Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.
Post installation access the keycloak using iam.mosip.net and get the credentials as per the post installation steps defined .

5.c. Keycloak - Rancher UI Integration

Login as admin user in Keycloak and make sure email and firstName fields are populated for the admin user. These are required for Rancher authentication to work properly.
In Keycloak (in the master realm), create a new client with the following values:

5.d. RBAC for Rancher using Keycloak

For users in Keycloak assign roles in Rancher - cluster and project roles. Under default project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects).
If you want to create custom roles, you can follow the steps given .
Add a member to cluster/project in Rancher:

6. MOSIP K8s Cluster setup

Pre-requisites:
- Install all the required tools mentioned in Pre-requisites for PC.
  - kubectl

7. MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup

7.a. Global configmap:

Global configmap contains the list of neccesary details to be used throughout the namespaces of the cluster for common details.
cd $K8_ROOT/mosip
Copy global_configmap.yaml.sample to global_configmap.yaml.

7.b. Ingress setup:

It is a service mesh for the MOSIP K8 cluster which provides transparent layers on top of existing microservices along with powerful features enabling a uniform and more efficient way to secure, connect, and monitor services.
- cd $K8_ROOT/mosip/on-prem/istio
- ./install.sh

7.c. Storage classes

Multiple storage classes options are available for onprem K8's cluster. In this reference deployment will continue to use NFS as a storage class.

Move to nfs directory in your personel computer.
Create a copy of hosts.ini.sample as hosts.ini.
Update the NFS machine details in hosts.ini file.
Note :

8. Import MOSIP Cluster into Rancher UI

Login as admin in Rancher console
Select Import Existing for cluster addition.
Select Generic as cluster type to add.

Wait for few seconds after executing the command for the cluster to get verified.
Your cluster is now added to the rancher management server.

9. MOSIP K8 cluster Nginx server setup

9.a. SSL certificates creation

For Nginx server setup, we need ssl certificate, add the same into Nginx server.
Incase valid ssl certificate is not there generate one using letsencrypt:
- SSH into the nginx server

9.b. Nginx server setup for MOSIP K8's cluster

Move to nginx directory in your local:
cd $K8_ROOT/mosip/on-prem/nginx/
Open required ports :

9.c. Check Overall nginx and istio wiring

Install httpbin: This utility docker returns http headers received inside the cluster.
httpbin can be used for general debugging - to check ingress, headers etc.
- To see what is reaching the httpbin (example, replace with your domain name):

10. Monitoring module deployment

Note :
Monitoring in the sandbox environment is optional and can be deployed if required.
For production environments, alternative monitoring tools can be used.
These steps can also be skipped in development environments if monitoring is not needed.

Prometheus and Grafana and Alertmanager tools are used for cluster monitoring.
Select 'Monitoring' App from Rancher console -> Apps & Marketplaces.
In Helm options, open the YAML file and disable Nginx Ingress.

11. Alerting setup

Note :
Alerting in the sandbox environment is optional and can be deployed if required.
For production environments, alternative alerting tools can be used.
These steps can also be skipped in development environments if alerting is not needed.

Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel.
Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager.
Create .

Install Default alerts along some of the defined custom alerts:

Alerting is installed.

12. Logging module setup and installation

Note :
Logging in the sandbox environment is optional and can be deployed if required.
For production environments, alternative logging tools can be used.
These steps can also be skipped in development environments if logging is not needed.

MOSIP uses and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard.

Install Rancher FluentD system : Required for screpping logs outs of all the microservices from MOSIP k8 cluster.
- Install Logging from Apps and marketplace within the Rancher UI.
- Select Chart Version 100.1.3+up3.17.7 from Rancher console -> Apps & Marketplaces.

Open kibana dashboard from https://kibana.sandbox.xyz.net.

Kibana --> Menu (on top left) --> Dashboard --> Select the dashboard.

13. MOSIP External Dependencies setup

External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, HSM etc.

Click to check the detailed installation instructions of all the external components.

Note:
Connect to mosip_pms DB in postgres and execute the query to change valid_to_date for mpolicy-default-mobile in pms.auth_policy table.
Open the terminal.

14. MOSIP Modules Deployment

Now that all the Kubernetes cluster and external dependencies are already installed, will continue with MOSIP service deployment.

Note:
In case of failure on execution of install-all.sh follow installation steps from the failure point.
Installation for config-server and admin service is facing delay in this version hence need to execute below mentioned commands respectively for config-server and admin-service once the script fails.

15. API Testrig

MOSIP’s successfull deployment can be verified by comparing the results of api testrig with testrig benchmark.

Navigate to the Infra Root Directory:
Clone the Functional Tests Repository
After cloned successfully try to install apitestrig
Make script executable

16. DSL Rig

Install Packetcreater
- Navigate to the packetcreator directory:
- Run the installation script:

AWS Installation Guidelines

Note:

This version of the guide is currently under revision and enhancement. For any support, please reach out to us on .

Overview

MOSIP modules are deployed in the form of microservices in a Kubernetes cluster.
is used as a trust network extension to access the admin, control, and observation pane
It is also used for on-the-field registrations.
MOSIP uses AWS load balancers for:
- SSL termination
- Reverse Proxy
Kubernetes cluster is administered using the and
In V3, we have two Kubernetes clusters:
- Observation Cluster - This cluster is a part of the observation plane and it helps in administrative tasks. By design, this is kept independent of the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or its services should be internal and should never be exposed to the external world.
  - is used for managing the Mosip cluster.

Deployment Repos

: contains scripts to install and configure Kubernetes cluster with required monitoring, logging and alerting tools.
: contains deployment scripts to run charts in defined sequence.
: contains all the configuration files required by the MOSIP modules.

Pre-requisites:

Hardware Requirements

VM’s required have any Operating System and can be selected as per convenience. In this installation guide, we are referring to Ubuntu OS throughout.

Network Requirements

All the VM's should be able to communicate with each other.
Need stable Intra network connectivity between these VM's.
All the VM's should have stable internet connectivity for docker image download (in case of local setup ensure to have a locally accesible docker registry).

DNS Requirements

Note:

Only proceed to DNS mapping after the ingressgateways are installed and the load balancer is already configured.
The above table is just a placeholder for hostnames, the actual name itself varies from organisation to organisation.

Certificate requirements

As only secured https connections are allowed via nginx server, you will need the below mentioned valid ssl certificates:

One valid wildcard ssl certificate related to domain used for accesing Observation cluster which will be created using ACM (Amazon certificate manager). In above e.g. *. is the similiar example domain.
One valid wildcard ssl certificate related to domain used for accessing MOSIP cluster which will be created using ACM (Amazon certificate manager). In above e.g. *. is the similiar example domain.

Prerequisite for complete deployment in Personal Computer

client version 1.23.6
client version 3.8.2 and add below repos as well :

Installation

A Wireguard bastion host (Wireguard server) provides secure private channel to access MOSIP cluster. The host restricts public access, and enables access to only those clients who have their public key listed in Wireguard server. Wireguard listens on UDP port 51820.

Architecture diagram

Setup Wirguard VM and wireguard bastion server:

Create a Wireguard server VM in aws console with above mentioned Hardware and Network requirements.
Edit the security group and add the following inbound rules in aws console
- type ‘custom TCP', port range ‘51820’ and source '0.0.0.0/0’

Setup Wireguard Client in your PC

Install in your PC.
Assign wireguard.conf:
- SSH to the wireguard server VM.

Observation K8s Cluster setup and configuration

Observation K8s Cluster setup

Setup rancher cluster,
- cd $K8_ROOT/rancher/aws
- Copy rancher.cluster.config.sample to rancher.cluster.config

Observation K8s Cluster’s Ingress and Storage class setup

Once the rancher cluster is ready we need ingress and storage class to be set for other applications to be installed.

: used for ingress in rancher cluster.
- The above will automatically spawn an .
Check the following on AWS console:

Domain name

Create the following domain names:

Rancher: rancher.xyz.net
Keycloak: keycloak.xyz.net

Point the above to internal ip address of the NLB. This assumes that you have a has been installed. On AWS this is done on Route 53 console.

Rancher K8s Cluster Apps Installation

Rancher UI : Rancher provides full CRUD capability of creating and managing kubernetes cluster.
- Install rancher using Helm, update hostname in rancher-values.yaml and run the following command to install.

MOSIP K8s Cluster setup

Setup mosip cluster
- cd $K8_ROOT/mosip/aws
- Copy cluster.config.sample to mosip.cluster.config

Import Mosip Cluster into Rancher UI

Login as admin in Rancher console
Select Import Existing for cluster addition.
Select the Generic as cluster type to add.

Wait for few seconds after executing the command for the cluster to get verified.
Your cluster is now added to the rancher management server.

MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup

Global configmap: Global configmap contains list of necesary details to be used throughout the namespaces of the cluster for common details.
- cd $K8_ROOT/mosip
- Copy global_configmap.yaml.sample

Monitoring Module deployment

Prometheus and Grafana and Alertmanager tools are used for cluster monitoring.

Select 'Monitoring' App from Rancher console -> Apps & Marketplaces.
In Helm options, open the YAML file and disable Nginx Ingress.

Alerting Setup

Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel.

Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager.
Create .
After setting slack incoming webhook update slack_api_url and slack_channel_name in alertmanager.yml

Logging Module Setup and Installation

Mosip uses Rancher Fluentd and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard.

Install Rancher FluentD system : for screpping logs outs of all the microservices from Mosip k8 cluster.
- Install Logging from Apps and marketplace within the Rancher UI.
- Select Chart Version 100.1.3+up3.17.7 from Rancher console -> Apps & Marketplaces.

Mosip External Dependencies setup

External Dependencies: are set of external requirements needed for funtioning of MOSIP’s core services like DB, object store, hsm etc.
Check detailed installation instruction of all the .

MOSIP Modules Deployment

Now that all the Kubernetes cluster and external dependencies are already installed, you can continue with MOSIP service deployment.
Check the detailed MOSIP Modules Deployment steps.

API Testrig

MOSIP’s successfull deployment can be verified by comparing the results of api testrig with testrig benchmark.
- - When prompted input the hour of the day to execute the api-testrig.

Create copy of hosts.ini.sample as hosts.ini and update the required details for wireguard VM

Install and start wireguard server using docker as given below:

sudo docker run -d \
--name=wireguard \
--cap-add=NET_ADMIN \
--cap-add=SYS_MODULE \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Calcutta \
-e PEERS=30 \
-p 51820:51820/udp \
-v /home/ubuntu/wireguard/config:/config \
-v /lib/modules:/lib/modules \
--sysctl="net.ipv4.conf.all.src_valid_mark=1" \
--restart unless-stopped \
ghcr.io/linuxserver/wireguard

cd /home/ubuntu/wireguard/config

assign one of the PR for yourself and use the same from the PC to connect to the server.

create assigned.txt file to assign the keep track of peer files allocated and update everytime some peer is allocated to someone.

peer1 :   peername
peer2 :   xyz

use ls cmd to see the list of peers.
get inside your selected peer directory, and add mentioned changes in peer.conf:
- cd peer1
- nano peer1.conf

Generate keys on your PCssh-keygen -t rsa

Make sure Kubeconfig file is set correctly to point to required Observation cluster.

kubectl config view

Note:

Output should show the cluster name to confirm you are pointing to right kubernetes cluster.
If not pinting to right K8 cluster change the kubeconfig to connect to right K8 cluster.

Execute script to install nfs server:

sudo ./install-nfs-server.sh

Note:

Script prompts for below mentioned user inputs:
.....
Please Enter Environment Name: <envName>
.....
.....
.....
[ Export the NFS Share Directory ] 
exporting *:/srv/nfs/mosip/<envName>
NFS Server Path: /srv/nfs/mosip/<envName>
envName: env name eg. dev/qa/uat...

Switch to your personel computer and excute below mentioned commands:

cd $K8_ROOT/mosip/nfs/

./install-nfs-client-provisioner.sh

Note:

Script prompts for:
NFS Server: NFS server ip for persistence.
NFS Path : NFS path for storing the persisted data. eg. /srv/nfs/mosip/

Post installation check:

Check status of NFS Client Provisioner.

kubectl -n nfs get deployment.apps/nfs-client-provisioner

Check status of nfs-client storage class.

 kubectl get storageclass
 NAME                 PROVISIONER                            RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
 longhorn (default)   driver.longhorn.io                     Delete          Immediate           true                   57d
 nfs-client           cluster.local/nfs-client-provisioner   Delete          Immediate           true                   40s

Set nfs-client storage class as default:

Use any editor to create new hosts.ini file:

Add below mentioned lines with updated details of nginx server to the hosts.ini and save.

[nginx]
node-nginx ansible_host=<internal ip> ansible_user=root ansible_ssh_private_key_file=<pvt .pem file>

As result of rke config command cluster.ymlfile will be generated inside same directory, update the below mentioned fields:

nano cluster.yml
Remove the default Ingress install
```
ingress:
provider: none
```
Update the name of the kubernetes cluster in cluster.yaml
For production deplopyments edit the cluster.yml, according to this .

Make sure Kubeconfig file is set correctly to point to required mosip cluster.

kubectl config view

Note:

Output should show the cluster name to confirm you are pointing to right kubernetes cluster.
If not pinting to right K8 cluster change the kubeconfig to connect to right K8 cluster.

Execute script to install nfs server:

sudo ./install-nfs-server.sh

Note:

Script prompts for below mentioned user inputs:
.....
Please Enter Environment Name: <envName>
.....
.....
.....
[ Export the NFS Share Directory ] 
exporting *:/srv/nfs/mosip/<envName>
NFS Server Path: /srv/nfs/mosip/<envName>
envName: env name eg. dev/qa/uat...

Switch to your personel computer and excute below mentioned commands:

cd $K8_ROOT/mosip/nfs/

./install-nfs-client-provisioner.sh

Note:

Script prompts for:
NFS Server: NFS server ip for persistence.
NFS Path : NFS path for storing the persisted data. eg. /srv/nfs/mosip/

Post installation check:

Check status of NFS Client Provisioner.

kubectl -n nfs get deployment.apps/nfs-client-provisioner

Check status of nfs-client storage class.

 kubectl get storageclass
 NAME                 PROVISIONER                            RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
 longhorn (default)   driver.longhorn.io                     Delete          Immediate           true                   57d
 nfs-client           cluster.local/nfs-client-provisioner   Delete          Immediate           true                   40s

Use any editor to create new hosts.ini file:

Add below mentioned lines with updated details of nginx server to the hosts.ini and save.

[nginx]
node-nginx ansible_host=<internal ip> ansible_user=root ansible_ssh_private_key_file=<pvt .pem file>

Click on Install.

After setting slack incoming webhook update slack_api_url and slack_channel_name in alertmanager.yml.

Use the psql command to connect to the PostgreSQL server. The general syntax is:

psql -h <host> -p 5432 -U postgres -d mosip_pms

: The server address (e.g., localhost or an IP address).
Assuming other details remain same like port and user.

UPDATE pms.auth_policy SET valid_to_date = valid_to_date + interval '1 year' WHERE name = 'mpolicy-default-mobile';

For admin service:

Update failureThreshold for startupProbe to 60.

kubectl -n admin edit deployment admin-service

Once admin service is up and running re-execute install.sh script after commenting out below mentioned commands:

Run the Installer Script

./install.sh

Note:
Script prompts for below mentioned inputs please provide as and when needed:
Enter the time (hr) to run the cronjob every day (0–23): Specify the hour you want the cronjob to run (e.g., 6 for 6 AM)
Do you have a public domain and valid SSL certificate? (Y/n):
Y – If you have a public domain and valid SSL certificate
n – If you do not have one (recommended only for development environments)
Retention days to remove old reports (Default: 3): Press Enter to accept the default or specify another value (e.g., 5).
Provide Slack Webhook URL to notify server issues on your Slack channel: (change the URL to your channel one)
Is the eSignet service deployed? (yes/no):
no – If eSignet is not deployed, related test cases will be skipped.
Is values.yaml for the apitestrig chart set correctly as part of the prerequisites? (Y/n): * Enter Y if this step is already completed.
Do you have S3 details for storing API-Testrig reports? (Y/n):
Enter Y to proceed with S3 configuration.
S3 Host: eg. http://minio.minio:9000

Run the installation script:

* ./install.sh

Note: Before running install.sh, please ensure the following is added in the Helm install command:
--set dslorchestrator.configmaps.dslorchestrator.servicesNotDeployed="esignet" \
After adding the above, the full Helm installation command should look like this:
helm -n $NS install dslorchestrator mosip/dslorchestrator \
 --set crontime="0 $time * * *" \
 --version $CHART_VERSION \
 --set dslorchestrator.configmaps.s3.s3-host='http://minio.minio:9000' \
 --set dslorchestrator.configmaps.s3.s3-user-key='admin' \
 --set dslorchestrator.configmaps.s3.s3-region='' \
 --set dslorchestrator.configmaps.db.db-server="$DB_HOST" \
 --set dslorchestrator.configmaps.db.db-su-user="postgres" \
 --set dslorchestrator.configmaps.db.db-port="5432" \
 --set dslorchestrator.configmaps.dslorchestrator.USER="$USER" \
 --set dslorchestrator.configmaps.dslorchestrator.ENDPOINT="https://$API_INTERNAL_HOST" \
 --set dslorchestrator.configmaps.dslorchestrator.packetUtilityBaseUrl="$packetUtilityBaseUrl" \
 --set persistence.nfs.server="$NFS_HOST" \
 --set persistence.nfs.path="/srv/nfs/mosip/dsl-scenarios/$ENV_NAME" \
 --set dslorchestrator.configmaps.dslorchestrator.reportExpirationInDays="$reportExpirationInDays" \
 --set dslorchestrator.configmaps.dslorchestrator.NS="$NS" \
 --set dslorchestrator.configmaps.dslorchestrator.servicesNotDeployed="esignet" \
 $ENABLE_INSECURE
When prompted, provide the following inputs:
NFS Host:
NFS PEM :
User for SSH Login:ubuntu
Cronjob Time (hour of the day, 0–23): (e.g., enter 6 for 6 AM)
Do you have a public domain and valid SSL certificate? (Y/n):
Y: If you have a valid public domain and SSL certificate
n: Use only in development environments
Packet Utility Base URL:
Retention Days to Remove Old Reports (default is 3): (Press Enter to accept default or provide a custom value)

cd $K8_ROOT/rancher/on-prem
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install \                                                                                                             
  ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --version 4.0.18 \
  --create-namespace  \
  -f ingress-nginx.values.yaml

cd $K8_ROOT/rancher/rancher-ui
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update
helm install rancher rancher-latest/rancher \
--version 2.6.9 \
--namespace cattle-system \
--create-namespace \
-f rancher-values.yaml

sudo apt update -y
sudo apt-get install software-properties-common -y
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update -y
sudo apt-get install python3.8 -y
sudo apt install letsencrypt -y
sudo apt install certbot python3-certbot-nginx -y

sudo apt update -y
sudo apt-get install software-properties-common -y
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update -y
sudo apt-get install python3.8 -y
sudo apt install letsencrypt -y
sudo apt install certbot python3-certbot-nginx -y

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
kubectl create ns cattle-monitoring-system
helm -n cattle-monitoring-system install rancher-monitoring-crd mosip/rancher-monitoring-crd

assign one of the PR for yourself and use the same from the PC to connect to the server.

create assigned.txt file to assign the keep track of peer files allocated and update everytime some peer is allocated to someone.
- peer1 : peername peer2 : xyz
Use ls cmd to see the list of peers.
get inside your selected peer directory, and add mentioned changes in peer.conf:
- cd peer1
- nano peer1.conf

Open Rancher page https://rancher.org.net.

Get Bootstrap password using

kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}'

Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.

Keycloak : Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.

cd $K8_ROOT/rancher/keycloak
./install.sh <iam.host.name>

keycloak_client.json: Used to create SAML client on Keycloak for Rancher integration.

global_configmap.yaml.

Load Balancers setup for istio-ingress.

The above istio installation will automatically spawn an Internal AWS Network Load Balancer (L4).
These may be also seen with

kubectl -n istio-system get svc

You may view them on AWS console in Loadbalancer section.
TLS termination is supposed to be on LB. So all our traffic coming to ingress controller shall be HTTP.
Obtain AWS TLS certificate as given
Add the certificates and 443 access to the LB listener.
Update listener TCP->443 to TLS->443 and point to the certificate of domain name that belongs to your cluster.
Forward TLS->443 listner traffic to target group that corresponds to listener on port 80 of respective Loadbalancers. This is because after TLS termination the protocol is HTTP so we must point LB to HTTP port of ingress controller.
Update health check ports of LB target groups to node port corresponding to port 15021. You can see the node ports with

kubectl -n istio-system get svc

Enable Proxy Protocol v2 on target groups.
Make sure all subnets are included in Availability Zones for the LB. Description --> Availability Zones --> Edit Subnets
Make sure to delete the listeners for port 80 and 15021 from each of the loadbalancers as we restrict unsecured port 80 access over http.

Check Overall if nginx and istio wiring is set correctly

Install httpbin: This utility docker returns http headers received inside the cluster. You may use it for general debugging - to check ingress, headers etc.
- cd $K8_ROOT/utils/httpbin ./install.sh
- To see what's reaching httpbin (example, replace with your domain name):

Install

cd $K8_ROOT/monitoring/alerting/
nano alertmanager.yml

Update

global:
  resolve_timeout: 5m
  slack_api_url: <YOUR-SLACK-API-URL>
...
  slack_configs:
  - channel: '<YOUR-CHANNEL-HERE>'
    send_resolved: true

cd $K8_ROOT/rancher/aws
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install \                               
  ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace  \
-f nginx.values.yaml

cd $K8_ROOT/rancher/rancher-ui
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update
helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --create-namespace \
  -f rancher-values.yaml

sudo docker run -d \
   --name=wireguard \
   --cap-add=NET_ADMIN \
   --cap-add=SYS_MODULE \
   -e PUID=1000 \
   -e PGID=1000 \
   -e TZ=Asia/Calcutta\
   -e PEERS=30 \
   -p 51820:51820/udp \
   -v /home/ubuntu/wireguard/config:/config \
   -v /lib/modules:/lib/modules \
   --sysctl="net.ipv4.conf.all.src_valid_mark=1" \
   --restart unless-stopped \
   ghcr.io/linuxserver/wireguard

Once rke remove is executed sucessfully need to delete cluster related incomplete configuration using :

Groups Field: member

02-error-only-logs.ndjson contains a Search dashboard which shows only the error logs of the services, called MOSIP Error Logs dashboard.

INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.0.0.1]
INFO[0000] [network] Deploying port listener containers   
INFO[0000] [network] Pulling image [alpine:latest] on host [10.0.0.1]
...
INFO[0101] Finished building Kubernetes cluster successfully
```

INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.0.0.1]
INFO[0000] [network] Deploying port listener containers
INFO[0000] [network] Pulling image [alpine:latest] on host [10.0.0.1]
...
INFO[0101] Finished building Kubernetes cluster successfully

#echo Installing Admin-Proxy into Masterdata and Keymanager.
#kubectl -n $NS apply -f admin-proxy.yaml
#echo Installing admin hotlist service.
#helm -n $NS install admin-hotlist mosip/admin-hotlist --version $CHART_VERSION
#echo Installing admin service. Will wait till service gets installed.
#helm -n $NS install admin-service mosip/admin-service --set istio.corsPolicy.allowOrigins\[0\].prefix=https://$ADMIN_HOST --wait --version $CHART_VERSION

On-Premises Deployment Without DNS

1. Wireguard

Wireguard bastian server provides secure private channel to access MOSIP cluster.
Bastian server restricts public access, and enables access to only those clients who have their public key listed in Wireguard server.
Bastion server listens on UDP port 51820.
In case you already have VPN configured to access nodes privately please skip Wireguard installation and continue to use the same VPN.

Setup Wireguard VM and wireguard bastion server

Create a Wireguard server VM with mentioned '.
Open required ports in the Bastian server VM.
- cd $K8_ROOT/wireguard/

Note:
Permission of the pem files to access nodes should have 400 permission. sudo chmod 400 ~/.ssh/privkey.pem
These ports are only needed to be opened for sharing packets over UDP.
Take necessary measure on firewall level so that the Wireguard server can be reachable on 51820/udp.

Install docker
- execute docker.yml to install docker and add user to docker group:
Setup Wireguard server

Note:
Increase the no. of peers above in case more than 30 wireguard client confs (-e PEERS=30) are needed.
Change the directory to be mounted to wireguard docker as per need. All your wireguard confs will be generated in the mounted directory (-v /home/ubuntu/wireguard/config:/config).

Setup Wireguard Client in your PC

Install Wireguard client in your PC using .
Assign wireguard.conf:
- SSH to the wireguard server VM.

2. Observation K8s Cluster setup and configuration

Install all the required tools mentioned in '' section.
Setup Observation Cluster node VM’s hardware and network configuration as per ().
Setup passwordless SSH into the cluster nodes via pem keys. (Ignore if VM’s are accessible via pem’s).

Note:
Make sure the permission for privkey.pem for ssh is set to 400.

Open ports and install docker on Observation K8 Cluster node VM’s.
- cd $K8_ROOT/rancher/on-prem
- Copy hosts.ini.sample to hosts.ini and update required details.

3. Observation K8s Cluster Ingress and Storage class setup

Once the rancher cluster is ready, we need ingress and storage class to be set for other applications to be installed.

3.a. : used for ingress in rancher cluster.

Note:
This will install ingress in namespace ingress-nginx of Observation cluster.
Crosscheck using below mentioned command:
kubectl get all -n ingress-nginx

3.b. Storage classes

Multiple storage classes options are available for onprem K8's cluster. In this reference deployment will continue to use NFS as a storage class.

Move to nfs directory in your personel computer.
Create a copy of hosts.ini.sample as hosts.ini.
Update the NFS machine details in hosts.ini file.
Note :

4. Setting up nginx server for Observation K8s Cluster

4.a. SSL Certificate setup for TLS termination

For Nginx server setup we need ssl certificate, add the same into Nginx server.
SSL certificates can be generated in multiple ways, either via lets encrypt if you have public DNS or via openssl certs when you don't have Public DNS.
- Openssl : Generate wildcard ssl certificate using openssl in case you don't have public DNS using below steps. (Ensure to use this only in development env, not suggested for Production env).

4.b. Install Nginx :

Move to nginx directory in your local
cd $K8_ROOT/mosip/on-prem/nginx/
Open required ports :

5. Observation K8's Cluster Apps Installation

5.a. Rancher UI

Rancher provides full CRUD capability of creating and managing kubernetes cluster.
Install rancher using Helm, update hostname, & add privateCA to true in rancher-values.yaml, and run the following command to setup secrets for installation.

Create a secret containing the observation nginx self-signed public certificate (i.e. tls.crt ) generated in .

USe below command to install Rancher UI:

Login:
- Connect to Wireguard Bastion. (Incase using WSL make sure to connect using Windows Wireguard instead of WSL one).
- Open Rancher page.

Note: Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.

5.b. Keycloak

: Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.

Post installation access the keycloak using iam.mosip.net and get the credentials as per the post installation steps definedAdd commentMore actions .

5.c. Keycloak - Rancher UI Integration

Login as admin user in Keycloak and make sure email and firstName fields are populated for the admin user. These are required for Rancher authentication to work properly.
In Keycloak (in the master realm), create a new client with the following values:

5.d. RBAC for Rancher using Keycloak

For users in Keycloak assign roles in Rancher - cluster and project roles. Under default project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects).
In case your cluster is used by multiple peoples with multiple roles and you want to create custom roles, you can follow the steps given .
Add a member to cluster/project in Rancher:

6. MOSIP K8s Cluster setup

Pre-requisites:
Install all the required tools mentioned in Pre-requisites for PC.
- kubectl

ansible
rke (version 1.3.10)
Setup MOSIP K8 Cluster node VM’s as per '.
Run env-check.yaml

Alternatively

Test cluster access:
- kubect get nodes
- Command will result in details of the nodes of the rancher cluster.

7. MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup

7.a. Global configmap:

Global configmap Contains the list of neccesary details to be used throughout the namespaces of the cluster for common details.
cd $K8_ROOT/mosip
Copy global_configmap.yaml.sample to global_configmap.yaml.

7.b. Ingress setup:

It is a service mesh for the MOSIP K8 cluster which provides transparent layers on top of existing microservices along with powerful features enabling a uniform and more efficient way to secure, connect, and monitor services.
- cd $K8_ROOT/mosip/on-prem/istio
- ./install.sh

7.c. Storage classes

Multiple storage classes options are available for onprem K8's cluster. In this reference deployment will continue to use NFS as a storage class.

Move to nfs directory in your personel computer.
Create a copy of hosts.ini.sample as hosts.ini.
Update the NFS machine details in hosts.ini file.
Note :

8. Import MOSIP Cluster into Rancher UI

Login as admin in Rancher console
Select Import Existing for cluster addition.
Select Generic as cluster type to add.

9. MOSIP K8 cluster Nginx server setup

9.a. Openssl certificates creation

For Nginx server setup, we need ssl certificate, add the same into Nginx server.
SSL certificates can be generated in multiple ways. Either via lets encrypt if you have public DNS or via openssl certs when you don't have Public DNS.
- Openssl : Generate wildcard ssl certificate using openssl in case you don't have public DNS using below steps. (Ensure to use this only in development env, not suggested for Production env).

9.b. Nginx server setup for MOSIP K8's cluster

Move to nginx directory in your local:Add commentMore actions
cd $K8_ROOT/mosip/on-prem/nginx/
Open required ports :

9.c. Check Overall nginx and istio wiring

Install httpbin: This utility docker returns http headers received inside the cluster.
httpbin can be used for general debugging - to check ingress, headers etc.
- To see what is reaching the httpbin (example, replace with your domain name):

10. Monitoring module deployment

Note :
Monitoring in the sandbox environment is optional and can be deployed if required.
For production environments, alternative monitoring tools can be used.
These steps can also be skipped in development environments if monitoring is not needed.

Prometheus and Grafana and Alertmanager tools are used for cluster monitoring.

Select 'Monitoring' App from Rancher console -> Apps & Marketplaces.
In Helm options, open the YAML file and disable Nginx Ingress.
Click on Install.

11. Alerting setup

Note :
Alerting in the sandbox environment is optional and can be deployed if required.
For production environments, alternative alerting tools can be used.
These steps can also be skipped in development environments if alerting is not needed.

Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel.
Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager.
Create .

Install Default alerts along some of the defined custom alerts:

Alerting is installed.

12. Logging module setup and installation

Note :
Logging in the sandbox environment is optional and can be deployed if required.
For production environments, alternative logging tools can be used.
These steps can also be skipped in development environments if logging is not needed.

MOSIP uses and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard.

Install Rancher FluentD system : for scraping logs outs of all the microservices from MOSIP k8 cluster.
- Install Logging from Apps and marketplace within the Rancher UI.
- Select Chart Version 100.1.3+up3.17.7 from Rancher console -> Apps & Marketplaces.

Open kibana dashboard from https://kibana.sandbox.xyz.net.

Kibana --> Menu (on top left) --> Dashboard --> Select the dashboard.

13. MOSIP External Dependencies setup

External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, HSM etc.

Click to check the detailed installation instructions of all the external components.

Note:
Connect to mosip_pms DB in postgres and execute the query to change valid_to_date for mpolicy-default-mobile in pms.auth_policy table.
Open the terminal.

Configuration change in case using Openssl wildcard ssl certificate. (Only advised in development env, not recommended for Production setup)

Add/Update the below property in application-default.properties and comment on the below property in the *-default.properties file in the config repo.
Add/Update the below property in the esignet-default.properties file in the config repo.

14. MOSIP Modules Deployment

Now that all the Kubernetes cluster and external dependencies are already installed, will continue with MOSIP service deployment.
While installing a few modules, installation script prompts to check if you have public domain and valid SSL certificates on the server. Opt option n as we are using self-signed certificates. For example:
Start installing mosip modules:

Note:
In case of failure on execution of install-all.sh follow installation steps from the failure point.
Installation for config-server and admin service is facing delay in this version hence need to execute below mentioned commands respectively for config-server and admin-service once the script fails.

15. API Testrig

MOSIP’s successfull deployment can be verified by comparing the results of api testrig with testrig benchmark.

Navigate to the Infra Root Directory:
Clone the Functional Tests Repository
After cloned successfully try to install apitestrig
Make script executable

16. DSL Rig

Install Packetcreater
- Navigate to the packetcreator directory:
- Run the installation script:

Create copy of hosts.ini.sample as hosts.ini and update the required details for wireguard VM.

Install and start wireguard server using docker as given below:

sudo docker run -d \
--name=wireguard \
--cap-add=NET_ADMIN \
--cap-add=SYS_MODULE \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Calcutta \
-e PEERS=30 \
-p 51820:51820/udp \
-v /home/ubuntu/wireguard/config:/config \
-v /lib/modules:/lib/modules \
--sysctl="net.ipv4.conf.all.src_valid_mark=1" \
--restart unless-stopped \
ghcr.io/linuxserver/wireguard

cd /home/ubuntu/wireguard/config

assign one of the PR for yourself and use the same from the PC to connect to the server.

create assigned.txt file to assign the keep track of peer files allocated and update everytime some peer is allocated to someone.

peer1 :   peername
peer2 :   xyz

use ls cmd to see the list of peers.
get inside your selected peer directory, and add mentioned changes in peer.conf:
- cd peer1
- nano peer1.conf

Generate keys on your PCssh-keygen -t rsa

Make sure Kubeconfig file is set correctly to point to required Observation cluster.

kubectl config view

Note:

Output should show the cluster name to confirm you are pointing to right kubernetes cluster.
If not pinting to right K8 cluster change the kubeconfig to connect to right K8 cluster.

Execute script to install nfs server:

sudo ./install-nfs-server.sh

Note:

Script prompts for below mentioned user inputs:
.....
Please Enter Environment Name: <envName>
.....
.....
.....
[ Export the NFS Share Directory ] 
exporting *:/srv/nfs/mosip/<envName>
NFS Server Path: /srv/nfs/mosip/<envName>
envName: env name eg. dev/qa/uat...

Switch to your personel computer and excute below mentioned commands:

cd $K8_ROOT/mosip/nfs/

./install-nfs-client-provisioner.sh

Note:

Script prompts for:
NFS Server: NFS server ip for persistence.
NFS Path : NFS path for storing the persisted data. eg. /srv/nfs/mosip/

Post installation check:

Check status of NFS Client Provisioner.

kubectl -n nfs get deployment.apps/nfs-client-provisioner

Check status of nfs-client storage class.

 kubectl get storageclass
 NAME                 PROVISIONER                            RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
 longhorn (default)   driver.longhorn.io                     Delete          Immediate           true                   57d
 nfs-client           cluster.local/nfs-client-provisioner   Delete          Immediate           true                   40s

Set nfs-client as default storage class.

Execute the following command to generate a self-signed SSL certificate. Prior to execution, kindly ensure to update environmental variables & rancher domain passed to openssl command:

mkdir -p /etc/ssl/certs/
export VALIDITY=700 
export COUNTRY=IN 
export STATE=KAR
export LOCATION=BLR
export ORG=MOSIP
export ORG_UNIT=MOSIP
export COMMON_NAME=*.xyz.net


openssl req -x509 -nodes -days $VALIDITY \
 -newkey rsa:2048 -keyout /etc/ssl/private/tls.key -out /etc/ssl/certs/tls.crt \
 -subj "/C=$COUNTRY/ST=$STATE/L=$LOCATION/O=$ORG/OU=$ORG_UNIT/CN=$COMMON_NAME" \
 -addext "subjectAltName = DNS:rancher.xyz.net, DNS:*.xyz.net"

Use any editor to create new hosts.ini file:

Add below mentioned lines with updated details of nginx server to the hosts.ini and save.

[nginx]
node-nginx ansible_host=<internal ip> ansible_user=root ansible_ssh_private_key_file=<pvt .pem file>

ssh -i ~/.ssh/<pem to ssh> ubuntu@<nginx server ip>

Clone k8s-infra

cd $K8_ROOT/rancher/on-prem/nginx
sudo ./install.sh

Provide below mentioned inputs as and when promted
- Rancher nginx ip : internal ip of the nginx server VM.
- SSL cert path : path of the ssl certificate to be used for ssl termination.
- SSL key path : path of the ssl key to be used for ssl termination.
- Cluster node IPs : IPs of the rancher cluster node

As result of rke config command cluster.ymlfile will be generated inside same directory, update the below mentioned fields:

nano cluster.yml
Remove the default Ingress install
```
ingress:
  provider: none
```
Add the name of the kubernetes cluster
For production deployments edit the cluster.yml, according to this .

In case not having Public DNS system add the custom DNS configuration for all the hostnames to be used for testing in Cluster's coredns configmap.

Check whether coredns pods are up and running in your cluster via the below command:
```
kubectl -n kube-system get pods -l k8s-app=kube-dns
```
To update the coredns configmap, use the below command.
```
kubectl -n kube-system edit cm coredns
```
Note: Default editor in WSL and Ubuntu is vi. Incase not familiar with vi change the editor to your prefered one:export EDITOR=<prefered editor>
Update the IP address and domain name in the below DNS hosts template and add it in the coredns configmap Corefile key in the kube-system namespace.
Check whether the DNS changes are correctly updated in coredns configmap.
Restart the coredns pod in the kube-system namespace.
Check status of coredns restart.
Note:
- Since this deployment is without proper DNS and using self signed certificate please be informed that while accessing the UI components in browsr the sites will be in Not-secure state.

Make sure Kubeconfig file is set correctly to point to required mosip cluster.

kubectl config view

Note:

Output should show the cluster name to confirm you are pointing to right kubernetes cluster.
If not pinting to right K8 cluster change the kubeconfig to connect to right K8 cluster.

Execute script to install nfs server:

sudo ./install-nfs-server.sh

Note:

Script prompts for below mentioned user inputs:
.....
Please Enter Environment Name: <envName>
.....
.....
.....
[ Export the NFS Share Directory ] 
exporting *:/srv/nfs/mosip/<envName>
NFS Server Path: /srv/nfs/mosip/<envName>
envName: env name eg. dev/qa/uat...

Switch to your personel computer and excute below mentioned commands:

cd $K8_ROOT/mosip/nfs/

./install-nfs-client-provisioner.sh

Note:

Script prompts for:
NFS Server: NFS server ip for persistence.
NFS Path : NFS path for storing the persisted data. eg. /srv/nfs/mosip/

Post installation check:

Check status of NFS Client Provisioner.

kubectl -n nfs get deployment.apps/nfs-client-provisioner

Check status of nfs-client storage class.

 kubectl get storageclass
 NAME                 PROVISIONER                            RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
 longhorn (default)   driver.longhorn.io                     Delete          Immediate           true                   57d
 nfs-client           cluster.local/nfs-client-provisioner   Delete          Immediate           true                   40s

You will get the kubectl commands to be executed in the kubernetes cluster. Copy the command and execute from your PC (make sure your kube-config file is correctly set to MOSIP cluster).

e.g.:
kubectl apply -f https://rancher.e2e.mosip.net/v3/import/pdmkx6b4xxtpcd699gzwdtt5bckwf4ctdgr7xkmmtwg8dfjk4hmbpk_c-m-db8kcj4r.yaml

docker volume create --name gensslcerts --opt type=none --opt device=/etc/ssl --opt o=bind docker run -it --mount type=volume,src='gensslcerts',dst=/home/mosip/ssl,volume-driver=local \ -e VALIDITY=700 \ -e COUNTRY=IN \ -e STATE=KAR \ -e LOCATION=BLR \ -e ORG=MOSIP \ -e ORG_UNIT=MOSIP \ -e COMMON_NAME=<*.sandbox.xyz.net> \ mosipdev/openssl:latest

Use any editor to create new hosts.ini file:

Add below mentioned lines with updated details of nginx server to the hosts.ini and save.

[nginx]
node-nginx ansible_host=<internal ip> ansible_user=root ansible_ssh_private_key_file=<pvt .pem file>

ssh -i ~/.ssh/<pem to ssh> -i ubuntu@

Clone k8s-infra.

cd $K8_ROOT/mosip/on-prem/nginx
sudo ./install.sh

Provide below mentioned inputs as and when prompted
- MOSIP nginx server internal ip
- MOSIP nginx server public ip
- Publically accessible domains (comma separated with no whitespaces)
- SSL cert path
- SSL key path
- Cluster node ip's (comma separated no whitespace)

When utilizing an openssl wildcard SSL certificate, please add the following server block to the nginx server configuration within the http block. Disregard this if using SSL certificates obtained through letsencrypt or for publicly available domains. Please note that this should only be used in a development environment and is not recommended for production environments.

nano /etc/nginx/nginx.conf

server{
listen <cluster-nginx-internal-ip>:80;
server_name iam.sandbox.xyz.net;
location /auth/realms/mosip/protocol/openid-connect/certs {
  proxy_pass                      http://myInternalIngressUpstream;
  proxy_http_version              1.1;
  proxy_set_header                Upgrade $http_upgrade;
  proxy_set_header                Connection "upgrade";
  proxy_set_header                Host $host;
  proxy_set_header                Referer $http_referer;
  proxy_set_header                X-Real-IP $remote_addr;
  proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header                X-Forwarded-Proto $scheme;
  proxy_pass_request_headers      on;
  proxy_set_header  Strict-Transport-Security "max-age=0;";
  }
location / { return 301 https://iam.sandbox.xyz.net; }
  }

Note: HTTP access is enabled for IAM because MOSIP's keymanager expects to have valid SSL certificates. Ensure to use this only for development purposes, and it is not recommended to use it in production environments.

Restart nginx service.

After setting slack incoming webhook update slack_api_url and slack_channel_name in alertmanager.yml.

Use the psql command to connect to the PostgreSQL server. The general syntax is:

psql -h <host> -p 5432 -U postgres -d mosip_pms

: The server address (e.g., localhost or an IP address).
Assuming other details remain same like port and user.

UPDATE pms.auth_policy SET valid_to_date = valid_to_date + interval '1 year' WHERE name = 'mpolicy-default-mobile';

For admin service:

Update failureThreshold for startupProbe to 60.

kubectl -n admin edit deployment admin-service

Once admin service is up and running re-execute install.sh script after commenting out below mentioned commands:

Run the Installer Script

./install.sh

Note:
Script prompts for below mentioned inputs please provide as and when needed:
Enter the time (hr) to run the cronjob every day (0–23): Specify the hour you want the cronjob to run (e.g., 6 for 6 AM)
Do you have a public domain and valid SSL certificate? (Y/n):
Y – If you have a public domain and valid SSL certificate
n – If you do not have one (recommended only for development environments)
Retention days to remove old reports (Default: 3): Press Enter to accept the default or specify another value (e.g., 5).
Provide Slack Webhook URL to notify server issues on your Slack channel: (change the URL to your channel one)
Is the eSignet service deployed? (yes/no):
no – If eSignet is not deployed, related test cases will be skipped.
Is values.yaml for the apitestrig chart set correctly as part of the prerequisites? (Y/n): * Enter Y if this step is already completed.
Do you have S3 details for storing API-Testrig reports? (Y/n):
Enter Y to proceed with S3 configuration.
S3 Host: eg. http://minio.minio:9000

Run the installation script:

* ./install.sh

Note: Before running install.sh, please ensure the following is added in the Helm install command:
--set dslorchestrator.configmaps.dslorchestrator.servicesNotDeployed="esignet" \
After adding the above, the full Helm installation command should look like this:
helm -n $NS install dslorchestrator mosip/dslorchestrator \
 --set crontime="0 $time * * *" \
 --version $CHART_VERSION \
 --set dslorchestrator.configmaps.s3.s3-host='http://minio.minio:9000' \
 --set dslorchestrator.configmaps.s3.s3-user-key='admin' \
 --set dslorchestrator.configmaps.s3.s3-region='' \
 --set dslorchestrator.configmaps.db.db-server="$DB_HOST" \
 --set dslorchestrator.configmaps.db.db-su-user="postgres" \
 --set dslorchestrator.configmaps.db.db-port="5432" \
 --set dslorchestrator.configmaps.dslorchestrator.USER="$USER" \
 --set dslorchestrator.configmaps.dslorchestrator.ENDPOINT="https://$API_INTERNAL_HOST" \
 --set dslorchestrator.configmaps.dslorchestrator.packetUtilityBaseUrl="$packetUtilityBaseUrl" \
 --set persistence.nfs.server="$NFS_HOST" \
 --set persistence.nfs.path="/srv/nfs/mosip/dsl-scenarios/$ENV_NAME" \
 --set dslorchestrator.configmaps.dslorchestrator.reportExpirationInDays="$reportExpirationInDays" \
 --set dslorchestrator.configmaps.dslorchestrator.NS="$NS" \
 --set dslorchestrator.configmaps.dslorchestrator.servicesNotDeployed="esignet" \
 $ENABLE_INSECURE
When prompted, provide the following inputs:
NFS Host:
NFS PEM :
User for SSH Login:ubuntu
Cronjob Time (hour of the day, 0–23): (e.g., enter 6 for 6 AM)
Do you have a public domain and valid SSL certificate? (Y/n):
Y: If you have a valid public domain and SSL certificate
n: Use only in development environments
Packet Utility Base URL:
Retention Days to Remove Old Reports (default is 3): (Press Enter to accept default or provide a custom value)

cd $K8_ROOT/rancher/on-prem
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--version 4.0.18 \
--create-namespace  \
-f ingress-nginx.values.yaml

./install.sh
Do you have public domain & valid SSL? (Y/n) 
Y: if you have public domain & valid ssl certificate
n: If you don't have a public domain and a valid SSL certificate. Note: It is recommended to use this option only in development environments.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
kubectl create ns cattle-monitoring-system
helm -n cattle-monitoring-system install rancher-monitoring-crd mosip/rancher-monitoring-crd

On-Prem Installation Guidelines

Note:

This version of the guide is currently under revision and enhancement. For any support, please reach out to us on .

Overview

MOSIP modules are deployed in the form of microservices in kubernetes cluster.
is used as a trust network extension to access the admin, control, and observation pane.
It is also used for the on-the-field registrations.
MOSIP uses server for:
- SSL termination
- Reverse Proxy
Kubernetes cluster is administered using the and tools.
In V3, we have two Kubernetes clusters:

Observation cluster - This cluster is a part of the observation plane and it helps in administrative tasks. By design, this is kept independent of the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or it's services should be internal and should never be exposed to the external world.

is used for managing the MOSIP cluster.
in this cluster is used for cluster user access management.
It is recommended to configure log monitoring and network monitoring in this cluster.

MOSIP cluster - This cluster runs all the MOSIP components and certain third party components to secure the cluster, API’s and data.

Architecture

Deployment repos

: contains the scripts to install and configure Kubernetes cluster with required monitoring, logging and alerting tools.
: contains the deployment scripts to run charts in defined sequence.
: contains all the configuration files required by the MOSIP modules.

Pre-requisites

Hardware requirements

VM’s required can be with any OS as per convenience.
Here, we are referting to Ubuntu OS throughout this installation guide.

Sl no.

Purpose

vCPU's

RAM

Storage (HDD)

no. ofVM's

Network requirements

All the VM's should be able to communicate with each other.
Need stable Intra network connectivity between these VM's.
All the VM's should have stable internet connectivity for docker image download (in case of local setup ensure to have a locally accessible docker registry).

Sl no.

Purpose

Network Interfaces

DNS requirements

Domain Name

Mapping details

Purpose

Certificate requirements

As only secured https connections are allowed via nginx server will need below mentioned valid ssl certificates:

One valid wildcard ssl certificate related to domain used for accessing Observation cluster, this needs to be stored inside the nginx server VM for Observation cluster. In above e.g.: *.org.net is the similar example domain.
One valid wildcard ssl certificate related to domain used for accessing Mosip cluster, this needs to be stored inside the nginx server VM for mosip cluster. In above e.g.: *.sandbox.xyz.net is the similar example domain.

Tools to be installed in Personal Computers for complete deployment

- any client version above 1.19
- any client version above 3.0.0 and add below repos as well:

: version: 1.15.0
: version:
[Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html: version > 2.12.4

Installation

Wireguard

Setup Wireguard VM and wireguard bastion server

Create a Wireguard server VM with above mentioned Hardware and Network requirements.
Open ports and Install docker on Wireguard VM.
- cd $K8_ROOT/wireguard/

Note:
Permission of the pem files to access nodes should have 400 permission. sudo chmod 400 ~/.ssh/privkey.pem
These ports are only needed to be opened for sharing packets over UDP.

Setup Wireguard server
- SSH to wireguard VM
- Create directory for storing wireguard config files. mkdir -p wireguard/config

Note:

Increase the no. of peers above in case more than 30 wireguard client confs (-e PEERS=30) are needed.
Change the directory to be mounted to wireguard docker as per need. All your wireguard confs will be generated in the mounted directory (-v /home/ubuntu/wireguard/config:/config).

Setup Wireguard Client in your PC

Install Wireguard client in your PC.
Assign wireguard.conf:
- SSH to the wireguard server VM.

Once connected to wireguard, you should be now able to login using private IP’s.

Observation K8s Cluster setup and configuration

Observation K8s Cluster setup

Install all the required tools mentioned in pre-requisites for PC.
- kubectl
- helm

Note:
Make sure the permission for privkey.pem for ssh is set to 400.

Run env-check-setup.yaml to check if cluster nodes are fine and do not have known issues in it.
- cd $K8_ROOT/rancher/on-prem
- create copy of hosts.ini.sample

Observation K8s Cluster Ingress and Storage class setup

Once the rancher cluster is ready, we need ingress and storage class to be set for other applications to be installed.

: used for ingress in rancher cluster.

this will install ingress in ingress-nginx namespace of rancher cluster.

Storage classes

The following storage classes can be used:

: If you are already using VMware virtual machines, you can proceed with the vSphere storage class.

Setting up nginx server for Observation K8s Cluster

For Nginx server setup we need ssl certificate, add the same into Nginx server.
Incase valid ssl certificate is not there generate one using letsencrypt:
- SSH into the nginx server

Observation K8's Cluster Apps Installation

Rancher UI

Rancher provides full CRUD capability of creating and managing kubernetes cluster.
Install rancher using Helm, update hostname in rancher-values.yaml and run the following command to install.

Login:
- Open page.
- Get Bootstrap password using
Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.

Keycloak

is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.

keycloak_client.json: Used to create SAML client on Keycloak for Rancher integration.

Keycloak - Rancher UI Integration

Login as admin user in Keycloak and make sure an email id, and first name field is populated for admin user. This is important for Rancher authentication as given below.
Enable authentication with Keycloak using the steps given .
In Keycloak add another Mapper for the rancher client (in Master realm) with following fields:

RBAC for Rancher using Keycloak

For users in Keycloak assign roles in Rancher - cluster and project roles. Under default project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects).
If you want to create custom roles, you can follow the steps given .
Add a member to cluster/project in Rancher:

Certificates expiry

In case you see certificate expiry message while adding users, on local cluster run these commands:

MOSIP K8s Cluster setup

Pre-requisites:
Install all the required tools mentioned in Pre-requisites for PC.
- kubectl

ansible
rke (version 1.3.10)
Setup MOSIP K8 Cluster node VM’s as per the hardware and network requirements as mentioned above.

Alternatively

Test cluster access:
- kubect get nodes
- Command will result in details of the nodes of the rancher cluster.

MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup

Global configmap: Global configmap contains the list of neccesary details to be used throughout the namespaces of the cluster for common details.

cd $K8_ROOT/mosip
Copy global_configmap.yaml.sample to global_configmap.yaml.
Update the domain names in

Storage classes

The following storage classes can be used:

: If you are already using VMware virtual machines, you can proceed with the vSphere storage class.

The NFS is used to store the Kubernetes pod's persistence data.

Import MOSIP Cluster into Rancher UI

Login as admin in Rancher console
Select Import Existing for cluster addition.
Select Generic as cluster type to add.

Wait for few seconds after executing the command for the cluster to get verified.
Your cluster is now added to the rancher management server.

MOSIP K8 cluster Nginx server setup

For Nginx server setup, we need ssl certificate, add the same into Nginx server.
Incase valid ssl certificate is not there generate one using letsencrypt:
- SSH into the nginx server

Monitoring module deployment

Prometheus and Grafana and Alertmanager tools are used for cluster monitoring.
Select 'Monitoring' App from Rancher console -> Apps & Marketplaces.
In Helm options, open the YAML file and disable Nginx Ingress.

Alerting setup

Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel.

Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager.
Create .
After setting slack incoming webhook update slack_api_url and slack_channel_name in alertmanager.yml

Install Default alerts along some of the defined custom alerts:

Alerting is installed.

Logging module setup and installation

MOSIP uses and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard.

Install Rancher FluentD system : for screpping logs outs of all the microservices from MOSIP k8 cluster.
- Install Logging from Apps and marketplace within the Rancher UI.
- Select Chart Version 100.1.3+up3.17.7 from Rancher console -> Apps & Marketplaces.

Open kibana dashboard from https://kibana.sandbox.xyz.net.

Kibana --> Menu (on top left) --> Dashboard --> Select the dashboard.

MOSIP External Dependencies setup

External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, HSM etc.

Click to check the detailed installation instructions of all the external components.

MOSIP Modules Deployment

Now that all the Kubernetes cluster and external dependencies are already installed, will continue with MOSIP service deployment.

Check detailed installation steps.

On-Prem without DNS Installation Guidelines

Note:

This version of the guide is currently under revision and enhancement. For any support, please reach out to us on .

Overview

MOSIP modules are deployed in the form of microservices in kubernetes cluster.
is used as a trust network extension to access the admin, control, and observation pane.
It is also used for the on-the-field registrations.
MOSIP uses server for:
- SSL termination
- Reverse Proxy
Kubernetes cluster is administered using the and tools.
In V3, we have two Kubernetes clusters:

Observation cluster - This cluster is a part of the observation plane, and it helps in administrative tasks. By design, this is kept independent of the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or it's services should be internal and should never be exposed to the external world.

is used for managing the MOSIP cluster.
in this cluster is used for cluster user access management.
It is recommended to configure log monitoring and network monitoring in this cluster.

MOSIP cluster - This cluster runs all the MOSIP components and certain third party components to secure the cluster, APIs and data.

Architecture diagram

Deployment repos

: contains the scripts to install and configure Kubernetes cluster with required monitoring, logging and alerting tools.
: contains the deployment scripts to run charts in defined sequence.
: contains all the configuration files required by the MOSIP modules.

Pre-requisites

Hardware requirements

VM’s required can be with any OS as per convenience.
Here, we are referring to Ubuntu OS throughout this installation guide.

Sl no.

Purpose

vCPU's

RAM

Storage (HDD)

no. ofVM's

Network requirements

All the VM's should be able to communicate with each other.
Need stable Intra network connectivity between these VM's.
All the VM's should have stable internet connectivity for docker image download (in case of local setup ensure to have a locally accessible docker registry).

Sl no.

Purpose

Network Interfaces

DNS requirements

Domain Name

Mapping details

Purpose

Certificate requirements

As only secured https connections are allowed via nginx server will need below mentioned valid ssl certificates:

One valid wildcard ssl certificate related to domain used for accessing Observation cluster, this needs to be stored inside the nginx server VM for Observation cluster. In above e.g.: *.org.net is the similar example domain.
One valid wildcard ssl certificate related to domain used for accessing Mosip cluster, this needs to be stored inside the nginx server VM for mosip cluster. In above e.g.: *.sandbox.xyz.net is the similar example domain.

Tools to be installed in Personal Computers for complete deployment

- any client version above 1.19
- any client version above 3.0.0 and add below repos as well:
: version: 1.15.0

Installation

Wireguard

Setup Wireguard VM and wireguard bastion server

Create a Wireguard server VM with above-mentioned Hardware and Network requirements.
Open ports and Install docker on Wireguard VM.
- cd $K8_ROOT/wireguard/

Note:
Permission of the pem files to access nodes should have 400 permission. sudo chmod 400 ~/.ssh/privkey.pem
These ports are only needed to be opened for sharing packets over UDP.
Take necessary measure on firewall level so that the Wireguard server can be reachable on 51820/udp.

Execute docker.yml to install docker and add user to docker group: ansible-playbook -i hosts.ini docker.yaml
Setup Wireguard server
- SSH to wireguard VM

Note:
Increase the no. of peers above in case more than 30 wireguard client confs (-e PEERS=30) are needed.
Change the directory to be mounted to wireguard docker as per need. All your wireguard confs will be generated in the mounted directory (-v /home/ubuntu/wireguard/config:/config).

Setup Wireguard Client in your PC

Install Wireguard client in your PC.
Assign wireguard.conf:
- SSH to the wireguard server VM.

Observation K8s Cluster setup and configuration

Observation K8s Cluster setup

Install all the required tools mentioned in pre-requisites for PC.
- kubectl
- helm

Note:
Make sure the permission for privkey.pem for ssh is set to 400.

Run env-check.yaml to check if cluster nodes are fine and do not have known issues in it.
- cd $K8_ROOT/rancher/on-prem/
- create copy of hosts.ini.sample

Observation K8s Cluster Ingress and Storage class setup

Once the rancher cluster is ready, we need ingress and storage class to be set for other applications to be installed.

: used for ingress in rancher cluster.
this will install ingress in ingress-nginx namespace of rancher cluster.
Storage class setup: creates a storage class in the cluster for creating pv (persistence volume) and pvc (persistence volume claim).
- Pre-requisites:

Setting up nginx server for Observation K8s Cluster

For Nginx server setup we need ssl certificate, add the same into Nginx server.
SSL certificates can be generated in multiple ways. Either via lets encrypt if you have public DNS or via openssl certs when you don't have Public DNS.
- Letsencrypt: Generate wildcard ssl certificate having 3 months validity when you have public DNS system using below steps.

Observation K8's Cluster Apps Installation

Rancher UI: Rancher provides full CRUD capability of creating and managing kubernetes cluster.

Install rancher using Helm, update hostname, & add privateCA to true in rancher-values.yaml, and run the following command to install.
- cd $K8_ROOT/rancher/rancher-ui

keycloak_client.json: Used to create SAML client on Keycloak for Rancher integration.

Keycloak - Rancher UI Integration

Login as admin user in Keycloak and make sure an email id, and first name field is populated for admin user. This is important for Rancher authentication as given below.
Enable authentication with Keycloak using the steps given .
In Keycloak add another Mapper for the rancher client (in Master realm) with following fields:

RBAC :

For users in Keycloak assign roles in Rancher - cluster and project roles. Under default project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects).
If you want to create custom roles, you can follow the steps given .
Add a member to cluster/project in Rancher:

Certificates expiry

In case you see certificate expiry message while adding users, on local cluster run these commands:

MOSIP K8s Cluster setup

Pre-requisites:
Install all the required tools mentioned in Pre-requisites for PC.
- kubectl

ansible
rke (version 1.3.10)
Setup MOSIP K8 Cluster node VM’s as per the hardware and network requirements as mentioned above.

Alternatively

Test cluster access:
- kubect get nodes
- Command will result in details of the nodes of the rancher cluster.

MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup

Global configmap: Global configmap contains the list of neccesary details to be used throughout the namespaces of the cluster for common details.

cd $K8_ROOT/mosip
Copy global_configmap.yaml.sample to global_configmap.yaml.
Update the domain names in global_configmap.yaml

Import MOSIP Cluster into Rancher UI

Login as admin in Rancher console
Select Import Existing for cluster addition.
Select Generic as cluster type to add.

MOSIP K8 cluster Nginx server setup

For Nginx server setup, we need ssl certificate, add the same into Nginx server.
SSL certificates can be generated in multiple ways. Either via lets encrypt if you have public DNS or via openssl certs when you don't have Public DNS.
- Letsencrypt: Generate wildcard ssl certificate having 3 months validity when you have public DNS system using below steps.

Monitoring module deployment

Prometheus and Grafana and Alertmanager tools are used for cluster monitoring.
Select 'Monitoring' App from Rancher console -> Apps & Marketplaces.
In Helm options, open the YAML file and disable Nginx Ingress.

Alerting setup

Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel.

Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager.
Create .
After setting slack incoming webhook update slack_api_url and slack_channel_name in alertmanager.yml

Logging module setup and installation

MOSIP uses and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard.

Install Rancher FluentD system : for scraping logs outs of all the microservices from MOSIP k8 cluster.
- Install Logging from Apps and marketplace within the Rancher UI.
- Select Chart Version 100.1.3+up3.17.7 from Rancher console -> Apps & Marketplaces.

Open kibana dashboard from https://kibana.sandbox.xyz.net.

Kibana --> Menu (on top left) --> Dashboard --> Select the dashboard.

MOSIP External Dependencies setup

External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, HSM etc.

Click to check the detailed installation instructions of all the external components.

Configuration change in case using Openssl wildcard ssl certificate. (Only advised in development env, not recommended for Production setup)

Add/Update the below property in application-default.properties and comment on the below property in the *-default.properties file in the config repo.
Add/Update the below property in the esignet-default.properties file in the config repo.

MOSIP Modules Deployment

Now that all the Kubernetes cluster and external dependencies are already installed, will continue with MOSIP service deployment.
While installing a few modules, installation script prompts to check if you have public domain and valid SSL certificates on the server. Opt option n as we are using self-signed certificates. For example:
Start installing mosip modules:

Check detailed installation steps.

Once rke remove is executed sucessfully need to delete cluster related incomplete configuration using :

In case not having Public DNS system add the custom DNS configuration for all the hostnames to be used for testing in Cluster's coredns configmap.

Check whether coredns pods are up and running in your cluster via the below command:
```
kubectl -n kube-system get pods -l k8s-app=kube-dns
```
To update the coredns configmap, use the below command.
```
kubectl -n kube-system edit cm coredns
```
Note: Default editor in WSL and Ubuntu is vi. Incase not familiar with vi change the editor to your prefered one:export EDITOR=<prefered editor>
Update the IP address and domain name in the below DNS hosts template and add it in the coredns configmap Corefile key in the kube-system namespace.
Check whether the DNS changes are correctly updated in coredns configmap.
Restart the coredns pod in the kube-system namespace.
Check status of coredns restart.

Groups Field: member

02-error-only-logs.ndjson contains a Search dashboard which shows only the error logs of the services, called MOSIP Error Logs dashboard.

INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.0.0.1]
INFO[0000] [network] Deploying port listener containers   
INFO[0000] [network] Pulling image [alpine:latest] on host [10.0.0.1]
...
INFO[0101] Finished building Kubernetes cluster successfully

INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.0.0.1]
INFO[0000] [network] Deploying port listener containers
INFO[0000] [network] Pulling image [alpine:latest] on host [10.0.0.1]
...
INFO[0101] Finished building Kubernetes cluster successfully

hosts {
  <PUBLIC_IP_OF_MOSIP_NGINX_NODE>    api.sandbox.xyz.net resident.sandbox.xyz.net esignet.sandbox.xyz.net prereg.sandbox.xyz.net healthservices.sandbox.xyz.net
  <INTERNAL_IP_OF_MOSIP_NGINX_NODE>  sandbox.xyz.net api-internal.sandbox.xyz.net activemq.sandbox.xyz.net kibana.sandbox.xyz.net regclient.sandbox.xyz.net admin.sandbox.xyz.net minio.sandbox.xyz.net iam.sandbox.xyz.net kafka.sandbox.xyz.net postgres.sandbox.xyz.net pmp.sandbox.xyz.net onboarder.sandbox.xyz.net smtp.sandbox.xyz.net compliance.sandbox.xyz.net
        
  ## Observation 
  <INTERNAL_IP_OF_OBS_NGINX_NODE>    rancher.xyz.net keycloak.xyz.net
  fallthrough
}

 <PUBLIC_IP>    api.sandbox.xyz.net resident.sandbox.xyz.net esignet.sandbox.xyz.net prereg.sandbox.xyz.net healthservices.sandbox.xyz.net
 <INTERNAL_IP>  sandbox.xyz.net api-internal.sandbox.xyz.net activemq.sandbox.xyz.net kibana.sandbox.xyz.net regclient.sandbox.xyz.net admin.sandbox.xyz.net minio.sandbox.xyz.net iam.sandbox.xyz.net kafka.sandbox.xyz.net postgres.sandbox.xyz.net pmp.sandbox.xyz.net onboarder.sandbox.xyz.net smtp.sandbox.xyz.net compliance.sandbox.xyz.net

#echo Installing Admin-Proxy into Masterdata and Keymanager.
#kubectl -n $NS apply -f admin-proxy.yaml
#echo Installing admin hotlist service.
#helm -n $NS install admin-hotlist mosip/admin-hotlist --version $CHART_VERSION
#echo Installing admin service. Will wait till service gets installed.
#helm -n $NS install admin-service mosip/admin-service --set istio.corsPolicy.allowOrigins\[0\].prefix=https://$ADMIN_HOST --wait --version $CHART_VERSION

assign one of the PR for yourself and use the same from the PC to connect to the server.

create assigned.txt file to assign the keep track of peer files allocated and update everytime some peer is allocated to someone.

peer1 :   peername
peer2 :   xyz

use ls cmd to see the list of peers.
get inside your selected peer directory, and add mentioned changes in peer.conf:
- cd peer1
- nano peer1.conf

hosts.ini

Run env-check-setup.yaml to check if cluster nodes are fine and dont have known issues in it.

As result of rke config command cluster.ymlfile will be generated inside same directory, update the below mentioned fields:

nano cluster.yml
Remove the default Ingress install

ingress:
provider: none

Add the name of the kubernetes cluster

For production deplopyments edit the cluster.yml, according to this .

global_configmap.yaml

Install Pre-requisites:

sudo apt update -y
sudo apt-get install software-properties-common -y
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update -y
sudo apt-get install python3.8 -y
sudo apt install letsencrypt -y
sudo apt install certbot python3-certbot-nginx -y

Check Overall if nginx and istio wiring is set correctly

Install httpbin: This utility docker returns http headers received inside the cluster. You may use it for general debugging - to check ingress, headers etc.

cd $K8_ROOT/utils/httpbin
./install.sh

To see what is reaching the httpbin (example, replace with your domain name):

curl https://api.sandbox.xyz.net/httpbin/get?show_env=true
curl https://api-internal.sandbox.xyz.net/httpbin/get?show_env=true

cd $K8_ROOT/mosip/on-prem
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install \                                                                                                             
  ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --version 4.0.18 \
  --create-namespace  \
  -f ingress-nginx.values.yaml

cd $K8_ROOT/rancher/rancher-ui
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo update
helm install rancher rancher-latest/rancher \
--namespace cattle-system \
--create-namespace \
-f rancher-values.yaml

sudo docker run -d \
--name=wireguard \
--cap-add=NET_ADMIN \
--cap-add=SYS_MODULE \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Calcutta \
-e PEERS=30 \
-p 51820:51820/udp \
-v /home/ubuntu/wireguard/config:/config \
-v /lib/modules:/lib/modules \
--sysctl="net.ipv4.conf.all.src_valid_mark=1" \
--restart unless-stopped \
ghcr.io/linuxserver/wireguard

sudo apt update -y
sudo apt-get install software-properties-common -y
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update -y
sudo apt-get install python3.8 -y
sudo apt install letsencrypt -y
sudo apt install certbot python3-certbot-nginx -y

Install and start wireguard server using docker as given below:

sudo docker run -d \
--name=wireguard \
--cap-add=NET_ADMIN \
--cap-add=SYS_MODULE \
-e PUID=1000 \
-e PGID=1000 \
-e TZ=Asia/Calcutta \
-e PEERS=30 \
-p 51820:51820/udp \
-v /home/ubuntu/wireguard/config:/config \
-v /lib/modules:/lib/modules \
--sysctl="net.ipv4.conf.all.src_valid_mark=1" \
--restart unless-stopped \
ghcr.io/linuxserver/wireguard

cd /home/ubuntu/wireguard/config

Assign one of the PR for yourself and use the same from the PC to connect to the server.

Create assigned.txt file to assign the keep track of peer files allocated and update every time some peer is allocated to someone.
```
peer1 :   peername
peer2 :   xyz
```
Use ls cmd to see the list of peers.
Get inside your selected peer directory, and add mentioned changes in peer.conf:
- cd peer1
- nano peer1.conf

Setup passwordless SSH into the cluster nodes via pem keys. (Ignore if VM’s are accessible via pem’s).

Generate keys on your PC
```
ssh-keygen -t rsa
```
Copy the keys to remote observation node VM’s
```
ssh-copy-id <remote-user>@<remote-ip>
```
SSH into the node to check password-less SSH

hosts.ini

Note: Values of below mentioned parameters are set as by default Longhorn installation script:

PV replica count is set to 1. Set the replicas for the storage class appropriately.
```
persistence.defaultClassReplicaCount=1
defaultSettings.defaultReplicaCount=1
```
Total available node CPU allocated to each instance-manager pod in the longhorn-system namespace.
```
guaranteedEngineManagerCPU: 5
guaranteedReplicaManagerCPU: 5   
```
- The value 5 means 5% of the total available node CPU.
- This value should be fine for sandbox and pilot, but you may have to increase the default to "12" for production.
- The value can be updated on Longhorn UI after installation.

Install Pre-requisites

sudo apt update -y
sudo apt-get install software-properties-common -y
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update -y
sudo apt-get install python3.8 -y
sudo apt install letsencrypt -y
sudo apt install certbot python3-certbot-nginx -y

Openssl : Generate wildcard ssl certificate using openssl in case you don't have public DNS using below steps. (Ensure to use this only in development env, not suggested for Production env).

Generate a self-signed certificate for your domain, such as *.sandbox.xyz.net.

Execute the following command to generate a self-signed SSL certificate. Prior to execution, kindly ensure to update environmental variables & rancher domain passed to openssl command:

mkdir -p /etc/ssl/certs/
export VALIDITY=700 
export COUNTRY=IN 
export STATE=KAR
export LOCATION=BLR
export ORG=MOSIP
export ORG_UNIT=MOSIP
export COMMON_NAME=*.xyz.net

openssl req -x509 -nodes -days $VALIDITY \
 -newkey rsa:2048 -keyout /etc/ssl/certs/tls.key -out /etc/ssl/certs/tls.crt \
 -subj "/C=$COUNTRY/ST=$STATE/L=$LOCATION/O=$ORG/OU=$ORG_UNIT/CN=$COMMON_NAME" \
 -addext "subjectAltName = DNS:rancher.xyz.net, DNS:*.xyz.net"

Above command will generate certs in below specified location. Use it when prompted during nginx installation.
- fullChain path: /etc/ssl/certs/tls.crt.
- privKey path: /etc/ssl/private/tls.key.

Install nginx:

Clone k8s-infra

cd $K8_ROOT/rancher/on-prem/nginx
sudo ./install.sh

Provide below mentioned inputs as and when promted
- Rancher nginx ip : internal ip of the nginx server VM.
- SSL cert path : path of the ssl certificate to be used for ssl termination.
- SSL key path : path of the ssl key to be used for ssl termination.
- Cluster node IPs : IPs of the rancher cluster node

Create a secret containing the observation nginx self-signed public certificate (i.e. tls.crt ) generated in openssl section.

kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem=./tls.crt
helm install rancher rancher/rancher --version 2.6.3 \
--namespace cattle-system \
--create-namespace \
--set privateCA=true \
-f rancher-values.yaml

Open Rancher page.

Get Bootstrap password using

kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}'

Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.

Keycloak: Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.

cd $K8_ROOT/apps/keycloak
./install.sh <iam.host.name>

Run env-check.yaml to check if cluster nodes are fine and don't have known issues in it.

As result of rke config command cluster.ymlfile will be generated inside same directory, update the below mentioned fields:

nano cluster.yml
Remove the default Ingress install
```
ingress:
  provider: none
```
Add the name of the kubernetes cluster
For production deployments edit the cluster.yml, according to this .

In case not having Public DNS system add the custom DNS configuration for the cluster.

Check whether coredns pods are up and running in your cluster via the below command:
```
kubectl -n kube-system get pods -l k8s-app=kube-dns
```

Update the IP address and domain name in the below DNS hosts template and add it in the coredns configmap Corefile key in the kube-system namespace.

hosts {
  <PUBLIC_IP_OF_MOSIP_NGINX_NODE>    api.sandbox.xyz.net resident.sandbox.xyz.net esignet.sandbox.xyz.net prereg.sandbox.xyz.net healthservices.sandbox.xyz.net
  <INTERNAL_IP_OF_MOSIP_NGINX_NODE>  sandbox.xyz.net api-internal.sandbox.xyz.net activemq.sandbox.xyz.net kibana.sandbox.xyz.net regclient.sandbox.xyz.net admin.sandbox.xyz.net minio.sandbox.xyz.net iam.sandbox.xyz.net kafka.sandbox.xyz.net postgres.sandbox.xyz.net pmp.sandbox.xyz.net onboarder.sandbox.xyz.net smtp.sandbox.xyz.net compliance.sandbox.xyz.net
  
  ## Observation 
  <INTERNAL_IP_OF_OBS_NGINX_NODE>    rancher.xyz.net keycloak.xyz.net
  fallthrough
}

To update the coredns configmap, use the below command.
Check whether the DNS changes are correctly updated in coredns configmap.
Restart the coredns pod in the kube-system namespace.
Check status of coredns restart.

Storage class setup: Longhorn creates a storage class in the cluster for creating pv (persistence volume) and pvc (persistence volume claim).

Pre-requisites:
```
cd $K8_ROOT/longhorn
./pre_install.sh
```
Install Longhorn via helm
- ./install.sh
- Note: Values of below mentioned parameters are set as by default Longhorn installation script:
  - PV replica count is set to 1. Set the replicas for the storage class appropriately.
  - Total available node CPU allocated to each instance-manager pod in the longhorn-system namespace.

You will get the kubectl commands to be executed in the kubernetes cluster. Copy the command and execute from your PC (make sure your kube-config file is correctly set to MOSIP cluster).

e.g.:
kubectl apply -f https://rancher.e2e.mosip.net/v3/import/pdmkx6b4xxtpcd699gzwdtt5bckwf4ctdgr7xkmmtwg8dfjk4hmbpk_c-m-db8kcj4r.yaml

Install Pre-requisites

sudo apt update -y
sudo apt-get install software-properties-common -y
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update -y
sudo apt-get install python3.8 -y
sudo apt install letsencrypt -y
sudo apt install certbot python3-certbot-nginx -y

Openssl : Generate wildcard ssl certificate using openssl in case you don't have public DNS using below steps. (Ensure to use this only in development env, not suggested for Production env).

Install docker on nginx node.

sudo apt-get update --fix-missing
sudo apt install docker.io -y
sudo systemctl restart docker

Generate a self-signed certificate for your domain, such as *.sandbox.xyz.net.
Execute the following command to generate a self-signed SSL certificate. Prior to execution, kindly ensure that the environmental variables passed to the OpenSSL Docker container have been properly updated:
Above command will generate certs in below specified location. Use it when prompted during nginx installation.
- fullChain path: /etc/ssl/certs/nginx-selfsigned.crt.
- privKey path: /etc/ssl/private/nginx-selfsigned.key.

Install nginx:

Clone k8s-infra

cd $K8_ROOT/mosip/on-prem/nginx
sudo ./install.sh

Provide below mentioned inputs as and when prompted
- MOSIP nginx server internal ip
- MOSIP nginx server public ip
- Publically accessible domains (comma separated with no whitespaces)
- SSL cert path

nano /etc/nginx/nginx.conf

server{
   listen <cluster-nginx-internal-ip>:80;
   server_name iam.sandbox.xyz.net;
   location /auth/realms/mosip/protocol/openid-connect/certs {
      proxy_pass                      http://myInternalIngressUpstream;
      proxy_http_version              1.1;
      proxy_set_header                Upgrade $http_upgrade;
      proxy_set_header                Connection "upgrade";
      proxy_set_header                Host $host;
      proxy_set_header                Referer $http_referer;
      proxy_set_header                X-Real-IP $remote_addr;
      proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header                X-Forwarded-Proto $scheme;
      proxy_pass_request_headers      on;
      proxy_set_header  Strict-Transport-Security "max-age=0;";
   }
   location / { return 301 https://iam.sandbox.xyz.net; }
}

Note: HTTP access is enabled for IAM because MOSIP's keymanager expects to have valid SSL certificates. Ensure to use this only for development purposes, and it is not recommended to use it in production environments.
Restart nginx service.

Check Overall if nginx and istio wiring is set correctly

Install httpbin: This utility docker returns http headers received inside the cluster. You may use it for general debugging - to check ingress, headers etc.
```
cd $K8_ROOT/utils/httpbin
./install.sh
```

To see what is reaching the httpbin (example, replace with your domain name):

curl https://api.sandbox.xyz.net/httpbin/get?show_env=true
curl https://api-internal.sandbox.xyz.net/httpbin/get?show_env=true

Click on Install.

cd $K8_ROOT/monitoring/alerting/
nano alertmanager.yml

Update:

global:
  resolve_timeout: 5m
  slack_api_url: <YOUR-SLACK-API-URL>
...
...
receivers:
- name: slack
  slack_configs:
  - channel: '<YOUR-CHANNEL-HERE>'
    send_resolved: true

cd $K8_ROOT/rancher/on-prem
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--version 4.0.18 \
--create-namespace  \
-f ingress-nginx.values.yaml

./install.sh
Do you have public domain & valid SSL? (Y/n) 
Y: if you have public domain & valid ssl certificate
n: If you don't have a public domain and a valid SSL certificate. Note: It is recommended to use this option only in development environments.

INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.0.0.1]
INFO[0000] [network] Deploying port listener containers   
INFO[0000] [network] Pulling image [alpine:latest] on host [10.0.0.1]
...
INFO[0101] Finished building Kubernetes cluster successfully

INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.0.0.1]
INFO[0000] [network] Deploying port listener containers
INFO[0000] [network] Pulling image [alpine:latest] on host [10.0.0.1]
...
INFO[0101] Finished building Kubernetes cluster successfully

docker volume create --name gensslcerts --opt type=none --opt device=/etc/ssl --opt o=bind
docker run -it --mount type=volume,src='gensslcerts',dst=/home/mosip/ssl,volume-driver=local \
-e VALIDITY=700        \
-e COUNTRY=IN          \
-e STATE=KAR           \
-e LOCATION=BLR        \
-e ORG=MOSIP           \
-e ORG_UNIT=MOSIP      \
-e COMMON_NAME=*.sandbox.xyz.net \
mosipdev/openssl:latest

 <PUBLIC_IP>    api.sandbox.xyz.net resident.sandbox.xyz.net esignet.sandbox.xyz.net prereg.sandbox.xyz.net healthservices.sandbox.xyz.net
 <INTERNAL_IP>  sandbox.xyz.net api-internal.sandbox.xyz.net activemq.sandbox.xyz.net kibana.sandbox.xyz.net regclient.sandbox.xyz.net admin.sandbox.xyz.net minio.sandbox.xyz.net iam.sandbox.xyz.net kafka.sandbox.xyz.net postgres.sandbox.xyz.net pmp.sandbox.xyz.net onboarder.sandbox.xyz.net smtp.sandbox.xyz.net compliance.sandbox.xyz.net

export KUBECONFIG="$HOME/.kube/<cluster_name>_config

INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.0.0.1]
INFO[0000] [network] Deploying port listener containers   
INFO[0000] [network] Pulling image [alpine:latest] on host [10.0.0.1]
...
INFO[0101] Finished building Kubernetes cluster successfully
```
* The last line should read `Finished building Kubernetes cluster` successfully to indicate that your cluster is ready to use.

INFO[0000] Building Kubernetes cluster
INFO[0000] [dialer] Setup tunnel for host [10.0.0.1]
INFO[0000] [network] Deploying port listener containers
INFO[0000] [network] Pulling image [alpine:latest] on host [10.0.0.1]
...
INFO[0101] Finished building Kubernetes cluster successfully

V3 installation

1.2.0.3

Overview

How does the rapid deployment model improve upon previous approaches in terms of DevOps modernization?

Where and how do I start?

How it works (High-level Overview)

Complete Deployment Flow

Documentation

1.2.0.2

Overview and Architecture

Overview

Architecture diagram

Deployment repos

Pre-Requisites

Hardware requirements

Network requirements

DNS requirements

Certificate requirements

Personal computer requirements

Testrig

Authdemo Service

MOSIP External Dependencies

MOSIP Modules Deployment

V3 installation

1.2.0.2

Overview and Architecture

Overview

Architecture diagram

Deployment repos

1.2.0.3

Overview

How does the rapid deployment model improve upon previous approaches in terms of DevOps modernization?

Where and how do I start?

How it works (High-level Overview)

Complete Deployment Flow

Documentation

MOSIP External Dependencies

Testrig

Authdemo Service

Installation

Postgres

Keycloak

Setup SoftHSM

Setup Object store

MinIO installation

S3 Credentials setup

ClamAV setup

ActiveMQ setup

Kafka setup

BioSDK Server setup

ABIS

MSG Gateway

Docker Secrets

Captcha

Landing page setup

API-testrig

Packetcreator

DSLrig/ DSLOrchestrator

Pre-Requisites

Hardware requirements

Network requirements

DNS requirements

Certificate requirements

Personal computer requirements

MOSIP Modules Deployment

Installation

AWS Deployment

1. Wireguard

2. Observation K8s Cluster setup

3. Observation K8s Cluster’s Ingress and Storage class setup

4. Domain name mapping

5. Rancher K8s Cluster Apps Installation

6. MOSIP K8s Cluster setup

7. Import Mosip Cluster into Rancher UI

8. MOSIP K8 Cluster Global configmap, Storage Class, Ingress and Loadbalancer setup

9. Monitoring Module deployment

10. Alerting Setup

11. Logging Module Setup and Installation

12. Mosip External Dependencies setup

13. MOSIP Modules Deployment