MOSIP is deployed as a combination of microservices. MOSIP deployment as mircroservices adds more scalability, robustness, resilience and fault tolerance. MOSIP services are grouped as multiple modules like Kernel, prereg, regproc etc. These modules can be deployed independently with all its dependencies as per the business needs.

MOSIP releases the below mentioned artifacts in open source:

• Jars : Artifacts released into Maven Central Repository after successful compilation of tagged code from all Mosip repositories.

• Docker images : Docker images for all MOSIP services.

• MOSIP helm: Packaged charts for every MOSIP service.

  helm repo add mosip https://mosip.github.io/mosip-helm

• K8s Infra: Scripts to create kubernetes cluster and configure it for MOSIP.

• Deployment scripts: These scripts are a part of mosip-infra to install the helm charts in a desired sequence.

MOSIP installations

MOSIP by default supports two ways of installation:

• V2 - A simple sandbox-based implementation with very low resources to help understand the working modules (deprecated).

• V3 - A scalable deployment with a service mesh, HA and other security protection (supported from v1.2.0.1-B1).

Apart from these installation models, countries can adopt a model of their choice. We recommend using Kubernetes or equivalent container orchestration tools for better management of the microservices.

Welcome to the Setup section! This section will help you get started with installing, configuring, and maintaining the system. Whether you’re deploying for the first time, implementing specific configurations, or upgrading an existing setup, you’ll find step-by-step instructions in the following sections:

Overview

V3 installation uses Wireguard for secure access. Refer to the following documents for further details:

Overview

A Wireguard bastion host (Wireguard server) provides secure private channel to access MOSIP cluster. The host restrics public access, and enables access to only those clients who have their public key listed in Wireguard server. Wireguard listens on UDP port51820.

Server installation

• Install docker, and make sure you add $USER to docker group:

• If you already have a config file you may mount it with -v <your host path>:/config.

• You may increase the number of peers keeping the above mounted folders intact, stopping the docker and running it again with -e PEERS=<number of peers>

Client install

• Install a Wireguard app on your machine. For MacOS there is a Wireguard app on the App Store.

• Enter the server docker and cd to /config folder. Here you will find the config files for peers. You may add the corresponding peer.conf file in client Wireguard config.

• Make sure Endpoint mentioned for the client is Wireguard bastion hosts' IP adddress.

• Modify the Allowed IPs of the client to private IP addresses for Internal Load Balancers of your clusters. Here, we assumed that all your clusters are running in the same VPC so that bastion host is able to reach all of them.

The following versioning conventions are followed for repos related to deployment:

• • MOSIP release version: w.x.y.z. Example 1.2.0.1

  • Helm chart version: wx.y.z. Example 12.0.1 (as Helm follows 3 digit versioning).

  • In case of develop branch of mosip-helm version in Chart.yaml points to next planned release version of MOSIP (as Helm does not allow version like develop).

• Helm charts contain default compatible docker image tag.

• appVersion field in Charts.yaml is not used.

Glossary

Production installation

Production Grade Deployment V3

Hardware requirements

• Server Hardware Requirements

• Devices Calculator

Production hardening

Production Hardening Guide

Below table lists various checks that must be performed before actual roll out of a deployment. This list is not exhaustive and it is expected that SIs use this as a reference and augment their own hardening procedures.

Cluster administration

In V3 installation cluster can be administered by logging into organisation wide Rancher setup. Rancher is integrated with Keycloak for authentication. To provide cluster access to a user perform the following steps as administrator:

1. Login into organisation wide Keycloak e.g https://iam.xyz.net. It is assumed that you have admin role in Keycloak.

2. Create a new user.

3. Make sure a strong password is set for the same under Credentials tab.

4. On Details tab you should see Update Password flag under Required User Actions. This will prompt a user to change the password during first login. Disable the same only if you are sure you don't want user to change password.
5. Login to Rancher as administrator, e.g. https://rancher.xyz.net.

6. Select a cluster for which you would like to enable access to the user.

7. Add the user as member of the cluster.

8. Assign a role, e.g Cluster Owner, Cluster Viewer.

Overview

MOSIP deployment is split into two distinct parts:

1. ID lifecycle management

   • Pre-registration

   • Registration

2. ID Authentication

The server-side hardware estimates for the above are specified at a high level in terms of compute (Virtual CPU, RAM) and storage requirements. We provide estimates for MOSIP core modules only. External components are not in the scope. See Exclusions.

The variables that largely determine the hardware requirements are:

1. The population of the country

2. Rate of enrolment

3. Usage of foundation ID by various services

Pre-registration

Refer to Pre-registration Resource Calculator XLS

Allow for 20% additional compute and storage for monitoring and any overheads.

Registration (enrolment)

The registration compute resources are related to the max rate of enrolment desired. The processing throughput must match the enrolment rate to avoid a pile-up of pending registration packets.

The data here is based on actual field data of a MOSIP deployment.

Assumptions:

• Rate of enrolment: 216000 per day

• Average packet size: 2MB

• Biometric modalities: Finger, iris, face

• Pod replication as given here. (TBD)

Compute requirements for registration

• Configuration of compute node: 12 VCPU, 64GB RAM, 64GB disk store.

• Number of nodes: 21

Storage requirements for registration

Storage is dependent on the population of a country (i.e. the number of UINs to be issued). Storage requirements for various types of data are listed below.

Allow for 20% additional compute and storage for monitoring and any overheads.

ID authentication

Refer to IDA Resource Calculator XLS

Allow for 20% additional compute and storage for monitoring and any overheads.

Exclusions

The compute and storage estimates for the following components are not included:

Authdemo Service

• The Authdemo service is utilized to execute the IDA APIs that are employed by API-testrig and DSLrig.

• The purpose of the Authdemo Service is to showcase the functionality of authentication.

• It can be considered as a simplified iteration of an authentication service, serving as a mock or prototype for the purpose of testing.

• When prompted, input the NFS host, it's pem-key, ssh login user of NFS server.

• Install script will create the NFS directories i.e., /srv/nfs/mosip/packetcreator-authdemo-authcerts to store the certificates generated by Authdemo service.

These certificates will be used by API-testrig, orchestrator and packetcreator.

API-testrig

API-testRig tests the working of APIs of the MOSIP modules.

MOSIP’s successful deployment can be verified by comparing the results of API-testrig with the testrig benchmark.

• When prompted, input the hour of the day to execute the API-testrig.

• Daily API-testrig cron job will be executed at the very opted hour of the day.

• The reports will move to the object store ( i.e., s3/minio) under automationtests bucket.

Packetcreator

Packetcreator will create packets for DSL orchestrator.

Note: It is recommended to deploy the packetcreator on a separate server/ cluster from where the other DSL orchestrators can access this service.

• When prompted, input the NFS host, it's pem-key, ssh login user of NFS server.

• Install script will create two NFS directories i.e., /srv/nfs/mosip/packetcreator_data, /srv/nfs/mosip/packetcreator-authdemo-authcerts.

• Packetcreator_data contains biometric data which is used to create packets.

• Copy the packetcreator_data from the link mentioned above to the NFS directory /srv/nfs/mosip/packetcreator_data.

• Ensure to use the same NFS host and path i.e., /srv/nfs/mosip/packetcreator-authdemo-authcerts for Authdemo and packetcreator service.

• When prompted, input the kubernetes ingress type (i.e., Ingress/Istio) and DNS as required if you are using the Ingress-nginx.

DSLrig/ DSLOrchestrator

• DSLrig will test end-to-end functional flows involving multiple MOSIP modules.

• The Orchestrator utilizes the Packet Creator to generate packets according to the defined specifications. It then communicates with the Authdemo Service making REST API calls to perform authentication-related actions or retrieve the necessary information.

• When prompted, input the NFS host, it's pem-key, ssh login user of NFS server.

• Install script will create NFS directories, i.e., /srv/nfs/mosip/dsl-scenarios/sandbox.xyz.net to store the DSL scenario sheet.

• Copy the scenario csv from the above link to the NFS directory /srv/nfs/mosip/dsl-scenarios/sandbox.xyz.net. Make sure to rename the csv files by replacing env with your domain ex: sandbox.xyz.net.

• To run the dslorchestrator for sanity only, update the dslorchestrator configmap TESTLEVEL key to sanity.

• The reports will move to object store (i.e., s3/minio) under dslreports bucket.

External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, and HSM etc.

List of external dependencies:

• Event Publisher/ streamer: MOSIP uses Kafka for publishing events to it's internal as well as external partners modules.

• BioSDK: Biometric SDK for quality check and authentication purpose using biometrics.

• Message Gateway: This is for notifying residents about different OTPs and other information.

Installation

Postgres

• Install Postgres

• Initialize Postgres DB

Opt for yes and enter Y.

Keycloak

• Install Keycloak

• Initialize Keycloak

Setup SoftHSM

Setup Object store

• Opt 1 for MinIO

• Opt 2 for S3 (incase you are not going with MinIO installation and want s3 to be installed)

  • Enter the prompted details.

MinIO installation

S3 Credentials setup

ClamAV setup

ActiveMQ setup

Kafka setup

BioSDK Server setup

Reference implementation of Biometric SDK server will be installed separately in MOSIP service installation section as the same is dependent on artifactory which is a MOSIP component.

ABIS

ABIS is needed to be up and running outside the MOSIP cluster and should be able to connect to the activeMQ. For testing purpose, MOSIP has provided a mock stimulator for the same named as mock-abis which will be deployed as part of the MOSIP services installation.

MSG Gateway

• MOSIP provides mock smtp server which will be installed as part of default installation, opt for Y.

Docker Secrets

Incase the images are getting pulled from private repositories.

Captcha

To setup the captcha for pre-reg and resident domains.

Landing page setup

Below is the sequence of installation of MOSIP modules and the sequence must be followed to resolve all interdependencies.

Installation

• Conf secrets

• Config Server

• Artifactory

• Keymanager

• WebSub

• Mock-SMTP

• Kernel

• Masterdata-loader

• Mock-biosdk

• Packetmanager

• Datashare

• Pre-reg

• Idrepo

• Partner Management Services

• Mock ABIS

• Mock-mv

• Registration Processor

• Admin

• ID Authentication

• Print

• Partner Onboarder

• MOSIP File Server

• Resident services

• Registration Client

This document outlines the steps required for migrating the deployment architecture from V2 to V3.

Step 1: New environment setup with V3 Architecture

This is required for migration from V2 to V3 architecture

1. Make sure to have all the pre-requisites ready as per the details present in the section pre-requisites

2. Setup Wireguard Bastion host

3. Setup wireguard client in your local and complete the configuration

4. Setup Observation K8 cluster

5. Configure Observation k8 cluster

6. Observation cluster’s nginx setup

7. Observation cluster applications setup

8. Observation cluster keycloak-rancher integration

9. Setup new MOSIP k8 cluster

10. MOSIP k8 cluster configuration

11. MOSIP cluster nginx setup

12. Setting up Monitoring for MOSIP cluster

13. Setting up Alerting for MOSIP cluster

14. Setting up Logging for MOSIP cluster

Step 2: Deployment of external services

(Required for V2 to V3 architecture migration)

1. Setup postgres server

   Note:

   i. Deploy postgres server in a seperate node.

   ii. Make sure postgres initialisation is not done (only install postgres).

2. Setup Keycloak server

   Note: Make sure keycloak initialisation is not done (only install keycloak).

3. Setup Softhsm

4. Setup Minio server

5. Setup ClamAV

6. Setup ActiveMQ

7. Setup Message Gateway

8. Setup docker registry secrets if you are using private dockers.

   Note: These instructions are only applicable if you need to access Private Docker Registries. You may disregard them if all of your Docker containers are downloaded from the public Docker Hub.

9. Setup Captcha for the required domains.

10. Setup Landing page for new MOSIP cluster.

Step 3: Backup and restoration of external services

This step is required for V2 to V3 architecture migration.

1. Softhsm (only required if softhsm is used instead of real HSM)

   i. Backup keys

   ii. Restore old key

   iii. Update softhsm ida and softhsm kernel security pin

2. Postgres

   i. Export

   ii. Import

   iii. secret creation

   iv. Increase postgres max_connections to 1000

3. Keycloak

   i. Export

   ii. Import

4. Minio

   i. Export the existing Minio as directory

   ii. Clone Minio

5. Kafka

   i. setup external minio for backup.

   ii. backup kafka

   iii. restore kafka

6. Conf-secrets

Update the secrets in existing secrets in conf-secrets namspace.

1. Packets in landing to be copied from old environment to the upgraded environment or same NFS folder can be mounted to regproc packet server and group 1 stage groups. Refer here for more details.

• dmz-sc.yaml

• dmz-pkt-pv.yaml

• dmz-pkt-pvc.yaml

• dmz-landing-pv.yaml

• dmz-landing-pvc.yaml

Overview

Reference implementations of modules or components are non production grade implementations that are meant to showcase a reference design or architecture. They can be used as references or starting points for customization and actual implementations.

Web portals

1. Pre-registration portal

2. Admin portal

3. Partner Management portal

Desktop apps

1. Authentication Application

2. Registration Client

Common components

1. ID object validator (kernel-ref-idobjectvalidator)

2. Integration with the SMS service provider (kernel-smsserviceprovider-msg91)

3. Integration with a Virus Scanner (kernel-virusscanner-clamav)

4. HSM Keystore Implementation (hsm-keystore-impl)

5. IAM: Keycloak

Registration Processor

1. External Stage

2. Demo deduplication

3. Hazelcast Cache Provider

ID Authentication

1. Demographic authentication (normalization)

2. Child Authentication Filter

Pre-registration

1. Booking Service

Print Service

Overview

This is a guide to implement MOSIP for a country. It is advised that Government and System Integrators (SI) study the recommended steps to work out an appropriate implementation strategy. The items are in "near-chronological order" and may differ for an implementation.

Key decisions

1. Choice of deployment of Pre-registration.

2. Rate of enrolment desired.

3. Rate of authentication expected.

4. Languages.

5. Customisation and procurement of components as given here.

6. ID schema (as prescribed by the country's regulatory authority).

7. Hardware requirements estimate.

   • Server side

   • Devices

8. Credential choices.

9. ID Card print design.

10. MOSIP versions.

11. MOSIP support (scope).

12. Disaster recovery strategy.

13. A phased approach for rollout.

Procurement

1. Engagement with an SI - terms and conditions.

2. Procurement of biometrics and other external components.

3. HSM

4. Postgres

5. Object store

6. Computer hardware

Country specific data

1. Masterdata

2. Applicanttype MVEL

Customisation

1. Customisation of components as decided in step 5 of Key decisions above.

Deployment

1. Certifcate/key generation

2. Deployment design.

3. Disaster recover set up

4. Biometric thresholding

5. Phased implementation

6. Sandbox, staging, development setups.

7. IDA installation

Partners

1. Onboarding of trusted partners

2. Print partner

Rollout

1. Set up of registration centers

2. Onboarding of officers and supervisors

3. Training

MOSIP periodically releases new versions of the platform, tagged with their version numbers.

Asymmetric Amoeba is the latest stable Long Term Support (LTS 1.2.0) version. This release focuses on easy manageability, usability, enhanced performance, robustness, security, inclusivity, and comprehensive documentation. Additionally, multiple languages are now supported across modules.

LTS versions get patch releases with minor and major updates, as and when required.

SUPPORTED FOR A MINIMUM OF 5 YEARS

LTS 1.2.0 was released in February 2022 and will be supported at the minimum until February 2027.

What happens after five years?

• Support will be available for migration to the next LTS version for two years following its release.

• Adopter inputs and experiences will be factored in to fine-tune subsequent versions.

Benefits of Long Term Support

LTS releases offer:

• Completely implemented roadmap features

• Frozen API and data formats

• Tooling, add-ons and extensions

• Compatible components and solutions in the marketplace

• Compliance and certification programs

• Active support

  • Proactive security updates

  • Patches for bugs

  • Periodical cumulative updates of functional, non-functional fixes and patches

  • Support for ecosystem partners for integration and implementation

• Additional support to adopting countries under MOU for versions under active support

  • L3 support to adopting countries

  • Training and capacity building

  • Technical advisory on ID and use case implementations

What’s New in LTS 1.2.0

MOSIP's current LONG TERM SUPPORT RELEASE is v1.2.0. The section below highlights the key features, benefits, and new modules added.

Functional Benefits

• New Admin UI with robust APIs

• New Partner Management Portal UI with robust APIs

• New Resident Portal with robust APIs

Other Benefits

• Enhanced security

• Finer documentation

• Enhanced mechanism to evaluate performance

• Improved service-level performance

• Standalone stages of registration processing

Tools & Add-ons

• Anonymous profiling to cater to analytical needs

• Improved reporting for better forecasting and efficient decision-making

• V3 Deployment Architecture

• Dockerized test automation

Newer Modules Compatible with LTS

• eSignet

• Inji

• OpenG2P

• OpenCRVS

• Compliance Tool Kit

• Android RegClient

The existing adopters of pre-LTS stable versions can:

• Get access to the full feature set, latest tools, and add-on modules

• Get access to the best tier of support

• Get access to periodic updates and fixes

• Utilise the upgrade window to avoid falling into unsupported mode

Upgrading to LTS 1.2.0

Standard Procedure for LTS Migration

1. Discussing migration and communication strategies (relying parties, stakeholders etc.)

2. Prioritising a list of issues that must be fixed before migration

3. Understanding and analysing, in detail, latest changes and customised features

4. Identifying sequencing of components and infra to be migrated

5. Discussing ways and means to automate specific steps

6. Migrating using upgrade scripts: DB, template, config, seed data, etc.

   • Phased migration: sandbox, staging, production

7. Hardening security

8. Marking off checklist template items

Note:

• The time and effort involvement depends on customisation requirements.

• Without customisations, the time needed would be about 2-3 months.

  • Time to deploy: 2 weeks

  • Time to execute upgrade scripts: 2-3 weeks

  • Time taken to test/ verify:

    • Two days for environment sanity using test rig automation

    • 3-4 weeks for full blown testing for all the modules

For detailed information on all the enhancements in the LTS 1.2.0 version, refer the Release Notes.

See below for a brief overview of features and newly-added modules.

Admin Services

• LTS 1.2.0 features a new full-blown admin console with robust APIs

  • Admin application is a web-based application used by a privileged group of administrative personnel to manage various master data and resources like centres, devices, machines, users.

  • Along with the resource and data management, the admin can generate master keys, check registration status, retrieve lost RID, resume processing of paused packets etc.

To know more, read through the Admin Portal User Guide.

Pre-registration and Registration

• A control panel has been added in the Registration Client for biometric and non-biometric devices and an for configurable settings. These settings gives the operator/supervisor better control of the system.

• The Registration Client is now capable of handling multiple time zone including the local time zone.

• A feature to retrieve lost RID has been added.

To know more, read the Registration Client Settings page.

ID Authentication

• Authentication filters have been added to ID Authentication.

• IDA is externalized through data sync via WebSub. This allows the relying parties to create their own IDA module or extend the existing IDA module.

• The ID Authentication module has been enhanced to retrieve any missing data such as credentials, partner data, and policies, in the event of a crash or time-out.

• The IDA database now stores the partner details, which includes data like the partner ID, policy ID, MISP ID, and API keys, to authenticate the above entities directly from IDA.

Partner Management

A new Partner Management Portal is available with robust APIs. This portal can be used by the partner admin, policy admin, and various MOSIP partners like authentication partners, device providers, FTM providers, and credential partners.

The partner portal has the following features:

• Self-registration of partners

• Partner-specific certificate upload

• Partner can add device models, FTM models, policy mapping, SBI details

• Partner and policy admins can approve/reject partner requests, create policies, and can add MOSIP-compliant CA (certificate authorities)

To know more, read through the Partner Management Portal User Guide.

Resident Services

• Resident portal is an online ID management system where the residents can avail services like download their UIN card, view history, lock/ unlock authentication types, generate/revoke VID, grievance redressal etc.

• To know more, read through Resident Services.

Hassle-free Upgrades

• Upgrade scripts are available that help in hassle free migration from 1.1.5.x to 1.2.0.x. These scripts will include the db upgrade scripts, template upgrade scripts, config upgrade scripts, MOSIP seed data upgrade scripts etc.

Deployment Architecture

• In LTS, support for legacy V2 deployment has been deprecated and the updated deployment method is V3 deployment that promises:

  • Enhanced security

  • High availability (owing to better load balancing)

  • Highly recommended for production

  • Performance enhancements

To learn more, read through Deployment.

Improved Reporting

• LTS 1.2.0 provides a reporting framework for real-time streaming of data visualization on dashboards that gives a visual display of metrics and important data to track the status of various pre and post-enrolment processes. This data helps the ID issuers in improving efficiency, forecasting, and better decision-making. The framework has been used to create a set of default dashboards using Kibana.

• To know more, read through Reporting.

Anonymous Profiling

• We have curated a data set called anonymous profile to cater to the analytic needs of the adopting countries which will help to assess the progress of ID programs.

• It is accessible to search engines such as elasticsearch.

• To know more, read through the Anonymous Profiling Support.

Test Automation

• We now have a dockerized API test automation as opposed to the jar file execution due to which the entire process of automation has become faster owing to

  • Cron jobs that handle daily automation reports eliminating any need for manual trigger for test report generation.

  • Automatic language, environment, and secret key configuration.

To know more, read through Automation Testing.

Enhanced Security

• Security should be built-in and not bolt-on. Taking our security checks to the next level, we completed a security audit as part of MOSIP 1.2.0 LTS (Asymmetric Amoeba), which was certified by Aujas.

• The components have been significantly tested for scale and performance. The adopting countries can now cater to their millions of customers with confidence.

Finer Documentation

• Documentation has evolved, thereby making a huge difference in resolving the issues of accessibility and assistive technology.

• The availability of comprehensive and well-devised user guides for all modules has helped the community to move one step closer towards simplified ways of working autonomously.

Performance Enhancements

• With the LTS 1.2.0 version, performance has been significantly improved.

• To know more, read through Performance Test Reports.

Newer Modules Compatible with LTS

eSignet

• eSignet allows easy login to any of the government services using a single credential and passwordless login using the supported authentication factors

• To know more, read through the eSignet documentation.

Inji

• A safe, trusted & inclusive mobile wallet and authenticator, that enables you to carry your digital IDs, prove your presence, (offline and online), and avail services in a snap.

• To know more, read through the Inji documentation.

Compliance Tool Kit (CTK)

• the Compliance Tool Kit (CTK) is an online portal that can be used by MOSIP partners to test the compliance of their product developed as per specifications (specs) published/adopted by MOSIP.

• To know more, read through Compliance Tool Kit documentation.

OpenG2P

• OpenG2P is an open source platform upon which government-to-person (G2P) solutions can be built.

• The platform offers people facing processes such as onboarding into schemes, identity verification, and cash transfers to their bank accounts along with a self-serviced beneficiary portal.

• It also incorporates the government department facing features such as creation of registries and beneficiary lists, eligibility checks, scheme definition, payment disbursement and reconciliation.

• To know more, read through OpenG2P documentation.

OpenCRVS

• OpenCRVS is a digital platform for recording a person's major life events like births, deaths, marriages, and divorces.

• It is a customisable open source solution designed for civil registration and it's essential services such as social protection, health care, education and economic and social opportunities.

Version: 1.2.0.1 (Latest stable release)

• Release date: 19th March 2024

• Release notes

• On-Prem Installation Guidelines

• On-Prem without DNS Installation Guidelines

• AWS Installation Guidelines

Version: 1.2.0.1-B4

• Release date: 12th January 2024

• Release notes

• On-Prem Installation Guidelines

• On-Prem without DNS Installation Guidelines

• AWS Installation Guidelines

Version: 1.2.0.1-B3

• Release date: 14th April 2023

• Release notes

• On-Prem Installation Guidelines

• On-Prem without DNS Installation Guidelines

• AWS Installation Guidelines

Version: 1.2.0.1-B2

• Release date: 8th Jan 2023

• Release notes

• On-Prem Installation Guidelines

Version: 1.2.0.1-B1

• Release date: 14th Oct 2022

• Release notes

• Deployment Instructions

MOSIP has recently announced the release of its latest version, 1.2.0.1. This new release brings about an architectural upgrade and addresses multiple bugs. To assist users in migrating from the MOSIP version 1.1.5.x to the latest version 1.2.0.1, a detailed Runbook has been provided, offering comprehensive guidance.

To set up the new environment and deploy the upgraded version of MOSIP, carefully follow the procedures outlined below step-by-step.

Points to Consider

• The migration will only be supported if the latest version deployed is 1.1.5.x (mostly 1.1.5.5-P1 and above). Countries should ensure that MOSIP is updated to version 1.1.5.x before migrating to the newer version.

• Once the upgrade is completed, all the modules running on the server will be upgraded together.

• The IDA Database will not have any entries from version 1.1.4.

• Migration of websub event is not supported for countries transitioning from db-based websub to kafka-based websub.

• After migration, if a new machine is added for the Registration Client, it must be installed with the latest version of the Registration Client. We will not provide support for the old version.

• Partners who decrypt data encrypted by the key manager will support decryption with thumbprint support after upgrading to version 1.2.0.1. Therefore, backward support to encrypt without thumbprint is not available.

• Prior to migration, countries should prioritize packets stuck in between stages using the registration processor and complete the processing.

• Any third-party subsystems such as Manual adjudication/ABIS will not respond after migration for a request received before migration. Therefore, it is suggested that all subsystems, such as ABIS and manual adjudication, consume all the messages from ActiveMQ, complete the processing, mark all for reprocess, and respond back to registration processor before migration.

• Default resource allocation (CPU and memory) has been added to all the pods in version 1.2.0.1, so additional nodes may be required after the upgrade.

• The rollback process during the migration needs to be handled by the respective system integrators (SI).

• All the server services and registration clients will be running version 1.1.5.x.

• Registration packets available in MinIO and the landing zone can be version 1.1.4.x / 1.1.5.x.

• Existing Grafana and Kibana reports from version 1.1.5 are not required to be viewable in the upgraded environment.

• If there are additional attributes in the data share for manual adjudication, the manual adjudication system will ignore them and not fail.

• The first global admin user login does not need to be handled during the upgrade.

• Out of scope for migration:

  • Configuration of correction flow and related changes are not included in this migration.

  • Anonymous profiles for existing data are not handled in this upgrade.

  • Support for the Lost RID/AID search feature for packets synced before the migration is out of scope.

Upgrade Process

This comprehensive upgrade process entails the deployment architecture upgrade from V2 to V3, as well as the MOSIP platform upgrade from version 1.1.5.5-P1 to 1.2.0.1. The various tasks involved in this process are organized into the following categories.

Upgrade of Deployment Architecture from V2 to V3

1. Installation and configuration of new environment with V3 architecture.

2. Deployment of external services.

3. Backup and restoration of external services.

Upgrade of Platform from version 1.1.5.5-P1 to 1.2.0.1

1. Upgrade of necessary external services.

2. Migration of properties.

3. Upgrade of MOSIP services.

4. Execution of activities once all upgraded services are operational.

5. Carrying out activities after completion of initial round of testing.

Let us go through the processes discussed above in detail.

Overview

• MOSIP modules are deployed in the form of microservices in a Kubernetes cluster.

• Wireguard is used as a trust network extension to access the admin, control, and observation pane

• It is also used for on-the-field registrations.

• MOSIP uses AWS load balancers for:

  • SSL termination

  • Reverse Proxy

  • CDN/Cache management

  • Loadbalancing

• Kubernetes cluster is administered using the Rancher and EKS

• In V3, we have two Kubernetes clusters:

  • Observation Cluster - This cluster is a part of the observation plane and it helps in administrative tasks. By design, this is kept independent of the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or its services should be internal and should never be exposed to the external world.

    • Rancher is used for managing the Mosip cluster.

    • Keycloak in this cluster is used for cluster user access management.

    • It is recommended to configure log monitoring and network monitoring in this cluster.

    • In case you have a internal container registry, then it should run here.

  • MOSIP Cluster - This cluster runs all the MOSIP components and certain third party components to secure the cluster, API’s and Data.

    • MOSIP External Components

    • Mosip Services

Deployment Repos

• k8s-infra : contains scripts to install and configure Kubernetes cluster with required monitoring, logging and alerting tools.

• mosip-infra : contains deployment scripts to run charts in defined sequence.

• mosip-config : contains all the configuration files required by the MOSIP modules.

• mosip-helm : contains packaged helm charts for all the MOSIP modules.

Pre-requisites:

Hardware Requirements

VM’s required have any Operating System and can be selected as per convenience. In this installation guide, we are referring to Ubuntu OS throughout.

Network Requirements

• All the VM's should be able to communicate with each other.

• Need stable Intra network connectivity between these VM's.

• All the VM's should have stable internet connectivity for docker image download (in case of local setup ensure to have a locally accesible docker registry).

• During the process, we will be creating two loadbalancers as mentioned in the first table below:

• Server Interface requirement as mentioned in the second table:

DNS Requirements

Note:

• Only proceed to DNS mapping after the ingressgateways are installed and the load balancer is already configured.

• The above table is just a placeholder for hostnames, the actual name itself varies from organisation to organisation.

Certificate requirements

As only secured https connections are allowed via nginx server, you will need the below mentioned valid ssl certificates:

• One valid wildcard ssl certificate related to domain used for accesing Observation cluster which will be created using ACM (Amazon certificate manager). In above e.g. *.org.net is the similiar example domain.

• One valid wildcard ssl certificate related to domain used for accessing MOSIP cluster which will be created using ACM (Amazon certificate manager). In above e.g. *.sandbox.xyz.net is the similiar example domain.

Prerequisite for complete deployment in Personal Computer

• kubectl client version 1.23.6

• helm client version 3.8.2 and add below repos as well :

  • Copy

    helm repo add bitnami https://charts.bitnami.com/bitnami
    helm repo add mosip https://mosip.github.io/mosip-helm

• istioctl : version: 1.15.0

• eksctl : version: 0.121.0

• AWS account and credentials with permissions to create EKS cluster.

• AWS credentials in ~/.aws/ folder as given here.

• Save ~/.kube/config file with another name. (IMPORTANT. As in this process your existing ~/.kube/config file will be overridden).

• Save .pem file from AWS console and store it in ~/.ssh/ folder. (Generate a new one if you do not have this key file).

• Create a directory as mosip in your PC and

  • clone k8’s infra repo with tag : 1.2.0.1-B2 inside mosip directory. git clone https://github.com/mosip/k8s-infra -b v1.2.0.1-B2

  • clone mosip-infra with tag : 1.2.0.1-B2 inside mosip directory git clone https://github.com/mosip/mosip-infra -b v1.2.0.1-B2

  • Set below mentioned variables in bashrc

    • Copy

      export MOSIP_ROOT=<location of mosip directory>
      export K8_ROOT=$MOSIP_ROOT/k8s-infra
      export INFRA_ROOT=$MOSIP_ROOT/mosip-infra

    • source .bashrc Note: Above mentioned environment variables will be used throughout the installation to move between one directory to other to run install scripts.

Installation

Wireguard

A Wireguard bastion host (Wireguard server) provides secure private channel to access MOSIP cluster. The host restricts public access, and enables access to only those clients who have their public key listed in Wireguard server. Wireguard listens on UDP port 51820.

Architecture diagram

Setup Wirguard VM and wireguard bastion server:

• Create a Wireguard server VM in aws console with above mentioned Hardware and Network requirements.

• Edit the security group and add the following inbound rules in aws console

  • type ‘custom TCP', port range ‘51820’ and source '0.0.0.0/0’

  • type ‘custom UDP', port range ‘51820’ and source '0.0.0.0/0’

• Install docker in the Wireguard machine as given here.

• Setup Wireguard server

  • SSH to wireguard VM

  • Create directory for storing wireguard config files. mkdir -p wireguard/config

  • Install and start wireguard server using docker as given below:

    Copy

    sudo docker run -d \
       --name=wireguard \
       --cap-add=NET_ADMIN \
       --cap-add=SYS_MODULE \
       -e PUID=1000 \
       -e PGID=1000 \
       -e TZ=Asia/Calcutta\
       -e PEERS=30 \
       -p 51820:51820/udp \
       -v /home/ubuntu/wireguard/config:/config \
       -v /lib/modules:/lib/modules \
       --sysctl="net.ipv4.conf.all.src_valid_mark=1" \
       --restart unless-stopped \
       ghcr.io/linuxserver/wireguard

    Note: * Increase the no of peers above in case needed more than 30 wireguard client confs. (-e PEERS=30) * Change the directory to be mounted to wireguard docker in case needed. All your wireguard confs will be generated in the mounted directory. (-v /home/ubuntu/wireguard/config:/config)

Setup Wireguard Client in your PC

• Install Wireguard client in your PC.

• Assign wireguard.conf:

  • SSH to the wireguard server VM.

  • cd /home/ubuntu/wireguard/config

  • assign one of the PR for yourself and use the same from the PC to connect to the server.

    • create assigned.txt file to assign the keep track of peer files allocated and update everytime some peer is allocated to someone.

      • Copy

        peer1 :   peername
        peer2 :   xyz

    • Use ls cmd to see the list of peers.

    • get inside your selected peer directory, and add mentioned changes in peer.conf:

      • cd peer1

      • nano peer1.conf

        • Delete the DNS IP.

        • Update the allowed IP's to subnets CIDR ip . e.g. 10.10.20.0/23

      • Share the updated peer.conf with respective peer to connect to wireguard server from Personel PC.

• add peer.conf in your PC’s /etc/wireguard directory as wg0.conf.

• start the wireguard client and check the status:

  • Copy

    sudo systemctl start wg-quick@wg0
    sudo systemctl status wg-quick@wg0

• Once Connected to wireguard you should be now able to login using private ip’s.

Observation K8s Cluster setup and configuration

Observation K8s Cluster setup

• Setup rancher cluster,

  • cd $K8_ROOT/rancher/aws

  • Copy rancher.cluster.config.sample to rancher.cluster.config.

  • Review and update the below mentioned parameters of rancher.cluster.config carefully.

    • name

    • region

    • version: “1.24“

    • instance related details

      • instanceName

      • instanceType

      • desiredcapacity

      • volumeSize

      • volumeType

      • publicKeyName.

    • update the details of the subnets to be used from vpc

  • Install

    • eksctl create cluster -f rancher.cluster.config

  • Wait for the cluster creation to complete, generally it takes around 30 minutes to create or update cluster.

  • Once EKS K8 cluster is ready below mentioned output will be displayed in the console screen. EKS cluster "my-cluster" in "region-code" region is ready

  • The config file for the new cluster will be created on ~/.kube/config

  • Make sure to backup and store the ~/.kube/config with new name. e.g. ~/.kube/obs-cluster.config.

  • Change file permission using below command: chmod 400 ~/.kube/obs-cluster.config

  • Set the KUBECONFIG properly so that you can access the cluster. export KUBECONFIG=~/.kube/obs-cluster.config

  • Test cluster access:

    • kubect get nodes

      • Command will result in details of the nodes of the rancher cluster.

Observation K8s Cluster’s Ingress and Storage class setup

Once the rancher cluster is ready we need ingress and storage class to be set for other applications to be installed.

• Nginx Ingress Controller : used for ingress in rancher cluster.

  • Copy

    cd $K8_ROOT/rancher/aws
    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
    helm repo update
    helm install \                               
      ingress-nginx ingress-nginx/ingress-nginx \
      --namespace ingress-nginx \
      --create-namespace  \
    -f nginx.values.yaml

    The above will automatically spawn an Internal AWS Network Load Balancer (L4).

• Check the following on AWS console:

  • An NLB has been created. You may also see the DNS of NLB with

    Copy

    kubectl -n ingress-nginx get svc

  • Obtain AWS TLS certificate as given here

  • Edit listner "443". Select "TLS".

  • Note, the target group name of listner 80. Set target group of 443 to target group of 80. Basically, we want TLS termination at the LB and it must forward HTTP traffic (not HTTPS) to port 80 of ingress controller. So

    • Input of LB: HTTPS

    • Output of LB: HTTP --> port 80 of ingress nginx controller

  • Enable "Proxy Protocol v2" in the target group settings

  • Make sure all subnets are selected in LB -->Description-->Edit subnets.

  • Check health check of target groups.

  • Remove listner 80 from LB as we will receive traffic only on 443.

• Storage class setup:

  • Default storage class on EKS is gp2 . GP2 by default is in Delete mode which means if PVC is deleted, the underlying storage PV is also deleted.

  • To enable volume expansion for the existing gp2 storage class, modify the YAML configuration by adding allowVolumeExpansion: true to the gp2 storage class configuration.

    • kubectl edit sc gp2 : to edit the yaml configuration.

  • Create storage class gp2-retain by running sc.yaml for PV in Retain mode. Set the storage class as gp2-retain in case you want to retain PV.

    • Copy

      cd $K8_ROOT/utils/misc/
      kubectl apply -f sc.yaml

  • we need the EBS driver for our storage class to work, follow the steps here to setup EBS driver.

Domain name

Create the following domain names:

• Rancher: rancher.xyz.net

• Keycloak: keycloak.xyz.net

Point the above to internal ip address of the NLB. This assumes that you have a Wireguard Bastion Host has been installed. On AWS this is done on Route 53 console.

Rancher K8s Cluster Apps Installation

• Rancher UI : Rancher provides full CRUD capability of creating and managing kubernetes cluster.

  • Install rancher using Helm, update hostname in rancher-values.yaml and run the following command to install.

    • Copy

      cd $K8_ROOT/rancher/rancher-ui
      helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
      helm repo update
      helm install rancher rancher-latest/rancher \
        --namespace cattle-system \
        --create-namespace \
        -f rancher-values.yaml

  • Login:

    • Open Rancher page https://rancher.org.net.

    • Get Bootstrap password using

      Copy

      kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}'

    • Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.

• Keycloak : Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.

  • Copy

    cd $K8_ROOT/rancher/keycloak
    ./install.sh <iam.host.name>

  • keycloak_client.json: Used to create SAML client on Keycloak for Rancher integration.

• Keycloak - Rancher Integration

  • Login as admin user in Keycloak and make sure an email id, and first name field is populated for admin user. This is important for Rancher authentication as given below.

  • Enable authentication with Keycloak using the steps given here.

  • In Keycloak add another Mapper for the rancher client (in Master realm) with following fields:

    • Protocol: saml

    • Name: username

    • Mapper Type: User Property

    • Property: username

    • Friendly Name: username

    • SAML Attribute Name: username

    • SAML Attribute NameFormat: Basic

  • Specify the following mappings in Rancher's Authentication Keycloak form:

    • Display Name Field: givenName

    • User Name Field: email

    • UID Field: username

    • Entity ID Field: https://your-rancher-domain/v1-saml/keycloak/saml/metadata

    • Rancher API Host: https://your-rancher-domain

    • Groups Field: member

• RBAC :

  • For users in Keycloak assign roles in Rancher - cluster and project roles. Under default project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects).

  • If you want to create custom roles, you can follow the steps given here.

  • Add a member to cluster/project in Rancher:

    • Give member name exactly as username in Keycloak

    • Assign appropriate role like Cluster Owner, Cluster Viewer etc.

    • You may create new role with fine grained acccess control.

• Certificates expiry

  • In case you see certificate expiry message while adding users, on local cluster run these commands:

    https://rancher.com/docs/rancher/v2.6/en/troubleshooting/expired-webhook-certificates/

MOSIP K8s Cluster setup

• Setup mosip cluster

  • cd $K8_ROOT/mosip/aws

  • Copy cluster.config.sample to mosip.cluster.config.

  • Review and update the below mentioned parameters of cluster.config.sample carefully.

    • name

    • region

    • version: “1.24“

    • instance related details

      • instanceName

      • instanceType

      • desiredcapacity

      • volumeSize

      • volumeType

      • publicKeyName.

    • update the details of the subnets to be used from vpc

  • Install

  eksctl create cluster -f mosip.cluster.config

  • Wait for the cluster creation to complete, generally it takes around 30 minutes to create or update cluster.

  • Once EKS K8 cluster is ready below mentioned output will be displayed in the console screen. EKS cluster "my-cluster" in "region-code" region is ready

  • The config file for the new cluster will be created on ~/.kube/config

  • Make sure to backup and store the ~/.kube/config with new name. e.g. ~/.kube/mosip-cluster.config.

  • Change file permission using below command: chmod 400 ~/.kube/mosip-cluster.config

  • Set the KUBECONFIG properly so that you can access the cluster. export KUBECONFIG=~/.kube/mosip-cluster.config

  • Test cluster access:

    • kubect get nodes

      • Command will result in details of the nodes of the MOSIP cluster.

Import Mosip Cluster into Rancher UI

• Login as admin in Rancher console

• Select Import Existing for cluster addition.

• Select the Generic as cluster type to add.

• Fill the Cluster Name field with unique cluster name and select Create.

• You will get the kubectl commands to be executed in the kubernetes cluster. Copy the command and execute from your PC. (make sure your kube-config file is correctly set to Mosip cluster)

eg.
kubectl apply -f https://rancher.e2e.mosip.net/v3/import/pdmkx6b4xxtpcd699gzwdtt5bckwf4ctdgr7xkmmtwg8dfjk4hmbpk_c-m-db8kcj4r.yaml

• Wait for few seconds after executing the command for the cluster to get verified.

• Your cluster is now added to the rancher management server.

MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup

• Global configmap: Global configmap contains list of necesary details to be used throughout the namespaces of the cluster for common details.

  • cd $K8_ROOT/mosip

  • Copy global_configmap.yaml.sample to global_configmap.yaml.

  • Update the domain names in global_configmap.yaml and run.

  • kubectl apply -f global_configmap.yaml

• Storage class setup:

  • Default storage class on EKS is gp2 . GP2 by default is in Delete mode which means if PVC is deleted, the underlying storage PV is also deleted.

  • To enable volume expansion for the existing gp2 storage class, modify the YAML configuration by adding allowVolumeExpansion: true to the gp2 storage class configuration.

    • kubectl edit sc gp2 : to edit the yaml configuration.

  • Create storage class gp2-retain by running sc.yaml for PV in Retain mode. Set the storage class as gp2-retain in case you want to retain PV.

    • Copy

      cd $K8_ROOT/utils/misc/
      kubectl apply -f sc.yaml

  • we need the EBS driver for our storage class to work, follow the steps here to setup EBS driver.

  • also we need EFS CSI driver for the regproc services, because EBS driver only supports RWO but we need RWX, follow these steps to setup EFS CSI driver.

• Ingress and load balancer (LB) :

  • Ingress is not installed by default on EKS. We use Istio ingress gateway controller to allow traffic in the cluster. Two channels are created - public and internal. See architecture.

  • Install istioctl as given here in your system.

  • Install ingresses as given here:

  • Copy

    cd $K8_ROOT/istio
    ./install.sh

• Load Balancers setup for istio-ingress.

  • The above istio installation will automatically spawn an Internal AWS Network Load Balancer (L4).

  • These may be also seen with

  Copy

  kubectl -n istio-system get svc

  • You may view them on AWS console in Loadbalancer section.

  • TLS termination is supposed to be on LB. So all our traffic coming to ingress controller shall be HTTP.

  • Obtain AWS TLS certificate as given here

  • Add the certificates and 443 access to the LB listener.

  • Update listener TCP->443 to TLS->443 and point to the certificate of domain name that belongs to your cluster.

  • Forward TLS->443 listner traffic to target group that corresponds to listener on port 80 of respective Loadbalancers. This is because after TLS termination the protocol is HTTP so we must point LB to HTTP port of ingress controller.

  • Update health check ports of LB target groups to node port corresponding to port 15021. You can see the node ports with

• Copy

  kubectl -n istio-system get svc

  • Enable Proxy Protocol v2 on target groups.

  • Make sure all subnets are included in Availability Zones for the LB. Description --> Availability Zones --> Edit Subnets

  • Make sure to delete the listeners for port 80 and 15021 from each of the loadbalancers as we restrict unsecured port 80 access over http.

• DNS mapping:

  • Initially all the services will be accesible only over the internal channel.

  • Point all your domain names to internal LoadBalancers DNS/IP intially till testing is done.

  • On AWS this may be done on Route 53 console.

  • After Go live decision enable public access.

• Check Overall if nginx and istio wiring is set correctly

  • Install httpbin: This utility docker returns http headers received inside the cluster. You may use it for general debugging - to check ingress, headers etc.

    • Copy

      cd $K8_ROOT/utils/httpbin
      ./install.sh

    • To see what's reaching httpbin (example, replace with your domain name):

      Copy

      curl https://api-internal.sandbox.xyz.net/httpbin/get?show_env=true
      Once public access is enabled also check this:
      curl https://api.sandbox.xyz.net/httpbin/get?show_env=true

Monitoring Module deployment

Prometheus and Grafana and Alertmanager tools are used for cluster monitoring.

• Select 'Monitoring' App from Rancher console -> Apps & Marketplaces.

• In Helm options, open the YAML file and disable Nginx Ingress.

  • Copy

    ingressNginx:
    enabled: false

• Click on Install.

Alerting Setup

Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel.

• Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager.

• Create slack incoming webhook.

• After setting slack incoming webhook update slack_api_url and slack_channel_name in alertmanager.yml.

  • cd $K8_ROOT/monitoring/alerting/

  • nano alertmanager.yml

  • Update

    Copy

    global:
      resolve_timeout: 5m
      slack_api_url: <YOUR-SLACK-API-URL>
    ...
      slack_configs:
      - channel: '<YOUR-CHANNEL-HERE>'
        send_resolved: true

• Update Cluster_name in patch-cluster-name.yaml.

  • cd $K8_ROOT/monitoring/alerting/

  • nano patch-cluster-name.yaml

  • Update

    Copy

    spec:
      externalLabels:
        cluster: <YOUR-CLUSTER-NAME-HERE>

• Install Default alerts along some of the defined custom alerts:

  • Copy

    cd $K8_ROOT/monitoring/alerting/
    ./install.sh

• Alerting is Installed.

Logging Module Setup and Installation

Mosip uses Rancher Fluentd and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard.

• Install Rancher FluentD system : for screpping logs outs of all the microservices from Mosip k8 cluster.

  • Install Logging from Apps and marketplace within the Rancher UI.

  • Select Chart Version 100.1.3+up3.17.7 from Rancher console -> Apps & Marketplaces.

• Configure Rancher FluentD

  • Create clusteroutput :

    • kubectl apply -f clusteroutput-elasticsearch.yaml

  • start clusterFlow

    • kubectl apply -f clusterflow-elasticsearch.yaml

• Install elasticsearch, kibana and Istio addons

  • cd $K8_ROOT/logging

  • ./intall.sh

• set min_age in elasticsearch-ilm-script.sh and execute the same. min_age : is the min no. of days for which indices will be stored in elasticsearch.

  • cd $K8_ROOT/logging

  • ./elasticsearch-ilm-script.sh

• Mosip provides set of Kibana Dashboards for checking logs and throughputs .

  • Brief description of these dashboards are as follows:

    • 01-logstash.ndjson contains the logstash Index Pattern required by the rest of the dashboards.

    • 02-error-only-logs.ndjson contains a Search dashboard which shows only the error logs of the services, called MOSIP Error Logs dashboard.

    • 03-service-logs.ndjson contains a Search dashboard which show all logs of a particular service, called MOSIP Service Logs dashboard.

    • 04-insight.ndjson contains dashboards which show insights into MOSIP processes, like the number of UINs generated (total and per hr), the number of Biometric deduplications processed, number of packets uploaded etc, called MOSIP Insight dashboard.

    • 05-response-time.ndjson contains dashboards which show how quickly different MOSIP Services are responding to different APIs, over time, called Response Time dashboard.

  • Import dashboards:

    • cd K8_ROOT/logging/dashboard

    • ./load_kibana_dashboards.sh ./dashboards <cluster-kube-config-file>

  • View dashbords

    • Open kibana dashboard from: https://kibana.sandbox.xyz.net.

    • Kibana --> Menu (on top left) --> Dashboard --> Select the dashboard.

Mosip External Dependencies setup

• External Dependencies: are set of external requirements needed for funtioning of MOSIP’s core services like DB, object store, hsm etc.

• Copy

  cd $INFRA_ROOT/deployment/v3/external/all
  ./install-all.sh

• Check detailed installation instruction of all the external componets.

MOSIP Modules Deployment

• Now that all the Kubernetes cluster and external dependencies are already installed, you can continue with MOSIP service deployment.

  • Copy

    cd $INFRA_ROOT/deployment/v3/mosip/all
    ./install-all.sh

• Check the detailed MOSIP Modules Deployment MOSIP Modular installation steps.

API Testrig

• MOSIP’s successfull deployment can be verified by comparing the results of api testrig with testrig benchmark.

  • Copy

    cd $INFRA_ROOT/deployment/v3/apitestrig
    ./install.sh

    • When prompted input the hour of the day to execute the api-testrig.

    • Daily api testrig cron jon will be executed at the very opted hour of the day.

Overview

• MOSIP modules are deployed in the form of microservices in kubernetes cluster.

• It is also used for the on-the-field registrations.

• • SSL termination

  • Reverse Proxy

  • CDN/Cache management

  • Loadbalancing

• In V3, we have two Kubernetes clusters:

Observation cluster - This cluster is a part of the observation plane and it helps in administrative tasks. By design, this is kept independent of the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or it's services should be internal and should never be exposed to the external world.

• It is recommended to configure log monitoring and network monitoring in this cluster.

• In case you have an internal container registry, then it should run here.

MOSIP cluster - This cluster runs all the MOSIP components and certain third party components to secure the cluster, API’s and data.

Architecture

Deployment repos

Pre-requisites

Hardware requirements

• VM’s required can be with any OS as per convenience.

• Here, we are referting to Ubuntu OS throughout this installation guide.

Network requirements

• All the VM's should be able to communicate with each other.

• Need stable Intra network connectivity between these VM's.

• All the VM's should have stable internet connectivity for docker image download (in case of local setup ensure to have a locally accessible docker registry).

• Server Interface requirement as mentioned in below table:

DNS requirements

Certificate requirements

As only secured https connections are allowed via nginx server will need below mentioned valid ssl certificates:

• One valid wildcard ssl certificate related to domain used for accessing Observation cluster, this needs to be stored inside the nginx server VM for Observation cluster. In above e.g.: *.org.net is the similar example domain.

• One valid wildcard ssl certificate related to domain used for accessing Mosip cluster, this needs to be stored inside the nginx server VM for mosip cluster. In above e.g.: *.sandbox.xyz.net is the similar example domain.

Tools to be installed in Personal Computers for complete deployment

• [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html: version > 2.12.4

• Create a directory as mosip in your PC and:

  • clone k8’s infra repo with tag : 1.2.0.1 (whichever is the latest version) inside mosip directory. git clone https://github.com/mosip/k8s-infra -b v1.2.0.1

  • clone mosip-infra with tag : 1.2.0.1 (whichever is the latest version) inside mosip directory. git clone https://github.com/mosip/mosip-infra -b v1.2.0.1

  • Set below mentioned variables in bashrc

  source .bashrc

  Note: Above mentioned environment variables will be used throughout the installation to move between one directory to other to run install scripts.

Installation

Wireguard

A Wireguard bastion host (Wireguard server) provides secure private channel to access MOSIP cluster. The host restricts public access, and enables access to only those clients who have their public key listed in Wireguard server. Wireguard listens on UDP port51820.

Setup Wireguard VM and wireguard bastion server

• Create a Wireguard server VM with above mentioned Hardware and Network requirements.

• Open ports and Install docker on Wireguard VM.

  • cd $K8_ROOT/wireguard/

  • create copy of hosts.ini.sample as hosts.ini and update the required details for wireguard VM\

    cp hosts.ini.sample hosts.ini

  • execute ports.yml to enable ports on VM level using ufw:

    ansible-playbook -i hosts.ini ports.yaml

Note:

• Permission of the pem files to access nodes should have 400 permission. sudo chmod 400 ~/.ssh/privkey.pem

• These ports are only needed to be opened for sharing packets over UDP.

• Take necessary measure on firewall level so that the Wireguard server can be reachable on 51820/udp.

• Setup Wireguard server

  • SSH to wireguard VM

  • Create directory for storing wireguard config files. mkdir -p wireguard/config

  • Install and start wireguard server using docker as given below:

Note:

• Increase the no. of peers above in case more than 30 wireguard client confs (-e PEERS=30) are needed.

• Change the directory to be mounted to wireguard docker as per need. All your wireguard confs will be generated in the mounted directory (-v /home/ubuntu/wireguard/config:/config).

Setup Wireguard Client in your PC

• Install Wireguard client in your PC.

• Assign wireguard.conf:

  • SSH to the wireguard server VM.

  • cd /home/ubuntu/wireguard/config

  • assign one of the PR for yourself and use the same from the PC to connect to the server.

    • create assigned.txt file to assign the keep track of peer files allocated and update everytime some peer is allocated to someone.

    • use ls cmd to see the list of peers.

    • get inside your selected peer directory, and add mentioned changes in peer.conf:

      • cd peer1

      • nano peer1.conf

        • Delete the DNS IP.

        • Update the allowed IP's to subnets CIDR ip . e.g. 10.10.20.0/23

      • Share the updated peer.conf with respective peer to connect to wireguard server from Personel PC.

• add peer.conf in your PC’s /etc/wireguard directory as wg0.conf.

• start the wireguard client and check the status:

• Once connected to wireguard, you should be now able to login using private IP’s.

Observation K8s Cluster setup and configuration

Observation K8s Cluster setup

• Install all the required tools mentioned in pre-requisites for PC.

  • kubectl

  • helm

  • ansible

  • rke (version 1.3.10)

• Setup Observation Cluster node VM’s as per the hardware and network requirements as mentioned above.

• Setup passwordless SSH into the cluster nodes via pem keys. (Ignore if VM’s are accessible via pem’s).

  • Generate keys on your PC ssh-keygen -t rsa

  • Copy the keys to remote observation node VM’s ssh-copy-id <remote-user>@<remote-ip>

  • SSH into the node to check password-less SSH ssh -i ~/.ssh/<your private key> <remote-user>@<remote-ip>

Note:

• Make sure the permission for privkey.pem for ssh is set to 400.

• Run env-check-setup.yaml to check if cluster nodes are fine and do not have known issues in it.

  • cd $K8_ROOT/rancher/on-prem

  • create copy of hosts.ini.sample as hosts.ini and update the required details for Observation k8 cluster nodes.

    • cp hosts.ini.sample hosts.ini

    • ansible-playbook -i hosts.ini env-check-setup.yaml

    • This ansible checks if localhost mapping is already present in /etc/hosts file in all cluster nodes, if not it adds the same.

• Open ports and install docker on Observation K8 Cluster node VM’s.

  • cd $K8_ROOT/rancher/on-prem

  • Ensure that hosts.ini is updated with nodal details.

  • Update vpc_ip variable in ports.yaml with vpc CIDR ip to allow access only from machines inside same vpc.

  • Execute ports.yml to enable ports on VM level using ufw:

    • ansible-playbook -i hosts.ini ports.yaml

  • Disable swap in cluster nodes. (Ignore if swap is already disabled)

    • ansible-playbook -i hosts.ini swap.yaml

  • execute docker.yml to install docker and add user to docker group:

    • ansible-playbook -i hosts.ini docker.yaml

• Creating RKE Cluster Configuration file

  • rke config

  • Command will prompt for nodal details related to cluster, provide inputs w.r.t below mentioned points:

    • SSH Private Key Path :

    • Number of Hosts:

    • SSH Address of host :

    • SSH User of host :

    • Make all the nodes Worker host by default.

    • To create an HA cluster, specify more than one host with role Control Plane and etcd host.

  • Network Plugin Type : Continue with canal as default network plugin.

  • For rest of other configurations, opt the required or default value.

• As result of rke config command cluster.yml file will be generated inside same directory, update the below mentioned fields:

  • nano cluster.yml

    • Remove the default Ingress install
  • Add the name of the kubernetes cluster

    • cluster_name: sandbox-name

• Setup up the cluster:

  • Once cluster.yml is ready, you can bring up the kubernetes cluster using simple command.

    • This command assumes the cluster.yml file is in the same directory as where you are running the command.

    • rke up
  • As part of the Kubernetes creation process, a kubeconfig file has been created and written at kube_config_cluster.yml, which can be used to start interacting with your Kubernetes cluster.

  • Copy the kubeconfig files

  • To access the cluster using kubeconfig file use any one of the below method:

    • cp $HOME/.kube/<cluster_name>_config $HOME/.kube/config Alternatively

    • export KUBECONFIG="$HOME/.kube/<cluster_name>_config

  • Test cluster access:

    • kubectl get nodes

      • Command will result in details of the nodes of the Observation cluster.

  • Save your files

    • Save a copy of the following files in a secure location, they are needed to maintain, troubleshoot and upgrade your cluster.

      • cluster.yml: The RKE cluster configuration file.