GitHubCommunityInitiativesWhat's NewContact Us
Edit

On-Premises Deployment Without DNS

1. Wireguard

  • Wireguard bastian server provides secure private channel to access MOSIP cluster.

  • Bastian server restricts public access, and enables access to only those clients who have their public key listed in Wireguard server.

  • Bastion server listens on UDP port 51820.

  • In case you already have VPN configured to access nodes privately please skip Wireguard installation and continue to use the same VPN.

Setup Wireguard VM and wireguard bastion server

  • Create a Wireguard server VM with mentioned 'Hardware and Network Requirements'.

  • Open required ports in the Bastian server VM.

    • cd $K8_ROOT/wireguard/

    • Create copy of hosts.ini.sample as hosts.ini and update the required details for wireguard VM.

    • cp hosts.ini.sample hosts.ini

    Note :

    • Remove [Cluster] complete section from copied hosts.ini file.

    • Add below mentioned details:

      • ansible_host : public IP of Wireguard Bastion server. eg. 100.10.20.56

      • ansible_user : user to be used for installation. In this ref-impl we use Ubuntu user.

      • ansible_ssh_private_key_file : path to pem key for ssh to wireguard server. eg. ~/.ssh/wireguard-ssh.pemhosts.ini

    • Execute ports.yml to enable ports on VM level using ufw:

    • ansible-playbook -i hosts.ini ports.yaml

Note:

  • Permission of the pem files to access nodes should have 400 permission. sudo chmod 400 ~/.ssh/privkey.pem

  • These ports are only needed to be opened for sharing packets over UDP.

  • Take necessary measure on firewall level so that the Wireguard server can be reachable on 51820/udp.

  • Install docker

    • execute docker.yml to install docker and add user to docker group:

  • Setup Wireguard server

    • SSH to wireguard VM

    • ssh -i <path to .pem> ubuntu@<public ip of wireguard server VM>

    • Create directory for storing wireguard config files.mkdir -p wireguard/config

    • Install and start wireguard server using docker as given below:

Note:

  • Increase the no. of peers above in case more than 30 wireguard client confs (-e PEERS=30) are needed.

  • Change the directory to be mounted to wireguard docker as per need. All your wireguard confs will be generated in the mounted directory (-v /home/ubuntu/wireguard/config:/config).

Setup Wireguard Client in your PC

  • Install Wireguard client in your PC using steps.

  • Assign wireguard.conf:

    • SSH to the wireguard server VM.

    • cd /home/ubuntu/wireguard/config

    • assign one of the PR for yourself and use the same from the PC to connect to the server.

      • create assigned.txt file to assign the keep track of peer files allocated and update everytime some peer is allocated to someone.

      • use ls cmd to see the list of peers.

      • get inside your selected peer directory, and add mentioned changes in peer.conf:

        • cd peer1

        • nano peer1.conf

          • Delete the DNS IP.

          • Update the allowed IP's to subnets CIDR ip . e.g. 10.10.20.0/23.

          Note:

          • CIDR Range will be shared by the Infra provider.

          • Make sure all the nodes are covered in the provided CIDR range. (nginx server, K8 cluster nodes for observation as well as mosip).

        • Share the updated peer.conf with respective peer to connect to wireguard server from Personel PC.

    • Add peer.conf in your PC’s /etc/wireguard directory as wg0.conf.

    • Start the wireguard client and check the status:

  • Once connected to wireguard, you should be now able to login using private IP’s.

2. Observation K8s Cluster setup and configuration

  • Install all the required tools mentioned in 'Personel Computer Setup' section.

  • Setup Observation Cluster node VM’s hardware and network configuration as per (requirements).

  • Setup passwordless SSH into the cluster nodes via pem keys. (Ignore if VM’s are accessible via pem’s).

    • Generate keys on your PCssh-keygen -t rsa

    • Copy the keys to remote observation node VM’sssh-copy-id <remote-user>@<remote-ip>

    • SSH into the node to check password-less SSHssh -i ~/.ssh/<your private key> <remote-user>@<remote-ip>

Note:

  • Make sure the permission for privkey.pem for ssh is set to 400.

  • Open ports and install docker on Observation K8 Cluster node VM’s.

    • cd $K8_ROOT/rancher/on-prem

    • Copy hosts.ini.sample to hosts.ini and update required details.

    • cp hosts.ini.sample hosts.ini

    Note:

    • Ensure you are inside on-prem directory as mentioned above.

    • ansible_host : internal IP of nodes. eg. 100.10.20.56, 100.10.20.57 ...

    • ansible_user : user to be used for installation. In this ref-implementation we use Ubuntu user.

    • ansible_ssh_private_key_file : path to pem key for ssh to wireguard server. eg. ~/.ssh/nodes-ssh.pemhosts.ini

    • Update vpc_ip variable in ports.yaml with vpc CIDR ip to allow access only from machines inside same vpc.

      Note:

      • CIDR Range will be shared by the Infra provider.

      • Make sure all the nodes are covered in the provided CIDR range. (nginx server, K8 cluster nodes for observation as well as mosip).

    • Execute ports.yml to enable ports on VM level using ufw:

    • ansible-playbook -i hosts.ini ports.yaml

    • Disable swap in cluster nodes. (Ignore if swap is already disabled)

    • ansible-playbook -i hosts.ini swap.yaml

      Caution: Always verify swap status with swapon --show before running the playbook to avoid unnecessary operations.

    • execute docker.yml to install docker and add user to docker group:

    • ansible-playbook -i hosts.ini docker.yaml

  • Creating RKE Cluster Configuration file

    • rke config

    • Command will prompt for nodal details related to cluster, provide inputs w.r.t below mentioned points:

      • SSH Private Key Path :

      • Number of Hosts:

      • SSH Address of host :

      • SSH User of host :

      • Make all the nodes Worker host by default.

      • To create an HA cluster, specify more than one host with role Control Plane and etcd host.

    • Network Plugin Type : Continue with canal as default network plugin.

    • For rest of other configurations, opt the required or default value.

  • As result of rke config command cluster.yml file will be generated inside same directory, update the below mentioned fields:

    • nano cluster.yml

      • Remove the default Ingress install

    • Update the name of the kubernetes cluster in cluster.yml.

      • cluster_name: observation-cluster

  • Setup up the cluster:

    • Once cluster.yml is ready, you can bring up the kubernetes cluster using simple command.

      • This command assumes the cluster.yml file is in the same directory as where you are running the command.

      • rke up

      • The last line should read Finished building Kubernetes cluster successfully to indicate that your cluster is ready to use.

      Note:

      • Incase rke up command is unsucessfull due to any underline error then we need to fix the same by checking the logs.

      • Once the issue is fixed we need to remove the cluster using rke remove.

      • Once rke remove is executed sucessfully need to delete cluster related incomplete configuration using :

    • As part of the Kubernetes creation process, a kubeconfig file has been created and written at kube_config_cluster.yml, which can be used to start interacting with your Kubernetes cluster.

    • Copy the kubeconfig files

    • To access the cluster using kubeconfig file use any one of the below method:

      • cp $HOME/.kube/<cluster_name>_config $HOME/.kube/config Alternatively

      • export KUBECONFIG="$HOME/.kube/<cluster_name>_config

    • Test cluster access:

      • kubectl get nodes

        • Command will result in details of the nodes of the Observation cluster.

    • Save your files

      • Save a copy of the following files in a secure location, they are needed to maintain, troubleshoot and upgrade your cluster.

        • cluster.yml: The RKE cluster configuration file.

        • kube_config_cluster.yml: The Kubeconfig file for the cluster, this file contains credentials for full access to the cluster.

        • cluster.rkestate: The Kubernetes Cluster State file, this file contains credentials for full access to the cluster.

    • In case not having Public DNS system add the custom DNS configuration for all the hostnames to be used for testing in Cluster's coredns configmap.

      • Check whether coredns pods are up and running in your cluster via the below command:

      • To update the coredns configmap, use the below command.

        Note: Default editor in WSL and Ubuntu is vi. Incase not familiar with vi change the editor to your prefered one:export EDITOR=<prefered editor>

      • Update the IP address and domain name in the below DNS hosts template and add it in the coredns configmap Corefile key in the kube-system namespace.

      • Check whether the DNS changes are correctly updated in coredns configmap.

      • Restart the coredns pod in the kube-system namespace.

      • Check status of coredns restart.

3. Observation K8s Cluster Ingress and Storage class setup

Once the rancher cluster is ready, we need ingress and storage class to be set for other applications to be installed.

3.a. Nginx Ingress Controller: used for ingress in rancher cluster.

Note:

  • This will install ingress in namespace ingress-nginx of Observation cluster.

  • Crosscheck using below mentioned command:

  • kubectl get all -n ingress-nginx

  • Command should result with list of all the pods, deployments etc in ingress nginx namespace.

3.b. Storage classes

Multiple storage classes options are available for onprem K8's cluster. In this reference deployment will continue to use NFS as a storage class.

  • Move to nfs directory in your personel computer.

  • Create a copy of hosts.ini.sample as hosts.ini.

  • Update the NFS machine details in hosts.ini file.

    Note :

    • Add below mentioned details:

    • ansible_host : internal IP of NFS server. eg. 10.12.23.21

    • ansible_user : user to be used for installation, in this ref-impl we use Ubuntu user.

    • ansible_ssh_private_key_file : path to pem key for ssh to wireguard server. eg. ~/.ssh/wireguard-ssh.pem hosts.ini.

  • Make sure Kubeconfig file is set correctly to point to required Observation cluster.

    Note:

    • Output should show the cluster name to confirm you are pointing to right kubernetes cluster.

    • If not pinting to right K8 cluster change the kubeconfig to connect to right K8 cluster.

  • Enable firewall with required ports:

  • SSH to the nfs node:

  • Clone k8s-infra repo in nginx VM:

  • Move to the nfs directory:

  • Execute script to install nfs server:

    Note:

    • Script prompts for below mentioned user inputs:

    • envName: env name eg. dev/qa/uat...

  • Switch to your personel computer and excute below mentioned commands:

    Note:

    • Script prompts for:

    • NFS Server: NFS server ip for persistence.

    • NFS Path : NFS path for storing the persisted data. eg. /srv/nfs/mosip/

  • Post installation check:

    • Check status of NFS Client Provisioner.

    • Check status of nfs-client storage class.

    • Set nfs-client as default storage class.

4. Setting up nginx server for Observation K8s Cluster

4.a. SSL Certificate setup for TLS termination

  • For Nginx server setup we need ssl certificate, add the same into Nginx server.

  • SSL certificates can be generated in multiple ways, either via lets encrypt if you have public DNS or via openssl certs when you don't have Public DNS.

    • Openssl : Generate wildcard ssl certificate using openssl in case you don't have public DNS using below steps. (Ensure to use this only in development env, not suggested for Production env).

      • Generate a self-signed certificate for your domain, such as *.sandbox.xyz.net.

      • SSH to the nginx VM.

      • Execute the following command to generate a self-signed SSL certificate. Prior to execution, kindly ensure to update environmental variables & rancher domain passed to openssl command:

      • Above command will generate certs in below specified location. Use it when prompted during nginx installation.

        • fullChain path: /etc/ssl/certs/tls.crt.

        • privKey path: /etc/ssl/private/tls.key.

4.b. Install Nginx :

  • Move to nginx directory in your local

  • cd $K8_ROOT/mosip/on-prem/nginx/

  • Open required ports :

    • Use any editor to create new hosts.ini file:

    • Add below mentioned lines with updated details of nginx server to the hosts.ini and save.

    • Execute below mentoned command to open required ports:

  • Login to nginx server node.

    • Clone k8s-infra

    • Provide below mentioned inputs as and when promted

      • Rancher nginx ip : internal ip of the nginx server VM.

      • SSL cert path : path of the ssl certificate to be used for ssl termination.

      • SSL key path : path of the ssl key to be used for ssl termination.

      • Cluster node IPs : IPs of the rancher cluster node

  • Post installation check:

    • sudo systemctl status nginx

  • Steps to Uninstall nginx (in case required). sudo apt purge nginx nginx-common.

  • DNS mapping:

    • Once nginx server is installed successfully, create DNS mapping for rancher cluster related domains as mentioned in DNS requirement section. (rancher.org.net, keycloak.org.net)\

      • Add DNS entries in local hosts file of your system.

      • For example: /etc/hosts files for Linux machines.

      • nano /etc/hosts

      • Update the domain and IP address.

5. Observation K8's Cluster Apps Installation

5.a. Rancher UI

  • Rancher provides full CRUD capability of creating and managing kubernetes cluster.

  • Install rancher using Helm, update hostname, & add privateCA to true in rancher-values.yaml, and run the following command to setup secrets for installation.

  • Create a secret containing the observation nginx self-signed public certificate (i.e. tls.crt ) generated in openssl section.

  • USe below command to install Rancher UI:

  • Login:

    • Connect to Wireguard Bastion. (Incase using WSL make sure to connect using Windows Wireguard instead of WSL one).

    • Open Rancher page.

    • Get Bootstrap password using

Note: Assign a password. IMPORTANT: makes sure this password is securely saved and retrievable by Admin.

5.b. Keycloak

  • Keycloak: Keycloak is an OAuth 2.0 compliant Identity Access Management (IAM) system used to manage the access to Rancher for cluster controls.

  • Post installation access the keycloak using iam.mosip.net and get the credentials as per the post installation steps definedAdd commentMore actions keycloak-access.

5.c. Keycloak - Rancher UI Integration

  • Login as admin user in Keycloak and make sure email and firstName fields are populated for the admin user. These are required for Rancher authentication to work properly.

  • In Keycloak (in the master realm), create a new client with the following values:

    • Client ID: https://<your-rancher-host>/v1-saml/keycloak/saml/metadata

    • Client Protocol: saml

    • Root URL: (leave empty)

    • After saving, configure the client with:

      • Name: rancher

      • Enabled: ON

      • Login Theme: keycloak

      • Sign Documents: ON

      • Sign Assertions: ON

      • Encrypt Assertions: OFF

      • Client Signature Required: OFF

      • Force POST Binding: OFF

      • Front Channel Logout: OFF

      • Force Name ID Format: OFF

      • Name ID Format: username

      • Valid Redirect URIs: https://<your-rancher-host>/v1-saml/keycloak/saml/acs

      • IDP Initiated SSO URL Name: IdPSSOName

    • Save the client

  • In the same client, go to the Mappers tab and create the following:

    • Mapper 1:

      • Protocol: saml

      • Name: username

      • Mapper Type: User Property

      • Property: username

      • Friendly Name: username

      • SAML Attribute Name: username

      • SAML Attribute NameFormat: Basic

    • Mapper 2:

      • Protocol: saml

      • Name: groups

      • Mapper Type: Group List

      • Group Attribute Name: member

      • Friendly Name: (Leave empty)

      • SAML Attribute NameFormat: Basic

      • Single Group Attribute: ON

      • Full Group Path: OFF

    • Click Add Builtin → select all → Add Selected

  • Download the Keycloak SAML descriptor XML file from:

    • https://<your-keycloak-host>/auth/realms/master/protocol/saml/descriptor

  • Generate a self-signed SSL certificate and private key (if not already available):

  • Rancher UI Configuration

    • In Rancher UI, go to :

    Users & Authentication → Auth Providers → Keycloak (SAML)

    • Configure the fields as follows :

      Display Name Field: givenName

      User Name Field: email or uid

      UID Field: username

      Groups Field: member

      Entity ID Field: (Leave empty)

      Rancher API Host: https://<your-rancher-host>

    • Upload the following files:

    Private Key: myservice.key

    Certificate: myservice.cert

    SAML Metadata XML: (from the Keycloak descriptor link)

    • Click Enable to activate Keycloak authentication.

    • After successful integration, Rancher users should be able to log in using their Keycloak

5.d. RBAC for Rancher using Keycloak

  • For users in Keycloak assign roles in Rancher - cluster and project roles. Under default project add all the namespaces. Then, to a non-admin user you may provide Read-Only role (under projects).

  • In case your cluster is used by multiple peoples with multiple roles and you want to create custom roles, you can follow the steps given here.

  • Add a member to cluster/project in Rancher:

    • Navigate to RBAC cluster members

    • Add member name exactly as username in Keycloak

    • Assign appropriate role like Cluster Owner, Cluster Viewer etc.

    • You may create new role with fine grained acccess control.

  • Add group to to cluster/project in Rancher:

    • Navigate to RBAC cluster members

    • Click on Add and select a group from the displayed drop-down.

    • Assign appropriate role like Cluster Owner, Cluster Viewer etc.

    • To add groups, the user must be a member of the group.

  • Creating a Keycloak group involves the following steps:

    • Go to the "Groups" section in Keycloak and create groups with default roles.

    • Navigate to the "Users" section in Keycloak, select a user, and then go to the "Groups" tab. From the list of groups, add the user to the required group.

6. MOSIP K8s Cluster setup

  • Pre-requisites:

  • Install all the required tools mentioned in Pre-requisites for PC.

    • kubectl

    • helm

  • ansible

  • rke (version 1.3.10)

  • Setup MOSIP K8 Cluster node VM’s as per 'Hardware and Network Requirements'.

  • Run env-check.yaml to check if cluster nodes are fine and don't have known issues in it.

    • cd $K8_ROOT/rancher/on-prem

    • create copy of hosts.ini.sample as hosts.ini and update the required details for MOSIP k8 cluster nodes.

      • cp hosts.ini.sample hosts.ini

      Note:

      • Ensure you are inside on-prem directory as mentioned above.

      • ansible_host : internal IP of nodes. eg. 100.10.20.56, 100.10.20.57 ...

      • ansible_user : user to be used for installation. In this ref-implementation we use Ubuntu user.

      • ansible_ssh_private_key_file : path to pem key for ssh to wireguard server. eg. ~/.ssh/nodes-ssh.pemhosts.ini

      • ansible-playbook -i hosts.ini env-check.yaml

      • This ansible checks if localhost mapping is already present in /etc/hosts file in all cluster nodes, if not it adds the same.

  • Setup passwordless ssh into the cluster nodes via pem keys. (Ignore if VM’s are accessible via pem’s).

    • Generate keys on your PC

      • ssh-keygen -t rsa

    • Copy the keys to remote rancher node VM’s:

      • ssh-copy-id <remote-user>@<remote-ip>

    • SSH into the node to check password-less SSH

      • ssh -i ~/.ssh/<your private key> <remote-user>@<remote-ip>

    • Rancher UI : (deployed in Observation K8 cluster).

  • Open ports and Install docker on MOSIP K8 Cluster node VM’s.

    • cd $K8_ROOT/mosip/on-prem

    • create copy of hosts.ini.sample as hosts.ini and update the required details for wireguard VM.

      • cp hosts.ini.sample hosts.ini

    • Update vpc_ip variable in ports.yaml with vpc CIDR ip to allow access only from machines inside same vpc.

      Note:

      • CIDR Range will be shared by the Infra provider.

      • Make sure all the nodes are covered in the provided CIDR range. (nginx server, K8 cluster nodes for observation as well as mosip).

    • execute ports.yml to enable ports on VM level using ufw:

      • ansible-playbook -i hosts.ini ports.yaml

    • Disable swap in cluster nodes. (Ignore if swap is already disabled)

      • ansible-playbook -i hosts.ini swap.yaml

      Caution: Always verify swap status with swapon --show before running the playbook to avoid unnecessary operations.

    • execute docker.yml to install docker and add user to docker group:

      • ansible-playbook -i hosts.ini docker.yaml

  • Creating RKE Cluster Configuration file

    • rke config

    • Command will prompt for nodal details related to cluster, provide inputs w.r.t below mentioned points:

      • SSH Private Key Path :

      • Number of Hosts:

      • SSH Address of host :

      • SSH User of host :

      • Make all the nodes Worker host by default.

      • To create an HA cluster, specify more than one host with role Control Plane and etcd host.

    • Network Plugin Type : Continue with canal as default network plugin.

    • For rest for other configuration opt the required or default value.

  • As result of rke config command cluster.ymlfile will be generated inside same directory, update the below mentioned fields:

    • nano cluster.yml

    • Remove the default Ingress install

    • Add the name of the kubernetes cluster

    • For production deployments edit the cluster.yml, according to this RKE Cluster Hardening Guide.

  • Setup up the cluster:

    • Once cluster.yml is ready, you can bring up the kubernetes cluster using simple command.

      • This command assumes the cluster.yml file is in the same directory as where you are running the command.

        • rke up

      • The last line should read Finished building Kubernetes cluster successfully to indicate that your cluster is ready to use.

      • Copy the kubeconfig files

    • To access the cluster using kubeconfig filr use any one of the below method:

    • cp $HOME/.kube/<cluster_name>_config $HOME/.kube/config

Alternatively

  • Test cluster access:

    • kubect get nodes

    • Command will result in details of the nodes of the rancher cluster.

  • Save Your files

    • Save a copy of the following files in a secure location, they are needed to maintain, troubleshoot and upgrade your cluster.:

      • cluster.yml: The RKE cluster configuration file.

      • kube_config_cluster.yml: The Kubeconfig file for the cluster, this file contains credentials for full access to the cluster.

      • cluster.rkestate: The Kubernetes Cluster State file, this file contains credentials for full access to the cluster.

  • In case not having Public DNS system add the custom DNS configuration for all the hostnames to be used for testing in Cluster's coredns configmap.

    • Check whether coredns pods are up and running in your cluster via the below command:

    • To update the coredns configmap, use the below command.

      Note: Default editor in WSL and Ubuntu is vi. Incase not familiar with vi change the editor to your prefered one:export EDITOR=<prefered editor>

    • Update the IP address and domain name in the below DNS hosts template and add it in the coredns configmap Corefile key in the kube-system namespace.

    • Check whether the DNS changes are correctly updated in coredns configmap.

    • Restart the coredns pod in the kube-system namespace.

    • Check status of coredns restart.

      Note:

      • Since this deployment is without proper DNS and using self signed certificate please be informed that while accessing the UI components in browsr the sites will be in Not-secure state.

7. MOSIP K8 Cluster Global configmap, Ingress and Storage Class setup

7.a. Global configmap:

  • Global configmap Contains the list of neccesary details to be used throughout the namespaces of the cluster for common details.

  • cd $K8_ROOT/mosip

  • Copy global_configmap.yaml.sample to global_configmap.yaml.

  • Update the domain names in global_configmap.yaml and run.

  • kubectl apply -f global_configmap.yaml

7.b. Istio Ingress setup:

  • It is a service mesh for the MOSIP K8 cluster which provides transparent layers on top of existing microservices along with powerful features enabling a uniform and more efficient way to secure, connect, and monitor services.

    • cd $K8_ROOT/mosip/on-prem/istio

    • ./install.sh

    • This will bring up all the Istio components and the Ingress Gateways.

    • Check Ingress Gateway services:

      • kubectl get svc -n istio-system

      Note: Response should contain service names as mentioned below.

      • istio-ingressgateway: external facing istio service.

      • istio-ingressgateway-internal: internal facing istio service.

      • istiod: Istio daemon for replicating the changes to all envoy filters.

7.c. Storage classes

Multiple storage classes options are available for onprem K8's cluster. In this reference deployment will continue to use NFS as a storage class.

  • Move to nfs directory in your personel computer.

  • Create a copy of hosts.ini.sample as hosts.ini.

  • Update the NFS machine details in hosts.ini file.

    Note :

    • Add below mentioned details:

    • ansible_host : internal IP of NFS server. eg. 10.12.23.21

    • ansible_user : user to be used for installation, in this ref-impl we use Ubuntu user.

    • ansible_ssh_private_key_file : path to pem key for ssh to wireguard server. eg. ~/.ssh/wireguard-ssh.pem hosts.ini.

  • Make sure Kubeconfig file is set correctly to point to required mosip cluster.

    Note:

    • Output should show the cluster name to confirm you are pointing to right kubernetes cluster.

    • If not pinting to right K8 cluster change the kubeconfig to connect to right K8 cluster.

  • Enable firewall with required ports:

  • SSH to the nfs node:

  • Clone k8s-infra repo in nginx VM:

  • Move to the nfs directory:

  • Execute script to install nfs server:

    Note:

    • Script prompts for below mentioned user inputs:

    • envName: env name eg. dev/qa/uat...

  • Switch to your personel computer and excute below mentioned commands:

    Note:

    • Script prompts for:

    • NFS Server: NFS server ip for persistence.

    • NFS Path : NFS path for storing the persisted data. eg. /srv/nfs/mosip/

  • Post installation check:

    • Check status of NFS Client Provisioner.

    • Check status of nfs-client storage class.

8. Import MOSIP Cluster into Rancher UI

  • Login as admin in Rancher console

  • Select Import Existing for cluster addition.

  • Select Generic as cluster type to add.

  • Fill the Cluster Name field with unique cluster name and select Create.

  • You will get the kubectl commands to be executed in the kubernetes cluster. Copy the command and execute from your PC (make sure your kube-config file is correctly set to MOSIP cluster).

  • Wait for few seconds after executing the command for the cluster to get verified.

  • Your cluster is now added to the rancher management server.

9. MOSIP K8 cluster Nginx server setup

9.a. Openssl certificates creation

  • For Nginx server setup, we need ssl certificate, add the same into Nginx server.

  • SSL certificates can be generated in multiple ways. Either via lets encrypt if you have public DNS or via openssl certs when you don't have Public DNS.

    • Openssl : Generate wildcard ssl certificate using openssl in case you don't have public DNS using below steps. (Ensure to use this only in development env, not suggested for Production env).

    • SSH to the nginx VM.

    • Install docker on nginx node.

    • Generate a self-signed certificate for your domain, such as *.sandbox.xyz.net.

    • Execute the following command to generate a self-signed SSL certificate. Prior to execution, kindly ensure that the environmental variables passed to the OpenSSL Docker container have been properly updated:

    • Above command will generate certs in below specified location. Use it when prompted during nginx installation.

      • fullChain path: /etc/ssl/certs/tls.crt.

      • privKey path: /etc/ssl/private/tls.key.

9.b. Nginx server setup for MOSIP K8's cluster

  • Move to nginx directory in your local:Add commentMore actions

  • cd $K8_ROOT/mosip/on-prem/nginx/

  • Open required ports :

    • Use any editor to create new hosts.ini file:

    • Add below mentioned lines with updated details of nginx server to the hosts.ini and save.

    • Execute below mentoned command to open required ports:

  • Login to nginx server node.

    • Clone k8s-infra.

    • Provide below mentioned inputs as and when prompted

      • MOSIP nginx server internal ip

      • MOSIP nginx server public ip

      • Publically accessible domains (comma separated with no whitespaces)

      • SSL cert path

      • SSL key path

      • Cluster node ip's (comma separated no whitespace)

  • When utilizing an openssl wildcard SSL certificate, please add the following server block to the nginx server configuration within the http block. Disregard this if using SSL certificates obtained through letsencrypt or for publicly available domains. Please note that this should only be used in a development environment and is not recommended for production environments.

    • nano /etc/nginx/nginx.conf

    Note: HTTP access is enabled for IAM because MOSIP's keymanager expects to have valid SSL certificates. Ensure to use this only for development purposes, and it is not recommended to use it in production environments.

    • Restart nginx service.

  • Post installation check:sudo systemctl status nginx

  • Steps to Uninstall nginx (in case required)sudo apt purge nginx nginx-common

  • DNS mapping:

    • Once nginx server is installed successfully, create DNS mapping for rancher cluster related domains as mentioned in DNS requirement section. (rancher.org.net, keycloak.org.net)

    • In case used Openssl for wildcard ssl certificate add DNS entries in local hosts file of your system.

      • For example: /etc/hosts files for Linux machines.

9.c. Check Overall nginx and istio wiring

  • Install httpbin: This utility docker returns http headers received inside the cluster.

  • httpbin can be used for general debugging - to check ingress, headers etc.

    • To see what is reaching the httpbin (example, replace with your domain name):

10. Monitoring module deployment

Note :

  • Monitoring in the sandbox environment is optional and can be deployed if required.

  • For production environments, alternative monitoring tools can be used.

  • These steps can also be skipped in development environments if monitoring is not needed.

  • Incase skipping execute below commands to install monitoring crd as the same is required by mosip services:

Prometheus and Grafana and Alertmanager tools are used for cluster monitoring.

  • Select 'Monitoring' App from Rancher console -> Apps & Marketplaces.

  • In Helm options, open the YAML file and disable Nginx Ingress.

  • Click on Install.

11. Alerting setup

Note :

  • Alerting in the sandbox environment is optional and can be deployed if required.

  • For production environments, alternative alerting tools can be used.

  • These steps can also be skipped in development environments if alerting is not needed.

  • Alerting is part of cluster monitoring, where alert notifications are sent to the configured email or slack channel.

  • Monitoring should be deployed which includes deployment of prometheus, grafana and alertmanager.

  • Create slack incoming webhook.

  • After setting slack incoming webhook update slack_api_url and slack_channel_name in alertmanager.yml.

    • cd $K8_ROOT/monitoring/alerting/

    • nano alertmanager.yml

    • Update:

  • Update Cluster_name in patch-cluster-name.yaml.

  • cd $K8_ROOT/monitoring/alerting/

  • nano patch-cluster-name.yaml

  • Update:

  • Install Default alerts along some of the defined custom alerts:

  • Alerting is installed.

12. Logging module setup and installation

Note :

  • Logging in the sandbox environment is optional and can be deployed if required.

  • For production environments, alternative logging tools can be used.

  • These steps can also be skipped in development environments if logging is not needed.

MOSIP uses Rancher Fluentd and elasticsearch to collect logs from all services and reflect the same in Kibana Dashboard.

  • Install Rancher FluentD system : for scraping logs outs of all the microservices from MOSIP k8 cluster.

    • Install Logging from Apps and marketplace within the Rancher UI.

    • Select Chart Version 100.1.3+up3.17.7 from Rancher console -> Apps & Marketplaces.

  • Configure Rancher FluentD

    • Create clusteroutput

      • kubectl apply -f clusteroutput-elasticsearch.yaml

    • Start clusterFlow

      • kubectl apply -f clusterflow-elasticsearch.yaml

    • Install elasticsearch, kibana and Istio addons\

    • set min_age in elasticsearch-ilm-script.sh and execute the same.

    • min_age : is the minimum no. of days for which indices will be stored in elasticsearch.

    • MOSIP provides set of Kibana Dashboards for checking logs and throughput's.

      • Brief description of these dashboards are as follows:

        • 01-logstash.ndjson contains the logstash Index Pattern required by the rest of the dashboards.

        • 02-error-only-logs.ndjson contains a Search dashboard which shows only the error logs of the services, called MOSIP Error Logs dashboard.

        • 03-service-logs.ndjson contains a Search dashboard which show all logs of a particular service, called MOSIP Service Logs dashboard.

        • 04-insight.ndjson contains dashboards which show insights into MOSIP processes, like the number of UINs generated (total and per hr), the number of Biometric deduplications processed, number of packets uploaded etc, called MOSIP Insight dashboard.

        • 05-response-time.ndjson contains dashboards which show how quickly different MOSIP Services are responding to different APIs, over time, called Response Time dashboard.

  • Import dashboards:

    • cd K8_ROOT/logging

    • ./load_kibana_dashboards.sh ./dashboards <cluster-kube-config-file>

  • View dashboards

Open kibana dashboard from https://kibana.sandbox.xyz.net.

Kibana --> Menu (on top left) --> Dashboard --> Select the dashboard.

13. MOSIP External Dependencies setup

External Dependencies are set of external requirements that are needed for functioning of MOSIP’s core services like DB, Object Store, HSM etc.

Click here to check the detailed installation instructions of all the external components.

Note:

  • Connect to mosip_pms DB in postgres and execute the query to change valid_to_date for mpolicy-default-mobile in pms.auth_policy table.

    • Open the terminal.

    • Use the psql command to connect to the PostgreSQL server. The general syntax is:

      • : The server address (e.g., localhost or an IP address).

      • Assuming other details remain same like port and user.

Configuration change in case using Openssl wildcard ssl certificate. (Only advised in development env, not recommended for Production setup)

  • Add/Update the below property in application-default.properties and comment on the below property in the *-default.properties file in the config repo.

  • Add/Update the below property in the esignet-default.properties file in the config repo.

14. MOSIP Modules Deployment

  • Now that all the Kubernetes cluster and external dependencies are already installed, will continue with MOSIP service deployment.

  • While installing a few modules, installation script prompts to check if you have public domain and valid SSL certificates on the server. Opt option n as we are using self-signed certificates. For example:

  • Start installing mosip modules:

Note:

  • In case of failure on execution of install-all.sh follow MOSIP Modules Deployment installation steps from the failure point.

  • Installation for config-server and admin service is facing delay in this version hence need to execute below mentioned commands respectively for config-server and admin-service once the script fails.

    • For config server:

      • Update failureThreshold for startupProbe to 60.

    • For admin service:

      • Update failureThreshold for startupProbe to 60.

      • Once admin service is up and running re-execute install.sh script after commenting out below mentioned commands:

15. API Testrig

MOSIP’s successfull deployment can be verified by comparing the results of api testrig with testrig benchmark.

  • Navigate to the Infra Root Directory:

  • Clone the Functional Tests Repository

  • After cloned successfully try to install apitestrig

  • Make script executable

  • Run the Installer Script

    Note:

    • Script prompts for below mentioned inputs please provide as and when needed:

      • Enter the time (hr) to run the cronjob every day (0–23): Specify the hour you want the cronjob to run (e.g., 6 for 6 AM)

      • Do you have a public domain and valid SSL certificate? (Y/n):

        • Y – If you have a public domain and valid SSL certificate

        • n – If you do not have one (recommended only for development environments)

      • Retention days to remove old reports (Default: 3): Press Enter to accept the default or specify another value (e.g., 5).

      • Provide Slack Webhook URL to notify server issues on your Slack channel: (change the URL to your channel one)

      • Is the eSignet service deployed? (yes/no):

        • no – If eSignet is not deployed, related test cases will be skipped.

      • Is values.yaml for the apitestrig chart set correctly as part of the prerequisites? (Y/n): * Enter Y if this step is already completed.

      • Do you have S3 details for storing API-Testrig reports? (Y/n):

        • Enter Y to proceed with S3 configuration.

        • S3 Host: eg. http://minio.minio:9000

        • S3 Region:(Leave blank or enter your specific region, if applicable)

        • S3 Access Key:admin

16. DSL Rig

  • Install Packetcreater

    • Navigate to the packetcreator directory:

    • Run the installation script:

    • NFS server details provide the following inputs:

      • NFS Host:

      • NFS PEM File :

      • User for SSH Login: ubuntu

    • Select the Ingress Controller Type: *

      1. Ingress

        1. Istio (Choose 2)

  • Install DSLrig

    • Navigate to the dslrig directory:

  • Run the installation script:

    Note: Before running install.sh, please ensure the following is added in the Helm install command:

    After adding the above, the full Helm installation command should look like this:

    • When prompted, provide the following inputs:

      • NFS Host:

      • NFS PEM :

      • User for SSH Login:ubuntu

      • Cronjob Time (hour of the day, 0–23): (e.g., enter 6 for 6 AM)

      • Do you have a public domain and valid SSL certificate? (Y/n):

        • Y: If you have a valid public domain and SSL certificate

        • n: Use only in development environments

      • Packet Utility Base URL:

      • Retention Days to Remove Old Reports (default is 3): (Press Enter to accept default or provide a custom value)

PreviousOn-Premises DeploymentNextAWS Deployment

Last updated

Was this helpful?