Version: 1.0.0

• Name: MTS 1.0

• Date: 6th October, 2022

• Release Notes

Version: 1.0.1

• Name: MTS 1.0.1

• Date: 9th November, 2022

• Release Notes

Release Name: MTS 1.1

Release Version: Version 1.1.0

Upgrade From: MTS 1.0

Release Date: TBD

Overview

Features covered

• Delivery type: Callback

• Output formatting

• Configurable mandatory fields

• Processes records in millions

• Process multiple requests simultaneously

• Auth request data expiry

Compatibility

MTS 1.0.0 is compatible with following versions of MOSIP IDA:

• MOSIP v1.1.5.x

• MOSIP v1.2.0.x

Repositories released

Documentation (WIP)

• For a basic overview, refer architecture and high level design

• Functional user stories:

  • Authentication and token issuance using MTS with input/output formats as stated below:

    • <Input: JSON/CSV format> and <Output: JSON/CSV format> - #MOSIP-23029

    • <Input: ODK format> and <Output: JSON/CSV format> - #MOSIP-23224

• For API documentation, click here

• For installation, refer README

• Known Issues

Overview

MOSIP Token Seeder (MTS) is a standalone service that outputs MOSIP Token ID for a given input list of UIN/VIDs after performing authentication with IDA. The service is a convenience module that makes it easy for Relying parties to perform bulk authentication to onboard users to their systems. Refer use cases section for details on the usage of MTS.

Features of MTS

• Bulk upload

• Support for multiple inputs and outputs (see diagram below). For instance, a CSV file may be uploaded, and the downloaded file will contain a column with tokens populated.

• Support for multiple delivery types

• REST interface

• PII at rest is encrypted. Further, the PII is erased after processing

• Works in asynchronous mode - queues all the requests

• Processing:

  • Processes multiple records per request

  • Processes multiple requests simultaneously

• Enables configurations:

  • Output formatting of fields

  • Setting up fields as mandatory/optional

  • Defining data clean-up policy

Use cases

One of the intended use cases of MTS is to seed existing beneficiary registries for deduplication.

Inputs

1. CSV

2. ODK based

3. JSON

4. Google Sheets (TBD)

5. Form.IO Sheets (TBD)

6. Verifiable Credentials (VC) (TBD)

Outputs

1. CSV

2. JSON

Delivery type

1. Download

2. Callback

3. Synchronous response

4. WebSub (TBD)

5. SFTP (TBD)

Processing

Process multiple records per request

MTS is capable of processing millions of records per request. There is no specific limitation to the number of records it can handle per request. Refer here for details.

Process multiple requests simultaneously

MTS processes multiple requests in a simultaneous manner, rather than a sequential pattern. Refer here for details.

Configurations

Output formatting of fields

When MTS receives an authentication and tokenization request, it processes the request and sends out the output, in the requested format. Additionally, whilst formulating the output response, MTS is capable of sending the response back, based on the preferred field mapping. e.g. A request received may carry fields First name and Last name. However, the requesting party may need the response to carry fields as per the mapped naming convention: First name to FN and Last name to LN. MTS enables this requirement, through configuration of the output format template which provides the flexibility to define the field mapping as preferred. Refer here for details.

Setting up fields as mandatory/optional

Every request that MTS receives comprises of a set of fields. However, MTS provides the flexibility of defining which field is mandatory/optional as part of the request. This is a one time activity that will have to be carried out at the time of initial installation setup of MTS. Based on this definition, MTS validates each request for the presence of mandatory fields. If this config file is not setup distinctly, then the default IDA setup will be considered. Refer here for details. Refer <installation guide> for setup related information.

Defining data clean-up policy

For each request received and processed, the data is held in MTS. Whilst the data is held in-memory, MTS provides a feature to clean-up the data held and also define the timeframe based on which the data clean-up job may be run. This will help control the volume of data stored and also limit the availability of data for potential security threats. The process of defining the policy is a one time activity that will have to be carried out at the time of initial installation setup of MTS. Refer here for details. Refer <installation guide> for setup related information.

Design

Authtoken API

Authtoken API is a RESTful interface to accept various auth request input for the Token Seeder system. The API works in a complete asynchronous mode. Relying party is returned a request identifier when they make successful authtoken request. Status check API can be used to poll the status of the request placed. In case the status returns a processed state, the output can be accessed, as configured in the primary request for. Eg. If the request was for a file download, the file download API can be called to return the output file. Refer MOSIP Token Seeder API for a detailed API documentation.

Token seeder

Token seeder is a batch processing module which initiates the token authentication process. Once a new request is enqueued into the token seeder, it fetches the same and sends the request on a record level to the authenticator module. Token seeder is also responsible for updating the success and failure status to the database. There is also a expiry program for clearing off the request already processed from the system based on the expiry settings configured.

Authenticator

Authenticator process takes in a valid authentication request and performs the demographic authentication with the MOSIP IDA server. Each auth request is well formed, encrypted and signed before its sent to the MOSIP IDA. It passes on the response received to the caller regardless of the status received. Authenticator module can also be used as a individual library outside of MOSIP token seeder for any use case it applies to.

Other links

API documentation

Refer API

Source code

Refer GitHub

Test

Refer Test cases

Installation

Refer README

Overview

MTS Connector (MTS-C) is a OpenG2P module which will be an addon to Odoo. MOI will help in fetching the tokenized data from the ODK Central by calling the MOSIP Token Seeder (MTS) and store the same in beneficiary registry. This will be an important module in deduplication process when OpenG2P system is using MOSIP as its id platform.

Features of MTS-C

• Generates MOSIP token while fetching from the ODK

• Uses callback delivery type of MTS

• Completely asynchronous execution

• OpenG2P can schedule a daily job to fetch the delta for the day

• A manual import feature will also be provided

Input

In OpenG2P, the user can configure for following fields to setup an interface with MTS.

Name: A string to identify the connector

URL to reach MTS: URL for MTS API

MTS Input type: MTS-C connects over "ODK" which is the first option in this selection. OMC option could be proceeded by selecting "OpenG2P Registry".

Mapping: MTS Field mapping as required by the API. Please refer MTS Documentation**.** Format of Mapping would be JSON.

Output Type: MTS-C only supports JSON output type of MTS.

Output Format: Output format is a JQ string which will be used by MTS to format its output to suite the caller's requirement.

Delivery Type: Currently supporting only "Callback". Callback feature can be used to make MTS do a submission of results onto an API within Odoo. The output formatting will help in making the desired input for the api.

Job Type: MTS-C provide both recurring and one time execution. Recurring can be configured to do continuous pull from the ODK over MTS.

Interval in minutes: Interval at which the MTS-C job runs.

MOSIP Language: Mosip language setup. Default is "eng".

ODK Base URL: Base URL or the complete domain address for the ODK central installation

ODK Odata url : OData service (.svc) URL for the ODK form to fetch the submissions.

ODK User email: Email Id to authenticate MTS for accessing Odata URL

ODK User password: Password used to authenticate Odata URL

Callback URL: A URL end point which would be called upon successful processing at MTS

Callback HTTP Method: HTTP Method (POST/PUT/GET/PATCH) used while MTS makes the callback

Callback Timeout: Timeout awaited by the callback until acknowledged with a response.

Callback Auth Type: Type of authentication expected by callback url. MTS-C currently support Odoo type which uses the session-based authentication implemented by Odoo.

Callback Auth Database: DB instance used by Odoo.

Callback auth username: Username to access callback api

Callback auth password: Password to access callback api

This page details only on very specific areas of MOSIP Token Seeder. For a elaborate understanding on the MTS API, please refer the API documentation page.

API documentation

Refer API documentation.

CSV format

When using above format, you may not need any mapping configuration. But in case you change any column name in the csv, please do provide the same in the mapping configuration.

Output might have mix of successful and failed records except for the case where whole of the input throws error. If successful, the record would be having the token placed against the vid. And if there is error processing a record, the same is updated against the vid and the error code and description is mentioned along.

Status messages

submitted

submitted is the first status immediately after you have placed a authtoken request.

invalid

If in case there is basic validation error such that the request could not be processed, the request in marked as invalid.

submitted_with_errors

Once the system passes through the basic validation but has found that none of the records can go through due to varied record level validation issues, the system will update the request with submitted_with_errors status.

processing

This status is updated when the seeder enqueues the request for processing.

processed

Once the request is completed processing and ready for a result, the system updates the record with processed status. There can be error codes mentioned in the output line items in case few but not entire records have any processing error.

processed_with_errors

When the request is processed but every record in the request has some or other processing errors, the system updates the request with processed_with_errors status.

Delay in status update

In case there is a prior request placed with considerably higher number of records, and you have placed subsequent request submitted even before getting output for your earlier request, the system might take a while to update the status of your newer request. It might be still in the submitted state until the system finds a window to start processing.

Callback

Call back functionality enables MTS api to submit the output to a specified URI. Callback might also need authorization on most occasions. MTS currently supports following authorization types.

Bearer Token

If the caller can supply a token while calling the MTS api, it can be configured in the authStaticBearer attribute of callbackProperties section.

"authStaticBearer": {
        "token": "string"
      }

Odoo

The functionality is added to support the OpenG2P use case where in OpenG2P odoo modules can call the MTS and get the result directly imported to odoo database. Following are the parameters supported.

"authOdoo": {
        "authUrl": "http://example.com",
        "database": "string",
        "username": "string",
        "password": "string",
        "extraHeaders": {}
      },

OAuth

OAuth protocol is also supported in MTS callback functionality. This helps numerous systems which implements OAuth specification to integrate with MTS seamlessly. Below are the configurations available to setup OAuth based callback.

"authOauth": {
        "authUrl": "http://example.com",
        "username": "",
        "password": "",
        "clientId": "",
        "clientSecret": "",
        "extraHeaders": {}
      },

Error codes

There are cases where MTS might successfully pass on the request but IDA generates error based on the its implementation scenario. MTS will log such error directly to the output json/csv/file. In any case there are uncaught errors thrown by IDA, MTS will output the same as unknown error (ATS-REQ-100).

Mapping config

The names on the left side of the mapping config denotes the original expected names and the value part will hold the change of name, if any. The mapping config if not supplied, the system will assume that there are no changes in the names used. The same applies if any one or more element of the mapping is not provided.

"mapping": {
      "vid": "vid_1",                                          //default:"vid"
      "name": [                                                //default:"name"
        "firstname","middlename","lastname"
      ],
      "gender": "gender",                                      //default:"gender"
      "dob": "dob",                                            //default:"dob"
      "phoneNumber": "phoneNumber",                            //default:"phoneNumber"
      "emailId": "emailId",                                    //default:"emailId"
      "fullAddress": [                                         //default:"fullAddress"
        "addressline1", "addressline2","city","state","pincode"
      ]
    }

Except for name and full address, the majority of the fields in authdata are direct mapping. MTS expects that there can be exceptions for name and full address for which the mapping is configured as a string array. For example, if the calling application or program stores the address fields in separate variables or columns like addressline1, addressline2, street, area, or zipcode; the same can be supplied directly as authdata with mapping supplied as a list of variable or column names as in the calling program.

Output format

With the output formatting capability, MTS can give you the result exactly the way in which you would want it ready for your further processing of data. The output format string to be supplied follows the jq format. In case the output format is not supplied, the default output will be generated which will be the same as the previous versions. The below section details on an example case for output formatting.

Sample authdata Input for MTS

{
    "vid": "456564345214",
    "demographics":{
            "fname": "Peter",
            "lname": "Parker",
            "gender": "male"
        },
    "contacts": [{
        "email": "peter.parker@somewhere.com",
        "fulladdress":"Cecilia Chapman, 711-2880 Nulla St.,Mankato Mississippi 96522, (257) 563-7401"
    }]
}

Output format string

"{
    \"pcn\": .input.vid,
    \"firstName\": .input.demographics.fname,
    \"lastName\": .input.demographics.lname,
    \"gender\": .input.demographics.gender,
    \"email\": .input.contacts[0].lname,
    \"address\": .input.contacts[0].fulladdress,
    \"authToken\": .output.token,
    \"authTokenStatus\": .output.status,
    \"authTokenError\": (.output.errorCode + \"::\" + .output.errorMessage),
}"

Authtoken output

{
    "pcn": "456564345214",
    "firstName": "Peter",
    "lastName": "Parker",
    "gender": "male",
    "email": "peter.parker@somewhere.com",
    "address": "Cecilia Chapman, 711-2880 Nulla St.,Mankato Mississippi 96522, (257) 563-7401",
    "authToken": "2944061782623593820388819382121346429",
    "authTokenStatus": "error",
    "authTokenError": ""   
}

**input**keyword in the output format string is used to represent the data you input into MTS as authdata parameter. output keyword is used to identify that you intend to pick the data from the default auth request result. Please find below a sample for the default auth request output.

{
    "vid": "456564345214",
    "token": "2944061782623593820388819382121346429",
    "status": "success",
    "errorCode": "",
    "errorMessage": "",
}

Field formats

Release Name: MTS 1.0.1

Release Version: Version 1.0.1

Upgrade From: Version 1.0.0

Release Date: 9th November, 2022

Overview

The 1.0.1 version of MTS is the first patch release above version 1.0. This patch release mainly comprises of fixes for bugs identified in the previous release.

Issues fixed

For more information, refer .

Compatibility

MTS 1.0.1 is compatible with following versions of MOSIP IDA:

• MOSIP v1.1.5.x

• MOSIP v1.2.0.x

Release artifacts

Known issues

Release Name: MTS 1.0

Release Version: Version 1.0.0

Upgrade From: NA

Release Date: 6th October, 2022

Overview

The 1.0.0 version of MTS covers basic features of as listed below. This version is tested for functionality. Non-functional requirements (Performance, scale etc.) will be taken up in subsequent releases.

Features covered

Authentication and token issuance using MOSIP Token Seeder with input/output formats and specifications as stated below:

• Input type: CSV, JSON, ODK

• Output type: CSV, JSON

• Delivery type: Download

• Asynchronous token seeding

• Status API

• Download API

• Authfield Mapping

• Dockerization

• Helm chart

• Processes 5k entries in one request

Compatibility

MTS 1.0.0 is compatible with following versions of MOSIP IDA:

• MOSIP v1.1.5.x

• MOSIP v1.2.0.x

Repositories released

2. Test repositories

Documentation

• Functional user stories:

  • Authentication and token issuance using MTS with input/output formats as stated below:

This page details only on very specific areas of MOSIP Token Seeder. For a elaborate understanding on the MTS API, please refer the API documentation page.

API documentation

Refer .

CSV format

When using above format, you may not need any mapping configuration. But in case you change any column name in the csv, please do provide the same in the mapping configuration.

Output might have mix of successful and failed records except for the case where whole of the input throws error. If successful, the record would be having the token placed against the vid. And if there is error processing a record, the same is updated against the vid and the and description is mentioned along.

Status messages

submitted

submitted is the first status immediately after you have placed a authtoken request.

invalid

If in case there is basic validation error such that the request could not be processed, the request in marked as invalid.

submitted_with_errors

Once the system passes through the basic validation but has found that none of the records can go through due to varied record level validation issues, the system will update the request with submitted_with_errors status.

processing

This status is updated when the seeder enqueues the request for processing.

processed

Once the request is completed processing and ready for a result, the system updates the record with processed status. There can be error codes mentioned in the output line items in case few but not entire records have any processing error.

processed_with_errors

When the request is processed but every record in the request has some or other processing errors, the system updates the request with processed_with_errors status.

Delay in status update

In case there is a prior request placed with considerably higher number of records, and you have placed subsequent request submitted even before getting output for your earlier request, the system might take a while to update the status of your newer request. It might be still in the submitted state until the system finds a window to start processing.

Error codes

There are cases where MTS might successfully pass on the request but IDA generates error based on the its implementation scenario. MTS will log such error directly to the output json/csv/file. In any case there are uncaught errors thrown by IDA, MTS will output the same as unknown error (ATS-REQ-100).

Mapping config

The names on the left side of the mapping config denotes the original expected names and the value part will hold the change of name, if any. The mapping config if not supplied, the system will assume that there are no changes in the names used. The same applies if any one or more element of the mapping is not provided.

Except for name and full address, the majority of the fields in authdata are direct mapping. MTS expects that there can be exceptions for name and full address for which the mapping is configured as a string array. For example, if the calling application or program stores the address fields in separate variables or columns like addressline1, addressline2, street, area, or zipcode; the same can be supplied directly as authdata with mapping supplied as a list of variable or column names as in the calling program.

Field formats

Overview

OpenG2P-registry MTS Connector (OMC) is a OpenG2P module which will be an addon to . OMC can generate a MOSIP Auth Token against the any record in OpenG2P registry by calling (MTS) and store the same in beneficiary registry. This will be an important module in deduplication process when OpenG2P system uses MOSIP as its ID platform.

Features of OMC

• Generates MOSIP token against the OpenG2P registry by calling .

• Uses callback delivery type of MTS

• Completely asynchronous execution

• OpenG2P can schedule a daily job to fetch the delta for the day

• A manual import feature will also be provided

• Removes VID if MOSIP Auth Token is generated

Input

In OpenG2P, the user can configure for following fields to setup an interface with MTS through OMC.

Name: A string to identify the connector

URL to reach MTS: URL for MTS API

MTS Input type: OMC option could be proceeded by selecting "OpenG2P Registry".

Mapping: MTS Field mapping as required by the API. Please refer MTS Documentation. Format of Mapping would be JSON.

Output Type: MTS-C only supports JSON output type of MTS.

Delivery Type: Currently supporting only "Callback". Callback feature can be used to make MTS do a submission of results onto an API within Odoo. The output formatting will help in making the desired input for the api.

Job Type: MTS-C provide both recurring and one time execution. Recurring can be configured to do continuous pull from the ODK over MTS.

MOSIP Language: Mosip language setup. Default is "eng".

Interval in minutes: Interval at which the MTS-C job runs.

List of fields to be used: List of fields which will be supplied as auth data. This field list may be a superset of fields required for auth as it may contain data required by the callback API. This list should be a valid JSON string array.

Callback URL: A URL end point which would be called upon successful processing at MTS

Callback HTTP Method: HTTP Method (POST/PUT/GET/PATCH) used while MTS makes the callback

Callback Timeout: Timeout awaited by the callback until acknowledged with a response.

Callback Auth Type: Type of authentication expected by callback URL. MTS-C currently support Odoo type which uses the session-based authentication implemented by Odoo.

Callback Auth Database: DB instance used by Odoo.

Callback auth username: Username to access callback api

Callback auth password: Password to access callback api