Release Name: MTS 1.1

Release Version: Version 1.1.0

Upgrade From: MTS 1.0

Release Date: TBD

Overview

Features covered

• Delivery type: Callback

• Output formatting

• Configurable mandatory fields

• Processes records in millions

• Process multiple requests simultaneously

• Auth request data expiry

Compatibility

MTS 1.0.0 is compatible with following versions of MOSIP IDA:

• MOSIP v1.1.5.x

• MOSIP v1.2.0.x

Repositories released

Documentation (WIP)

• For a basic overview, refer architecture and high level design

• Functional user stories:

  • Authentication and token issuance using MTS with input/output formats as stated below:

    • <Input: JSON/CSV format> and <Output: JSON/CSV format> - #MOSIP-23029

    • <Input: ODK format> and <Output: JSON/CSV format> - #MOSIP-23224

• For API documentation, click here

• For installation, refer README

• Known Issues

Version: 1.0.0

• Name: MTS 1.0

• Date: 6th October, 2022

• Release Notes

Version: 1.0.1

• Name: MTS 1.0.1

• Date: 9th November, 2022

• Release Notes

Overview

MOSIP Token Seeder (MTS) is a standalone service that outputs for a given input list of UIN/VIDs after performing authentication with . The service is a convenience module that makes it easy for to perform bulk authentication to onboard users to their systems. Refer section for details on the usage of MTS.

Features of MTS

• Bulk upload

• Support for multiple and (see diagram below). For instance, a CSV file may be uploaded, and the downloaded file will contain a column with tokens populated.

• Support for multiple

• REST interface

• PII at rest is encrypted. Further, the PII is erased after processing

• Works in asynchronous mode - queues all the requests

• :

  • Processes multiple records per request

  • Processes multiple requests simultaneously

• Enables :

  • Output formatting of fields

  • Setting up fields as mandatory/optional

  • Defining data clean-up policy

Use cases

One of the intended use cases of MTS is to seed existing beneficiary registries for deduplication.

Inputs

4. Google Sheets (TBD)

5. Form.IO Sheets (TBD)

6. Verifiable Credentials (VC) (TBD)

Outputs

1. CSV

2. JSON

Delivery type

1. Download

3. Synchronous response

4. WebSub (TBD)

5. SFTP (TBD)

Processing

Process multiple records per request

Process multiple requests simultaneously

Configurations

Output formatting of fields

Setting up fields as mandatory/optional

Defining data clean-up policy

Design

Authtoken API

Token seeder

Token seeder is a batch processing module which initiates the token authentication process. Once a new request is enqueued into the token seeder, it fetches the same and sends the request on a record level to the authenticator module. Token seeder is also responsible for updating the success and failure status to the database. There is also a expiry program for clearing off the request already processed from the system based on the expiry settings configured.

Authenticator

Other links

API documentation

Source code

Test

Installation

Release Name: MTS 1.0

Release Version: Version 1.0.0

Upgrade From: NA

Release Date: 6th October, 2022

Overview

The 1.0.0 version of MTS covers basic features of MOSIP Token Seeder as listed below. This version is tested for functionality. Non-functional requirements (Performance, scale etc.) will be taken up in subsequent releases.

Features covered

Authentication and token issuance using MOSIP Token Seeder with input/output formats and specifications as stated below:

• Input type: CSV, JSON, ODK

• Output type: CSV, JSON

• Delivery type: Download

• Asynchronous token seeding

• Status API

• Download API

• Authfield Mapping

• Dockerization

• Helm chart

• Processes 5k entries in one request

Compatibility

MTS 1.0.0 is compatible with following versions of MOSIP IDA:

• MOSIP v1.1.5.x

• MOSIP v1.2.0.x

Repositories released

1. MTS repository

2. Test repositories

   • Functional tests

Documentation

• For a basic overview, refer architecture and high level design

• Functional user stories:

  • Authentication and token issuance using MTS with input/output formats as stated below:

    • <Input: JSON/CSV format> and <Output: JSON/CSV format> - #MOSIP-23029

    • <Input: ODK format> and <Output: JSON/CSV format> - #MOSIP-23224

• For API documentation, click here

• For installation, refer README

• Known issues

This page details only on very specific areas of MOSIP Token Seeder. For a elaborate understanding on the MTS API, please refer the API documentation page.

API documentation

Refer API documentation.

CSV format

When using above format, you may not need any mapping configuration. But in case you change any column name in the csv, please do provide the same in the mapping configuration.

Output might have mix of successful and failed records except for the case where whole of the input throws error. If successful, the record would be having the token placed against the vid. And if there is error processing a record, the same is updated against the vid and the error code and description is mentioned along.

Status messages

submitted

submitted is the first status immediately after you have placed a authtoken request.

invalid

If in case there is basic validation error such that the request could not be processed, the request in marked as invalid.

submitted_with_errors

Once the system passes through the basic validation but has found that none of the records can go through due to varied record level validation issues, the system will update the request with submitted_with_errors status.

processing

This status is updated when the seeder enqueues the request for processing.

processed

Once the request is completed processing and ready for a result, the system updates the record with processed status. There can be error codes mentioned in the output line items in case few but not entire records have any processing error.

processed_with_errors

When the request is processed but every record in the request has some or other processing errors, the system updates the request with processed_with_errors status.

Delay in status update

In case there is a prior request placed with considerably higher number of records, and you have placed subsequent request submitted even before getting output for your earlier request, the system might take a while to update the status of your newer request. It might be still in the submitted state until the system finds a window to start processing.

Error codes

There are cases where MTS might successfully pass on the request but IDA generates error based on the its implementation scenario. MTS will log such error directly to the output json/csv/file. In any case there are uncaught errors thrown by IDA, MTS will output the same as unknown error (ATS-REQ-100).

Mapping config

The names on the left side of the mapping config denotes the original expected names and the value part will hold the change of name, if any. The mapping config if not supplied, the system will assume that there are no changes in the names used. The same applies if any one or more element of the mapping is not provided.

"mapping": {
      "vid": "vid_1",                                          //default:"vid"
      "name": [                                                //default:"name"
        "firstname","middlename","lastname"
      ],
      "gender": "gender",                                      //default:"gender"
      "dob": "dob",                                            //default:"dob"
      "phoneNumber": "phoneNumber",                            //default:"phoneNumber"
      "emailId": "emailId",                                    //default:"emailId"
      "fullAddress": [                                         //default:"fullAddress"
        "addressline1", "addressline2","city","state","pincode"
      ]
    }

Except for name and full address, the majority of the fields in authdata are direct mapping. MTS expects that there can be exceptions for name and full address for which the mapping is configured as a string array. For example, if the calling application or program stores the address fields in separate variables or columns like addressline1, addressline2, street, area, or zipcode; the same can be supplied directly as authdata with mapping supplied as a list of variable or column names as in the calling program.

Field formats

This page details only on very specific areas of MOSIP Token Seeder. For a elaborate understanding on the MTS API, please refer the API documentation page.

API documentation

Refer API documentation.

CSV format

When using above format, you may not need any mapping configuration. But in case you change any column name in the csv, please do provide the same in the mapping configuration.

Output might have mix of successful and failed records except for the case where whole of the input throws error. If successful, the record would be having the token placed against the vid. And if there is error processing a record, the same is updated against the vid and the error code and description is mentioned along.

Status messages

submitted

submitted is the first status immediately after you have placed a authtoken request.

invalid

If in case there is basic validation error such that the request could not be processed, the request in marked as invalid.

submitted_with_errors

Once the system passes through the basic validation but has found that none of the records can go through due to varied record level validation issues, the system will update the request with submitted_with_errors status.

processing

This status is updated when the seeder enqueues the request for processing.

processed

Once the request is completed processing and ready for a result, the system updates the record with processed status. There can be error codes mentioned in the output line items in case few but not entire records have any processing error.

processed_with_errors

When the request is processed but every record in the request has some or other processing errors, the system updates the request with processed_with_errors status.

Delay in status update

In case there is a prior request placed with considerably higher number of records, and you have placed subsequent request submitted even before getting output for your earlier request, the system might take a while to update the status of your newer request. It might be still in the submitted state until the system finds a window to start processing.

Callback

Call back functionality enables MTS api to submit the output to a specified URI. Callback might also need authorization on most occasions. MTS currently supports following authorization types.

Bearer Token

If the caller can supply a token while calling the MTS api, it can be configured in the authStaticBearer attribute of callbackProperties section.

"authStaticBearer": {
        "token": "string"
      }

Odoo

The functionality is added to support the OpenG2P use case where in OpenG2P odoo modules can call the MTS and get the result directly imported to odoo database. Following are the parameters supported.

"authOdoo": {
        "authUrl": "http://example.com",
        "database": "string",
        "username": "string",
        "password": "string",
        "extraHeaders": {}
      },

OAuth

OAuth protocol is also supported in MTS callback functionality. This helps numerous systems which implements OAuth specification to integrate with MTS seamlessly. Below are the configurations available to setup OAuth based callback.

"authOauth": {
        "authUrl": "http://example.com",
        "username": "",
        "password": "",
        "clientId": "",
        "clientSecret": "",
        "extraHeaders": {}
      },

Error codes

There are cases where MTS might successfully pass on the request but IDA generates error based on the its implementation scenario. MTS will log such error directly to the output json/csv/file. In any case there are uncaught errors thrown by IDA, MTS will output the same as unknown error (ATS-REQ-100).

Mapping config

The names on the left side of the mapping config denotes the original expected names and the value part will hold the change of name, if any. The mapping config if not supplied, the system will assume that there are no changes in the names used. The same applies if any one or more element of the mapping is not provided.

"mapping": {
      "vid": "vid_1",                                          //default:"vid"
      "name": [                                                //default:"name"
        "firstname","middlename","lastname"
      ],
      "gender": "gender",                                      //default:"gender"
      "dob": "dob",                                            //default:"dob"
      "phoneNumber": "phoneNumber",                            //default:"phoneNumber"
      "emailId": "emailId",                                    //default:"emailId"
      "fullAddress": [                                         //default:"fullAddress"
        "addressline1", "addressline2","city","state","pincode"
      ]
    }

Except for name and full address, the majority of the fields in authdata are direct mapping. MTS expects that there can be exceptions for name and full address for which the mapping is configured as a string array. For example, if the calling application or program stores the address fields in separate variables or columns like addressline1, addressline2, street, area, or zipcode; the same can be supplied directly as authdata with mapping supplied as a list of variable or column names as in the calling program.

Output format

With the output formatting capability, MTS can give you the result exactly the way in which you would want it ready for your further processing of data. The output format string to be supplied follows the jq format. In case the output format is not supplied, the default output will be generated which will be the same as the previous versions. The below section details on an example case for output formatting.

Sample authdata Input for MTS

{
    "vid": "456564345214",
    "demographics":{
            "fname": "Peter",
            "lname": "Parker",
            "gender": "male"
        },
    "contacts": [{
        "email": "peter.parker@somewhere.com",
        "fulladdress":"Cecilia Chapman, 711-2880 Nulla St.,Mankato Mississippi 96522, (257) 563-7401"
    }]
}

Output format string

"{
    \"pcn\": .input.vid,
    \"firstName\": .input.demographics.fname,
    \"lastName\": .input.demographics.lname,
    \"gender\": .input.demographics.gender,
    \"email\": .input.contacts[0].lname,
    \"address\": .input.contacts[0].fulladdress,
    \"authToken\": .output.token,
    \"authTokenStatus\": .output.status,
    \"authTokenError\": (.output.errorCode + \"::\" + .output.errorMessage),
}"

Authtoken output

{
    "pcn": "456564345214",
    "firstName": "Peter",
    "lastName": "Parker",
    "gender": "male",
    "email": "peter.parker@somewhere.com",
    "address": "Cecilia Chapman, 711-2880 Nulla St.,Mankato Mississippi 96522, (257) 563-7401",
    "authToken": "2944061782623593820388819382121346429",
    "authTokenStatus": "error",
    "authTokenError": ""   
}

**input**keyword in the output format string is used to represent the data you input into MTS as authdata parameter. output keyword is used to identify that you intend to pick the data from the default auth request result. Please find below a sample for the default auth request output.

{
    "vid": "456564345214",
    "token": "2944061782623593820388819382121346429",
    "status": "success",
    "errorCode": "",
    "errorMessage": "",
}

Field formats

Overview

OpenG2P-registry MTS Connector (OMC) is a OpenG2P module which will be an addon to Odoo. OMC can generate a MOSIP Auth Token against the any record in OpenG2P registry by calling MOSIP Token Seeder (MTS) and store the same in beneficiary registry. This will be an important module in deduplication process when OpenG2P system uses MOSIP as its ID platform.

Features of OMC

• Generates MOSIP token against the OpenG2P registry by calling MTS.

• Uses callback delivery type of MTS

• Completely asynchronous execution

• OpenG2P can schedule a daily job to fetch the delta for the day

• A manual import feature will also be provided

• Removes VID if MOSIP Auth Token is generated

Input

In OpenG2P, the user can configure for following fields to setup an interface with MTS through OMC.

Name: A string to identify the connector

URL to reach MTS: URL for MTS API

MTS Input type: OMC option could be proceeded by selecting "OpenG2P Registry".

Mapping: MTS Field mapping as required by the API. Please refer MTS Documentation. Format of Mapping would be JSON.

Output Type: MTS-C only supports JSON output type of MTS.

Output Format: Output format is a JQ string which will be used by MTS to format its output to suite the caller's requirement.

Delivery Type: Currently supporting only "Callback". Callback feature can be used to make MTS do a submission of results onto an API within Odoo. The output formatting will help in making the desired input for the api.

Job Type: MTS-C provide both recurring and one time execution. Recurring can be configured to do continuous pull from the ODK over MTS.

MOSIP Language: Mosip language setup. Default is "eng".

Interval in minutes: Interval at which the MTS-C job runs.

Filters to apply to Registry: A domain filter can be used to identify the records for tokenization. For. eg. Only records which have VID associated with it and is not tokenized need to be picked for tokenization.

List of fields to be used: List of fields which will be supplied as auth data. This field list may be a superset of fields required for auth as it may contain data required by the callback API. This list should be a valid JSON string array.

Callback URL: A URL end point which would be called upon successful processing at MTS

Callback HTTP Method: HTTP Method (POST/PUT/GET/PATCH) used while MTS makes the callback

Callback Timeout: Timeout awaited by the callback until acknowledged with a response.

Callback Auth Type: Type of authentication expected by callback URL. MTS-C currently support Odoo type which uses the session-based authentication implemented by Odoo.

Callback Auth Database: DB instance used by Odoo.

Callback auth username: Username to access callback api

Callback auth password: Password to access callback api

Release Name: MTS 1.0.1

Release Version: Version 1.0.1

Upgrade From: Version 1.0.0

Release Date: 9th November, 2022

Overview

The 1.0.1 version of MTS is the first patch release above version 1.0. This patch release mainly comprises of fixes for bugs identified in the previous release.

Issues fixed

For more information, refer .

Compatibility

MTS 1.0.1 is compatible with following versions of MOSIP IDA:

• MOSIP v1.1.5.x

• MOSIP v1.2.0.x

Release artifacts

Known issues

Overview

MTS Connector (MTS-C) is a OpenG2P module which will be an addon to . MOI will help in fetching the tokenized data from the ODK Central by calling the (MTS) and store the same in beneficiary registry. This will be an important module in deduplication process when OpenG2P system is using MOSIP as its id platform.

Features of MTS-C

• Generates MOSIP token while fetching from the ODK

• Uses callback delivery type of MTS

• Completely asynchronous execution

• OpenG2P can schedule a daily job to fetch the delta for the day

• A manual import feature will also be provided

Input

In OpenG2P, the user can configure for following fields to setup an interface with MTS.

Name: A string to identify the connector

URL to reach MTS: URL for MTS API

MTS Input type: MTS-C connects over "ODK" which is the first option in this selection. OMC option could be proceeded by selecting "OpenG2P Registry".

Output Type: MTS-C only supports JSON output type of MTS.

Delivery Type: Currently supporting only "Callback". Callback feature can be used to make MTS do a submission of results onto an API within Odoo. The output formatting will help in making the desired input for the api.

Job Type: MTS-C provide both recurring and one time execution. Recurring can be configured to do continuous pull from the ODK over MTS.

Interval in minutes: Interval at which the MTS-C job runs.

MOSIP Language: Mosip language setup. Default is "eng".

ODK Base URL: Base URL or the complete domain address for the ODK central installation

ODK Odata url : OData service (.svc) URL for the ODK form to fetch the submissions.

ODK User email: Email Id to authenticate MTS for accessing Odata URL

ODK User password: Password used to authenticate Odata URL

Callback URL: A URL end point which would be called upon successful processing at MTS

Callback HTTP Method: HTTP Method (POST/PUT/GET/PATCH) used while MTS makes the callback

Callback Timeout: Timeout awaited by the callback until acknowledged with a response.

Callback Auth Type: Type of authentication expected by callback url. MTS-C currently support Odoo type which uses the session-based authentication implemented by Odoo.

Callback Auth Database: DB instance used by Odoo.

Callback auth username: Username to access callback api

Callback auth password: Password to access callback api