Menu
28 March Blog

Distributed Data Exchange Resource Server

Smart cities produce huge amounts of data from a diverse set of sources with data providers ranging from various public, government and private entities. For example, IoT sensor data for various monitoring tasks, location data for buses, crowd-sourced issue reporting data etc. Exchanging data via centralised access service may lead to performance bottlenecks, data storage and scalability issues. Further, data providers may not want to route their data through external servers and would only want to serve data to the interested consumers directly. To enable efficient exchange of data in such scenarios India Urban Data Exchange (IUDX) admits a federated data-access service architecture and allows for third party resource servers to coexist with other IUDX services. In this article we are going to look at the architectural constructs involved in getting an external resource server integrated with an existing IUDX system.

Components of IUDX System

Figure 1: Data Exchange architecture

Data Exchange consists of a Catalogue Server; used for discovery of resources, an Authorization Server; which helps in user and consent management and Resource Server(s); which acts as a data plane for publishing and consuming data. Out of the three components, IUDX has implemented Authorization service and Catalogue service as centralised servers and allows the resource access services to operate in either centralised (or) distributed fashion.

An IUDX compliant external resource server will need to serve data as per the standard IUDX resource access APIs and the resource access should be as per the access control policies set by the provider. To achieve easy integration of external data sources with the exchange ecosystem, while not requiring the data to be routed through a central resource server, IUDX provides a ‘Resource Server Proxy’ component that abstracts out most of the standardised data exchange work flows and reduces the effort of integration drastically. We now describe the functions and interfaces of Resource Server Proxy in a bit more detail.

Resource Server Proxy

The Resource Server Proxy is a software layer that enables an external data source a.k.a non IUDX system to become IUDX compliant. It can be integrated with external data sources made available via direct access to the database or data APIs for serving relevant data. The Resource Server Proxy consists of the following components:

  • Resource Access Layer
  • Integration with Authorization Server
  • Integration with Catalogue Server
  • Metering and Auditing Layer

Resource Access Layer

A data provider a.k.a data owner can implement a layer called the Resource Access Layer on top of their database system (or) API servers. This layer allows the users (consumers) to fetch data using the IUDX Resource Server APIs. The resources served through the Resource Access Layer shall be registered in the centralised catalogue server where the consumers can understand the meta-data of the resource. The consent management for the resource shall also be managed by the centralised Authorization server.

Integration with Authorization Server

In order to integrate with the centralised Authorization Server a.k.a Consent Management Server, the Resource Access Layer should integrate with the Authorization Server APIs for downloading the JWT certificates to be used to authenticate/validate the Java Web Tokens (JWT) presented by the consumers.

Integration with Catalogue Server

In order to integrate with the centralised Resource Meta-Data (Catalogue) Server, the Resource Access Layer should integrate with the Resource Meta-Data Server API for listing the resources and their identities (ID’s). With this API, the Resource Access Server can restrict the resource identities (ID’s) a consumer can present to it. 

Metering and Auditing Layer

In order to understand the usage of a resource and APIs, the Resource Access Layer with the help of a metering and auditing layer, integrates with an immutable database for storing auditing information. This information helps: IUDX Administrators (for planning of new APIs based on usage), Data Exchange Consumers (for usage planning) and Data Providers (for enabling flexible policies, understanding data consumption and usage).

The above discussion can be summarised with the help of following figure:

Figure 1: Data Exchange architecture

The integration effort required to incorporate external data sources is reduced to implementing the data integration layer by the external parties and all other interactions are taken care of by the resource server proxy layer. IUDX resource server proxy layer is available as an open source and pluggable component to enable ease of integration with proprietary databases and APIs. The distributed resource servers enable a scalable and efficient exchange of data between two parties without the need of intermediate routing of data. IUDX resource server proxy layer is a vital component that will accelerate the process of more and more resource providers joining the IUDX platform thereby enriching the diversity of data available with the platform and eventually leading to development of richer data based applications.

Distributed Resource Server On Field

The concepts described above have been put to test on the field. The Smart City Living Lab at the International Institute of Information Technology Hyderabad (IIITH) in collaboration with IUDX, have developed, tested and productized the above idea of distributed Resource Server after understanding the on field constraints and challenges.  The Smart City Research Centre of IIITH is set up with support from MEITY (Government of India), Smart City Mission and Government of Telangana. IIITH’s Living Lab has been collaborating with the IUDX for seamless data exchange between IIITH’s data monitoring systems and data consumers across the country using Resource Server Proxy.