Building Block View

Following is a break-down of the system and its components to show interactions with and within the system.

Users

The web-based user-interface (website) is the main method for interacting with the GDI node.

When a researcher visits the GDI User Portal, the Portal will use the underlying APIs of the GDI node for presenting node-specific information to the user, however, this interaction is not part of this view.

For using management components, users need a session established through national authentication service.

For using secure processing environment (SPE, a.k.a SAPU in HPC), users need to log in using two factor authentication (currently evaluating: MyAccessID). However, SPE is not part of the GDI node website.

Microservices

Frontend modules will call the general REST API endpoints for requesting or updating the system data. Since the API is extensive, it is divided into sections of microservices, though to the frontend, it is all one big uniform API to use (i.e. the domain-name is the same, but request-paths vary). There are also some microservices (data-pipeline) that are only used privately, i.e. their API is not publicly accessible.

Central Services

As shown in chapter 3, some central GDI services interact with the GDI nodes. Following image depicts the central services interacting with the microservices (through HTTP-based API endpoints). The image just shows only those interactions that are affected by the central GDI services.

Note that the central services interact only with data-catalogue (for retrieving/filtering metadata).

Storage

All microservices rely on PostgreSQL (a.k.a Postgres) for storing their information in tables. In addition, Kafka is used for publishing reacting to system- and user-events.

In some cases, local file-systems can be used for storing temporary or quickly accessible files:

• data-pipeline may store produced catalogue and Beacon genomic variations info in files for the data-catalogue.
• data-catalogue may read catalogue and Beacon genomic variations info from files to quickly respond to searches (details about this will be clarified in 2025).

Within storages, the data are organised into catalogues (Postgres schemas, Minio buckets) and storage units (Postgres tables, Minio objects). The following image depicts potential data grouping solution.

Genomic variations of the Beacon could be stored in Parquet files (to be evaluated and confirmed in 2025) in the Minio storage.

The image also includes external data-sources where the system can pull data from (on demand) to a pipeline or SPE.