Constraints

This section describes important constraints that have been identified and should be considered in the design of the architecture.

Existing infrastructure

  1. Estonian Biobank (research institution) is using High-Performance Computing Center (HPC) of University of Tartu for handling high-scale data-storage and data-processing.
  2. TEHIK is hosting the national health-data (but not yet genomic data), which is protected by the law, contracts, and technical means.
  3. Tartu University Hospital has its own technical infrastructure for sequencing and storing full genomes (clinical case).
  4. Kubernetes is currently the most common deployment environment. Minio is the most-likely storage service. Both of them are already in place and in active use.

Data Scope

  1. The GDI node needs to be able to incorporate full genomic data from both clinical and research institutions.
  2. Consistent pseudonyms need to be used for linking data to individuals.
  3. Data providers maintain but they can also use their data within the node.
  4. Data providers can update their data (more samples, new reference genomes).
  5. VCF is the most preferred data-format for research (and for extracting genomic variations for data-discovery).
  6. Other preferred file formats: BAM, CRAM.
  7. System should limit the size of files based on the storage system.
  8. Internally, system must store its data-files encrypted.

Interfaces

  1. The GDI node must provide a web-based user-interface: website with documentation, contacts, and administration portal.
    • User Portal functionality at GDI node level is initially not in the scope.
  2. The GDI node must support user authentication through the national authentication service (e.g. TARA in Estonia).
  3. The GDI node must provide following interfaces to the central User Portal:
    • Data Catalogue
    • Beacon v2 API
  4. The GDI node must provide an SPE (e.g. SAPU in HPC) for approved research.
  5. The GDI node should interface with existing storage solutions (S3).
  6. At this stage, other interfaces are not yet planned, but are provisioned (TEHIK, hospitals).

System Management

  1. The system must support more than one organisations who can manage their users.
  2. The system must provide a set of permissions for managing permitted actions per user.
  3. The system and user activity must be auditable.
  4. Background processes need to be visible in the system (may require a permission).
  5. Help-desk needs elevated management permissions.

Data Management

  1. Data providers manage their data as datasets, which consist of genomic and metadata.
  2. Genomic data is stored at the Data provider in their own storage, where the GDI node system is given read-access.
  3. Datasets are formed by dataset manifests that reference the dataset files by path.
  4. Datasets support versioning (through a naming convention).
  5. Visibility and accessibility of dataset-versions (but not content) must be user-controlled through the GDI node system.
  6. The system may support data-processing pipelines for various needs. For example: generating Beacon-compatible data from a VCF.
  7. The GDI node system must enable schema manangement for dataset manifests, as there can be more than one valid schemas depending on the data use case.

Data Access

  1. Before requesting access, a data-cohort must be defined in the system.
  2. The data providers having their data in the cohort must be notified of the data-access request.
  3. The data providers may veto the use of their data (within a predefined time-frame) before approved access becomes effective.
  4. Data providers may use user-permissions to permit data processing within the GDI node for their members.
  5. In case of external researchers, the system must check GA4GH Passport & Visa information in the JWT for checking user’s permission to read data.
  6. The genomic data within the GDI node storage cannot be exported from the system.

Data Analysis

  1. Setting up an SPE for data analysis is a manual interactive process between the researcher and the node help-desk. It includes signing a legal contract.
  2. The GDI node may charge the researcher for consumed resources (e.g. data storage, CPU, and memory).
  3. Data provider may require review of analysis results before the researcher can view them.
  4. Data provider has read-only access to the research project, output-files and logs.