# Home
Welcome to the official page for OpenCGA documentation.
OpenCGA is the most advanced big data genomic analysis platform. It is implemented as an open-source project that implements a high-performance, scalable and secure platform for Genomic data analysis and visualisation.
OpenCGA provides the most advanced and complete genomic data platform. The performance, scalability and huge number features makes of OpenCGA an unique full-stack solution today. OpenCGA takes care of security and implements a high-performance query engine and analysis frameworks for *Big Data* analysis and visualisation in current genomics. OpenCGA uses the most modern and advanced technologies, and has been designed and implemented to scale to hundreds of thousands if genomes accounting for petabytes of variant data. It is built on top of three main components: *Catalog Metadata Database*, *Variant Storage Engine* and *Analysis Framework*.
## Main Features
* **Authenticated** and **secure** platform to query and visualise data. An advanced **permission** system has been implemented to ensure data privacy.
* A **metadata database** to keep track of registered users, projects, studies, files, samples, families, jobs and other entities.
* **Advanced Clinical Data** database implemented, users can define their data models for samples, patients or families.
* **Alignment storage** allows to index BAM/CRAM, calculate index and query data and coverage
* The most advanced, high-performance and scalable **Variant Storage Engine** solution today. Variant Storage Engine can normalise, load, index, aggregate, annotate and precompute variant stats for hundreds of thousands of whole genomes.
* **Analysis Framework** implemented on top of variant and alignment storage engines. OpenCGA comes with many analysis already implemented such as GWAS. Users can easily extend OpenCGA functionality by implementing a plugin or connecting to a external binary.
* Real **Big Data Analytics** supported, you can use different computing frameworks such as MapReduce or Spark on top HBase or Parquet files.
* Full **Clinical Analysis Solution** implemented, you can create the cases and run different clinical interpretations algorithms from your scripts or from a web application.
* Rich and comprehensive [**RESTful Web Services API**](http://docs.opencb.org/display/opencga/RESTful+Web+Services) with more than 160 endpoints to manage, query and analyse metadata, variants, alignments and clinical data.
* Easy **programmatic access** and **pipeline integration** thanks to the four different **client libraries** developed in **Java,** [**Python**](http://docs.opencb.org/display/opencga/Python)**,** [**R**](http://docs.opencb.org/display/opencga/R) and **Javascript**
* Interactive **web-based application** to query, analyse and visualise variants, alignments and clinical data
{% hint style="info" %}
**Zetta Genomics** is a start-up launched in 2019 to offer official support and customisation of your OpenCB applications.
Zetta offers advanced data management systems for precision medicine based on the OpenCB applications. Find more information about this consolidated and demanding initiative at
{% endhint %}
\*\*\*\*
\*\*\*\*
# Overview
OpenCGA is an open-source platform that aims to provide a full stack solution for big data analysis and visualisation of genomic data. OpenCGA has been designed to provide a secure, high-performance and scalable solution for genomics analysis and visualisation.
OpenCGA implements a complete solution that covers all aspects of genomic analysis: metadata database, authentication and security, variant normalisation and aggregation, variant storage and annotation, highly scalable variant NoSQL storage engine, alignment and coverage, big data variant analysis, RESTful web services, visualisation
OpenCGA is developed and maintained in the University of Cambridge and it is currently used by several big data projects such as GEL (Genomics England).
## Main Features
OpenCGA provides a complete solution for genomics data analysis:
* **Authenticated** and **secure** platform to query and visualise data, advanced **permission** system
* A **metadata database** to keep track of registered users, projects, studies, files, samples, families, jobs,
* **Clinical data** from sample, patients or families
* **Alignment storage** allows to index BAM/CRAM, calculate index and query data and coverage
* The most advanced, high-performance and scalable **Variant storage** solution, you can normalise, load, index and aggregate thousands of whole genomes per day
* **Genomic Analysis** implemented on top of variant and alignment storage layer using advanced technologies such as Spark
* Full **clinical analysis platform** implemented, you can create the cases and run different clinical interpretations algorithms from your scripts or from a web application
* Comprehensive **RESTful web service API** with more than 150 endpoints to fully query and manage all metadata and clinical data
* Four different **client libraries** implemented in Java, Python, R and Javascript
* Interactive **web-based application** for the analysis and visualisation of variants and reads
## Projects
OpenCGA is used by several projects being the most important [Genomics England (NHS)](https://www.genomicsengland.co.uk/).
# Features
Overview of main features
In this section, you will find a summary of the main features of OpenCGA.
## Metadata Catalog and Security
OpenCGA Catalog is one of the most important components. Catalog implements the data models, allow custom annotations, implement permissions, ... An audit system has also been implemented.
### Catalog Data Models and Annotations
* **Rich data models** implemented for studies, files, samples, individuals, families, ...
* Advanced **free data model** implemented for storing **custom annotations** such as stats or clinical data from patients. Users can define **confidential annotations** as only visible for authorised users.
### Catalog Database
* Catalog database has been implemented using MongoDB to provide a **high-performance** and **scalable query engine**.
* Catalog can use Solr as a secondary index to calculate **complex annotations** and **stats**.
### Authentication and Permissions
* OpenCGA comes with a built-in **authentication** system. Other systems are also supported such as LDAP or Microsoft Azure AD (*under development*). Authentication **tokens** use [**JWT**](https://jwt.io/) standard which facilitates the creation of federated systems.
* Advanced and efficient resource **permission** system implemented in Catalog. You can define different permissions such as VIEW, WRITE or DELETE at study level or at any specific document. This allow to **share** data with other users. More information at [Sharing and Permissions](http://docs.opencb.org/display/opencga/Sharing+and+Permissions).
## Variant Storage
OpenCGA provides a framework for implementing *big data* variant storage engines which support: real-time queries, interactive complex data aggregations, full-text search, variant analysis, ... The framework takes care of several common operations such as variant normalisation, sample genotype aggregation, variant stats calculation, variant annotation, secondary indexing or in-memory cache. Two different engines are implemented using NoSQL databases: MongoDB and HBase. A secondary index using Solr is nicely integrated with the two implementations. By implementing variant storage engines with NoSQL databases we ensure a fast response time and high concurrent queries.
#### Data Management
* Advanced **variant normalisation** implemented supporting multi-allelic split or left-alignment of INDELs among others.
* High quality **sample genotype aggregation** supporting multi-allelic variants, overlapping SNV-INDEL or structural variants. HBase storage engine can aggregate tens of thousands of samples efficiently. Current design and implementation should scale to hundreds of thousands of samples.
* **Dynamic** variant storage, you can add or remove samples dynamically from the variant storage efficiently
* Rich and efficient **variant data model** implemented. Variant data models support different studies, file information, sample information, and rich variant annotation. **Sample genotypes** are **efficiently** stored to scale to hundreds of thousands of genotypes, this allows to optimise analysis by minimising the disk usage and memory consumption.
* **Structural variants** are fully supported incliuding SNV, INDEL, insertion, deletions, CNV, ...
* Multi-cohort **variant stats** supported. Users can define different cohorts (group of samples) and precompute and index their variant stats, this allows a real-time queries or aggregations. A default cohort called ***all*** is managed automatically.
* [CellBase](http://docs.opencb.org/display/cellbase/CellBase+Home) high-performance **variant annotation** tool is integrated providing rich variant annotations which are stored and indexed, this allows a real-time queries or aggregations. Variant annotation data is returned with the variants since it is part of the data model. Multiple variant annotation can be stored and fetched.
* Custom **variant scores** from external analysis tools such as GWAS association can be loaded, indexed and queried by.
* **Export** variant data in different formats such as VCF or Parquet. You can filter which variants and samples are exported.
#### Query Engine
* OpenCGA implements a **very sophisticated query engine** supporting the combination of more than 25 filters: region, genes, type, file attributes, sample genotypes, consequence types, population frequencies, biotype, conservation scores, variant and gene clinical traits, mode of inheritance, disease panels, ... **Full-text search** is also implemented.
* Other **query options** supported such as *include, exclude, limit, skip, count,* ...
* Some **basic analysis implemented** such as compound heterozygous, *de novo* variants, sex imputation, unique variant saturation, ...
* Variant query engine supports filtering by **sample clinical data** thanks to the integration with *Catalog.*
* MongoDB or HBase are fully integrated with **Solr secondary indexes** to provide a real-time query engine for all queries and use cases.
#### Aggregation and Stats
* Solr integration allows the execution of **complex aggregations** (faceted search) interactively. Nested and range aggregations are supported. For instance, you can aggregate variants by chromosome and type over 46 million variants in just 2 seconds:
* **Variant query filters** – for filtering variants – and **aggregation** analysis can be combined to calculate the aggregation of any variant query result.
* **Aggregation stats** such as average, median, percentile, min, max, ... are also supported
#### Big Data Analysis
* Variants can be exported to **parquet** file which is an efficient columnar file format. This parquet file can be used by Hive or Spark big data technologies.
* Some **complex analysis** such as IBS are implemented using a custom Spark library to extend the number of uses cases supported. Note that this analysis can take some time and Spark is not a highly concurrent technology, therefore this analysis are queued by OpenCGA.
* Variant data model store genotypes efficiently ensuring we can execute analysis with tens of thousands of samples.
#### Performance and scalability
* HBase storage engine have been implemented to provide **real-time queries** and **interactive aggregations** *(faceted)* even with tens of thousands of whole genomes.
* **Google gRPC** is used as an alternative to REST (JSON) to improve performance.
* Some benchmarks with more than 11,000 whole genomes accounting for 25TB show that we can load more than 2,000 files a day and execute most queries in less than 1-2 seconds in a small Hadoop cluster of 20 nodes.
* You can go to [HGVA](http://hgva.opencb.org/#home/reference_grch37/UK10K) to test OpenCGA query engine performance. HGVA uses OpenCGA and IVA and load about 700 million unique variants from different human studies.
## Clinical Analysis
OpenCGA aims to provide a full solution for Clinical Genomics analysis, this covers patient clinical data, interpretation algorithms and a pathogenic variant database.
### Clinical Data
* Catalog can **store and index any clinical data model** for samples, individuals or families. Models are defined by users.
* User can configure the **permission** and **visibility** of clinical data using *Catalog* permissions.
### Clinical Interpretation Analysis
* Open a **patient case** study by creating a *clinical analysis*, this contains all the patient and family data from Catalog at that moment, the phenotype to be analysed or the files among other information. A rich **interpretation data model** has also been modelled – combining GEL and other data models – to capture all the relevant information from the interpretation.
* Complete **disease panel** management implemented: create, update and delete disease panels. You can also import them automatically from [**PanelApp (GEL)**](https://panelapp.genomicsengland.co.uk/). Updated panels are **versioned** to keep track of existing interpreted analysis.
* Several **rare disease interpretation analysis** implemented such as TEAM or Tiering which is based on GEL RD Tiering tool (Cancer interpretation analysis coming soon). You can use one or more disease panels in the interpretation analysis.
* You can save more than one interpretation analysis result in the *clinical analysis* to create one or more **clinical reports**.
* Together with a tier classification a **semi-automatic ACMG classification** has been also implemented.
### Pathogenic Variant Database
* Interpreted variants – and their variant annotation – can be indexed in a high-performance **pathogenic variant database**. Clinical data from *catalog,* the *clinical analysis* and *interpretation* are also indexed together with interpreted variants.
* Real-time **queries** and **complex aggregations** have been implemented.
## **Alignment Storage**
OpenCGA can manage alignment data. BAM files can be indexed and coverage calculated.
### **Fetching alignments**
* **Query indexed BAM files**, allowed filters include by region, mapping quality, number of mismatches, properly paired, ...
* GA4GH data model used for alignments
* **Google gRPC** is used as an alternative to REST (JSON) to improve performance.
### **Coverage**
* **Coverage** can be calculated and stored in a *BigWig* file.
* **Coverage queries** at any window size or zoom.
## RESTful Web Services
OpenCGA implements more than 150 RESTful web services to allow users to manipulate and query Catalog metadata and data such as *alignment*, *variants* and *pathogenic variants*. REST web services are documented using [Swagger](https://swagger.io/), you can see OpenCGA Swagger documentation at . To facilitate the usage all of these web services we have implemented different client libraries and a command line (see below in *Usability).*
REST web services can be grouped in different categories: *Catalog, Alignment, Variant, Clinical* and *Admin.*
### Catalog
* Catalog **data manipulation**, you can create, update, delete change permission of data.
* **Advanced search** web services to query any resource (file, samples, ...)
### Alignment
* You can **index** BAM files to query reads and calculate **coverage** in BigWig format
* **Query** endpoint to fetch alignments in GA4GH format from several files. Filters implemented include: region, mapping quality, number of mismatches, number of hits, properly paired, ...
### Variant
* Query variant endpoint allows to query variants by any variant filter. Full control of which fields are returned
* Aggregation stats implemented.
* Others: fetch old variant annotation, variant study metadata, ...
### Clinical
* Several web services to create clinical analysis, execute interpretations or query pathogenic variant database.
### Admin
* Administrative web services, only OpenCGA root user can execute them
## Usability
### REST Clients
* Four REST clients have been implemented in different programming language: Java, Pythong, R and JavaScript.
### Command-line Interface (CLI)
* A fully functional command-line has been implemented
## Visualisation
### OpenCGA web catalog
* Web-based application to query and aggregate metadata from catalog
### IVA
* Web-based application for Intercative Variant Analysis
* Highly customisable
* Plugin oriented
### Genome Browser
* Genome browser for NGS
# Architecture Diagram
# Security
# Metadata and Clinical Data
## Metadata
OpenCGA implements an advanced Metadata Database ([OpenCGA Catalog](/develop-2/components-1/catalog)) that keeps track of ...
## Clinical Data
OpenCGA provides a rich set of data models covering the most commons needs for any genomic analysis...
# Data Models
## Implementation
### Diagram
## Common Data Models
### Annotation Set
| Field | Description |
| ----- | ----------- |
| | |
### Phenotype
Describe a phenotype following an OBO ontology.
| Field | Description |
| -------------------------------------------------------- | ----------- |
|
id
String
| |
|
name
String
| |
|
source
String
| |
|
ageOfOnset
String
| |
|
status
Status
| |
|
attributes
Map
| |
### Creation and Modification Date
### Status
### File ID
File IDs contain the path using `:`
# Sample
## Overview
Sample data model ...
## Data Models
### Sample
This is the main data model, it stores the most basic and important information. You can find the Java [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/sample/Sample.java).
| Field | Description |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
id
String
|
Unique Sample ID in the study, this can be repeated across different studies. This is a mandatory parameter in the creation and cannot be changed at the moment.
Constraints: Mandatory, Unique, Immutable
|
|
uuid
String
|
Global unique ID in any study of any OpenCGA installation. This is created during the sample creation and cannot be changed.
Constraints: Internal, Unique, Immutable
|
|
individualId
String
| A reference to the [Individual](/develop-2/overview/data-models/individual) containing this sample. Notice that samples can exist without and Individual ID, this field is not mandatory. |
|
fileIds
List\
| List of [File ID](/develop-2/overview/data-models#file-id) containing this sample, eg BAM, VCF, QC images, ... |
|
| |
### SampleProcessing
This object describes how the sample was processed in the lab.
| Field | Description |
| --------------------------------------------------------------- | -------------------------------------------------------------- |
|
product
String
| Type of product sequenced, this can be DNA or RNA for instance |
|
preparationMethod
String
| |
|
extractionMethod
String
| |
|
labSampleId
String
| |
|
quantity
String
| |
|
date
String
| |
### SampleCollection
### SampleQualityControl
## Example
This is a full JSON example:
```javascript
{
id: "ISDBM322015",
uuid: "eba13afe-0172-0004-0001-d4c92fd95e0a",
individualId: "ISDBM322015",
fileIds: [
"data:quartet.variants.annotated.vcf.gz",
"SonsAlignedBamFile.bam"
],
annotationSets: [],
description: "",
somatic: false,
qualityControl: {
fileIds: [],
comments: [],
alignmentMetrics: [
{
bamFileId: SonsAlignedBamFile.bam,
fastQc: {13 items},
samtoolsFlagstats: {14 items},
geneCoverageStats: [2 items]
}
],
variantMetrics: {
variantStats: [1 item],
signatures: [],
vcfFileIds: []
}
},
release: 1,
version: 5,
creationDate: "20200625131831",
modificationDate: "20200709003738",
phenotypes: [
{
id: "HP:0000545",
name: "Myopia",
source: "HPO"
}
],
status: {
name: "",
description: "",
date: ""
},
internal: {
status: {
name: "READY",
date: "20200625131831",
description: ""
}
},
attributes: {
OPENCGA_INDIVIDUAL: {
id: "ISDBM322015",
name: "ISDBM322015",
uuid: "eba13738-0172-0006-0001-283471b7ae69",
father: {4 items},
mother: {4 items},
location: {},
qualityControl: {4 items},
sex: "MALE",
karyotypicSex: "XY",
ethnicity: "",
population: {},
release: 1,
version: 6,
creationDate: "20200625131830",
modificationDate: "20201027004616",
lifeStatus: "ALIVE",
phenotypes: [2 items],
disorders: [1 item],
parentalConsanguinity: false,
status: {3 items},
internal: {1 item},
attributes: {}
}
}
}
```
# Individual
# Alignment and Coverage
# Variant Query and Analysis
# Clinical Data Analysis
# Running Analysis
# Scalability and Performance
# Sizing OpenCGA
# OpenCGA Architecture
# Catalog
Metadata and Clinical Data Database
## Overview
A genomic data analysis platform need to keep track of different resources such as metadata of files, sample annotations or jobs. OpenCGA Catalog aims to collect and integrate all the information needed for executing genomic analysis. This information is organized in nine main entities: *users*, *studies*, *files*, *samples*, *datasets*, *cohorts*, *individuals*, *disease panels* and *jobs*.
## Main Features
The main tasks of Catalog are to provide:
* Authentication and authorization to the different resources.
* A collaborative environment.
* File audit to keep track of files and metadata.
* Analysis and Jobs.
* Sample, individual and cohort annotation.
* Security
* Versioning
## Data Models
This section describes the most relevant entities. For more detailed information about the data models such as Java source code, examples or the JSON Schemas you can visit [OpenCGA Catalog Data Models](https://github.com/opencb/opencga/wiki/OpenCGA-Catalog-Data-Models) page. You can see an overview of the data model in this picture:
The most relevant entities in OpenCGA Catalog are:
* User: Contains the data related to the user account.
* Project: Contains information of a project, covering as many related studies as necessary.
* Study: Main space set environment. Contain files, samples, individuals, jobs...
* File: Information regarding a submitted or generated file.
* Sample: Information regarding the sample. Closely related to file entity.
* Individual: Contain the information regarding the individual from whom the sample has been taken.
* Cohort: Group sets of samples with some common feature(s).
* Disease panel: Define a disease panel containing the variants, genes and/or regions of interest.
* Job: Job analysis launched using any of the files or samples.
## RESTful web services
All this information can be stored and retrieved using our Java and RESTful web services API.
# Alignment and Coverage
OpenCGA Alignment Engine provides a solution to storage and process sequence alignment data from Next-Generation Sequencing (NGS) projects. The Alignment Engine supports the most common alignment file formats, i.e.: [SAM, BAM and CRAM](https://github.com/samtools/hts-specs), and takes the alignment data model specification from [**GA4GH**](http://ga4gh.org/#/) and the implementation from [**OpenCB GA4GH**](https://github.com/opencb/ga4gh). See a full description at [Alignment Data Model](http://docs.opencb.org/display/opencga/Alignment+Data+Model).
We do not define or endorse any dedicated unaligned sequence data format. Instead we recommend storing such data in one of the alignment formats (SAM, BAM, or CRAM) with the unmapped flag set.
OpenCGA alignment engine provides the following analysis:
* **Index analysis** To index a coordinate-sorted alignment file (BAM or CRAM) for fast random access. This index is needed when region parameters are used to limit the query analysis to particular regions of interest.
* **Query analysis** This analysis outputs those alignments matching the specified filters, such as minimum mapping quality, maximum insert size, maximum number of mismatches in the alignment, properly paired alignments,... In addition, users may specify one or more comma-separated regions to restrict output to only those alignments which overlap the specified region(s). Note that use of region specifications requires a coordinate-sorted and indexed input file (in BAM or CRAM format).
* **Coverage analysis** This analysis takes a coordinate-sorted and indexed alignment file (in BAM or CRAM format) as input and generates a coverage file (in BigWig format). The coverage is calculated as the number of reads per window of a user-defined size, if window size is equal to 1, the coverage is the number of reads per position. Once coverage is computed, the read coverage over multiple genomic regions can be fetched quite quickly.
* **Statistics analysis** OpenCGA computes statistics for a given alignment file by using the [samtools stats command](http://www.htslib.org/doc/samtools-stats.1.html). Alignment statistics are indexed in order to allow users to query for alignment files according to those statistics.
In addtion, OpenCGA provides wrappers to the following third-party alignment software packages:
* [**FastQC**](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/): a quality control tool for high throughput sequence data.
* [**BWA**](http://bio-bwa.sourceforge.net/): a software package for mapping low-divergent sequences against a large reference genome.
* [**Samtools**](http://www.htslib.org/): a program for interacting with high-throughput sequencing data in SAM, BAM and CRAM formats.
* [**deepTools**](https://deeptools.readthedocs.io/en/develop/): a suite of python tools particularly developed for the efficient analysis of high-throughput sequencing data, such as ChIP-seq, RNA-seq or MNase-seq.
## OpenCGA Alignment User Interfaces
OpenCGA provides two interfaces to allow users execute the alignment tools and analysis:
* Command line inteface
* RESTful web services interface
### OpenCGA command line interface
The OpenCGA command line interface to manage alignment data is accessible through the script *opencga.sh* using the command *alignments*:
| |
| - |
The tutorial [Working with Alignment Data](http://docs.opencb.org/display/opencga/Working+with+Alignment+Data) shows how to use the OpenCGA alignment commandline.
### OpenCGA RESTful web services interface
Next image shows the OpenCGA RESTfull web services to manage alignment data:
# Variant Storage Engine
## Overview
## Data Model
{% content-ref url="/pages/-MHE6uwM-UWt-O5amZL-" %}
[Broken mention](broken://pages/-MHE6uwM-UWt-O5amZL-)
{% endcontent-ref %}
# Clinical Analysis
# Analysis Framework
# Data Models
## Overview
This section describes the most relevant entities. For more detailed information about the data models such as *Java* source code, examples or the *JSON Schemas* you can visit the official [OpenCGA Catalog Data Models](https://github.com/opencb/opencga/wiki/OpenCGA-Catalog-Data-Models) code in GitHub.
A schematic diagram with the relation between OpenCGA Catalog Data Models is shown below:
### Catalog Entities
The most relevant entities in OpenCGA Catalog are:
* **User**:
Users represent the physical persons that are granted access to the database. Contains the data related to the user account.
**Project**
Projects represent the first physical separation of the data in OpenCGA. A Project entity contains information of a project, covering as many related studies as necessary.
**Study**
Studies represent the main space set environment. The Study is the parent of all the entities except project and user . It is important because most entities are defined at a study level (see the diagram above).
**File:**
Files represent the metadata about the files uploaded or linked to OpenCGA. This entity contains information regarding a submitted or generated file.
**Sample**
Information regarding the sample. Closely related to file entity.
**Individual**
Contain the information regarding the individual from whom the sample has been taken.
**Cohort**
Group sets of samples with some common feature(s).
**Disease panel:**
Define a disease panel containing the variants, genes and/or regions of interest.
**Job**
Job analysis launched using any of the files or samples.
## Design Principles
All OpenCGA Data Models have been designed to follow a list of principles. This principles are agnostic (not entity-dependant) and thus apply to all the entities. Knowing these principles allows you understand the mechanism used by Catalog to represent real world clinical metadata and to infer the structure of the data associated with any entity:
1. **Parent-Child List Relationship**
2. **Chid-Parent reference**
3. **Annotation Sets:** Catalog offers the option is to define user-custom annotation sets at any entity level.
# User
## Overview
User data model hosts information about any user.
### Summary
| Field | Create | Update | Unique | Required |
| -------------- | :-----------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------: |
| id |  |  |  |  |
| name |  |  |  |  |
| email |  |  |  |  |
| organization |  |  |  |  |
| account |  |  |  |  |
| internal |  |  |  |  |
| quota |  |  |  |  |
| projects |  |  |  |  |
| sharedProjects |  |  |  |  |
| configs |  |  |  |  |
| filters |  |  |  |  |
| attributes |  |  |  |  |
## Data Model
### User
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/user/User.java).
| Field | Description |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
|
id String
| Object ID is a mandatory parameter when creating a new one, this ID cannot be changed at the moment. |
|
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### UserFilter
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/user/UserFilter.java).
| Field | Description |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
id String
| Object ID is a mandatory parameter when creating a new one, this ID cannot be changed at the moment. |
|
description String
| Users may provide a description for the entry. |
|
resource Resource
| User resource can have the values AUDIT, USER, PROJECT, STUDY, FILE, SAMPLE, JOB, INDIVIDUAL, COHORT, DISEASE\_PANEL, FAMILY, CLINICAL\_ANALYSIS, INTERPRETATION, VARIANT, ALIGNMENT, CLINICAL, EXPRESSION, RGA and FUNCTIONAL. |
|
query Query
| User filter query |
|
query QueryOptions
| User filter query options |
### UserQuota
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/user/UserQuota.java).
### UserInternal
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/user/UserInternal.java).
### Account
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/user/Account.java).
| Field | Description |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
|
type AccountType
| User account type can have the values GUEST, FULL and ADMINISTRATOR. |
|
creationDate String
| Autogenerated date following the format YYYYMMDDhhmmss containing the date when the entry was first registered. |
|
| An integer describing the current data release. |
|
attributes Map\
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### ProjectOrganism
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/project/ProjectOrganism.java).
### ProjectInternal
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/project/ProjectInternal.java).
| Field | Description |
| ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
|
A map with related permission rules. The key of the map can have the values SAMPLES, FILES, COHORTS, INDIVIDUALS, FAMILIES, JOBS, CLINICAL\_ANALYSES and DISEASE\_PANELS. The value is a List of permission rules
| Dictionary that can be customised by users to store any additional information users may require. |
|
attributes Map\
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### Panel
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/panel/Panel.java).
| Field | Description |
| ---------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
|
uuid String
| Unique 32-character identifier assigned automatically by OpenCGA. |
|
release int
| An integer describing the current data release. |
|
version int
| OpenCGA version of this panel, this is incremented when the panel is updated. |
|
| Panel status can have the values READY or DELETED. |
|
studyUid long
| Panel reference to study. |
|
uid long
| Panel reference to study. |
### Group
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/study/Group.java).
### StudyInternal
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/study/StudyInternal.java).
| Field | Description |
| -------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
|
| Date of the last modification of the internal object. |
### StudyNotification
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/study/StudyNotification.java).
| Field | Description |
| ----------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
|
| Url of the study notification. |
### PermissionRule
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/study/PermissionRule.java).
| Field | Description |
| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
|
id String
| Object ID is a mandatory parameter when creating a new one, this ID cannot be changed at the moment. |
|
query Query
| PermissionRule query. |
|
members List\
| List of members of the permission rule. |
|
permissions List\
| List of permissions of the permission rule. |
### Status
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/common/Status.java).
### Enum Entity
*Enumeration class.* You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/common/Enums/Entity.java).
### AdditionalInfo
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/common/AdditionalInfo.java).
| Field | Description |
| ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
|
id String
| Object ID is a mandatory parameter when creating a new one, this ID cannot be changed at the moment. |
|
name String
| Name of the . |
|
description String
| Users may provide a description for the entry. |
|
type String
| Type of the additional info. |
|
attributes Map\
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### ExternalSource
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/common/ExternalSource.java).
| Field | Description |
| ---------------------------------------------------------- | --------------------- |
|
id String
| Source ID... |
|
name String
| Source name... |
|
description String
| Source description... |
|
source String
| Source ... |
|
url String
| Source ID |
### URI
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/java/net/URI.java).
### VariableSet
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/study/VariableSet.java).
### StudyType
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/study/StudyType.java).
| Field | Description |
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
|
id String
| Object ID is a mandatory parameter when creating a new one, this ID cannot be changed at the moment. |
|
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### FileExperiment
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileExperiment.java).
### FileRelatedFile
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileRelatedFile.java).
### Software
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/interpretation/Software.java).
| Field | Description |
| ------------------------------------------------------------------ | ------------------- |
|
name String
| Software name |
|
version String
| Software version |
|
repository String
| Software repository |
|
commit String
| Software commit |
|
website String
| Software website |
|
params Map\
| Software params |
### Status
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/common/Status.java).
### URI
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/java/net/URI.java).
### FileInternal
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileInternal.java).
| Field | Description |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
|
| Date of the last modification of the internal object. |
### FileQualityControl
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileQualityControl.java).
### MissingSamples
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/file/MissingSamples.java).
| Field | Description |
| ----------------------------------------------------------------- | ----------------------------- |
|
existing List\
| List of existing samples. |
|
nonExisting List\
| List of non existing samples. |
### FileStatus
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileStatus.java).
### FileInternalAlignment
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileInternalAlignment.java).
### InternalStatus
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/common/InternalStatus.java).
### FileInternalVariant
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileInternalVariant.java).
# Sample
## Overview
Sample data model hosts information about any biological material, normally extracted from an *Individual*, that is used for a particular analysis. This is the main data model, it stores the most basic and important information.
### Summary
| Field | Create | Update | Unique | Required |
| ---------------- | :-----------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------: |
| id |  |  |  |  |
| uuid |  |  |  |  |
| source |  |  |  |  |
| processing |  |  |  |  |
| collection |  |  |  |  |
| qualityControl |  |  |  |  |
| creationDate |  |  |  |  |
| modificationDate |  |  |  |  |
| description |  |  |  |  |
| somatic |  |  |  |  |
| phenotypes |  |  |  |  |
| individualId |  |  |  |  |
| fileIds |  |  |  |  |
| cohortIds |  |  |  |  |
| status |  |  |  |  |
| attributes |  |  |  |  |
| release |  |  |  |  |
| version |  |  |  |  |
| internal |  |  |  |  |
## Data Model
### Sample
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/sample/Sample.java).
| Field | Description |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
id String
| Sample data model hosts information about any biological material, normally extracted from an *Individual*, that is used for a particular analysis. This is the main data model, it stores the most basic and important information. |
|
uuid String
| Unique 32-character identifier assigned automatically by OpenCGA. |
|
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
|
release int
| An integer describing the current data release. |
|
version int
| Autoincremental version assigned to the registered entry. By default, updates does not create new versions. To enable versioning, users must set the `incVersion` flag from the /update web service when updating the document. |
|
| Describes which method was used to collect the sample. |
|
date String
| Date when the sample was collected. |
|
attributes Map\
| Attributes of the sample collection. |
### Status
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/common/Status.java).
### SampleProcessing
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/sample/SampleProcessing.java).
| Field | Description |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
|
| Describes which product was used to process the sample in the lab. |
|
preparationMethod String
| Describes which preparation method was used to process the sample in the lab. |
|
preparationMethod String
| Describes which extraction method was used to process the samplein the lab. |
|
labSampleId String
| Original id has the sample in the lab. |
|
quantity String
| Number of process has done the sample. |
|
date String
| Date when the sample was processed in the lab. |
|
attributes Map\
| Attributes of the processing. |
### SampleInternal
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/sample/SampleInternal.java).
| Field | Description |
| --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
|
| Date of the last modification of the internal object. |
### SampleQualityControl
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/sample/SampleQualityControl.java).
| Field | Description |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- |
|
files List\
| Files used for the quality control of the sample. |
|
| Describes variant quality control. |
### Phenotype
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/Phenotype.java).
| Field | Description |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
id String
| Id to identify the object |
|
ageOfOnset String
| Indicates the age of on set of the phenotype |
|
status Status
| Status of phenotype OBSERVED, NOT\_OBSERVED, UNKNOWN |
|
name String
| Object name |
|
description String
| Users may provide a description for the entry. |
|
source String
| Ontology source |
|
url String
| Ontology url |
|
attributes Map\
| Dictionary that can be customised by users to store any additional information users may require.. |
### ExternalSource
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/common/ExternalSource.java).
| Field | Description |
| ---------------------------------------------------------- | --------------------- |
|
id String
| Source ID... |
|
name String
| Source name... |
|
description String
| Source description... |
|
source String
| Source ... |
|
url String
| Source ID |
### ClinicalComment
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/ClinicalComment.java).
| Field | Description |
| ---------------------------------------------------------- | ------------------------------------- |
|
author String
| Clinical comment author |
|
message String
| Clinical comment message |
|
tags List\
| List of tags for the clinical comment |
|
date String
| Date of the clinical comment |
### RgaIndex
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/common/RgaIndex.java).
| Field | Description |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
|
status Status
| Status of the Rga index NOT\_INDEXED, INDEXED, INVALID\_PERMISSIONS, INVALID\_METADATA, INVALID. |
|
date String
| Date of Rga index. |
### InternalStatus
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/common/InternalStatus.java).
### OntologyTermAnnotation
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/core/OntologyTermAnnotation.java).
| Field | Description |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
id String
| Id to identify the object |
|
name String
| Object name |
|
description String
| Users may provide a description for the entry. |
|
source String
| Ontology source |
|
url String
| Ontology url |
|
attributes Map\
| Dictionary that can be customised by users to store any additional information users may require.. |
### SampleVariantQualityControlMetrics
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/sample/SampleVariantQualityControlMetrics.java).
| Field | Description |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
|
| Genome plot for the quality control of the sample. |
|
files List\
| File for the quality control metrics of the sample. |
### GenomePlot
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/qc/GenomePlot.java).
| Field | Description |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
|
id String
| Id to identify the object |
|
description String
| Users may provide a description for the entry. |
|
| File of the genomePlot |
### SampleQcVariantStats
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/qc/SampleQcVariantStats.java).
| Field | Description |
| ----------------------------------------------------------------- | ---------------------------------------------- |
|
id String
| Id to identify the object |
|
description String
| Users may provide a description for the entry. |
|
query Map\
| Map for query |
|
stats SampleVariantStats
| Stats result set |
|
sampleId String
| Stats result set |
### Signature
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/qc/Signature.java).
| Field | Description |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
|
id String
| Id to identify the object |
|
description String
| Users may provide a description for the entry. |
|
| List of GenomePlotTrack |
### SignatureFitting
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/qc/SignatureFitting.java).
| Field | Description |
| --------------------------------------------------------------- | ------------------------------------------ |
|
method String
| Method used to fit the signature |
|
signatureSource String
| Source of the fitting signature |
|
signatureVersion String
| Signature version of the fitting signature |
|
scores List\
| Scores of the fitting signature |
|
coeff double
| Coefficient of the fitting signature |
|
file String
| Files of the fitting signature |
### GenomeContextCount
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/qc/Signature/GenomeContextCount.java).
| Field | Description |
| ------------------------------------------------------ | ----------------------- |
|
context String
| Genome context to count |
|
total int
| Counted integer |
### GenomePlotTrack
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/qc/GenomePlotTrack.java).
| Field | Description |
| ----------------------------------------------------------------- | ------------------------------- |
|
| Individual ID in the study, this must be unique in the study but can be repeated in different studies. This is a mandatory parameter when creating a new Individual, this ID cannot be changed at the moment. |
|
uuid String
| Unique 32-character identifier assigned automatically by OpenCGA. |
|
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
|
release int
| An integer describing the current data release. |
|
version int
| Autoincremental version assigned to the registered entry. By default, updates does not create new versions. To enable versioning, users must set the `incVersion` flag from the /update web service when updating the document. |
### Location
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/individual/Location.java).
| Field | Description |
| --------------------------------------------------------- | --------------------- |
|
address String
| Location address. |
|
postalCode String
| Location postal code. |
|
city String
| Location city. |
|
state String
| Location state. |
|
country String
| Location country. |
### Status
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/common/Status.java).
### SexOntologyTermAnnotation
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/core/SexOntologyTermAnnotation.java).
| Field | Description |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
id String
| Id to identify the object |
|
name String
| Object name |
|
description String
| Users may provide a description for the entry. |
|
source String
| Ontology source |
|
url String
| Ontology url |
|
attributes Map\
| Dictionary that can be customised by users to store any additional information users may require.. |
### OntologyTermAnnotation
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/core/OntologyTermAnnotation.java).
| Field | Description |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
id String
| Id to identify the object |
|
name String
| Object name |
|
description String
| Users may provide a description for the entry. |
|
source String
| Ontology source |
|
url String
| Ontology url |
|
attributes Map\
| Dictionary that can be customised by users to store any additional information users may require.. |
### IndividualPopulation
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/individual/IndividualPopulation.java).
| Field | Description |
| ------------------------------------------------------------ | ------------------------------------------- |
|
name String
| Name of the individual population. |
|
subpopulation String
| Subpopulation of the individual population. |
|
description String
| Description of the individual population. |
### Phenotype
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/Phenotype.java).
| Field | Description |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
id String
| Id to identify the object |
|
ageOfOnset String
| Indicates the age of on set of the phenotype |
|
status Status
| Status of phenotype OBSERVED, NOT\_OBSERVED, UNKNOWN |
|
name String
| Object name |
|
description String
| Users may provide a description for the entry. |
|
source String
| Ontology source |
|
url String
| Ontology url |
|
attributes Map\
| Dictionary that can be customised by users to store any additional information users may require.. |
### Disorder
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/Disorder.java).
| Field | Description |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
id String
| Id to identify the object |
|
name String
| Object name |
|
description String
| Users may provide a description for the entry. |
|
source String
| Ontology source |
|
url String
| Ontology url |
|
attributes Map\
| Dictionary that can be customised by users to store any additional information users may require.. |
### IndividualQualityControl
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/individual/IndividualQualityControl.java).
| Field | Description |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
| Comments related to the quality control. |
### IndividualInternal
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/individual/IndividualInternal.java).
| Field | Description |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
|
| Date of the last modification of the internal object. |
### SampleRelatednessReport
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/qc/SampleRelatednessReport.java).
### MendelianErrorReport
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/qc/MendelianErrorReport.java).
### InferredSexReport
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/qc/InferredSexReport.java).
### ClinicalComment
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/ClinicalComment.java).
| Field | Description |
| ---------------------------------------------------------- | ------------------------------------- |
|
| Map of members ids and enum of roles (FATHER, MOTHER, IDENTICAL\_TWIN, SON, UNCLE, PATERNAL\_GRANDFATHER.) . |
|
attributes Map\
since: 1.0
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
|
release int
| An integer describing the current data release. |
|
version int
| Autoincremental version assigned to the registered entry. By default, updates does not create new versions. To enable versioning, users must set the `incVersion` flag from the /update web service when updating the document. |
### Status
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/common/Status.java).
### FamilyInternal
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/family/FamilyInternal.java).
| Field | Description |
| --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
|
| Date of the last modification of the internal object. |
### Phenotype
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/Phenotype.java).
| Field | Description |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
id String
| Id to identify the object |
|
ageOfOnset String
| Indicates the age of on set of the phenotype |
|
status Status
| Status of phenotype OBSERVED, NOT\_OBSERVED, UNKNOWN |
|
name String
| Object name |
|
description String
| Users may provide a description for the entry. |
|
source String
| Ontology source |
|
url String
| Ontology url |
|
attributes Map\
| Dictionary that can be customised by users to store any additional information users may require.. |
### FamilyQualityControl
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/family/FamilyQualityControl.java).
| Field | Description |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------- |
|
| Comments related to the quality control. |
### Disorder
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/Disorder.java).
| Field | Description |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
|
id String
| Id to identify the object |
|
name String
| Object name |
|
description String
| Users may provide a description for the entry. |
|
source String
| Ontology source |
|
url String
| Ontology url |
|
attributes Map\
| Dictionary that can be customised by users to store any additional information users may require.. |
### RelatednessReport
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/qc/RelatednessReport.java).
| Field | Description |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
|
method String
| Method of the relatedness report |
|
maf String
| Minor allele frequency to filter variants, e.g.: 1kg\_phase3:CEU>0.35, cohort:ALL>0.05 |
|
scores List\
| Relatedness scores for pair of samples |
|
files List\
| List of files of Relatedness Report |
### InternalStatus
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/common/InternalStatus.java).
### ClinicalComment
You can find the Java code [here](https://github.com/opencb/biodata/tree/develop/biodata-models/src/main/java/org/opencb/biodata/models/clinical/ClinicalComment.java).
| Field | Description |
| ---------------------------------------------------------- | ------------------------------------- |
|
| Object ID is a mandatory parameter when creating a new one, this ID cannot be changed at the moment. |
|
uuid String
| Unique 32-character identifier assigned automatically by OpenCGA. |
|
type CohortType
| Cohort type that can have the values CASE\_CONTROL, CASE\_SET, CONTROL\_SET, PAIRED, PAIRED\_TUMOR, AGGREGATE, TIME\_SERIES, FAMILY, TRIO and COLLECTION. |
|
creationDate String
| Autogenerated date following the format YYYYMMDDhhmmss containing the date when the entry was first registered. |
|
modificationDate String
since: 1.0
| Autogenerated date following the format YYYYMMDDhhmmss containing the date when the entry was last modified. |
|
description String
| Users may provide a description for the entry. |
|
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### CohortInternal
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/cohort/CohortInternal.java).
| Field | Description |
| --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
|
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### ExecutionResult
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/tools/result/ExecutionResult.java).
| Field | Description |
| ----------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
|
id String
| Object ID is a mandatory parameter when creating a new one, this ID cannot be changed at the moment. |
|
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### JobStudyParam
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/job/JobStudyParam.java).
| Field | Description |
| ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
|
id String
| Object ID is a mandatory parameter when creating a new one, this ID cannot be changed at the moment. |
|
others List\
| List of strings. |
### ToolInfo
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/job/ToolInfo.java).
| Field | Description |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
id String
| Object ID is a mandatory parameter when creating a new one, this ID cannot be changed at the moment. |
|
description String
| Users may provide a description for the entry. |
|
scope Scope
| Tool info scope can have the values GLOBAL, PROJECT and STUDY. |
|
type Type
| Tool info type can have the values OPERATION and ANALYSIS. |
|
resource Resource
| Tool info resource can have the values AUDIT, USER, PROJECT, STUDY, FILE, SAMPLE, JOB, INDIVIDUAL, COHORT, DISEASE\_PANEL, FAMILY, CLINICAL\_ANALYSIS, INTERPRETATION, VARIANT, ALIGNMENT, CLINICAL, EXPRESSION, RGA and FUNCTIONAL. |
### JobInternal
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/job/JobInternal.java).
| Field | Description |
| --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
| Date of the last modification of the internal object. |
### Status
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/tools/result/Status.java).
### InternalStatus
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/common/InternalStatus.java).
### JobInternalWebhook
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/job/JobInternalWebhook.java).
| Field | Description |
| --------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
|
| Webhook status map can have the values SUCCESS or ERROR. |
### ToolStep
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/tools/result/ToolStep.java).
| Field | Description |
| --------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
|
id String
| Object ID is a mandatory parameter when creating a new one, this ID cannot be changed at the moment. |
|
| Executor status can have the values PENDING, RUNNING, DONE and ERROR. |
|
attributes ObjectMap
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### URI
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/java/net/URI.java).
### Date
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/java/util/Date.java).
### ExecutionStatus
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/models/common/Enums/ExecutionStatus.java).
### ExecutorInfo
You can find the Java code [here](https://github.com/opencb/opencga/tree/TASK-1587/opencga-core/src/main/java/org/opencb/opencga/core/tools/result/ExecutorInfo.java).
| Field | Description |
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
|
id String
| Object ID is a mandatory parameter when creating a new one, this ID cannot be changed at the moment. |
|
clazz String
| ExecutorInfo class. |
|
params ObjectMap
| ExecutorInfo params. |
|
source Source
| Executor info source can have the values FILE, PARQUET\_FILE, MONGODB, HBASE, STORAGE. |
|
An object describing the internal information of the ClinicalAnalysis. This is managed by OpenCGA.
Tags: internal
|
|
attributes Map<Object,String>
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
|
status Status
| An object describing the status of the ClinicalAnalysis. |
### File
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/File.java).
| Field | Description |
| ----------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
id String
|
File ID is a mandatory parameter when creating a new File, this ID cannot be changed at the moment.
Tags: required, immutable, unique
|
|
uuid String
|
Global unique ID at the whole OpenCGA installation. This is automatically created during the File creation and cannot be changed.
An object describing the internal information of the File. This is managed by OpenCGA.
Tags: internal
|
|
attributes Map<Object,String>
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### FileExperiment
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileExperiment.java).
| Field | Description |
| ---------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
|
| |
### Enum FileExperiment.Method
*Enumeration class.* You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileExperiment/Method.java).
| Field | Description |
| ----------------------------------------------------- | ----------- |
|
WHOLE\_EXOME\_SEQUENCING
| |
|
WHOLE\_GENOME\_SEQUENCING
| |
|
TARGETED\_DNA\_SEQUENCING
| |
|
AMPLICON\_SEQUENCING
| |
|
GENOTYPING\_MICROARRAY
| |
### Enum FileExperiment.NucleicAcidType
*Enumeration class.* You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileExperiment/NucleicAcidType.java).
| Field | Description |
| ------------------------------- | ----------- |
|
DNA
| |
|
RNA
| |
### Enum FileExperiment.Technology
*Enumeration class.* You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileExperiment/Technology.java).
| Field | Description |
| -------------------------------------- | ----------- |
|
SEQUENCING
| |
|
MICROARRAY
| |
### CustomStatus
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/common/CustomStatus.java).
| Field | Description |
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
name String
|
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed vestibulum aliquet lobortis. Pellentesque venenatis lacus quis nibh interdum finibus.
Tags: required, immutable
|
|
description String
|
Proin aliquam ante in ligula tincidunt, cursus volutpat urna suscipit. Phasellus interdum, libero at posuere blandit, felis dui dignissim leo, quis ullamcorper felis elit a augue.
Tags: required
|
|
date String
|
Nullam commodo tortor nec lectus cursus finibus. Sed quis orci fringilla, cursus diam quis, vehicula sapien. Etiam bibendum dapibus lectus, ut ultrices nunc vulputate ac.
Tags: internal, unique, immutable
|
### Enum File.Format
*Enumeration class.* You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/File/Format.java).
| Field | Description |
| ---------------------------------------------------- | ----------- |
|
VCF
| |
|
BCF
| |
|
GVCF
| |
|
TBI
| |
|
BIGWIG
| |
|
SAM
| |
|
BAM
| |
|
BAI
| |
|
CRAM
| |
|
CRAI
| |
|
FASTQ
| |
|
FASTA
| |
|
PED
| |
|
TAB\_SEPARATED\_VALUES
| |
|
COMMA\_SEPARATED\_VALUES
| |
|
XML
| |
|
PROTOCOL\_BUFFER
| |
|
JSON
| |
|
AVRO
| |
|
PARQUET
| |
|
IMAGE
| |
|
PLAIN
| |
|
BINARY
| |
|
NONE
| |
|
UNKNOWN
| |
### FileQualityControl
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileQualityControl.java).
| Field | Description |
| ------------------------------------------------------------------------------------------------------------------------ | ----------- |
|
| |
### FileIndex.TransformedFile
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileIndex/TransformedFile.java).
| Field | Description |
| ------------------------------------------------------- | ----------- |
|
id long
| |
|
metadataId long
| |
### FileIndex.LocalFileIndex
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileIndex/LocalFileIndex.java).
| Field | Description |
| ------------------------------------------------------ | ----------- |
|
fileId long
| |
|
indexer String
| |
### FileIndex.IndexStatus
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileIndex/IndexStatus.java).
| Field | Description |
| ----------------------------------------------------------- | ----------- |
|
NONE String
| |
|
TRANSFORMING String
| |
|
TRANSFORMED String
| |
|
LOADING String
| |
|
INDEXING String
| |
|
STATUS\_LIST List>
| |
### FileStatus
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/FileStatus.java).
| Field | Description |
| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
|
TRASHED String
| TRASHED name means that the object is marked as deleted although is still available in the database. |
|
STAGE String
| |
|
MISSING String
| |
|
PENDING\_DELETE String
| |
|
DELETING String
| |
|
REMOVED String
| |
|
MISSING\_SAMPLES String
| |
|
STATUS\_LIST List<String>
| |
### MissingSamples
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/file/MissingSamples.java).
| Field | Description |
| ---------------------------------------------------------------- | ----------- |
|
existing List<String>
| |
|
nonExisting List<String>
| |
### Family
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/family/Family.java).
| Field | Description |
| --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
id String
|
Family is a mandatory parameter when creating a new sample, this ID cannot be changed at the moment.
Tags: required, immutable, unique
|
|
uuid String
|
Global unique ID at the whole OpenCGA installation. This is automatically created during the Family creation and cannot be changed.
An object describing the internal information of the Family. This is managed by OpenCGA.
Tags: internal
|
|
roles Map<MapString,String,FamiliarRelationship>
| |
|
attributes Map<Object,String>
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### FamilyQualityControl
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/family/FamilyQualityControl.java).
| Field | Description |
| --------------------------------------------------------------------------- | ----------- |
|
relatedness List<RelatednessReport>
| |
|
fileIds List<String>
| |
|
comments List<ClinicalComment>
| |
### Individual
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/individual/Individual.java).
| Field | Description |
| ----------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
id String
|
Individual ID in the study, this must be unique in the study but can be repeated in different studies. This is a mandatory parameter when creating a new Individual, this ID cannot be changed at the moment.
Tags: required, immutable, unique
|
|
uuid String
|
Global unique ID at the whole OpenCGA installation. This is automatically created during the sample creation and cannot be changed.
| You can use this field to store any other information, keep in mind this is not indexed so you cannot search by attributes. |
### IndividualQualityControl
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/individual/IndividualQualityControl.java).
| Field | Description |
| ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
|
inferredSexReports List<InferredSexReport>
| List of inferred sex reports, it depends on the method (currently by coverage ratio) |
|
sampleRelatednessReport SampleRelatednessReport
| |
|
mendelianErrorReports List<MendelianErrorReport>
| Mendelian errors |
|
fileIds List<String>
| File IDs related to the quality control |
|
comments List<ClinicalComment>
| Comments related to the quality control |
### Sample
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/sample/Sample.java).
| Field | Description |
| -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
id String
|
Sample ID in the study, this must be unique in the study but can be repeated in different studies. This is a mandatory parameter when creating a new sample, this ID cannot be changed at the moment.
Tags: required, immutable, unique
|
|
uuid String
|
Generic: Unique 32-character identifier assigned automatically by OpenCGA.
Contains different metrics to evaluate the quality of the sample.
Note: The sample collection is a list of samples More info at: ZetaGenomics
|
|
release int
|
An integer describing the current data release.
Tags: immutable
|
|
version int
|
Generic: Autoincremental version assigned to the registered entry. By default, updates does not create new versions. To enable versioning, users must set the incVersion flag from the /update web service when updating the document.
Tags: immutable
|
|
creationDate String
|
Generic: Autogenerated date following the format YYYYMMDDhhmmss containing the date when the entry was first registered.
Tags: immutable
|
|
modificationDate String
|
Generic: Autogenerated date following the format YYYYMMDDhhmmss containing the date when the entry was last modified.
Tags: immutable
|
|
description String
| Generic: Users may provide a description for the entry. |
|
somatic boolean
| Indicates if the sample is somatic or germline (default) |
|
Generic: Field automatically managed by OpenCGA containing relevant information of the entry. This field is used for internal purposes and is visible for users.
Tags: immutable
|
|
attributes Map<Object,String>
|
Dictionary that can be customised by users to store any additional information users may require.
Note: This field is not meant to be queried. It should only contain extra information. To store additional information meant to be queried, please use annotationSets.
|
### SampleCollection
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/sample/SampleCollection.java).
| Field | Description |
| ------------------------------------------------------------------------------ | ----------- |
|
tissue String
| |
|
organ String
| |
|
quantity String
| |
|
method String
| |
|
date String
| |
|
attributes Map<Object,String>
| |
### SampleQualityControl
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/sample/SampleQualityControl.java).
| Field | Description |
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------- |
|
Nullam commodo tortor nec lectus cursus finibus. Sed quis orci fringilla, cursus diam quis, vehicula sapien. Etiam bibendum dapibus lectus, ut ultrices nunc vulputate ac.
Tags: internal, unique, immutable
|
|
date String
|
Nullam commodo tortor nec lectus cursus finibus. Sed quis orci fringilla, cursus diam quis, vehicula sapien. Etiam bibendum dapibus lectus, ut ultrices nunc vulputate ac.
Tags: internal, unique, immutable
|
### Enum RgaIndex.Status
*Enumeration class.* You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/common/RgaIndex/Status.java).
| Field | Description |
| ------------------------------------------------ | ----------- |
|
NOT\_INDEXED
| |
|
INDEXED
| |
|
INVALID\_PERMISSIONS
| |
|
INVALID\_METADATA
| |
|
INVALID
| |
### Location
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/individual/Location.java).
| Field | Description |
| --------------------------------------------------------- | ----------- |
|
address String
| |
|
postalCode String
| |
|
city String
| |
|
state String
| |
|
country String
| |
### IndividualPopulation
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/individual/IndividualPopulation.java).
| Field | Description |
| ------------------------------------------------------------ | ----------- |
|
name String
| |
|
subpopulation String
| |
|
description String
| |
### IndividualInternal
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/individual/IndividualInternal.java).
### FamilyInternal
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/family/FamilyInternal.java).
| Field | Description |
| --------------------------------------------------------------------------------------- | ----------- |
|
| |
### Status
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/common/Status.java).
| Field | Description |
| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
name String
| |
|
date String
| |
|
description String
| |
|
message String
| |
|
READY String
| READY name means that the object is being used. |
|
DELETED String
| DELETED name means that the object is marked as removed, so it can be completely removed from the database with a clean action. |
|
STATUS\_LIST List<String>
| |
### InterpretationInternal
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/clinical/InterpretationInternal.java).
| Field | Description |
| ------------------------------------------------------------------------------------------------------- | ----------- |
|
| |
### ClinicalAnalysisStatus
You can find the Java code [here](https://github.com/opencb/opencga/blob/master/opencga-core/src/main/java/org/opencb/opencga/core/models/clinical/ClinicalAnalysisStatus.java).
| Field | Description |
| ----------------------------------------------------------------------------------- | ----------- |
|
INCOMPLETE String
| |
|
READY\_FOR\_VALIDATION String
| |
|
READY\_FOR\_INTERPRETATION String
| |
|
INTERPRETATION\_IN\_PROGRESS String
| |
|
READY\_FOR\_INTEPRETATION\_REVIEW String
| |
|
INTERPRETATION\_REVIEW\_IN\_PROGRESS String
| |
|
READY\_FOR\_REPORT String
| |
|
REPORT\_IN\_PROGRESS String
| |
|
DONE String
| |
|
REVIEW\_IN\_PROGRESS String
| |
|
CLOSED String
| |
|
REJECTED String
| |
|
STATUS\_LIST List<String>
| |
# Case Studies
## Overview
OpenCGA is being used or deployed in several big data projects, here you can find some of the most relevant.
### Genomics England - Research Environment
A total of 64,078 genomes were loaded into a small Hadoop cluster GEL Research. You can find more info at [Genomics England Research](http://docs.opencb.org/display/opencga/Genomics+England+Research).
### Microsoft Azure
# Genomics England Research
## Background
One of the goals of [The 100,000 Genomes Project](https://www.genomicsengland.co.uk/about-genomics-england/the-100000-genomes-project/) from [Genomics England](https://www.genomicsengland.co.uk) is to enable new medical research. Researchers will study how best to use genomics in healthcare and how best to interpret the data to help patients. The causes, diagnosis and treatment of disease will also be investigated. This is currently the largest national sequencing project of its kind in the world.
To achieve this goal Genomics England set up a *Research* environment for researchers and clinicians. OpenCGA, CellBase and IVA from OpenCB were installed as data platform. We loaded **64,078 whole genomes** in OpenCGA, in total **about 1 billion unique variants** were loaded and indexed in [OpenCGA Variant Storage](http://docs.opencb.org/display/opencga/Variant+Storage+Engine), and all the metadata and clinical data for samples and patients were loaded in [OpenCGA Catalog](http://docs.opencb.org/display/opencga/Catalog+and+Security). **OpenCGA was able to load and index about 6,000 samples a day**, executing the variant annotation and computing different cohort stats for the all the data run in less than a week. In summary, all data was loaded, indexed, annotated and stats calculated in less than 2 weeks. Genomic variants were annotated using [CellBase](http://docs.opencb.org/display/cellbase/CellBase+Home) and the [IVA](http://docs.opencb.org/display/iva/Welcome+to+IVA) front-end was installed for researchers and clinicians to analyse and visualise the data. In this document you can find a full report of about the loading and analysis of the 64,078 genomes.
## Genomic and Clinical Data
Clinical data and genomic variants of **64,078 genomes** were loaded and indexed in OpenCGA. In total we loaded more than **30,000 VCF** files accounting for about 40TB of compressed disk space. Data was organised in four different datasets depending on the genome assembly *(GRCh37* or *GRCh38)* and the type of study *(germline* or *somatic)*, and this was organised in OpenCGA in three different *Projects* and four *Studies:*
| Project | Study ID and Name | Samples | VCF Files | VCF File Type | Samples/File | Variants |
| ------------------- | --------------------------------------------------------- | ------- | --------- | ------------- | ------------- | --------------- |
| **GRCh37 Germline** |
| 9,589 | 9,589 | Somatic | 1.00 | **398,402,166** |
| **Total** | 64,078 | 40,676 | | | 1,421,041,774 | |
[OpenCGA Catalog](http://docs.opencb.org/display/opencga/Catalog+and+Security) stores all the metadata and clinical data of **files, samples**, **individuals** and **cohorts**. Rare Disease studies also include pedigree metadata by defining **families**. Also, a **Clinical Analysis** were defined for each family. Several [Variable Sets](http://docs.opencb.org/display/opencga/AnnotationSets+1.4.0) have been defined to store GEL custom data for all these entities.
## Platform
For the Research environment we have used **OpenCGA v1.4** using the new Hadoop Variant Storage that use [**Apache HBase**](https://hbase.apache.org/) as back-end because of the huge amount of data and analysis needed. We have also used [**CellBase**](http://docs.opencb.org/display/cellbase/CellBase+Home) **v4.6** for the variant annotation. Finally we set up a [**IVA**](http://docs.opencb.org/display/iva/Welcome+to+IVA) **v1.0** web-based variant analysis tool.
The **Hadoop cluster** consists of about 30 nodes running [**Hortonworks**](https://hortonworks.com/) **HDP 2.6.5** (which comes with **HBase 1.1.2**) and a LSF queue for loading all the VCF files, see this table for more detail:
| Node | Nodes | Cores | Memory (GB) | Storage (TB) |
| --------------------- | ----- | ----- | ----------- | -------------- |
| **Hadoop Master** | 5 | 28 | 216 | 7.2 (6x1.2) |
| **Hadoop Worker** | 30 | 28 | 216 | 7.2 (6x1.2) |
| **LSF Loading Queue** | 10 | 12 | 364 | Isilon storage |
## Genomic Data Load
In order to improve the **loading performance,** we set up a small LSF queue of ten computing nodes. This configuration allowed us to load multiple files at the same time. We configured LSF to load up to 6 VCF files per node resulting in 60 files being loaded in HBase in parallel without any incidence, by doing this we observed a **50x in loading throughput**. This resulted in an average of 125 VCF files loaded per hour in studies RD37 and RD38, which is about **2 files per minute**. In the study CG38 the performance was 240 VCF files per hour or about **4 files per minute**.
### Rare Disease Loading Performance
The files from Rare Disease studies (RD38 & RD37) contain 2 samples per file on average. This results in larger files, increasing the loading time compared with single-sample files. As mentioned above the loading performance was about 125 files per hour or 3,000 files per day. In terms of number of samples it is about **250 samples per hour or 6,000 samples a day**.
The loading performance always depend on the number of variants and concurrent files being loaded, the performance was quite stable during the load and performance degradation was observed as can be seen here:
| Concurrent files loaded | 60 |
| ----------------------------- | -------- |
| Average files loaded per hour | 125.72 |
| Load time per file | 00:28:38 |
#### Saturation Study
As part of the data loading process we decided to study the number of unique variants added in each batch of 500 samples. We generated this saturation plot for RD38:
### Cancer Loading Performance
The files from Cancer Germline studies (CG38) contain one sample per file. Compared with the Rare Disease, these files are smaller in size, therefore, as expected the file load was almost 2x faster. As mentioned above, the loading performance was about 240 genomes per hour or 5,800 files per day. In terms of number of samples it is about **5,800 samples a day**, which is consistent with Rare Disease performance.
| Concurrent files loaded | 60 |
| ----------------------------- | -------- |
| Average files loaded per hour | 242.05 |
| Load time per file | 00:14:52 |
## Analysis Benchmark
In this section you can find information about the performance of main variant storage operations and most common queries and clinical analysis. Please, for data loading performance information go to section **Genomic Data Load** above.
### Variant Storage Operations
Variant Storage operations take care of preparing the data for executing queries and analysis. There are two main operations: **Variant Annotation** and **Cohort Stats Calculation**.
#### Variant Annotation
This operation uses the [**CellBase**](http://docs.opencb.org/display/cellbase/CellBase+Home) to annotate each unique variant in the database, this annotation include consequence types, population frequencies, conservation scores clinical info, ... and will be typically used for variant queries and clinical analysis. Variant annotation of the 585 million unique variants of project **GRCh38 Germline** took about 3 days, **about 200 million variants were annotated per day**.
#### Cohort Stats Calculation
Cohort Stats are used for filtering variants in a similar way as the population frequencies. A set of cohorts were defined in each study.
* **ALL** with all samples in the study
* **PARENTS** with all parents in the study (only for Rare Disease studies)
* **UNAFF\_PARENTS** with all unaffected parents in the study (only for Rare Disease studies)
Pre-computing stats for different cohort and ten of thousands of samples is a high-performance operation that run in **less than 2 hours** for each study.
### Query and Aggregation Stats
To study the performance we used **RD38** which the largest study with 438 million variants and 33,000 samples. We first run some queries to the aggregated data filtering by variant annotation and cohort stats. We were interested in the different index performance so we limit the results to be returned the first 10 variants excluding the genotypic data of the 33,000 samples, by doing this we remove the effect of reading from disk or transferring data through the network which is very variable across different clusters. For queries using patient data go to the next section. Here you can find some of the common queries executed.
| Filters | Results | Total Results | Time (sec) |
| ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------- | ---------- |
| **consequence type** = LoF + missense\_variant | 10 | 3704626 | 0.189 |
|
| 10 | 3157533 | 5.96 |
As can be observed most queries run below 1 second, you can combine as many filters as wanted.
### Clinical Analysis
We also use here **RD38** which is the largest study. Clinical queries, or sample queries, enforces queries to return variants of a specific set of samples. These queries can use all the filters from the general queries. The result here also includes a **pathogenic prediction** for each variant, which determines possible conditions associated to the variant.
| Filters | Results | Total Results | Time (sec) |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------- | ---------- |
|
mode of inheritance = recessive
filter = PASS
| 10 | 211787 | 0.420 |
|
mode of inheritance = recessive
consequence type = LoF + missense\_variant
| 10 | 710 | 1.95 |
|
mode of inheritance = recessive
consequence type = LoF + missense\_variant
filter = PASS
| 10 | 656 | 1.92 |
|
mode of inheritance = recessive
consequence type = LoF + missense\_variant
filter = PASS
panel = with 58 genes
| 10 | 7 | 2.11 |
|
de novoAnalysis
filter = PASS
consequence type = LoF + missense\_variant
| 24 | 24 | 0.680 |
|
Compound Heterozygous Analysis
filter = PASS
biotype = protein\_coding
consequence type = LoF + missense\_variant
| 417 | 417 | 10.930 |
As it can be observed most of the family clinical analysis run in less than 2 seconds in the largest study with 33,000 samples.
## User Interfaces
Several user interfaces have been developed to query and analyse data from OpenCGA: IVA web-based tool, Python and R clients, and a command line.
### IVA
[IVA](http://docs.opencb.org/display/iva/Welcome+to+IVA) v1.0.3 was installed to provide a friendly web-based analysis tool to browse variants and execute clinical analysis.
### Command line
You can also query variants efficiently using the command line built in. Performance depends on the number of samples fetched and the RPC used *(REST* or *gRPC),* in the best scenario you can fetch few thousands variants per second. You can see a simple example here producing a VCF file:
| |
| - |
## Support
OpenCB team is setting up **Zetta Genomics**, a start-up to offer support, consultancy and custom feature development. We have partnered with Microsoft Azure to ensure OpenCB Suite runs efficiently in **Microsoft Azure** cloud. We are running a proof-of-concept at the moment with GEL data to benchmark and test Azure.
## Acknowledgements
We would like to thank Genomics England very much for their support and for trusting in OpenCGA and the rest of OpenCB Suite for this amazing release. In particular, we would like to thank Augusto Rendon, Anna Need, Carolyn Tregidgo, Frank Nankivell and Chris Odhams for their support, test and valuable feedback.
# Microsoft Azure
Azure HDInisght
{% hint style="warning" %}
Work in progress
{% endhint %}
## Genomic and Clinical Data
For this proof of concept (PoC) we loaded all the genomic variants of about **4,700 genomes** from Genomics England**,** variants were loaded and indexed in the development version OpenCGA 2.0.0-beta. In total we loaded about **208 million unique variants** from **4,700 gVCF** files accounting for about 20TB of compressed disk space. It is worth noting that these files were generated using Dragen 2.x and the are unusually big, about 5-6GB per file.
## Platform
For this proof of concept (PoC) we used the development version **OpenCGA v2.0.0-beta** using the Hadoop Variant Storage Engine that uses [**Apache HBase**](https://hbase.apache.org/) as back-end. We also used [**CellBase**](http://docs.opencb.org/display/cellbase/CellBase+Home) **4.6** for the variant annotation.
For the platform we used a 10-nodes **\*\*\[**Azure HDInsight**]\(****)** 3.6 **cluster using \[**Data Lake Storage Gen2**]\(****). HDInsight 3.6 uses \[**Hortonworks HDP**]\(****)** 2.6.5 **(with** Hadoop 2.7.3 **and** HBase 1.1.2\*\*) and we used Azure Batch for loading concurrently all the VCF files which had been copied previously to a NFS server, you can see details here:
| Node Type | Nodes | Azure Type | Cores | Memory (GB) | Storage |
| --------------------- | ----- | ------------------ | ----- | ----------- | -------------- |
| **Hadoop Master** | 3 | Standard\_D12\_V2 | 4 | 28 | Data Lake Gen2 |
| **Hadoop Worker** | 10 | Standard\_DS13\_V2 | 8 | 56 | Data Lake Gen2 |
| **Azure Batch Queue** | 20 | Standard\_D4s\_v3 | 4 | 16 | NFS Server |
Initially, we evaluated the new HDInsight 4.0, although it worked quite well there were some few minor issue, so for this PoC we decided to use the more stable HDInsight 3.6 (*HDI3.6*) over Data Lake Gen2 (*DL2*), we will refer to this as *HDI3.6+DL2.* During the PoC we worked with Azure engineers to debug and fix all these issues, unfortunately we did no have time to repeat the benchmark.
As you will see below in the Analysis Benchmark section, once we completed the PoC we increased the size of HDInsight to 20 nodes and repeated some tests to study the performance improvement.
## Genomic Data Loading and Indexing
In order to study the **loading performance** we set up a Azure Batch queue of 20 computing nodes. This allowed us to load multiple files at the same time from different servers. We configured Azure Batch to load 1 VCF file per node resulting in 20 files being loaded in HBase simultaneously. During the load we studied two different configurations:
1. Loading VCF files vs. loading transformed files
2. Using HDD vs. SSD disk in the NFS server
You can observe the results in the following plot:
### Discussion
* As expected loading already transformed files is much faster since we only need to load and index data in HBase. See this link for more information [Indexing Genomic Variants](http://docs.opencb.org/display/opencga/Indexing+Genomic+Variants)
* Loading from SSD disks showed a better performance
The most typical scenario when indexing genomic data is to *transform+load* at the same time, so assuming SSD disk the observed performance was about **380 VCF files indexed a day, or about 2TB/day**. It is worth noting that:
1. *gVCF used were several times bigger than usual*
2. *the number of Hadoop worker nodes was just 10*
3. *we loaded up to 20 files concurrently but this could have been increased*
These variables have a huge impact in the indexing performance, so the expected performance with more real gVCF files and production cluster is more than 1,000 VCF a day.
## Analysis Benchmark
In this section you can find information about the performance of main variant storage engine operations, queries and analysis. Please, for data loading performance information go to section **Genomic Data Load** above.
### Variant Storage Operations
Variant Storage operations take care of preparing the data for executing queries and analysis. Some of the most important operations include: **Genotype Aggregation**, **Variant Annotation**, **Cohort Stats Calculation** and **Sample Variants Indexing**.
#### Genotype Aggregation
We executed an initial aggregation with the first batch 700 samples accounting for 74.096.015 variants. This run in about **4:10 hours** without any issue, this performance is expected to be quite stable. Also, the number of worker nodes affects the performance, unfortunately, because lack of time we could only test with 10 nodes.
#### Variant Annotation
This operation uses the **\*\*\[**CellBase**]\(****) to annotate each unique variant in the database, this annotation include consequence types, population frequencies, conservation scores clinical info, ... and will be typically used for variant queries and different analysis. We executed variant annotation during the variant indexing, in OpenCGA 2.0.0 you can load variants and annotate at the same time, the performance of variant annotation is affected mainly by CellBase installation, in this PoC we used a small CellBase installation from University of Cambridge which is far from ideal. The performance observed was about** 120 million variants annotated a day.\*\*
#### Cohort Stats Calculation
We pre-computed the variants stats for the 200 million variants across the 4,700 samples, this operation includes the calculation and indexing of the variant stats. We run this few times with 10 nodes and 20 nodes. The observed performance was:
* With **10-nodes**: **1:50 hours**
* With **20-nodes**: **53 min**
As expected the performance improves linearly (**2.07x speed-up**) with the number of nodes.
#### Sample Variant Indexing
This operation plays a crucial role when querying by sample genotype. This is also one of the most complex and intensive operation since we index here all the genotypes loaded, all the variants for each sample is indexed, in this PoC we have about **20 billion variants across the 4,700 samples**. We run this few times with 10 nodes and 20 nodes. The observed performance was:
* With **10-nodes**: **4:53 hours**
* With **20-nodes**: **2:37 min**
As expected the performance improves linearly (**1.86x speed-up**) with the number of nodes. **Note:** this operation is incremental so if we load new samples we only need to index variants from these samples, the runtime above was obtained indexing all sample genotypes in one single execution.
### Query Benchmark
To study the query performance we tried different configurations:
* **Variants table compression -** Either GZ (2.9TB) or SNAPPY (4,7TB)
* **Bucket cache size per node -** Using the 1TB premium disk to improve writes to extend the bucket size
* **Cache compression** - [HBASE-11331](https://issues.apache.org/jira/browse/HBASE-11331)
* **Cache warmup -** Custom MR job to read the whole hbase table and warm up the caches
#### Queries
* filter=PASS,region,sample(1, OR)
* filter=PASS,region,sample(2, OR)
* filter=PASS,region,sample(3, OR)
* filter=PASS,region,sample(4, OR)
* filter=PASS,region,sample(5, OR)
* filter=PASS,region,sample(1, AND)
* filter=PASS,region,sample(2, AND)
* filter=PASS,region,sample(3, AND)
* filter=PASS,region,sample(4, AND)
* filter=PASS,region,sample(5, AND)
* filter=PASS,region=15,sample(3, AND)
* filter=PASS,region=15,sample(3, OR)
* filter=PASS,region,ct=(lof,missense\_variant),sample(1, OR)
* filter=PASS,region,ct=(lof,missense\_variant),sample(2, OR)
* filter=PASS,region,ct=(lof,missense\_variant),sample(3, OR)
* filter=PASS,region,ct=(lof,missense\_variant),sample(4, OR)
* filter=PASS,region,ct=(lof,missense\_variant),sample(5, OR)
* filter=PASS,region,ct(6),sample(1, OR)
* filter=PASS,region,ct(6),sample(2, OR)
* filter=PASS,region,ct(6),sample(3, OR)
* filter=PASS,region,ct(6),sample(4, OR)
* filter=PASS,region,ct(6),sample(5, OR)
* filter=PASS,region,ct=(lof,missense\_variant),biotype=protein\_coding,sample(1, OR)
* filter=PASS,region,ct(9),biotype=protein\_coding,sample(1, OR)
#### Scenarios
| ID | Variants Table Compression | Bucket Cache size (GB/node) | Cache compression | Cache warmup | | Average Query Time (s) | Speed-up |
| -- | -------------------------- | --------------------------- | ----------------- | ------------ | ----- | ---------------------- | -------- |
| 1 | GZ | 45 | No | No | 5.487 | NA | |
| 2 | GZ | 45 | No | Yes | 5.126 | 1.1x | |
| 3 | GZ | 300 | No | Yes | 3.628 | 1.5x | |
| 4 | GZ | 300 | Yes | Yes | 2.115 | 2.6x | |
| 5 | SNAPPY | 300 | Yes | No | 2.412 | 2.3x | |
| 6 | SNAPPY | 300 | Yes | Yes | 1.816 | 3.0x | |
| 7 | SNAPPY | 45 | No | No | 3.750 | 1.5x | |
#### Results
| | **AZURE** | **Native on-prem** | | | | | | | | | | |
| ----------------------------------------------------------------------------------- | ---------------------------------- | ------------------ | -------------- | --------- | -------------- | ----------- | -------- | -------- | -------- | -------- | -------- | ------ |
| **HDI3** | **HDI4** | **HDP2.6** | | | | | | | | | | |
| **GZ** | **SNAPPY** | **SNAPPY** | **SNAPPY** | | | | | | | | | |
| **45GB** | **300GB** | **300GB** | **45GB** | **500GB** | **10GB** | | | | | | | |
| **Uncompressed** | **Uncomp.** | **Comp.** | **Compressed** | **Comp.** | **Compressed** | **Uncomp.** | | | | | | |
| **Cold** | **Warm** | **Warm** | **Warm** | **Cold** | **Warm** | **Cold** | **Cold** | **Warm** | **RD37** | **RD38** | **CG38** | |
| | **1** | **2** | **3** | **4** | **5** | **6** | **7** | **8** | **9** | **10** | **11** | **12** |
| filter=PASS,region,ct(6),sample(1, OR) | 5.172 | 5.729 | 4.263 | 1.792 | 2.950 | 1.534 | 3.561 | 4.446 | 1.495 | 0.829 | 2.729 | 1.246 |
| filter=PASS,region,ct(6),sample(2, OR) | 6.488 | 5.905 | 4.035 | 2.116 | 3.256 | 2.007 | 3.926 | 5.338 | 2.081 | 0.84 | 1.644 | 2.27 |
| filter=PASS,region,ct(6),sample(3, OR) | 7.657 | 5.366 | 4.383 | 2.715 | 3.299 | 2.725 | 4.355 | 6.427 | 2.285 | 1.947 | 2.205 | 1.99 |
| filter=PASS,region,ct(6),sample(4, OR) | 9.039 | 6.402 | 5.098 | 2.124 | 3.562 | 2.338 | 5.263 | 5.108 | 2.522 | 0.996 | 3.01 | 1.88 |
| filter=PASS,region,ct(6),sample(5, OR) | 8.688 | 8.546 | 4.880 | 3.161 | 3.432 | 2.455 | 5.778 | 7.006 | 3.067 | 1.628 | 1.961 | 1.962 |
| filter=PASS,region,ct(9),biotype=protein\_coding,sample(1, OR) | 4.138 | 6.090 | 2.891 | 1.837 | 2.544 | 2.605 | 3.906 | NA\* | NA\* | 1.43 | 1.345 | 1.345 |
| filter=PASS,region,ct=(lof,missense\_variant),biotype=protein\_coding,sample(1, OR) | 3.908 | 5.060 | 3.284 | 1.795 | 1.804 | 1.588 | 3.619 | 3.758 | 1.714 | 0.802 | 1.549 | 1.443 |
| filter=PASS,region,ct=(lof,missense\_variant),sample(1, OR) | 5.905 | 3.856 | 3.394 | 2.304 | 2.737 | 1.422 | 3.283 | 4.136 | 1.913 | 0.85 | 1.78 | 1.461 |
| filter=PASS,region,ct=(lof,missense\_variant),sample(2, OR) | 7.762 | 5.686 | 3.996 | 2.561 | 3.348 | 2.017 | 3.912 | 4.209 | 2.375 | 1.765 | 2.533 | 4.981 |
| filter=PASS,region,ct=(lof,missense\_variant),sample(3, OR) | 7.069 | 6.668 | 4.145 | 2.121 | 3.588 | 2.173 | 4.568 | 5.724 | 2.539 | 1.147 | 2.019 | 1.959 |
| filter=PASS,region,ct=(lof,missense\_variant),sample(4, OR) | 6.918 | 6.488 | 4.908 | 3.000 | 3.867 | 2.622 | 5.013 | 5.987 | 3.065 | 1.229 | 2.01 | 1.754 |
| filter=PASS,region,ct=(lof,missense\_variant),sample(5, OR) | 8.917 | 9.043 | 4.799 | 2.202 | 3.251 | 2.569 | 5.996 | 6.64 | 2.981 | 1.653 | 2.732 | 2.115 |
| filter=PASS,region,sample(1, AND) | 3.678 | 2.882 | 2.563 | 1.697 | 1.405 | 1.262 | 1.789 | 2.589 | 1.695 | 0.808 | 2.127 | 1.533 |
| filter=PASS,region,sample(1, OR) | 3.381 | 3.052 | 2.946 | 1.959 | 1.265 | 1.312 | 1.740 | 2.192 | 1.667 | 0.832 | 2.595 | 1.915 |
| filter=PASS,region,sample(2, AND) | 4.220 | 3.682 | 2.852 | 1.864 | 1.785 | 1.443 | 2.665 | 3.088 | 1.980 | 1.046 | 3.161 | 1.802 |
| filter=PASS,region,sample(2, OR) | 3.997 | 3.657 | 2.401 | 2.170 | 1.532 | 1.327 | 2.255 | 2.963 | 1.726 | 0.925 | 1.827 | 1.529 |
| filter=PASS,region,sample(3, AND) | 5.560 | 4.418 | 3.287 | 2.461 | 2.418 | 1.810 | 3.307 | 3.888 | 2.098 | 1.154 | 2.219 | 2.388 |
| filter=PASS,region,sample(3, OR) | 4.430 | 4.218 | 3.401 | 1.914 | 1.485 | 1.481 | 3.148 | 3.339 | 2.068 | 0.923 | 2.251 | 1.945 |
| filter=PASS,region,sample(4, AND) | 4.987 | 5.235 | 4.918 | 2.555 | 2.041 | 1.973 | 4.122 | 4.805 | 2.538 | 1.37 | 3.193 | 2.145 |
| filter=PASS,region,sample(4, OR) | 4.347 | 4.816 | 3.229 | 2.256 | 1.736 | 1.823 | 3.216 | 3.989 | 2.238 | 1.062 | 1.947 | 1.661 |
| filter=PASS,region,sample(5, AND) | 6.695 | 5.556 | 4.779 | 2.783 | 2.996 | 2.452 | 4.835 | 5.793 | 3.087 | 1.714 | 2.779 | 2.553 |
| filter=PASS,region,sample(5, OR) | 5.067 | 5.395 | 3.910 | 2.518 | 2.217 | 1.789 | 4.358 | 4.495 | 2.639 | 1.599 | 2.053 | 1.781 |
| filter=PASS,region=15,sample(3, AND) | 1.894 | 2.714 | 1.539 | 0.463 | 0.841 | 0.460 | 3.070 | 4.768 | 1.535 | 0.904 | 1.716 | 1.616 |
| filter=PASS,region=15,sample(3, OR) | 1.771 | 2.550 | 1.163 | 0.384 | 0.521 | 0.408 | 2.311 | 4.482 | 1.701 | 0.865 | 1.553 | 1.454 |
| | | | | | | | | | | | | |
| Average | 5.487 | 5.126 | 3.628 | 2.115 | 2.412 | 1.816 | 3.750 | 4.573 | 2.218 | 1.180 | 2.206 | 1.947 |
| Speed-up | 1.0 | 1.1 | 1.5 | 2.6 | 2.3 | 3.0 | 1.5 | 1.2 | 2.5 | 4.7 | 2.5 | 2.8 |
| | | | | | | | | | | | | |
| Average – no region 15 | 5.819 | 4.879 | 3.835 | 2.268 | 2.569 | 1.942 | 3.846 | 4.568 | 2.275 | 1.207 | 2.258 | 1.984 |
| Speed-up | 1.0 | 1.2 | 1.5 | 2.6 | 2.3 | 3.0 | 1.5 | 1.3 | 2.6 | 4.8 | 2.6 | 2.9 |
| | | | | | | | | | | | | |
| | \* Apache Phoenix not working in H | | | | | | | | | | | |
### GWAS
We defined two cohorts of 100 samples each and run a GWAS analysis over the 208 million variants. The performance observed was very similar to Cohort Stats Calculation.
The Stats calculation took about 2 hours with 10 nodes and about 1 hour with 20 nodes.
# Installation
## Overview
OpenCGA has been designed and implemented to run natively in both *cloud environments* using Kubernetes and *on-prem clusters*.
### Kubernetes
### Bare metal installation
### Docker Images
# Kubernetes Cluster
# Azure
# Configuration
# On-Premise HPC Cluster
You can install OpenCGA on a local HPC cluster. Installing and configuring OpenCGA consists of different steps, as you will see on this page you must first make sure that the server(s) have all dependencies installed, then you can configure and complete the installation.
## Installation and Configuration
### Step 1 - Configuring the Server
OpenCGA requires Java 8, Tomcat or MongoDB. We try to keep dependencies to the minimum to ease development, installation and administration, and divide dependencies into *required* and *optional.* You can learn about all the dependencies and how to install them at [**Installation Guide > Server Configuration**](http://docs.opencb.org/display/opencga/Server+Configuration).
### Step 2 - Getting OpenCGA
There are two main ways to get OpenCGA for installation:
* You can download binaries from [OpenCGA GitHub Releases](https://github.com/opencb/opencga/releases), notice that **only** *stable* and *pre-releases* are tagged and *built*, if you want to test a development version see next point.
* Or You can download the source code from GitHub and use Apache Maven to compile and build it.
Please visit [Getting OpenCGA](/develop-2/manual/installation/compute-cluster/building-from-source) to learn more about these two options.
### Step 3 - Install OpenCGA Binaries
These instructions assume that you have already downloaded or built openCGA binaries as described on [Using Binaries](http://docs.opencb.org/pages/createpage.action?spaceKey=opencga\&title=Using+Binaries\&linkCreation=true\&fromPageId=327810) and [Building from Source Code](http://docs.opencb.org/display/opencga/Building+from+Source+Code).
Create an installation directory called */opt/opencga* and copy the contents of opencga into this :
**Note:** In case of reinstallation, you must clean the installation directory (*/opt/opencga*)
```bash
mkdir /opt/opencga
cp -r build/* /opt/opencga
```
### Step 4 - Configure OpenCGA
Execute the following command line to install and initialise Catalog database:
```bash
cd /opt/opencga
./bin/opencga-admin.sh catalog install <<< admin_P@ssword
```
### Step 5 - Deploy WAR file
This is the main interface to perform any action with OpenCGA. User has two different options to start web services:
* You can deploy OpenCGA in a Web Server.
* Or you can use the OpenCGA admin command line.
Next you can learn more about these two options.
#### Deploying OpenCGA in a Web Server
Install [Apache Tomcat](https://tomcat.apache.org/download-80.cgi) and copy the deploy `opencga.war`. To do this, just copy it from the compilation directory (where you downloaded the OpenCGA repository) into the Tomcat `webapps` directory:
`cp /opt/opencga/opencga.war $(path_to_tomcat)/webapps`
`path_to_tomcat` is where you downloaded it, or probably`/var/lib/tomcat8` if you installed via apt-get. Then, you should be able to see the swagger page at [`http://localhost:8080/opencga/`](http://localhost:8080/opencga/). See [Using RESTful web services](https://github.com/opencb/opencga/wiki/Using-RESTful-web-services) for a tutorial.
Tomcat server will look for the configuration files in the installation directory, which can be changed in compilation time changing the property OPENCGA.INSTALLATION.DIR.
If the installation directory is empty at compilation time, the web services will search for the environment variable `OPENCGA_HOME`. If none of this is properly set, the web services will not work.
#### Using the OpenCGA admin command line
The OpenCGA admin command line allows users to run an embedded REST server. These web services will be served with Jetty as follows:
`/opencga-admin.sh server rest --start -p`
Warning: This method is still under development.
# Server Configuration
## Required Dependencies
### Java
OpenCGA needs **Java 8** to compile and run, we recommend using **Oracle JDK 1.8.0\_60+** since is the one we use, and therefore it is fully supported. There are two main ways of installing Oracle JDK in Linux: *via the package manager* and *manually*.
Installing ***via package manager*** can be more or less easy to do depending on the Linux distribution you use, for example for **Ubuntu 16.04** you can execute the following commands:
```bash
## You need root permissions
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
```
To install Java ***manually*** you can download the latest Java SDK for Linux x64 file (*linux-x64.tar.gz*) from Oracle at [**http://www.oracle.com/technetwork/es/java/javase/downloads/index.html**](http://www.oracle.com/technetwork/es/java/javase/downloads/index.html)
Then you have to uncompress it, move it to an installation folder such as */opt,* and create a symbolic link:
```bash
## You can uncompress from the Download directory
tar -zxvf jdk-8u91-linux-x64.tar.gz
## You need root permissions for the following
sudo mv jdk1.8.0_91 /opt
sudo ln -s /opt/jdk1.8.0_91 java
## Then you should have something like this:
lrwxrwxrwx 1 root root 11 Jun 13 2016 java -> jdk1.8.0_91/
drwxr-xr-x 8 imedina imedina 4.0K Jan 6 13:31 jdk1.8.0_91/
```
and finally, you must set the *JAVA\_HOME* variable in the system or user *bashrc* file:
```bash
## Add this to ~/.bashrc or /etc/bash.bashrc
export JAVA_HOME="/opt/java"
PATH="$JAVA_HOME/bin:$M2:$PATH"
```
Independently of the installation procedure followed above you can check is Java properly installed by executing *java -version*, you should get something like this:
```bash
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
```
### Tomcat
OpenCGA needs **Tomcat 8** to run RESTful web services, we recommend to use **Apache Tomcat 8.x** since is the one we use in production but we have also used [**Jetty**](https://eclipse.org/jetty/) without any problem. You can install Apache Tomcat in Linux either *via package manager* or *manually*.
Installing ***via package manager*** can be more or less easy to do depending on the Linux distribution you use, keep in mind that Tomcat should run with the same user that OpenCGA. We will assume that OpenCGA will be installed with user *opencga*. Run the following to set Tomcat to run with user opencga. for example for **Ubuntu 16.04** you can execute the following command:
```bash
## You need root permissions
sudo apt-get install tomcat8
## You can check that Tomcat is running by executing
sudo service tomcat8 status
## Run the following to set Tomcat to run with user opencga
cd /var/lib/tomcat8/
sudo service tomcat8 stop
sudo chown -RL opencga:opencga ./*
## Edit the files /etc/init.d/tomcat8 and /etc/default/tomcat8
TOMCAT8_USER=opencga
TOMCAT8_GROUP=opencga
## Then reload the service and restart
sudo systemctl daemon-reload
sudo service tomcat8 start
```
You can install Tomcat ***manually*** by downloading it from and executing as the OpenCGA installation user, execute the following commands:
We recommend to make same changes in the Tomcat configuration:
```bash
## You can uncompress from the Download directory
tar -zxvf apache-tomcat-8.5.3.tar.gz
## You need root permissions to move to /opt and create a symbolic link
sudo mv apache-tomcat-8.5.3 /opt
sudo ln -s /opt/apache-tomcat-8.5.3 tomcat
## You can start and stop Tomcat executing
/opt/tomcat/bin/startup.sh
/opt/tomcat/bin/shutdown.sh
```
* Increase Tomcat **memory**: you can do this by adding *JAVA\_OPTS=" $JAVA\_OPTS -Xms1024m -Xmx12288m "* to file *bin/catalina.sh*. In this example we have increased the memory to 12GB.
* Enable **compression** to save bandwidth: you can ask Tomcat to compress the HTML and JSON output, go to conf/server.xml and look for the connector 8080 and leave it like this:
```markup
```
### MongoDB 3.2
OpenCGA uses **MongoDB 4.2.x+** to store *Catalog* database and also as a possible backend for the *Variant Storage* engine. The best ways of installing MongoDB is from the Linux package managers, you can follow MongoDB tutorials for Ubuntu and RedHat/CentOS at:
* Ubuntu:
* RedHat/CentOS:
You can customise server configuration at */etc/mongodb.conf* or the client at *\~/.mongorc.js*. Some recommended configuration for MongoDB:
* Always use WiredTiger engine
* Make sure the *journal* is enabled
* Make sure to have one directory per db with: *directoryperdb=true*
* For cluster installation we recommend to set up a Replica Set, this will improve performance and will make the installation more robust. OpenCGA versions 2.x+ utilise MongoDB's transactions, therefore a Replica Set is required.
### Apache Solr
**Apache Solr 6.x** is used in OpenCGA as a complimentary search engine for improving the performance of some queries and aggregations, full-text search and *faceted* queries to the Variant database. Solr is a highly reliable, scalable, and fault-tolerant NoSQL database, it provides distributed indexing, replication, load-balanced querying, automated fail over, recovery, centralised configuration and more. You can learn how to install Solr at [**https://cwiki.apache.org/confluence/display/solr/Installing+Solr**](https://cwiki.apache.org/confluence/display/solr/Installing+Solr).
OpenCGA needs to be able to create new Solr *collections* and provide the Variant search schema. To do this you need to download the [Server Configuration](http://docs.opencb.org/pages/viewpage.action?pageId=37552132) if you are using OpenCGA version 1.2.x or 1.3.x, for next version 1.4.0 you must use OpenCGAConfSet-1.4.0.tar.gz, then you just need to uncompress it and copy it into your Solr folder installation (i.e., $SOLR\_HOME/server/solr/configsets) .
* For **Solr standalone installation**, you have to copy the OpenCGAConfSet-1.4.0 configuration into the folder *server/solr/configsets* for Solr standalone installation. Now, you can create new *core* or *collections* dynamically from Solr command line or from REST web services, e.g.:
[*http://localhost:8983/solr/admin/cores?action=CREATE\&name=my-new-core\&configSet=OpenCGAConfSet-1.4.0*](http://localhost:8983/solr/admin/cores?action=CREATE\&name=my-new-core\&configSet=OpenCGAConfSet-1.4.0)
* For **Solr cloud installation**, you have to upload the OpenCGAConfSet configuation to your cluster by running the Solr zookeeper command line. In the following example command line, the OpenCGAConfSet configuration is located in the folder *server/solr/configsets*:
```bash
./bin/solr zk upconfig -n OpenCGAConfSet-1.4.0 -d server/solr/configsets/OpenCGAConfSet-1.4.0 -z localhost:9983
```
Now we can create collections from our search command line or from REST services, e.g.:
For more information, [https://cwiki.apache.org/confluence/display/solr/Using+ZooKeeper+to+Manage+Configuration+File](https://cwiki.apache.org/confluence/display/solr/Using+ZooKeeper+to+Manage+Configuration+Files)
## Optional Dependencies
### Apache Maven
OpenCGA uses **Apache Maven 3.x** as a building tool, we use maven to compile, build, install and run tests of OpenCGA, so you do not need Maven unless you want to compile and build the source code for any reason, you can follow [**Installation Guide > Building from Source Code**](http://docs.opencb.org/display/opencga/Building+from+Source+Code) instructions to learn how to do it. Remember that you can always get stable OpenCGA binaries from [**Installation#GettingOpenCGA**](http://docs.opencb.org/display/opencga/Installation#Installation-GettingOpenCGA).
Maven can be easily installed in Linux **via the package manager**, you can execute with *sudo* command or as *root* the following commands in Ubuntu or CentOS:
```bash
## Ubuntu 16.04
sudo apt-get install maven
## CentOS 7.x
sudo yum install maven
```
Or you can also install Maven **manually** following this tutorial [**http://maven.apache.org/install.html**](http://maven.apache.org/install.html)
You can check that Maven is installed by executing *mvn -v* or *mvn -version*, you should get something like this:
```bash
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /opt/maven
Java version: 1.8.0_231, vendor: Oracle Corporation, runtime: /opt/jdk1.8.0_231/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-54-generic", arch: "amd64", family: "unix"
```
You need to add a Maven Profile to set up some variables that will be injected during the building, you can learn more about this at [**Installation Guide > Building from Source Code**](http://docs.opencb.org/display/opencga/Building+from+Source+Code).
### Hadoop
This is an optional backend plugin for OpenCGA storage. We have used both Hortonworks and Cloudera as Hadoop distribution. At the moment we recommend **Hortonwork 3.1+**.
## Cluster Dependencies
### HAProxy
OpenCGA uses **HAProxy 1.5+** in the cluster installation to balance all REST web services call to the different deployed Tomcats. You can easily install HAProxy **via package manager**:
```bash
## Ubuntu 16.04
sudo apt-get install haproxy
## CentOS 7.x
sudo yum install haproxy
```
Here you can find an easy example for a very simple configuration, go to [HAProxy](http://www.haproxy.org/) for more documentation:
```bash
frontend http-in
bind *:80
## Asign 'webservices' to paths starting with 'opencga'
acl webservices path_beg -i /opencga
## Redirect ACL 'webservices' to backend 'tomcat_prod'
use_backend tomcat_prod if webservices
## backend definition balancing two Tomcat instances
backend tomcat_prod
cookie SERVERID insert nocache indirect
balance roundrobin
option httpclose
option httpchk HEAD /
server webprod01 10.21.2.3:8080 cookie webprod01tc check port 8080 inter 5s rise 2 fall 3 maxconn 300
server webprod02 10.21.2.5:8080 cookie webprod02tc check port 8080 inter 5s rise 2 fall 3 maxconn 300
```
# Getting OpenCGA
## Download Binaries
You can download any *stable* or *pre-release* version from [OpenCGA GitHub Releases](https://github.com/opencb/opencga/releases). You will find a *tar.gz* file with the name of *opencga* and the *version,* for instance, to download OpenCGA 2.0.0-rc3 you have to go to GitHub Release page at and download the file ***opencga-2.0.0-rc3.tar.gz*** from the *Assets* section:
## Building from Source
### Prerequisites
In order to build OpenCGA from source code you must first get the source code of OpenCGA from GitHub, most of the dependencies - including OpenCB dependencies - will be fetched from [Maven Central Repository](http://search.maven.org/), however, in some scenarios, OpenCB dependencies will need to be built from GitHub source code. Compiling and building processes are carried out by [Apache Maven](https://maven.apache.org/). The following tools are required for successful build:
* Java 1.8.0\_60+
* Apache Maven
You can learn how to install them in this section at [Server Configuration](/develop-2/manual/installation/compute-cluster/server-configuration).
### Getting and Compiling Dependencies
OpenCGA as any other software has dependencies, some of them come from other OpenCB projects such as CellBase while others are third-party dependencies such as MongoDB. All OpenCGA ***stable*** releases are always merged and tagged at ***master*** branch (users are encouraged to use latest stable release for production), you can find all releases at [OpenCGA Releases](https://github.com/opencb/opencga/releases). We guarantee that all the dependencies needed for building stable releases are deployed at [Maven Central Repository](http://search.maven.org/), this is true for both OpenCB and third-party dependencies. Therefore for building a stable release you only need to clone OpenCGA repository itself since all the dependencies will be fetched form Maven Central repository.
This is different for development branches. Active OpenCGA development is carried out at ***develop*** branch, in this branch third-party dependencies will be still fetched from Maven Central Repository **but** this is not true for OpenCB dependencies since it is very likely they are still in development and therefore they are not deployed. Keep in mind that we only guarantee that *develop* compiles and that bugs are expected, use this branch for development or for testing new functionalities. So, for building ***develop*** branch you may require to download and install the following OpenCB repositories in this order:
* *java-common-libs:* (branch 'develop')
* *biodata*: (branch 'develop')
* *cellbase*: (branch 'develop')
* *hpg-bigdata*: (branch 'develop')
As you can see one of our rules is that *develop* branch of all major applications such as OpenCGA and CellBase always depend on *develop* branches. So, if you really want to build *develop* the you can clone and build dependencies by executing:
```bash
## Clone develop branc
git clone -b develop https://github.com/opencb/java-common-libs.git
git clone -b develop https://github.com/opencb/biodata.git
git clone -b develop https://github.com/opencb/cellbase.git
git clone -b develop https://github.com/opencb/oskar.git
## Now you can execute the following command in each of the folders the specified order above
mvn clean install -DskipTests
```
### Clone and Build OpenCGA
You can clone OpenCGA from GitHub by executing:
```bash
## Latest stable version
git clone -b master https://github.com/opencb/opencga.git
## Develop branch, for this to work remember to clone and build OpenCB dependencies (see above)
git clone -b develop https://github.com/opencb/opencga.git
```
### Building with Maven
OpenCGA allows to customise many variables in the configuration files, in order to make easier the building and configuration of OpenCGA we rely on Maven Properties that can be defined in file *`~/.m2/settings.xml`*. During the building all these properties will be injected automatically in the configuration files so users do not have to manually change all configuration values. **Note** this is only possible when building OpenCGA from source code, if you download the binary version you will have to manually set up all configuration variables.
An example of that file can be found in the [README](https://github.com/opencb/opencga#build) and below. The description of each property can be found below:
* **OPENCGA.CATALOG.DB.HOSTS**: This property should be configured with the host and port of the MongoDB installation. By default, for development purposes, we have it set with "localhost:27017".
* **OPENCGA.CATALOG.DB.USER**: This property should only be set if the MongoDB needs authentication. In this case, this property will contain the user name with permissions for the database. \*This can be left empty in any case. The admin will be able to set this credentials using the command line.
* **OPENCGA.CATALOG.DB.PASSWORD**: This property should only be set if the MongoDB needs authentication. In this case, this property will contain the password of the user with permissions for the database. \*Like in the user property, this can be left empty. The admin will be able to set this credentials using the command line.
* **OPENCGA.INSTALLATION.DIR**: This property is extremely important when using Tomcat to deploy the webservices. This property will have to point to the final OpenCGA installation directory after everything has been built. This property will be used by Tomcat to locate the configuration files. If this is not properly set, none of the webservices will work. Default: /opt/opencga.
* **OPENCGA.USER.WORKSPACE**: In Catalog, users are allowed to build their own directory structure, upload their own files, run analysis, etc. This path should be pointing to a physical location where Catalog will be storing those files and directory structure. By default, we normally put it in a folder called "sessions" within the installation directory (file:///opt/opencga/sessions/).
* **OPENCGA.STORAGE.VARIANT.DB.HOSTS**:
* **OPENCGA.STORAGE.VARIANT.DB.USER**:
* **OPENCGA.STORAGE.VARIANT.DB.PASSWORD**:
* **OPENCGA.ANALYSIS.EXECUTION.MANAGER**: OpenCGA Catalog allows users to run jobs. This property indicates how the jobs will be launched. At the moment we only support two types: LOCAL to run the jobs locally in a thread or SGE to run the jobs using Sun Grid Engine. More queuing systems will be supported soon.
* **OPENCGA.CLIENT.HOST**: This property should be pointing to the URL where the webservices will be available. For development purposes, the default is . This property is read by the command line opencga.sh in order to communicate with the webservices.
* **OPENCGA.CELLBASE.REST.HOST**: URL to be used for Variant Annotation.
You can copy this example to *\~/.m2/settings.xml*:
```markup
default-configtrueopencga-${opencga.version}/opt/opencga/opt/opencga/sessionsopencgaLOCALhttp://localhost:9090/${opencga.war.name}localhost:9091909090919092localhost:2701720http://localhost:8983/solr/300002000mongodblocalhost:6379localhost:27017http://localhost:8983/solr/opencga-variant-configset-${opencga.version}30000http://localhost:8983/solr/""30000localhost:2701720https://ws.opencb.org/cellbase/v4
```
After creating this and configuring a default profile, you can build OpenCGA by executing the following command from the root of the cloned repository:
```bash
$ mvn clean install -DskipTests
```
The first time this command can take some minutes since it has to fetched and store locally all the dependencies, the next builds will be much faster. After successful building, user should find the following file structure under a build folder:
```bash
build/
├── analysis
│ ├── clinical-variant-query.yml
│ ├── individual-qc
│ ├── interpretation-zetta
│ ├── R
│ └── resources
├── bin
│ ├── opencga-admin.sh
│ ├── opencga-internal.sh
│ ├── opencga.sh
│ └── utils
├── clients
│ ├── python
│ └── R
├── cloud
│ ├── aws
│ ├── azure
│ ├── docker
│ └── kubernetes
├── conf
│ ├── client-configuration.yml
│ ├── configuration.yml
│ ├── hadoop
│ ├── log4j2.internal.xml
│ ├── log4j2.service.xml
│ ├── log4j2.xml
│ ├── opencga-env.sh
│ └── storage-configuration.yml
├── libs
│ ├── accessors-smart-1.2.jar
│ ├── activation-1.1.jar
│ ├── adal4j-1.6.3.jar
│ ├── adapter-rxjava-2.4.0.jar
│ ├── animal-sniffer-annotations-1.17.jar
│ ├── annotations-2.0.1.jar
│ ├── annotations-4.1.1.4.jar
│ ├── ant-1.8.2.jar
│ ├── ant-launcher-1.8.2.jar
│ ├── aopalliance-repackaged-2.5.0-b32.jar
│ ├── asm-5.0.4.jar
│ ├── automaton-1.11-8.jar
│ ├── avro-1.7.7.jar
│ ├── avro-ipc-1.7.7.jar
│ ├── azure-1.13.0.jar
│ ├── azure-annotations-1.2.0.jar
│ ├── azure-batch-4.0.1.jar
│ ├── azure-client-authentication-1.5.4.jar
│ ├── azure-client-runtime-1.6.4.jar
│ ├── azure-keyvault-1.0.0.jar
│ ├── azure-keyvault-core-0.8.0.jar
│ ├── azure-keyvault-webkey-1.0.0.jar
│ ├── azure-mgmt-appservice-1.13.0.jar
│ ├── azure-mgmt-batch-1.13.0.jar
│ ├── azure-mgmt-batchai-1.13.0.jar
│ ├── azure-mgmt-cdn-1.13.0.jar
│ ├── azure-mgmt-compute-1.13.0.jar
│ ├── azure-mgmt-containerinstance-1.13.0.jar
│ ├── azure-mgmt-containerregistry-1.13.0.jar
│ ├── azure-mgmt-containerservice-1.13.0.jar
│ ├── azure-mgmt-cosmosdb-1.13.0.jar
│ ├── azure-mgmt-dns-1.13.0.jar
│ ├── azure-mgmt-eventhub-1.13.0.jar
│ ├── azure-mgmt-graph-rbac-1.13.0.jar
│ ├── azure-mgmt-keyvault-1.13.0.jar
│ ├── azure-mgmt-locks-1.13.0.jar
│ ├── azure-mgmt-monitor-1.13.0.jar
│ ├── azure-mgmt-msi-1.13.0.jar
│ ├── azure-mgmt-network-1.13.0.jar
│ ├── azure-mgmt-redis-1.13.0.jar
│ ├── azure-mgmt-resources-1.13.0.jar
│ ├── azure-mgmt-search-1.13.0.jar
│ ├── azure-mgmt-servicebus-1.13.0.jar
│ ├── azure-mgmt-sql-1.13.0.jar
│ ├── azure-mgmt-storage-1.13.0.jar
│ ├── azure-mgmt-trafficmanager-1.13.0.jar
│ ├── azure-storage-6.1.0.jar
│ ├── BigWig-1.0.jar
│ ├── biodata-formats-2.0.4.jar
│ ├── biodata-models-2.0.4.jar
│ ├── biodata-tools-2.0.4.jar
│ ├── biojava-alignment-4.1.0.jar
│ ├── biojava-core-4.1.0.jar
│ ├── biojava-phylo-4.1.0.jar
│ ├── caffeine-2.6.1.jar
│ ├── cellbase-client-4.8.0.jar
│ ├── cellbase-core-4.8.0.jar
│ ├── client-runtime-1.6.4.jar
│ ├── commons-codec-1.11.jar
│ ├── commons-collections-3.2.1.jar
│ ├── commons-compress-1.4.1.jar
│ ├── commons-datastore-core-4.0.4-20201126.222923-7.jar
│ ├── commons-datastore-mongodb-4.0.4-20201126.222937-7.jar
│ ├── commons-datastore-solr-4.0.4-20201126.222939-7.jar
│ ├── commons-io-2.5.jar
│ ├── commons-jexl-2.1.1.jar
│ ├── commons-lang-2.4.jar
│ ├── commons-lang3-3.8.1.jar
│ ├── commons-lib-4.0.4-20201126.222926-5.jar
│ ├── commons-logging-1.2.jar
│ ├── commons-math3-3.6.1.jar
│ ├── commons-rdf-api-0.5.0.jar
│ ├── converter-jackson-2.4.0.jar
│ ├── error_prone_annotations-2.3.2.jar
│ ├── forester-1.005.jar
│ ├── ga4gh-0.6.0a5.jar
│ ├── generex-1.0.2.jar
│ ├── google-auth-library-credentials-0.13.0.jar
│ ├── grpc-all-1.23.0.jar
│ ├── grpc-api-1.23.0.jar
│ ├── grpc-auth-1.23.0.jar
│ ├── grpc-context-1.23.0.jar
│ ├── grpc-core-1.23.0.jar
│ ├── grpc-netty-1.23.0.jar
│ ├── grpc-okhttp-1.23.0.jar
│ ├── grpc-protobuf-1.23.0.jar
│ ├── grpc-protobuf-lite-1.23.0.jar
│ ├── grpc-stub-1.23.0.jar
│ ├── grpc-testing-1.23.0.jar
│ ├── gson-2.8.0.jar
│ ├── guava-19.0.jar
│ ├── hk2-api-2.5.0-b32.jar
│ ├── hk2-locator-2.5.0-b32.jar
│ ├── hk2-utils-2.5.0-b32.jar
│ ├── hppcrt-0.7.5.jar
│ ├── htsjdk-2.21.3.jar
│ ├── http2-client-9.4.19.v20190610.jar
│ ├── http2-common-9.4.19.v20190610.jar
│ ├── http2-hpack-9.4.19.v20190610.jar
│ ├── http2-http-client-transport-9.4.19.v20190610.jar
│ ├── httpclient-4.5.6.jar
│ ├── httpcore-4.4.10.jar
│ ├── httpmime-4.5.6.jar
│ ├── jackson-annotations-2.10.1.jar
│ ├── jackson-core-2.10.1.jar
│ ├── jackson-core-asl-1.9.13.jar
│ ├── jackson-databind-2.10.1.jar
│ ├── jackson-dataformat-cbor-2.10.1.jar
│ ├── jackson-dataformat-yaml-2.10.1.jar
│ ├── jackson-datatype-joda-2.10.1.jar
│ ├── jackson-jaxrs-base-2.10.1.jar
│ ├── jackson-jaxrs-json-provider-2.10.1.jar
│ ├── jackson-mapper-asl-1.9.13.jar
│ ├── jackson-module-jaxb-annotations-2.10.1.jar
│ ├── jakarta.activation-api-1.2.1.jar
│ ├── jakarta.xml.bind-api-2.3.2.jar
│ ├── jansi-1.18.jar
│ ├── javassist-3.20.0-GA.jar
│ ├── javax.annotation-api-1.2.jar
│ ├── javax.el-3.0.1-b11.jar
│ ├── javax.inject-1.jar
│ ├── javax.inject-2.5.0-b32.jar
│ ├── javax.mail-1.6.1.jar
│ ├── javax.servlet-api-3.1.0.jar
│ ├── javax.ws.rs-api-2.0.1.jar
│ ├── jcip-annotations-1.0-1.jar
│ ├── jcl-over-slf4j-1.7.24.jar
│ ├── jcommander-1.64.jar
│ ├── jersey-client-2.25.1.jar
│ ├── jersey-common-2.25.1.jar
│ ├── jersey-container-servlet-2.25.1.jar
│ ├── jersey-container-servlet-core-2.25.1.jar
│ ├── jersey-entity-filtering-2.25.1.jar
│ ├── jersey-guava-2.25.1.jar
│ ├── jersey-media-jaxb-2.25.1.jar
│ ├── jersey-media-json-jackson-2.25.1.jar
│ ├── jersey-media-multipart-2.25.1.jar
│ ├── jersey-server-2.25.1.jar
│ ├── jetty-6.1.26.jar
│ ├── jetty-alpn-client-9.4.17.v20190418.jar
│ ├── jetty-alpn-java-client-9.4.17.v20190418.jar
│ ├── jetty-client-9.4.17.v20190418.jar
│ ├── jetty-http-9.4.17.v20190418.jar
│ ├── jetty-io-9.4.17.v20190418.jar
│ ├── jetty-security-9.4.17.v20190418.jar
│ ├── jetty-server-9.4.17.v20190418.jar
│ ├── jetty-servlet-9.4.17.v20190418.jar
│ ├── jetty-util-6.1.26.jar
│ ├── jetty-util-9.4.17.v20190418.jar
│ ├── jetty-webapp-9.4.17.v20190418.jar
│ ├── jetty-xml-9.4.17.v20190418.jar
│ ├── jjwt-0.9.1.jar
│ ├── joda-time-2.1.jar
│ ├── json-smart-2.3.jar
│ ├── jsr305-3.0.2.jar
│ ├── jul-to-slf4j-1.7.26.jar
│ ├── kryo-2.23.0.jar
│ ├── kubernetes-client-4.3.0.jar
│ ├── kubernetes-model-4.3.0.jar
│ ├── kubernetes-model-common-4.3.0.jar
│ ├── lang-tag-1.5.jar
│ ├── log4j-api-2.13.3.jar
│ ├── log4j-core-2.13.3.jar
│ ├── log4j-slf4j-impl-2.13.3.jar
│ ├── logging-interceptor-3.12.0.jar
│ ├── mail-1.4.7.jar
│ ├── microsoft-graph-1.4.0.jar
│ ├── mimepull-1.9.6.jar
│ ├── minlog-1.2.jar
│ ├── mongo-java-driver-3.11.2.jar
│ ├── netty-3.4.0.Final.jar
│ ├── netty-buffer-4.1.29.Final.jar
│ ├── netty-codec-4.1.29.Final.jar
│ ├── netty-codec-http2-4.1.38.Final.jar
│ ├── netty-codec-http-4.1.38.Final.jar
│ ├── netty-codec-socks-4.1.38.Final.jar
│ ├── netty-common-4.1.29.Final.jar
│ ├── netty-handler-4.1.29.Final.jar
│ ├── netty-handler-proxy-4.1.38.Final.jar
│ ├── netty-resolver-4.1.29.Final.jar
│ ├── netty-transport-4.1.29.Final.jar
│ ├── netty-transport-native-epoll-4.1.29.Final.jar
│ ├── netty-transport-native-unix-common-4.1.29.Final.jar
│ ├── ngs-java-2.9.0.jar
│ ├── nimbus-jose-jwt-9.1.3.jar
│ ├── oauth2-oidc-sdk-5.64.4.jar
│ ├── objenesis-2.1.jar
│ ├── okhttp-2.5.0.jar
│ ├── okhttp-3.12.0.jar
│ ├── okhttp-urlconnection-3.11.0.jar
│ ├── okio-1.15.0.jar
│ ├── opencensus-api-0.21.0.jar
│ ├── opencensus-contrib-grpc-metrics-0.21.0.jar
│ ├── opencga-analysis-2.0.0.jar
│ ├── opencga-app-2.0.0.jar
│ ├── opencga-catalog-2.0.0.jar
│ ├── opencga-client-2.0.0.jar
│ ├── opencga-core-2.0.0.jar
│ ├── opencga-master-2.0.0.jar
│ ├── opencga-server-2.0.0.jar
│ ├── opencga-storage-app-2.0.0.jar
│ ├── opencga-storage-core-2.0.0.jar
│ ├── opencga-storage-mongodb-2.0.0.jar
│ ├── opencga-storage-server-2.0.0.jar
│ ├── osgi-resource-locator-1.0.1.jar
│ ├── oskar-analysis-0.2.0-20191017.131623-2.jar
│ ├── oskar-core-0.2.0-20191017.131625-2.jar
│ ├── owlapi-api-5.1.13.jar
│ ├── owlapi-oboformat-5.1.13.jar
│ ├── owlapi-parsers-5.1.13.jar
│ ├── paranamer-2.3.jar
│ ├── perfmark-api-0.17.0.jar
│ ├── protobuf-java-3.9.0.jar
│ ├── protobuf-java-util-3.9.0.jar
│ ├── proto-google-common-protos-1.12.0.jar
│ ├── reactive-streams-1.0.0.jar
│ ├── reactor-core-2.0.7.RELEASE.jar
│ ├── reactor-stream-2.0.7.RELEASE.jar
│ ├── redisson-2.2.0.jar
│ ├── reflections-0.9.10.jar
│ ├── retrofit-2.4.0.jar
│ ├── rocksdbjni-4.5.1.jar
│ ├── rxjava-1.2.4.jar
│ ├── slf4j-api-1.7.25.jar
│ ├── snakeyaml-1.24.jar
│ ├── snappy-java-1.0.5.jar
│ ├── solr-solrj-8.4.1.jar
│ ├── stax2-api-3.1.4.jar
│ ├── swagger-annotations-1.5.22.jar
│ ├── swagger-core-1.5.22.jar
│ ├── swagger-jaxrs-1.5.22.jar
│ ├── swagger-jersey2-jaxrs-1.5.22.jar
│ ├── swagger-models-1.5.22.jar
│ ├── validation-api-1.1.0.Final.jar
│ ├── velocity-1.7.jar
│ ├── woodstox-core-asl-4.4.1.jar
│ ├── xz-1.8.jar
│ ├── zjsonpatch-0.3.0.jar
│ ├── zookeeper-3.5.5.jar
│ └── zookeeper-jute-3.5.5.jar
├── LICENSE
├── misc
│ ├── catalog
│ ├── clients
│ ├── completion
│ ├── demo
│ ├── examples
│ ├── migration
│ ├── scripts
│ └── solr
├── monitor
│ └── README.md
├── opencga-client-2.0.0.tar.gz
├── opencga.war
├── README.md
└── test
├── bin
├── dependencies
├── fitnesse
└── README.md
32 directories, 257 files
```
# Choosing Variant Storage Engine
HBase vs. MongoDB
OpenCGA implements a [Variant Storage Engine](/develop-2/components-1/variant-storage) framework ...
# Configuration
There are two configuration files:
* [configuration.yml](/develop-2/manual/installation/compute-cluster/configuration#main-configuration): this is the main configuration file for OpenCGA Catalog and Analysis
* [storage-configuration.yml](/develop-2/manual/installation/compute-cluster/configuration#storage-configuration): storage configuration file
For Hadoop configuration ...
## configuration.yml
## storage-configuration.yml
# Running Docker
## Overview
OpenCGA produces **four** different Docker images for runnning OpenCGA in two main different modes: *cluster* and *local*. All of them run Java 8 and can be found in [OpenCB Docker Hub](https://hub.docker.com/u/opencb), the docker images are:
* \*\*\*\*[opencga-base](https://hub.docker.com/r/opencb/opencga-base): base image for building the other images, this image contains the OpenCGA basic installation folder.
* [opencga-init](https://hub.docker.com/r/opencb/opencga-init): image running REST web services
* [opencga-demo](https://hub.docker.com/r/opencb/opencga-demo): all-in-one image with all OpenCGA components running
* [opencga-ext-tools](https://hub.docker.com/r/opencb/opencga-ext-tools): image that contains external tools (e.g., samtools, fastqc, R, ...) used by OpenCGA analysis
You can use OpenCGA Docker image to run a complete OpenCGA platform locally. You can use the other images to deploy a whole cluster solution in a cloud environment using Kubernetes. Docker Images are deployed in [Docker Hub OpenCB organisation](https://hub.docker.com/orgs/opencb).
### Implementation
OpenCGA publishes a number of images into [DockerHub](https://hub.docker.com/u/opencb) for user ease. These images are based on *Alpine JRE* images to keep sizes as small as possible, contains OpenCGA binaries, for complete contents of image, please have a look at Dockerfile in github. A typical image name will follow the following structure :
* opencga:{*OPENCGA\_VERSION\_NUMBER*}-{*VARIANT\_STORAGE\_FLAVOUR*}
* OPENCGA\_VERSION\_NUMBER \_\_ will be like 1.4.0, 1.4.2, 2.0 etc
* VARIANT\_STORAGE\_FLAVOUR can be mongoDB 4.0, hdinshigh, emr etc
* e.g. **opencga:1.4.0-mongo4.0**
OpenCGA has published a docker image for quick testing and playing without going through hassle to learn, wait and install each and every OpenCGA components. The docker image is available at the public docker registry under the repository ***opencb/opencga-demo**.*
The OpenCGA demo docker image contains the following components:
* OpenCGA binaries
* MongoDB 4.0
* Solr 6.6 (*default*)
* ***init.sh,*** a bash script to :
1. Install OpenCGA catalog
2. Populate data (optional)
To download OpenCGA demo image, use the command [*docker pull*](https://docs.docker.com/engine/reference/commandline/pull/) with the OpenCB enterprise, opencga-demo repository and tag.
## Run OpenCGA Cluster
Kubernetes, Ansible, OpenStack, ....
## Run OpenCGA Local
This mode is not intended for production but for a user demo.
## Build Docker Images
# Login
Testing REVIEWER permission
# Using OpenCGA
You can interact with your data in OpenCGA in several different ways, find the one that suits you most and get started!
## Overview
OpenCGA is implemented to provide users with multiple resources to manage and query variant and phenotypic information.
Each method comes with specific advantages: whilst ones are more versatile and allow users to do almost anything, others are less limited in the use but quicker to configure and use.
OpenCGA is a very versatile piece of software and thus we've dedicated special care to allow the final user to explore every possibility. Anyone can explore and choose the client methods which adapts better to their own specific use case.
### REST Web Services
All the clients operate with OpenCGA data through a comprehensive and optimised REST web service Application Programming Interface (API), which consists of more than 200 web services.The web services are organised based on the main OpenCGA entities (see [Data Models](/develop-2/overview/data-models)); each entity then displays different layers of operations with a comprehensive set of parameters.
The design of the web services has allowed to implement **three different ways** to query and operate with the multiple data stores that compose OpenCGA through the REST API, providing an intuitive way for the user to access the big data variant store.
* [REST Client Libraries](https://app.gitbook.com/@opencb/s/opencga/~/drafts/-MgLnDk3roHbBW3IRKmL/manual/using-opencga/client-libraries): four different client libraries have been implemented to ease the use of REST web services. This allows users to easily integrate OpenCGA in any pipeline. The four libraries are equally functional and fully maintained, these are [*Java*](http://docs.opencb.org/display/opencga/Java)*,* [*Python*](http://docs.opencb.org/display/opencga/Python) (available at [PyPI](https://pypi.org/project/pyopencga/)), [*R*](http://docs.opencb.org/display/opencga/R) and [*JavaScript*](http://docs.opencb.org/display/opencga/JavaScript)
* [Command Line](http://docs.opencb.org/display/opencga/Command+Line): users and administrators can use ***`opencga.sh`*** command line to query and operate OpenCGA.
* [IVA Web Application](http://docs.opencb.org/display/opencga/IVA+Web+App): an interactive web application called IVA has been developed to query and visualisation OpenCGA data.
# REST Web Service API
Understanding REST web service API
## REST API Design
### Understanding the URL
The general format of the REST API web services is:
```http
https://HOST_URL/APPLICATION/webservices/rest/{apiVersion}/{resource}/{ids}/{endpoint}?{options}
```
where **HOST\_URL** is the URL pointing to the host server and **APPLICATION** is the name of Java ***war*** file deployed in web server (eg. Tomcat), for example,
Entities inside the curly braces `{ }` are the web service parameters, and they are treated as variables. For example the following URL:
```http
http://bioinfo.hpc.cam.ac.uk/opencga-prod/webservices/rest/v1/samples/HG01879,HG01880/info?study=1000g
```
As it is explained later in this documentation, this RESTful web service will return the information stored in OpenCGA of the user ***demo***.
* **apiVersion (**v2**):** indicates OpenCGA version to retrieve information from, data models and API may change between versions.
* **resource:** specifies the data type of what the user wants to query by, in this example the resources are *samples*. This is one of the different *resources* listed below. \_\_
* **id:** the ID of the resources we want to query by*.* In this example are HG01879 and HG01880. Path parameters are limited to 100 IDs.
* **endpoint (**info**):** these parameters must be specified depending on the nature of your input data. For instance, info is used to fetch the information stored in the database regarding the id's passed.
* **options (**study=1000g**) :** variables in key-value pair form, passed as query parameters.
### REST Params
#### *apiVersion*
*apiVersions* are numbered as *v1*, *v2*, etc. At this moment we are heading to the second stable *apiVersion* which will be ***v2***.
#### resource
There are several metadata *resources* implemented such as *users, samples, individuals,* ... see below for more info.
#### *IDs*
This is the unique identifier(s) corresponding to the *resource* we want to interact with. Plural means a comma-separated list of IDs can be passed to improve performance with a single REST call rather than multiple calls. OpenCGA preserves the order of the results with corresponding IDs. A Boolean variable, **silent,** can be set to indicate, in case of a failure (resource doesn't exist, permission denied, etc), whether the user is interested in receiving partial results (true) with the information that could be successfully retrieved or just a failure with no results. As a trade-off between performance and ease of use a maximum of **100** IDs are allowed in one web service.
#### *options*
These query parameters can modify the behavior of the query (*exclude*, *include*, *limit*, *skip* and *count*) or add some filters to some specific endpoints to add useful functionality. The following image shows some typical options for a certain web service.
## REST Response
{% hint style="warning" %}
REST web services return the response wrapped in a ***RestResponse*** object ([view data model](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/response/RestResponse.java)). This consists of some *metadata* and a list of ***OpenCGAResult*** objects ([view data model](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/response/OpenCGAResult.java)) called ***responses*** containing the data ***results*** and *metadata* requested. The first response of the list will always contain the response of the OpenCGA federation being directly queried. Any additional response in the list will belong to other federated servers that could be connected. Each federated response will contain a list of ***results*** **\*\*(\_**OpenCGAResult\*\*\_) containing the data that has been queried.
{% endhint %}
```javascript
{
"apiVersion": "v2",
"time": 23,
"params": {
"include": "id",
"study": "study1",
"limit": "3"
},
"events": [
{
"type": "WARNING",
"message": "This is a development version OpenCGA 2.0.0-RC"
}
],
"responses": [
{
"time": 16,
"events": [],
"numResults": 3,
"results": [
{
"id": "HG01879"
},
{
"id": "HG01880"
},
{
"id": "HG01881"
}
],
"resultType": "org.opencb.opencga.core.models.Sample",
"numMatches": 3502,
"numInserted": 0,
"numUpdated": 0,
"numDeleted": 0
}
]
}
```
where:
* **Line 1**: single *RestResponse* object
* **Lines 2 and 3**: show the version and the duration time (ms)
* **Lines 4-8**: show all the parameters that have been provided.
* **Line 9-14:** show an events array where *info, warning* and *error* messages will be shown: For instance, when having network issues you could get "Catalog database not accessible".
* **Line 15**: list of *DataResults* called ***responses***. In this example, because federation is disabled, it only contains a single *DataResult.*
* **Line 17**: database duration time (ms) for each *DataResult.*
* **Line 18**: list of *events* where *info, warning* and *error* messages will be shown. For instance, it can show messages such as "Permission denied to access sample xxx". \_\_
* **Line 19**: number of elements returned in the ***results*** list.
* **Line 20-30**: List of ***results*** for this query.
* **Line 31**: resource type of ***results***.
* **Line 32**: total number of records found in the database for the given query.
* **Line 33-35**: Number of elements inserted, updated and deleted in the database. These counters only make sense for *create, updated* and *delete* operations.
### OpenCGA 1.x
Most web services return the results encapsulated in a single ***QueryResponse*** object ([view data model](https://github.com/opencb/java-common-libs/blob/develop/commons-datastore/commons-datastore-core/src/main/java/org/opencb/commons/datastore/core/QueryResponse.java)) consisting of some metadata and a list of ***QueryResult*** objects ([view data model](https://github.com/opencb/java-common-libs/blob/develop/commons-datastore/commons-datastore-core/src/main/java/org/opencb/commons/datastore/core/QueryResult.java)) called ***response*** containing the data and metadata requested. The reason for this two-level response is that some REST web services allow to pass multiple IDs as input parameter, this improves significantly the performance by reducing the number of calls, for instance a calling ***/info*** method with three sample IDs will return a ***QueryResponse*** object with three **QueryResults*****.*** Then, each ***QueryResult*** can contain multiple ***results,*** for instance when getting all samples from an individual or when fetching all variants from a gene.
However, most of the web services will return a ***QueryResponse*** with one single ***QueryResult*** with one or more ***result.*** In general the response object looks like:
```javascript
{
"apiVersion": "v1",
"time": 19,
"warning": "",
"error": "",
"queryOptions": {
"metadata": true,
"skipCount": false,
"limit": 10
},
"response": [
{
"id": "search",
"dbTime": 18,
"numResults": 10,
"numTotalResults": 56,
"warningMsg": "",
"errorMsg": "",
"resultType": "",
"result": [
{
// result 1
},
{
// result 2
},
// ...
{
// result 10
}
]
}
]
}
```
where:
* **Line 1**: single *QueryResponse* object
* **Lines 2 and 3**: show the version and the duration time (ms)
* **Lines 4 and 5**: show warning and error messages, for instance when having network issues you could get "Catalog database not accessible"
* **Line 6**: summary of all option parameters provided
* **Line 11**: list of *QueryResults* called ***response***. In this example, and in most of calls, there is only one *QueryResult*.
* **Line 14**: database duration time (ms) for each *QueryResult.*
* **Line 15 and 16**: number of elements returned in the list ***result*** (see below) and total number of records found in the database for a given query.
* **Line 17 and 18**: specific warning and error messages for each QueryResult
* **Line 19**: type of result such as resource.
* **Line 20**: list of results for this query, this can be *samples, variants,* ...
## Resources and Endpoints
REST API is organised into two main groups of web services, one to work with metadata and a different one to run some analyses: ***Catalog*** and ***Analysis***. See below a description of the web services.
### Catalog Web Services
Contains all endpoints for managing and querying metadata and permission.
| Resource | Path | Description | Main Endpoints |
| ----------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |
| Users | /users | Different methods to work with users | info, create, login, ... |
| Projects | /projects | Projects are defined for each user and contains studies | info, create, studies, ... |
| Studies | /studies | Studies are the main component of OpenCGA Catalog. They can be shared with other users and are the containers of the data (files, samples, cohorts, jobs...). | info, create, groups, ... |
| Files | /files | Files are added to the study and can be indexed to be queried | info, create, index, share, ... |
| Jobs | /jobs | Jobs are used to execute analyses. | info, create, ... |
| Families | /families | Family is a connected collection of individuals based on their relationship. | info, create, ... |
| Individuals | /individuals | Individual is the member from which a sample was taken. | info, create, ... |
| Samples | /samples | Samples are each of the experiment samples, typically matches a NGS BAM file or VCF sample. | info, create, annotate, share, ... |
| Cohorts | /cohorts | Cohort is a group of samples that share some common properties. These are used for data analysis. | info, create, stats, samples, ... |
| Clinical Analysis | /clinical | This handles creating and search of a clinical analyses. | info, create, ... |
| Meta | /meta | Contains basic information about the status of an OpenCGA installation instance. | ping, about, status |
| GA4GH | /ga4gh | GA4GH standard web services to search genomics data in OpenCGA | variant search, reads search, responses |
### Analysis Web Services
Different endpoint for running the alignment, variant and clinical analysis
| Category | Path | Description | Main Endpoints |
| ------------------ | ------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| Alignment Analysis | /analysis/alignment | Operations over Read Alignments to facilitate complete analysis with different tools. | index, query, stats, coverage |
| Variant Analysis | /analysis/variant | Operations over Genomic Variants to facilitate complete analysis with different tools. | index, stats, query, validate, ibs, facet, samples, metadata |
| Clinical Analysis | /analysis/clinical | You can manage Clinical Analysis metadata (e.g create a case, set permissions) or run a genome interpretation | execute |
### Swagger
OpenCGA has been documented using [Swagger](http://swagger.io/) project. Detailed information about *resources*, *endpoints* and *options* is available at:
## Client Libraries
Currently OpenCGA implements the following four client libraries:
1. [Java](http://docs.opencb.org/display/opencga/Java)
2. [Python](http://docs.opencb.org/display/opencga/Python)
3. [R](http://docs.opencb.org/display/opencga/R)
4. [JavaScript](http://docs.opencb.org/display/opencga/JavaScript)
## Deprecation Policy
Certain APIs are deprecated over the period of time as OpenCGA is a live project and continuously improved and new features are implemented. The deprecation cycle consists of a **warning** period to let make the user aware that these services are considered for change and highly likely will be replaced followed by a **deprecated** message**.** OpenCGA supports deprecated services for two releases (Deprecated and Next one). Deprecated services are hidden from Swagger in the following release and completely removed in the next one.
```javascript
Warning (working) --> Deprecated (working) --> Hidden (working) --> Removed (not working)
```
# IVA Web App
A rich web app front-end to work with OpenCGA
## Overview
## Uses Cases
## Gallery
# Client Libraries
Java, Python, R and Javascript clients are implemented
OpenCGA implements four client libraries:
* [Java](/develop-2/manual/using-opencga/client-libraries/java)
* [Python](/develop-2/manual/using-opencga/client-libraries/python)
* [R](/develop-2/manual/using-opencga/client-libraries/r)
* [JavaScript](/develop-2/manual/using-opencga/client-libraries/javascript)
The source code of all these client libraries is autogenerated by OpenCGA, this ensures all the four clients are functionally equivalent and 100% of REST API.
# pyopencga - Python Library
If you're a Python user and you're keen of running OpenCGA through Python, you're in the right place!
## Overview
OpenCGA implements a Python REST client library called **PyOpenCGA** to execute any query or operation through the REST web services API. **PyOpenCGA** provides programmatic access to **all the implemented** REST web services, providing an easy, lightweight, fast and intuitive solution to access OpenCGA data. The library offers the convenience of an object-oriented scripting language and provides the ability to integrate the obtained results into other Python applications.
Some of the main features include:
* full RESTful web service API implemented, all endpoints are supported including new alignment or clinical functionality.
* data is returned in a new *RestResponse* object which contains metadata and the results, some handy methods and iterators implemented.
* it uses the OpenCGA *client-configuration.yml* file.
* several Jupyter Notebooks implemented.
[**PyOpenCGA**](https://pypi.org/project/pyopencga/) has been implemented by Daniel Perez, Pablo Marin and David Gomez and it is based on a previous library called *pyCGA* implemented by Antonio Rueda and Daniel Perez from Genomics England. The code is open-source and can be found at . It can be installed using [PyPI](https://pypi.org/) and . Please, find more details on how to use the python library at [Using the Python client](http://docs.opencb.org/display/opencga/Using+the+Python+client).
## Installation
Python client requires at least **Python 3.x,** although most of the code is fully compatible with Python 2.7. You can install PyOpenCGA either from [PyPI](https://pypi.org/project/pyopencga/) repository or from source code.
### PyPI
PyOpenCGA client is available at PyPI repository at . Installation is as simple as running the following command line:
```
## Latest stable version
pip install pyopencga
```
### Source Code
From OpenCGA v2.0.0 the Python client source code can be found at GitHub Release at . You can easily install *pyOpenCGA* using the *setup.py* file.
```
## Get latest stable version from https://github.com/opencb/opencga/releases. You can use wget from the terminal
wget https://github.com/opencb/opencga/releases/download/v2.0.0/opencga-2.0.0.tar.gz
## Decompress
tar -zxvf opencga-2.0.0.tar.gz
## Move to the pyOpenCGA client folder
cd opencga-2.0.0/clients/python
## Install the library
python setup.py install
```
## Getting started
### Client Configuration
Configuration is handled by the *ClientConfiguration* class. You can create a *ClientConfiguration* using either the *conf/client-configuration.yml* file or by passing a dictionary.
```
## Import ClientConfiguration class
from pyopencga.opencga_config import ClientConfiguration
## You can create a ClientConfiguration by using the path to the client-configuration.yml file (it can also accept a JSON file)
config = ClientConfiguration('opencga-2.0.0/conf/client-configuration.yml')
## Additionally, you can pass a dictionary using the same structure as the client-configuration.yml (the only required parameter is REST host)
config = ClientConfiguration({"rest": {"host": "http://bioinfo.hpc.cam.ac.uk/opencga-prod"}})
```
### OpenCGA Client
*OpencgaClient* is the main class in *pyOpenCGA*. It manages login/logout authentication, REST clients initialisation and provides a set of other utilities.
To create an *OpencgaClient* instance, a *ClientConfiguration* instance must be passed as an argument. You can authenticate in two different ways. First, you can login by providing the user and optionally the password. Second, you can provide a valid token when creating *OpencgaClient*. Remember that tokens are only valid for a period of time.
```
## Import ClientConfiguration and OpencgaClient class
from pyopencga.opencga_config import ClientConfiguration
from pyopencga.opencga_client import OpencgaClient
## Create an instance of OpencgaClient passing the configuration
config = ClientConfiguration('opencga-2.0.0/conf/client-configuration.yml')
oc = OpencgaClient(config)
### Two authentication options:
## Option 1. If the user has a valid token, it can be passed to start doing calls as an authenticated user
oc = OpencgaClient(config, token='TOKEN')
## Option 2. If no token is provided, the user must login with valid credentials. Password is optional (if it is not passed to the login method, it will be prompted to the user)
oc.login(user='USER') ## The password will be asked
# or
oc.login(user='USER', password='PASSWORD')
## You can logout by executing the following command, the token will be deleted.
oc.logout()
```
The *OpencgaClient* class works as a *client factory* containing all the different *clients,* one per REST resource, that are necessary to call any REST web service. Below is a list of available clients:
```
## Create main clients
users = oc.users
projects = oc.projects
studies = oc.studies
files = oc.files
jobs = oc.jobs
families = oc.families
individuals = oc.individuals
samples = oc.samples
cohorts = oc.cohorts
panels = oc.panels
## Create analysis clients
alignments = oc.alignment
variants = oc.variant
clinical = oc.clinical
ga4gh = oc.ga4gh
## Create administrative clients
admin = oc.admin
meta = oc.meta
variant_operations = oc.variant_operations
```
### Client API
Clients implements **all** available REST API endpoints, one method has been implemented for each REST web service. The list of available actions that can be performed with all those clients can be checked in Swagger as explained in [RESTful Web Services#Swagger](http://docs.opencb.org/display/opencga/RESTful+Web+Services#RESTfulWebServices-Swagger). Each particular client has a method defined for each available web service implemented for the resource. For instance, the whole list of actions available for the **Sample** resource are shown below.
For all those actions, there is a method available in the sample client. For instance, to search for samples using the */search* web service, you need to execute:
```
## Look for the first 5 sample IDs of the study "study"
sample_result = oc.samples.search(study='study', limit=5, include='id')
```
### Working with the *RestResponse*
As described in [RESTful Web Services#RESTResponse](http://docs.opencb.org/display/opencga/RESTful+Web+Services#RESTfulWebServices-RESTResponse), all REST web services return a *RestResponse* object containing some metadata and a list of *OpenCGAResults*. Each of these *OpenCGAResults* contain some other metadata and the actual data results.
To work with these REST responses in an easier way, *RestResponse* class has been implemented to wrap the web service *RetResponse* object and to offer some useful methods to process the results. For instance, the *sample\_result* variable from the example above is a *RestResponse* instance. This object defines several methods to navigate through the data.
The implemented *RestResponse* methods are:
```
## Returns the list of results for the response in position "response_pos" (response_pos=0 by default)
sample_response.get_results(response_pos)
## Returns the result in position "result_pos" for the response in position "response_pos" (response_pos=0 by default)
sample_response.get_result(result_pos, response_pos)
## Returns the list of responses
sample_response.get_responses()
## Returns the response in position "response_pos" (response_pos=0 by default)
sample_response.get_response(response_pos)
## Returns all results from the response in position "response_pos" as an iterator (response_pos=None returns all results for all QueryResponses)
sample_response.result_iterator(response_pos)
## Returns all response events by type "event_type" ('INFO', 'WARNING' or 'ERROR') (event_type=None returns all types of event)
sample_response.get_response_events(event_type)
## Returns all response events by type "event_type" ('INFO', 'WARNING or 'ERROR') for the response in position "response_pos" (event_type=None returns all types of event; response_pos=0 by default)
sample_response.get_result_events(event_type, response_pos)
## Return number of matches for the response in position "response_pos" (response_pos=None returns the number for all QueryResponses)
sample_response.get_num_matches(response_pos)
## Return number of results for the response in position "response_pos" (response_pos=None returns the number for all QueryResponses)
sample_response.get_num_results(response_pos)
## Return number of insertions for the response in position "response_pos" (response_pos=None returns the number for all QueryResponses)
sample_response.get_num_inserted(response_pos)
## Return number of updates for the response in position "response_pos" (response_pos=None returns the number for all QueryResponses)
sample_response.get_num_updated(response_pos)
## Return number of deletions for the response in position "response_pos" (response_pos=None returns the number for all QueryResponses)
sample_response.get_num_deleted(response_pos)
```
To explore the data in an easier way, a method named ***print\_results*** has also been implemented to show the response in a more human-readable format.
```
## Print results of the query for the response in position "response_pos" (response_pos=None returns the results for all QueryResponses)
sample_response.print_results(fields='id', response_pos=0, limit=5, separator='\t', metadata=True, outfile='path/to/output.tsv')
```
## Examples and tutorials
### Setting up OpencgaClient and logging in
```
# First, we need to import both the ClientConfiguration and the OpencgaClient
from pyopencga.opencga_config import ClientConfiguration
from pyopencga.opencga_client import OpencgaClient
# Second, we need to set up the configuration
# The main client-configuration.yml file has a "host" section to point to the REST OpenCGA endpoints
# We need to either pass the path to the configuration file or a dictionary with the same structure of the file
config = ClientConfiguration({'rest': {'host': 'http://bioinfo.hpc.cam.ac.uk/opencga-prod'}})
# Third, we create an instance of the OpencgaClient passing the configuration
oc = OpencgaClient(config)
# Finally, we need to authenticate.
oc.login(user='demouser', password='demouser')
# Additionally, we can check that we've logged in successfully by printing the obtained token
print(oc.token)
```
### Getting ID's for available projects, studies, families and samples
```
# We can get the ID of all the available projects in this OpenCGA installation
for project in oc.projects.search().get_results():
print(project['id'])
# We can get the ID of all the available studies in the project
for study in oc.studies.search(project='family').get_results():
print(study['id'])
# We can get the ID for all the available families in the study
for family in oc.families.search(study='corpasome').get_results():
print(family['id'])
# We can get the ID for all the available samples in the study
for sample in oc.samples.search(study='corpasome').get_results():
print(sample['id'])
```
### Getting gene variants for individuals with a particular disorder
```
# We are interested in looking for all the individuals containing a particular disorder: "OMIM:611597"
individuals_query_response = oc.individuals.search(
study='corpasome', # name of the study where the families are stored
disorders='OMIM:611597', # id of the disorders of interest
include='id' # retrieve only these fields from the results
)
# If we want to know exactly the number of individuals obtained, we can run:
print(individuals_query_response.get_num_results())
# Now we fetch all the variants falling in the "BFSP2" gene for those individuals
# In this case, we will limit the variant query to a maximum of 10 results
# We also exclude sample information (includeSample='none') as it can be huge and would make this query much slower
for individual in individuals_query_response.get_results():
print('Individual: ' + individual['id'])
samples = ','.join([sample['id'] for sample in individual['samples']])
variant_response = oc.variants.query(study='corpasome', sample=samples, gene='BFSP2', includeSample='none', limit=10)
if variant_response.get_num_results() > 0:
for variant in variant_response.get_results():
print('{}:{}-{}\t{}'.format(variant['chromosome'], str(variant['start']), str(variant['end']), variant['type']))
else:
print('No variant results found')
```
### Getting sample variant ID's
```
# Now we are interested in getting the rs IDs for the first 10 variants for a particular sample
for variant in oc.variants.query(sample='ISDBM322015', study='corpasome', limit=10).get_results():
print(variant['names'])
# We can also get rs IDs for multiple samples
for variant in oc.variants.query(sample='ISDBM322015,ISDBM322016,ISDBM322017,ISDBM322018', study='corpasome', limit=10).get_results():
print(variant['names'])
```
### Getting all samples containing a variant
```
# If we have an ID for a variant, we can obtain its ID in OpenCGA (chromosome:position:reference:alternate)
variant_id = oc.variants.query(study='corpasome', xref='rs1851943').get_result(0)['id']
# Now we are interested in getting all the samples that have that particular ID
for variant in oc.variants.query_sample(study='corpasome', variant=variant_id, debug=True).get_results():
for study in variant['studies']:
for sample in study['samples']:
print(sample['sampleId'])
```
Additionally, there are several notebooks defined in with more real examples.
# opencgaR - R library
If you're a R user and you're keen of running OpenCGA through R, you're in the right place!
## Overview
**opencgaR** is an R library that interacts with OpenCGA REST API to execute any operation supported by the web services through R code.
The client offers programmatic access to the implemented REST web services, facilitating the access and analysis of data stored in OpenCGA. From version 2.0.0 data is returned in a new *RestResponse* object which contains metadata and the results. The client also implements some handy methods to return information from this object.
{% hint style="info" %}
**opencgaR code** has been implemented by *Marta Bleda.* It's open-source and can be found at . It can be installed easily by downloading the pre-build package. Please, find more details on how to use the R library at [Using the R client](http://docs.opencb.org/display/opencga/Using+the+R+client).
{% endhint %}
## Installation
### Requisites
* An operating machine with R installed and functional. **opencgaR** requires at least **R version 3.4 (**although most of the code is fully compatible with earlier versions).
* Have the pre-build R package stored in your local machine. Alternatively you can also provide the URL to the pre-build R package file.
The pre-build R package of **opencgaR** can be downloaded from the OpenCGA v2.0.0 GitHub Release at .
Once requirements have been fulfilled, installing opencgaR becomes as easy as use the `install.packages` function in R. `install.packages` can also install a source package from a remote `.tar.gz` file by providing the URL to such file.
### Installation from the R terminal
```
## Install opencgaR by providing the URL to the package
> install.packages("opencgaR_2.0.0.tar.gz", repos=NULL, type="source")
```
# Java
# JavaScript
The OpenCGA JavaScript Client is provided as part of [JSorolla](https://github.com/opencb/jsorolla).\
Some examples of basic usage can be found in [examples](https://github.com/opencb/jsorolla/tree/develop/src/core/clients/opencga/examples) directory.
### Example:
#### Fetching the first 10 variants of the Study of interest using a token
```
import {OpenCGAClient} from "./opencga-client.js";
const HOST = ""; // add your host
const STUDY = ""; // add your study of interest
const TOKEN = ""; // add a valid token
const client = new OpenCGAClient({
host: HOST,
version: "v2",
cookies: {active: false},
token: TOKEN
});
(async () => {
try {
const restResponse = await client.variants().query({study: STUDY, limit:10});
console.table(restResponse.getResults());
} catch (response) {
if (response instanceof RestResponse) {
console.error(response.getEvents())
} else {
console.error(response)
}
}
})();
```
#### Fetching the first 10 variants of the Study of interest using OpenCGA credentials.
In this case an Opencga Session is created. The Opencga Study being used is the default one for the user.
```
import {OpenCGAClient} from "./opencga-client.js";
const HOST = ""; // add your host
const STUDY = ""; // add your study of interest
const USERNAME = ""; // add your username
const PASSWORD = ""; // add your username
const client = new OpenCGAClient({
host: HOST,
version: "v2",
cookies: {active: false}
});
(async () => {
try {
await client.login(USERNAME, PASSWORD)
const session = await client.createSession();
const restResponse = await session.opencgaClient.variants().query({limit:10, study: session.study.fqn});
console.table(restResponse.getResults());
} catch (e) {
console.error(e)
}
})();
```
# Command Line
## Overview
**opencga.sh** is the officially recommended command line tool for users. It implements most of the functionality with many different *commands* and *subcommands.* These *commands* are a one-to-one mapping of *Resources* from REST web services and *subcommands* are mapping to end-points. All the operations that can be performed using the command line internally create one or several REST calls, so access to REST machine/cluster is required.
## Installation
OpenCGA command line can be downloaded from the main GitHub repository:
For a detailed description of all the steps required to download and configure the CLI, refer to the xxx section of the USER MANUAL.
## Correlation Between REST and CLI
In the following URL, "*samples"* is the resource and "*search"* is the endpoint:
[https://ws.opencb.org/opencga-demo/webservices/rest/v1/**samples**/**search**](https://ws.opencb.org/opencga-demo/webservices/rest/v1/samples/search)
the corresponding command in the command line is :
| `./opencga.sh samples` |
| ---------------------- |
and the corresponding subcommand is :
| `./opencga.sh samples search` |
| ----------------------------- |
## CLI Session Management
Generally, unless we are pointing to a public OpenCGA installation, users will first need to log in using the "users login" command line. Once the user has successfully logged in, a session file will be generated in their home folder:
| `~/.opencga/session.json` |
| ------------------------- |
This session file contains the following information:
This makes it easier for users to login only once and execute any number of commands till the session token is expired. Session expiration is set by OpenCGA server independently from the client. Once the token is expired, the user has to login again and can perform desired operations as normal.
# Configuration
## **How to configure the OpenCGA command line for my installation?**
OpenCGA offers various ways to interact with the variant data stored in the project/studies, including the IVA web application, R, Python and Java clients libraries and a powerful Command line interface (CLI).
This document describes how to download, configure and execute the OpenCGA CLI assuming that:
* You have a Linux or Mac work station with internet access and a Java Runtime installed. This tutorial is performed using Ubuntu 20.04.2 LTS. If you have problems installing OpenCGA CLI on other platforms, please contact at or create a ticket at[ Zetta Service Desk](https://zettagenomics.atlassian.net/servicedesk/customer/portal/1).
* You have access to an OpenCGA server. This tutorial uses the public server at .
* You already have a user. This tutorial uses the readonly demouser:demouser user. Having a user with admin credentials will allow you to perform more advanced tasks like populating metadata, providing permissions or ingesting new datasets.
## **Procedure**
### **1. Download the OpenCGA from the GitHub official repository**
In a web browser navigate to the official OpenCGA GitHub repository at /releases. Access the latest release page by clicking the release tag at the right side of the main screen shown below:
{% hint style="warning" %}
**Note:** [this page](https://github.com/opencb/opencga/releases) lists all of the OpenCGA releases. We recommend using the latest stable version, especially if it’s your first time interacting with OpenCGA and its CLI. Newer rc (release candidate) releases may be available but these require compiling which is outside \
the scope of these instructions.
{% endhint %}
On the release page click the `opencga-client-x.x.x.tar.gz` link, the package download should start automatically:
### **2. Folder organisation:**
Once the file is downloaded, you should be able to find it in your local Download folder:
Unzip and place the folder in a proper location within your file system. One possible example of organisation could be as follows.
First, move the folder to your desire location:
```
$ mv $HOME/Downloads/opencga-client-2.0.3.tar.gz $HOME/
```
Now, decompress the folder containing the OpenCGA CLI using the next command:
```
$ tar -xvzf $HOME/opencga-client-2.0.3.tar.gz -C $HOME/
```
### **3. CLI configuration:**
Once the folder has been decompressed and placed in a path where you can make proper use of the CLI tool, the next step consists in editing the config file so it points to the URL of the OpenCGA installation in which we’d like to use the CLI.
For that, open the `client-configuration.yml` file in the conf folder with your usual text editor:
```
$ nano $HOME/opencga-client-2.0.3/conf/client-configuration.yml
```
To change the host to your OpenCGA installation, go to the host parameter in the REST client configuration options and change the default URL to point to your installation. Save the changes and exit the editor.
**Example:** let’s suppose we’d like to use the command line in the university demo installation at .
### **4. Start using the CLI: opencga.sh**
You can call the command line with the program opencga.sh (note that for convenience you can add the bin folder in your $PATH variable in bashrc or similar):
```
$ $HOME/opencga-client-2.0.3/bin/opencga.sh
```
To test that the command line works properly, you can try to login into the demo installation with the demouser credentials: **user**: `demouser`, **password**: `demouser`.
```
$ echo "demouser" | $HOME/opencga-client-2.0.3/bin/opencga.sh users login -u demouser
```
**`You have been logged in correctly. This is your new token eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJkZW1vdXNlciIsImF1ZCI6Ik9wZW5DR0EgdXNlcnMiLCJpYXQiOjE2MjQzMDg2NzMsImV4cCI6MTYyNDMxMjI3M30.YVNlvEDdqR02QJr6GLYCCB6WeGs7h8fyrscSsjyBSSM`**
If the login is successful, the token associated with your session will appear on the screen.
Now you should be ready to start using the CLI for querying, updating and ingesting data in OpenCGA!
# Files
opencga.sh files
## Upload
Upload a physical local file to one specific study in OpenCGA.
| Name | Usage |
| -------------- | ----------------------------------------------------------------------------------- |
| -i --input | Input file |
| -s --study | Study \[\[user@]project:]study where study and project can be either the ID or UUID |
| --catalog-path | Path within catalog where the file will be located |
| --file-name | Name of the file by which it will be stored in catalog |
| --description | Description of the file |
| --bioformat | Bioformat of the file (VARIANT, ALIGNMENT, SEQUENCE, PEDIGREE...NONE) |
| --file-format | Format of the file (VCF, BCF, GVCF, SAM, BAM, BAI...UNKNOWN) |
# Public Demo
## Overview
We have installed a public ***demo*** at the University of Cambridge to facilitate the testing and development for all users. We have loaded and indexed **five different datasets** organised in 3 *projects* and 5 *studies*, these cover the most typical data use cases today such as multi-sample VCF, family exomes, and genomes; or cancer somatic data. All documentation examples and tutorials use this *demo* installation.
## Connecting to the public *demo*
OpenCGA public *demo* REST URL is available at . You can check REST API and documentation at .
### Credentials
We have created a read-only user called ***demouser*** with password ***demouser***. As in most OpenCGA installations where normal users are not the owners of the data, *demouser* has been given VIEW access to all *demo* user data, this is a very common configuration in OpenCGA where the owner of the data grant access to other users. In this *demo* installation the owner of the data is *demo* user, while *demouser* user is the public user created to query data.
## Datasets
### Genomic Data
In this *demo* we have indexed 5 different genomic datasets. Data has been organised in three *projects* and five *studies*. These represents different assemblies and data types such as multi sample VCF, aggregated VCF or family genome or exome. The data is organised in 3 *projects* and 5 studies. You can find some useful information in this table:
| Project ID - Name | Study ID - Name | VCF File Type | Samples | Variants |
| ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | ---------------------- | ------- | -------------- |
|
| GWS Family Multisample | 17 | **12,263,246** |
### Clinical Data
In order to make this *demo* more useful to users we have loaded or simulated some clinical data, this allows to exploit OpenCGA analysis such as GWAS or clinical interpretation. You can find clinical data for each study in the following sections.
#### 1000g
We loaded the 1000 Genomes pedigree file, you can find a copy at
#### uk10k
There is no possible clinical data in this study. This is a *WGS aggregated* dataset so no samples or genotypes were present in the dataset and, therefore, no *Individuals* or *Samples* have been created.
#### corpasome
We simulated two different disorders and few phenotypes for the different members of the family. To be documented soon.
#### platinum
To be documented soon.
#### rams\_cml
To be documented soon.
# Managing Data
This section provides instructions on how data is managed (loaded, updated, removed) in OpenCGA.
1. [Working with Projects and Studies](broken://pages/-MlPg7YeWGVlhpSZm3wr)
2. [Loading of VCF files](broken://pages/-MlPg7YfX7RHGqy33pdu)
3. [Working with metadata](/develop-2/manual/data-management/working-with-files)
4. [Controlling access and setting permissions](/develop-2/manual/data-management/sharing-and-permissions)
5. [Data versioning](/develop-2/manual/data-management/population-of-metadata/data-versioning)
# Working with Projects and Studies
## Introduction
The project/study organisation is key in order to optimise the usability of the data in OpenCGA.
**Projects** provide physical separation of data into different database tables.\
**Studies** provide logical separation of data within a Project.
### Guidelines
* You MUST store your data in different projects when they are on different genome assemblies (e.g you should create a project for data from *GRCh37* and other for data from *GRCh38*)
* You CAN store your data in different projects when there is no foreseeable need to process them jointly.
* You may divide your data in studies corresponding to different independent datasets that may be used together in some analysis, with the aim of having homogeneous datasets for each study.
### Owner user
The owner is the user who creates the project/study where the new data will be loaded. The users with permission to perform data ingestion into a concrete study in OpenCGA are the owner user, and other users with admin privileges for the specific study (provided by the owner).
After deciding structure, the new projects and studies may need to be created. This step must be performed by the owner of the new created elements.
## **Creating new projects**
The first step is [login](/develop-2/manual/login) into OpenCGA with a `FULL` account (See [Data Management](/develop-2/manual/data-management/sharing-and-permissions)). Then, a Project can be created using the next command:
```
$ ./opencga.sh projects create --id
-n
--organism-scientific-name hsapiens
--organism-assembly
```
Optionally, you can add other parameters like `--description` . You can get the full list of parameters by adding to the command.
## **Creating new studies**
Similar to the project creation, studies are created with this command:
```
$ ./opencga.sh studies create --project
--id
-n
```
{% hint style="info" %}
You don’t need to provide the organism assembly again, as it’s inherited from the project. Remember that all studies from the same project will share the same assembly.
{% endhint %}
To get the list of all projects and studies belonging to one specific user, run:
```
$ ./opencga.sh users info
```
### \*\*\*\*
# Loading of VCF files
This page describes the process that an operator follows to load their VCF files into an OpenCGA Variant Store.
## **Introduction**
This page describes the typical process that an operator will follow to load their VCF files into an OpenCGA using the OpenCGA command line tools. Loading of VCF files can be done either before (normally) or after the loading of sample and clinical metadata.
It typically takes a few minutes to load a VCF from a single exome but it can take several days (or even weeks) to load many thousands of whole genomes. For more information on data load times see \[Data Load Benchmarks].
The process is divided into 5 steps:
1. "Register" the source VCF files with OpenCGA; this creates basic File, Sample and Individual entries in Catalog.
2. "Index" each VCF file; this loads data into the Variant Store
3. "Annotate" all newly-created variants against the CellBase knowledge base.
4. "Summarise"; re-calculate all variant statistics.
5. "Secondary Index" to include annotations and summaries.
\[TODO: update the figure below to follow the process described above]
## **Prerequisites**
This document assumes that:
* The source VCF files are accessible (e.g. via shared filesystem) on the target OpenCGA server.
* The operator has access to a workstation with network access to the web services on the OpenCGA server.
* Compatible OpenCGA client software is installed on the workstation. Find [here](/develop-2/manual/using-opencga/command-line) the instructions on how to install the client software.
* The destination Study has been created on the OpenCGA server. Find [here](broken://pages/-MlPg7YeWGVlhpSZm3wr) instructions for creating Projects and Studies.
* The operator has login credentials on the OpenCGA server with appropriate permissions; i.e. write access to the destination Study.
## **Catalog file register**
This step presents the data to OpenCGA and registers the new files into the system. Samples will be created automatically after linking the file, by reading the VCF Header. This step can be further extended with extra annotations, defining individuals, creating cohorts or even families.
It is important to note that this step is a synchronous operation that does not upload the genomic data (e.g:VCFs) into OpenCGA, instead, the files will only be “linked” (registered) with OpenCGA. Therefore, the files to link must be in a location that is accessible by the OpenCGA server (REST servers and the Master service).
### **Catalog Path Structure**
**I**nternally, the Catalog metadata holds a logical tree view of the linked files that can easily be explored or listed. Try using:
```
$ ./opencga.sh files tree --study --folder
```
New folders can be created with this command:
```
$ ./opencga files create --study --path
```
Being `` the directory that you’d like to create within catalog.
### Linking files **synchronously** vs. **asynchronously**
{% hint style="warning" %}
**Note** that for VCF files with more than 5000 samples linking should be launched as an asynchronous job
{% endhint %}
There are two different commands depending on the type of VCF that needs to be loaded. Aggregated VCF files with many samples need to be linked by launching an asynchronous job.
#### **- Linking files synchronously (\~less than 5000 samples)**
Files are registered into OpenCGA Catalog using this command line:
```
$ ./opencga.sh files link --study
--path
--input
```
Multiple files can be linked using the same command typing multiple input files separated by space or comma.
#### **- Linking files asynchronously (more than 5000 samples)**
For VCFs containing more than 5000 samples, the linking steps needs to be performed as an asynchronous job. In this case, a different command needs to be run:
```
$ ./opencga.sh files link-run --study
--path
--input
```
**Full example: This example includes creating a directory and link of VCF file in the new path.**
```
## Create one folder “data/” in study “myStudy”
$ ./opencga.sh files create --study --path
## Link the files “myFirstFile.vcf.gz” and “mySecondFile.vcf.gz” in the newly created folder
$ ./opencga.sh files link --study
--path
-i /data/myFirstFile.vcf.gz /data/mySecondFile.vcf.gz
```
## **Variant storage index**
This operation will read the content of the file, run some simple validations to detect possible errors or data corruptions, and ingest the variants into the Hadoop system, building some high performance indexes.
Each file index operation will be run by an asynchronous job, to be executed by the OpenCGA Master service.
Contrary to the Catalog File Register step, only one file should be provided as input in the Variant storage index command line. This will create separate asynchronous indexing jobs for each one of the files. This is important in order to avoid failure of the jobs.
Use this command to launch a variant index job:
```
$ ./opencga.sh operations variant-index --study
--file
```
All the jobs along with their current status can be either inspected from IVA, or running this command line:
```
$ ./opencga.sh jobs top ---study
```
**Special scenarios**
{% hint style="danger" %}
**Note**: Be aware that the misuse of the parameters described below may lead to data corruption. Please, ask for support at or create a ticket in the [Zetta Service Desk](https://zettagenomics.atlassian.net/servicedesk/customer/portal/1) if you are not sure about what option adjust to your dataset.
{% endhint %}
* **Samples data split by chromosome or region**
By default, OpenCGA doesn’t allow you to index a VCF file if any of its samples is already indexed as part of another VCF file. This restriction is to avoid accidental data duplications. In case of having one dataset split by chromosome or region, this restriction can be bypassed by adding the param `--load-split-data ` to the variant index command line.
* **Multiple files for the same samples**
Similarly to the previous scenario, a dataset may contain multiple files from the same set of samples that may want to be indexed together, for example, when using multiple VCF callers for the same sample. In this case, you can bypass the restriction by adding the param `--load-multi-file-data.`
* **Family or Somatic callers**
When using special callers it is important to specify it in the command line with either`--family / --somatic.`
## **Variant Annotation**
Once all the data is loaded, we need to run the Variant Annotation. This is a key enrichment operation that will attach CellBase Variant Annotations with the loaded data, allowing filtering by a large number of fields.
Find more information at**:** [**http://docs.opencb.org/display/cellbase/Variant+Annotation**](http://docs.opencb.org/display/cellbase/Variant+Annotation)
The Variant Storage Engine will run the annotation just for the new variants, being able to reuse the existing annotations to save time and disk usage. This operation is executed at the project level, so shared variants between studies won’t need to be annotated twice.
```
$ ./opencga.sh operations variant-annotation-index --project
--study
```
Similar to the variant-index process, this command line will queue an asynchronous job to be executed by the OpenCGA Master service.
## **Variant Statistics calculation**
The second enrichment operation is the Variant Statistics Calculation. After defining a cohort, you might decide to compute the Variant Stats for that cohort. These statistics include the most typical values like allele and genotype frequencies, MAF, QUAL average, FILTER count...
```
$ ./opencga.sh operations variant-stats-index --study
--cohort ,..,
```
For updating the stats of all the cohorts, or when there are no cohorts in the study apart from the default `ALL cohort`:
```
$ ./opencga.sh operations variant-stats-index --study --cohort ALL
```
#### **Aggregated VCFs**
{% hint style="warning" %}
**This section is under current development.**
{% endhint %}
In case of having computed stats codified in the INFO column of a VCF using standard or non-standard keys, these values can be converted into `VariantStats` models, and be used for filtering.
To extract the statistics, you need to create a mapping file between the INFO keys containing the information, and it’s meaning. Each line will have this format:\
**\.\=\**
Then, this file needs to be linked in catalog, and referred when computing the stats.
OpenCGA supports 3 different “ways” of codifying the information, known as “aggregation method”. Some of these are named after public studies that started using them. Each one defines a set of known keys that will be used to parse the statistics.
* **BASIC : Using standard vcf-spec INFO keys.**
* **AN : Total number of alleles in called genotypes**
* **AC : Total number of alternate alleles in called genotypes**
* **AF : Allele Frequency, for each ALT allele, in the same order as listed**
* **EXAC**
* **HET: Count of HET genotypes. For multi allelic variants, the genotype order is 0/1, 0/2, 0/3, 0/4... 1/2, 1/3, 1/4... 2/3, 2/4... 3/4...**
* **HOM : Count of HOM genotypes. For multi allelic variants, the genotype order is 1/1, 2/2, …**
* **EVS**
* **GTS: List of Genotypes**
* **GTC: Genotypes count, ordered according to “GTS”**
* **GROUPS\_ORDER: Order of cohorts for key “MAF”**
* **MAF: Minor allele frequency value for each cohort, ordered according to “GROUPS\_ORDER”**
**e.g. Single cohort variant stats**\
**custom\_mapping.properties**
|
ALL.AC =AC
ALL.AN =AN
ALL.AF =AF
ALL.HET=AC\_Het
ALL.HOM=AC\_Hom/2
#Key “HEMI” is not supported
#ALL.HEMI=AC\_Hemi
|
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
```
$ ./opencga.sh operations variant-stats-index --study
--cohort
--aggregation EXAC
--aggregation-mapping-file custom_mapping.properties
```
## **Variant Secondary Index Build**
Secondary indexes are built using the search engine Apache Solr for improving the performance of some queries and aggregations, allowing full text search and faceted queries to the Variant database.
This secondary index will include the Variant Annotation and all computed Variant Stats. Therefore, this step needs to be executed only once all annotations and statistics are finished.
```
$ ./opencga.sh operations variant-secondary-index --project
--study
```
## **Enrichment Operations**
This steps are optional operations, that can be indexed to enrich the data displayed at the IVA web application:
### **Sample Variant Stats**
Sample Variant Stats will contain a set of aggregated statistics values for each sample.
```
$ ./opencga.sh variant sample-stats-run --study --sample all
```
These aggregated values can be computed across all variants from each sample, or using a subset of variants using a variant filter query. e.g:
```
$ ./opencga.sh variant sample-stats-run --study
--sample all
--variant-query ct=missense_variant
--variant-query biotype=protein_coding
```
By default, this analysis will produce a file, and optionally, the result can be indexed in the catalog metadata store, given an ID.
```
./opencga.sh variant sample-stats-run --study
--sample all
--index
--index-id missense_variants
--variant-query ct=missense_variant
--variant-query biotype=protein_coding
```
The ID ALL can only be used if without any variant query filter.
```
$ ./opencga.sh variant sample-stats-run --study
--sample all
--index
--index-id ALL
```
### **Cohort Variant Stats**
{% hint style="warning" %}
**This section is under current development.**
{% endhint %}
### **Family Index**
{% hint style="warning" %}
**This section is under current development.**
{% endhint %}
# Working with Files
# Sharing and Permissions
## Overview
OpenCGA *Catalog* provides an **authenticated** **environment** to manage data. It counts with a powerful mechanism of custom-built and secure permission system defined by a mechanism of **Access Control Lists (ACLs).**
The system enables to define a list of permissions at any entity level. The permissions are granted to members, being a member either a isolate user or a defined group of users.
## Users
A [*User*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/User.java) is generally a person who will be using OpenCGA. The idea in OpenCGA is that every single person have its corresponding user created in OpenCGA. Every user should be authenticated (see [Authentication](http://docs.opencb.org/display/opencga/Authentication) section) to be able to perform any action. However, to be able to perform any actions, users will need to be granted some specific permissions or to have a specific category within the *Study* (see [Sharing and Permissions](http://docs.opencb.org/display/opencga/Sharing+and+Permissions) section).
There are two default types of user accounts:
* **full**: these users have permission to create *projects* and *studies.*
* **guest**: users that will not have the possibility to create their own *projects* and *studies.* Despite this, these users will still be able to collaborate (view, write...) in other user's studies as long as they have been granted the proper permissions.
### Groups
You can create *group* of users, this will simplify data permission management. Groups are defined at *study* level, i.e. each *study* contains different groups. *Groups* can only be created by the study *owner* or the study *admins.* A [*Group*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/Group.java) of users will generally bring together users that have something in common. Groups are strongly related to permissions in Catalog (see [Sharing and Permissions](http://docs.opencb.org/display/opencga/Sharing+and+Permissions) section). For example, let's imagine that we have 5 different departments in our institution and each department requires different permissions to the data. In that case, we could think of creating as many groups of users as different departments we have in our institution and give the specific permissions to those groups (not to the users) that have been created in OpenCGA. Doing it this way have lots of benefits:
* A user belonging to different departments (groups) will have the permissions from all the groups he/she belongs to.
* If one user leaves the department, we would just need to remove that user from the corresponding group. That user will automatically lose the permissions the group has\*.
* If one user starts in the department, we would just need to add that user to the corresponding group. That user will automatically gain the permissions the group has.
\* Unless the user had some or all the permissions granted to the group defined in a different group he/she might still belong to or assigned directly to the user.
All *studies* have always two administrative groups that cannot be deleted or renamed, these are *admins* and *members*.
## Projects
Any ***full*** user can create any number of *projects* (and *studies).* A [*Project*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/Project.java) is *a piece of planned work or an activity that is finished over a period of time and intended to achieve a particular purpose* ([Cambridge dictionary](https://dictionary.cambridge.org/dictionary/english/project) definition). A *Project* in Catalog is understood as a scientific project for one concrete species. Any project in Catalog will contain at least a name, an alias (project identifier) and the species organism. But it can also contain the organisation and a description of the project.
Projects are used as the central piece for variants storage.
## Studies
Projects are composed by a set of studies. A [*Study*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/Study.java) is *the activity of examining a subject in detail in order to discover new information* ([Cambridge dictionary](https://dictionary.cambridge.org/dictionary/english/study) definition) Any project owner can create as many studies as necessary. Most of the Catalog data models, except for *User* and *Project* belong to a particular *Study,* so it can be seen as the central piece in OpenCGA Catalog. A *Study* contains, similarly to *Project*, a name and an alias (study identifier). Optionally, it can have a description as well.
### Groups in a Study
Despite the explanation of *Groups* from a previous section, *Groups* are actually defined within a *Study*. Different studies can have different groups of users that are basically defined by the *Study* owner or administrators. By default, every *Study* is created with two reserved groups (*admins* and *members*). The roles of these two groups is described in [Sharing and Permissions](http://docs.opencb.org/display/opencga/Sharing+and+Permissions) section.
### Variable Sets and Annotation Sets
One of the most outstanding features of OpenCGA Catalog is the ability to not only store any type of data in the database, but also filter and query by any of the values populated by the researchers. This can be achieved with what we have called *Variable Sets* and *Annotation Sets.*
A [*Variable Set*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/VariableSet.java) is a set of [*Variables*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/Variable.java), understanding as a *Variable* the complete definition of a field that need to be populated. In other words, a *Variable Set* could be seen as a template of a form that is given to the patient containing the points the patient should fill in. A *Variable Set* will look similar to the table shown above. That *Variable Set* is composed of four well described *Variables*:
| ***Variable*** | Sex | Categorical | Yes | MALE, FEMALE, UNKNOWN |
| -------------- | ----------- | ----------- | --- | --------------------- |
| ***Variable*** | Age | Integer | Yes | NA |
| ***Variable*** | Mother name | Text | Yes | NA |
| ***Variable*** | Affected | Boolean | Yes | NA |
Every *Study* can have as many different *Variable Set* definitions as necessary.
The values defined for each of the *Variables* are called [*Annotations*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/Annotation.java), and the population of a whole *Variable Set* is called [*Annotation Set*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/AnnotationSet.java)*.* This means that an *Annotation Set* only makes sense and is always related to one concrete *Variable Set.*
There are four *Annotable* data models: *Sample, Individual, Family* and *Cohort*. Each entry from these data models can have *Annotation Sets* as can be seen in the diagram in the right margin. An *Annotation Set* will look to something similar to:
| *Annotation* | Sex | MALE |
| ------------ | ----------- | ---- |
| *Annotation* | Age | 60 |
| *Annotation* | Mother name | Jane |
| *Annotation* | Affected | Yes |
OpenCGA allows querying by any of these key-value pairs.
### Files
OpenCGA Catalog keeps track of all the files and folders containing the relevant data. Every [*File*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/File.java) registry contains the physical path where the files/folders are stored in the file system (uri). Besides this, Catalog creates a virtual file structure so no matter what the real location of the files are, users can organise and work with those files differently. Everything related to *Files* can be found in the [File Management](http://docs.opencb.org/display/opencga/File+Management) section.
### Individuals and Families
We understand an [*Individual*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/Individual.java) as a subject (typically a person) for which some analysis will be made. A group of *Individuals* with any parental or blood relationship is called [*Family*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/Individual.java). Any of these two data models can have *Annotation Sets* defined.
### Samples and Cohorts
A [*Sample*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/Sample.java) is any biological material, normally extracted from an *Individual*, that is used for a particular analysis. [*Cohorts*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/Cohort.java) contains groups of samples sharing some particular conditions such as "healthy" vs "infected". Any of these two data models can also have *Annotation Sets* defined.
### Clinical Analysis
A [*Clinical Analysis*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/ClinicalAnalysis.java) contains all the information of the *Individuals* and *Samples* involved to perform a real clinical analysis. It also allows storing the interpretations derived from the results.
### Jobs
OpenCGA Catalog allows running different tools. This tools can be any of the ones built in OpenCGA, but also any external tool the user might need to use. Every time the user calls to a analysis web service to run anything, a new [*Job*](https://github.com/opencb/opencga/blob/develop/opencga-core/src/main/java/org/opencb/opencga/core/models/Job.java) is created. This jobs contain the essential information of the task that needs to be run. A daemon is in charge of checking whether there are any prepared, queued or running jobs and update the information.
{% hint style="info" %}
OpenCGA supports SGE (Sun Grid Engine) that accepts, schedules, dispatches, and manages the remote and distributed execution of large numbers of standalone or parallel jobs.
{% endhint %}
# Users and Groups
## Users in OpenCGA
Any study has intrinsically defined three different membership levels or user groups:
### Owner
The owner of an study is the user that creates the study. Because of that an study will always have one and only one owner, who will be able to perform any action over the data contained in the study. There are two actions that are only possible for the owner user: ***delete the study*** and ***assign or remove users to/from admins*** **groups** *(see next section).*
### Administrative groups
OpenCGA defines two reserved groups that will have some special behaviour.
#### Admins
Every *Study* in OpenCGA contains a special group called ***admins***. This group will contain a list of users that will be able to do most of the administrative work the owner might want other users to do. Users belonging to this group will be able to perform almost any action except for the two ones that are only allowed for the owner of the study. Special operations that only these users will be able to perform are ***create/update/delete groups of*** ***users***, ***create/update/delete variable sets*** and ***assign/remove permissions to other users/groups.***
#### Members
Apart from *admins,* there is also an special group called *members*. Any user with any kind of granted access to the study will automatically belong to this group. The main aim of this group is to keep track of the users with any access to the study, but it also has other advantages such as:
* The *admin* users might want to predefined some permissions any *member* of a study will have. In such a case, *admin* users will just add new users to that group and those users will automatically be granted the permissions the group has.
* If an *admin* user wants to completely revoke any permission to one user, by removing that user from the *members* group, OpenCGA will automatically search for any permissions set for that user in any entity and remove it.
### Decision Algorithm
The next schema provides a visual explanation of the algorithm implemented in Catalog for deciding whether the user has or not access to the data in the context of a study.
There are two circumstances under which the algorithm behaves as follows:
* If the user and any of the groups where the user belongs to have permissions defined for one entry, the permissions that will be actually used will be the user's.
* In case the user belongs to more than one group and those groups are assigned different permissions for one concrete entry, the effective permissions that will be used will be the union of the permissions found in all those groups.
# Study ACLs
{% hint style="info" %}
Remember that you can always check the Catalog code that implements OpenCGA's ACLs permission system in our official publicly available [GitHub repository](https://github.com/opencb/opencga/blob/9b00edc7b556898d6b65527a333ecdd62aea3791/opencga-core/src/main/java/org/opencb/opencga/core/models/study/StudyAclEntry.java).
{% endhint %}
Here you will find the list of permissions that can be granted to a **member** (user or group) in OpenCGA. Thanks to the **ACLs** system, permissions can be assigned to almost any entry level (except for *User* and *Project).*
## How it works
A list of the basic permissions and their explanations can be found in the list below:
* VIEW*: Give permission to access in read-only mode to the entry (study, file, sample...). WRITE: Give permission to create and update that kind of entries within the study. This do not include permissions to modify annotation and/or annotation sets. Those actions will need additional permissions.*
* *DELETE:* Give permission to delete that kind of entries.
* ANNOTATIONS: In Sample, Individual, Family and Cohort we have three additional permissions to deal with annotations.
Files deserve a special treatment as they not only exist in the database, but also physically in the file system. The special permissions added for files are the following:
*VIEW\_FILE\_HEADER*: Give permission to retrieve just the header of a file. DOWNLOAD\_FILES or *DOWNLOAD*: Give permission to download the whole file.
{% tabs %}
{% tab title="SAMPLES" %}
* **VIEW\_SAMPLES**
* **WRITE\_SAMPLES** *(implies: VIEW\_SAMPLES)*
* **DELETE\_SAMPLES** *(implies:VIEW\_SAMPLES, WRITE\_SAMPLES)*
* **VIEW\_SAMPLE\_ANNOTATIONS** *(implies: VIEW\_SAMPLES)*
* **WRITE\_SAMPLE\_ANNOTATIONS** *(implies: VIEW\_SAMPLES, VIEW\_SAMPLE\_ANNOTATIONS)*
* **DELETE\_SAMPLE\_ANNOTATIONS** *(implies: VIEW\_SAMPLES, VIEW\_SAMPLE\_ANNOTATIONS, WRITE\_SAMPLE\_ANNOTATIONS)*
* **VIEW\_AGGREGATED\_VARIANTS**
* **VIEW\_SAMPLE\_VARIANTS** *(implies: VIEW\_SAMPLES, VIEW\_SAMPLE\_ANNOTATIONS, VIEW\_AGGREGATED\_VARIANTS)*
{% endtab %}
{% tab title="INDIVIDUALS" %}
* **VIEW\_INDIVIDUALS**
* **WRITE\_INDIVIDUALS** *(implies: VIEW\_INDIVIDUALS)*
* **DELETE\_INDIVIDUALS** *(implies:VIEW\_INDIVIDUALS, WRITE\_INDIVIDUALS)*
* **VIEW\_INDIVIDUAL\_ANNOTATIONS** *(implies: VIEW\_INDIVIDUALS)*
* **WRITE\_INDIVIDUAL\_ANNOTATIONS** *(implies: VIEW\_INDIVIDUALS, VIEW\_INDIVIDUAL\_ANNOTATIONS)*
* **DELETE\_INDIVIDUAL\_ANNOTATIONS** *(implies: VIEW\_INDIVIDUALS, VIEW\_INDIVIDUAL\_ANNOTATIONS, WRITE\_INDIVIDUAL\_ANNOTATIONS)*
{% endtab %}
{% tab title="FILES" %}
* **VIEW\_FILES**
* **VIEW\_FILE\_HEADER** *(implies: VIEW\_FILES)*
* **VIEW\_FILE\_CONTENT** *(implies: VIEW\_FILES)*
* **WRITE\_FILES** (*implies: VIEW\_FILES*)
* **DELETE\_FILES** *\*\*(implies: VIEW\_FILES, WRITE\_FILES)*
* **DOWNLOAD\_FILES** *(implies: VIEW\_FILES)*
* **UPLOAD\_FILES** *\*\*(implies: WRITE\_FILES, VIEW\_FILES)*
* **VIEW\_FILE\_ANNOTATIONS** *(implies: VIEW\_FILES)*
* **WRITE\_FILE\_ANNOTATIONS** *(implies: VIEW\_FILE\_ANNOTATIONS, VIEW\_FILES)*
* **DELETE\_FILE\_ANNOTATIONS** *(implies: WRITE\_FILE\_ANNOTATIONS, VIEW\_FILE\_ANNOTATIONS, VIEW\_FILES)*
{% endtab %}
{% tab title="JOBS" %}
* **EXECUTE\_JOBS**
* **VIEW\_JOBS**
* **WRITE\_JOBS** *(implies: VIEW\_JOBS)*
* **DELETE\_JOBS** *(implies: VIEW\_JOBS, WRITE\_JOBS)*
{% endtab %}
{% tab title="FAMILIES" %}
* **VIEW\_FAMILIES**
* **WRITE\_FAMILIES** *(implies: VIEW\_FAMILIES)*
* **DELETE\_FAMILIES** *(implies: VIEW\_FAMILIES, WRITE\_FAMILIES)*
* **VIEW\_FAMILY\_ANNOTATIONS** *(implies: VIEW\_FAMILIES)*
* **WRITE\_FAMILY\_ANNOTATIONS** *(implies:VIEW\_FAMILIES, VIEW\_FAMILY\_ANNOTATIONS)*
* **DELETE\_FAMILY\_ANNOTATIONS** *(implies: VIEW\_FAMILIES, VIEW\_FAMILY\_ANNOTATIONS, WRITE\_FAMILY\_ANNOTATIONS)*
{% endtab %}
{% tab title="COHORTS" %}
* **VIEW\_COHORTS**
* **WRITE\_COHORTS** *(implies: VIEW\_COHORTS)*
* **DELETE\_COHORTS** *(implies: VIEW\_COHORTS, WRITE\_COHORTS)*
* **VIEW\_COHORT\_ANNOTATIONS** *(implies: VIEW\_COHORTS)*
* **WRITE\_COHORT\_ANNOTATIONS** *(implies: VIEW\_COHORTS, VIEW\_COHORT\_ANNOTATIONS)*
* **DELETE\_COHORT\_ANNOTATIONS** *(implies: VIEW\_COHORTS, VIEW\_COHORT\_ANNOTATIONS, WRITE\_COHORT\_ANNOTATIONS)*
{% endtab %}
{% tab title="PANELS" %}
* **VIEW\_PANELS**
* **WRITE\_PANELS** *(implies: VIEW\_PANELS)*
* **DELETE\_PANELS** *(implies: VIEW\_PANELS, WRITE\_PANELS)*
{% endtab %}
{% tab title="CLINICAL" %}
* **VIEW\_CLINICAL\_ANALYSIS**
* **WRITE\_CLINICAL\_ANALYSIS** *(implies: VIEW\_CLINICAL\_ANALYSIS)*
* **DELETE\_CLINICAL\_ANALYSIS** *(implies: VIEW\_CLINICAL\_ANALYSIS, WRITE\_CLINICAL\_ANALYSIS)*
{% endtab %}
{% endtabs %}
## Permissions Templates
OpenCGA Catalog implements two Permissions templates: predefined generic roles that capture a list of defined permissions. The permission templates can be granted to either users or groups.
* **analyst**: The member (user or group) will be given full READ and WRITE (not DELETE) permissions for all the entries related to the study. These users will be able to view and do modifications on all the data that is related to the study.
* **view\_only**: The member (user or group) will be given full READ permissions.
## Special cases
Permissions can be given to any concrete entity (file, sample, cohort...) to deny or grant access to just one concrete entry. This is always true except for a few exceptions in which we might propagate those same permissions to other entries:
### Files
File entry might be of type file or folder (directory). Permissions assigned in folders are propagated to all the children (files and folders) recursively.
### Individuals/Samples
Individuals are really strongly related with samples. So every time permissions are given to an individual, the same permissions can be applied to all the related samples if the user sets the 'propagate' field to True, and vice-versa.
### Give public access to non-existing users
Catalog has one special user for this purpose represented with *\** symbol. Anytime a user tries to fetch anything and no session id is provided, Catalog will treat that user as *\**. By default, only authorised users will have access to data. However, study managers can still define permissions for non-authenticated users assigning permissions to the "user" *\*.*
# Population of metadata
{% hint style="info" %}
We recommend to
{% endhint %}
# Data Versioning
# Adding Custom Metadata
# Alignment Engine
# BAM Index
# Alignment Read Query
# Working with Coverage
# Alignment Analysis
# Variant Storage Engine
# Variant Query
# Variant Aggregation Stats
# Variant Analysis
# Clinical Genomics
# Clinical Interpretation
# Clinical Analysis
# Using JupyterLab
WIP
# Administrator
# User Management
# Templates / Manifest
## Templates / Manifest
Since OpenCGA v2.1.0, OpenCGA users with administration roles have the possibility to work with templates. Templates are a set of files with a defined specification that allow the user to perform a series of different operations, related to the ingestion of metadata in OpenCGA, e.g: define the samples, individuals, permission groups, etc. For more information on how OpenCGA stores metadata in Catalog [here](/develop-2/components-1/catalog).
Templates are defined at a study level and could be provided in different formats accordingly to the user's needs. The file format and some common use cases are illustrated in the following section.
Remember that OpenCGA is highly configurable, and the use of templates constitutes a useful resource to reduce some common artifacts on the ingestion of metadata, but you can always use the OpenCGA clients ([Client Libraries](/develop-2/manual/using-opencga/client-libraries)), command line ([Command Line](/develop-2/manual/using-opencga/command-line)) or [REST Web Service API](/develop-2/manual/using-opencga/restful-web-service-api) to perform different operations in OpenCGA.
### How it Works
The templates define a way to easily ingest metadata into OpenCGA. You need different things:
* **Manfiest**: There is only one required file that you'd need to provide to use the template-related operations. This is a `json` OR `yml` file named `manifest.{json|yaml}` containing the specific configuration applied to the template. This file will define the root (i.e: the study where you will perform the operation). An example is provided below
```
manifest.yaml
```
* **Metadata and Clinical Data**: You might need to provide a file per entity, where entities corresponds to one of the different comprehensive data models supported by OpenCGA Catalog (**individuals, samples, files, families, cohorts, clinical\_analysis**). Each file will contain the entity-related information that you want to ingest into Catalog. For usability purposes two main specifications will be accepted. You can find the file structures accepted below:
{% hint style="warning" %}
NOTE: All the fields within each entity that are subjected to be manipulable by the users are clearly stated in the documentation of the entity data model. Please refer to [Data Models](/develop-2/manual/administrator/templates-manifest).
{% endhint %}
#### JSON/YAML Files
You might want to provide a single JSON or YAML file per entity. In the case of using JSON you should write one JSON per line, if YAML is used then you can just concat them separating by '---'.
The following entities are supported.
* For [Individual](/develop-2/manual/administrator/templates-manifest): `individuals.{json|yaml}`
* For [Sample](/develop-2/manual/administrator/templates-manifest): `samples.{json|yaml}`
* For [File](/develop-2/manual/administrator/templates-manifest): `files.{json|yaml}`
* For [Family](/develop-2/manual/administrator/templates-manifest): `families.{json|yaml}`
* For [Cohort](/develop-2/manual/administrator/templates-manifest): `cohorts.{json|yaml}`
* For [Clinical Analysis](/develop-2/manual/administrator/templates-manifest): `clinical.{json|yaml}`
#### TAB Text Files
You can load data for the entities using **TAB separated .txt files**. There are some simple construction rules that the user needs to follow for the ingestion to be successful:
* First line starting with # symbol containing the exact name of the corresponding data model
* The column name must correspond to the field reserved in the entity data model. Refer to each entity data model documentation for checking the accepted fields.
* The order of the columns is not relevant.
```
samples.txt
#id individualId somatic
s1 NA001 true
s2 NA001 true
s3 NA002 true
individuals.txt:
#id name sex status.name
NA001 uk1 female READY
NA002 uk2 male READY
individuals.phenotypes.txt
#Individualid id name source
NA001 HP:0002664 Neoplasm HPO
NA001 HP:0002664 Lung Adenocarcinoma HPO
individuals.samples.txt
#IndividualId id
NA001 s1
NA001 s2
NA002 s3
```
# Roadmap
In this section, you can find only the main top-level features planned for major releases. For a more detailed list, you can go to GitHub Issues at .
## OpenCGA 2.x Releases
{% hint style="info" %}
From OpenCGA version 2.0.0 we follow **time-based releases**, two minor releases a year will be scheduled in April and October.
{% endhint %}
### 2.1.0 (Apr 2021)
You can track GitHub issues at [GitHub Issues 2.1.0](https://github.com/opencb/opencga/issues?q=is%3Aopen+is%3Aissue+milestone%3Av2.1.0). You can follow the development at [GitHub Projects](https://github.com/opencb/opencga/projects).
#### General
* Implement a **Centralised Log** analytic solution, we are planning to use Kibana *\*\**
#### Catalog
* Implement a new **Action** system, Catalog will notify to a message queue *(RabbitMQ, Apache Kafka),* this will allow other applications to know what's going on
* Improve **RESTful** web services by adding standardise **error codes** to the response, this will improve debugging
**Variant Storage Engine**
* Extend consequence type and population frequency filter in the sample genotype index
* Improve sample genotype index for clinical and cancer by filtering by cosmic or VAF
* Allow the index of custom INFO or FORMAT fields
* Implement a new **Cache** functionality, some sample and family-based variant queries and analysis can take up to few seconds, since this data is read-only this could be easily cached
#### Clinical
#### Analysis Framework
#### Others
* Implement **FIHR Genomics** API, this will allow FIHR applications to query genomic variants in OpenCGA
### 2.0.0 (Oct 2020)
You can track GitHub issues at [GitHub Issues 2.0.0](https://github.com/opencb/opencga/issues?q=is%3Aopen+is%3Aissue+milestone%3Av2.0.0). You can follow the development at [GitHub Projects](https://github.com/opencb/opencga/projects).
#### General
* Improve **Docker** images, now stable versions with the different variant storage are pushed to Docker Hub
* Upgrade **dependencies**: MongoDB 4.2, Solr 8.1.1, JUnit 5.5.1, ...
* **Clean ups** and **remove** deprecated code and APIs
#### Catalog
* Add **ACID Transactions** to all database operations
* Improve **Audit**, extend audit data model and ensure all actions are now audited. Also, make audit *queryable*.
* Implement a new **Task** system, this will be used internally by OpenCGA to schedule some jobs, this new functionality can be also used by external applications
* Improve **RESTful** web services response and **warning/error** notifications
* Prepare OpenCGA for supporting **Federation** in next releases
* Improve **performance** and **test coverage**
#### Storage Engines
**Alignment**
* Support CRAM file
**Variant**
* Implement **structural variant imprecise** queries
* Implement new **Variant Score** to store results from analysis such as GWAS, this can be used when filtering
* Remove any **blocking variant operation**, any variant operation should be able to run at any time in a consistent way
* Improve **HBase sample index**, this will improve the **performance** of some **queries and** **analysis**
* Implement HBase-based **aggregations**
* Support new **HBase 2.0** version
* Improve **testing** and **benchmark** module
#### Analysis
**Framework**
* Develop an **Analysis Framework**, this will allow users to extend and customise OpenCGA with their own analysis
* Implement a **WrappedAnalysis** functionality in this framework to make easy to use any external tool such as Plink (see below in *Varlant Analysis* section)
**Variant**
* Implement on-demand **Variant Stats** and **Variant Sample Stats**
* Add GWAS **variant analysis**, this can optionally be stored and indexed in the new **Variant Score** object
* Add *Plink* as **wrapped analysis**
**Clinical Interpretation**
* Implement **Cancer Tiering** interpretation analysis algorithm
* Network-based clinical interpretation algorithm *(experimental)*
* Implement **Secondary Findings** analysis
#### Clinical
* Network-based clinical interpretation algorithm *(experimental)*
#### Cloud
* Full support for **Microsoft Azure and HDInsight 4.0,** this also includes **Azure AD, Azure Blob** and **Azure Batch**. We would like to **thank very much Microsoft Azure** for their amazing support and help here.
* Add **Kubernetes** for deployment and orchestration
**Note**: some of these features might be released in the Enterprise version coming soon
## OpenCGA 1.x Releases
### 1.4.0 (March 2019)
#### General
* Implement the new **HTSGET 1.0** protocol
* **IVA 0.9.0** will implement a full study and clinical analysis among many other features
* Add many more negative and variant **functional tests**
* **Documentation** improvements with new diagrams and tutorials
#### Catalog
* Complete and test all **delete** operations and implement *delete by queries* to make easier to delete batches of resources, with this the **REST API** can be considered complete
* Implement a new **admin** REST API, this will allow OpenCGA administrator to execute administrative tasks remotely
* New **PermissionRule** feature, you can define rules for assigning permissions automatically when new data is created, e.g. *set VIEW permission to USER to all samples where HOSPITAL = 'X'*
* New implementation of how **clinical data** (*annotation sets*) are store in the database, this new physical schema significantly improves querying annotations (even with nested objects or arrays), *group by* aggregations, *include/exclude* filtering and allow to *flatten* the annotations
* Complete ***ClinicalAnalysis and*** ***ClinicalInterpretation*** data models and functionality
* Add **DiseasePanel** entity to manage panels
#### Variant Storage
* Final **HBase variant storage** implementation. New architecture should scale to few million of genomes and billion of variants.
* Support the last pending structural variant: **Translocation**. With this all structural variants are properly represented and stored
* Improve **variant stats** and add **simple variant analysis** such as association or Hardy-Weinberg test, this will be stored and indexed in the new ***VariantScore*** object
* Add INDEL **left-alignment** normalisation to *VariantNormaliser*
* **Variant Benchmark suite** to study scalability and performance
* Add a native implementation of Genomics England Tiering analysis
### 1.3.0 (November 2017)
#### General
* CLI **autocompletion** implemented
* New single CLI for execute **migrations** automatically
* New and fully functional **R client library** for REST web services, with this the four client libraries are completed
* New **IVA 0.9.0** is developed coordinately to exploit all the new features, they will be released together
* Many more **functional tests** added to test all new functionality described below
* Review and improve **Swagger** documentation and descriptions
* **Documentation** improvements with new diagrams and tutorials
#### Catalog
* New ***Family*** data model finished, now it is production ready, this completes and integrates three related data models: *Sample, Individual* and *Family*
* New ***Versioning*** feature implemented for *Sample, Individual* and *Family*. Now you can track any change in those data models, users can query o review any *version* of those documents
* New ***Export*** functionality implemented, this allows to export a *Project* as it was at any specific release, this can then imported in a new OpenCGA server
* New Study administrative group called ***admins***, all users in this group will be granted some special permissions at Study level such as *create groups* or *share* data, this will make Study administration much easier
* New ***Confidential*** permission for Variable Sets, now you can make some clinical data private for some users
* New ***ClinicalAnalysis*** data model added, this allows to define and stored different clinical interpretation analysis, this is still experimental and it should not be used in production
* Improvements in ***Group By*** queries, now you can pass a ***count*** parameter and aggregations only use data you can view, this can be useful for summarising data. Also, this has been added to *Individual* and *Family*
* Ensure that all query **GET** REST web services accept **comma-separated list of IDs**, at the moment only few of them accept ID lists, this will reduce the number of REST calls needed improving the performance
* New REST web service to **execute remote scripts** for Catalog, for instance "*move samples from Study*"
* **Performance improvements** when checking permissions (ACL) in *create* and *update* methods, now on average 50% less database queries are needed
#### Variant Storage
* Improve support for **Structural Variants**, in this release we will fully support *Insertion, Deletion* and *Copy Number* variants
* New ***VariantMetadata*** implemented, this is *exported* together with the variant data to be further analysed with other OpenCB projects using Spark
* New ***VariantScore*** object added to Variant data model, this will allow to store variant scores from cohort-related analysis such as association or Hardy-Weinberg tests in the next release
* Implement some **HBase** physical schema improvements and a better integration with Solr
* Support ***Amazon EMR*** Hadoop cluster
* **Performance improvements** when querying variants from samples, this will have a big impact in clinical interpretation analysis
#### Alignment Storage
* Major improvements in **BAM query** engine. New **server-side** filters added, this is a more efficient implementation since the data sent through the network is reduced. The available filters now are: *region, minMapQ, maxNumberMismatches, maxNumberHits, properlyPaired, maxInsertSize, unmmapped* and *duplicated.*
* New **coverage** calculator using **BigWig**. Now coverage is calculated and stored in BigWig format, the *windowSize* is configurable. Also, coverage can now be queried for a *region* and optionally a *windowSize,* the server will **aggregate and compute the average** in *windowSizes.*
* New **REST** and **gRPC** APIs implementing the new query filters and coverage functionality. When using **REST** a JSON string is returned using GA4GH data model. When **gRPC** is used a binary stream is obtained. Note that in both protocols the filters are applied in the server.
## Unscheduled features
The following features have been accepted but no release version has been assigned:
* Add test for the CLI
* Support Slurm
* Add **Reactive Programming** (RxJava) and **Events**, this will allow to be easily integrated into other custom Java-based applications
* New **Gene Expression** database, this will include a Gene Annotation based on CellBase
You can find detailed information for some of them at
# Release Notes
You can find more detailed information at [GitHub Issues](https://github.com/opencb/opencga/issues).
## 2.0.0-RC1 (June 2020)
### Catalog
* \[**FEATURE**] Improve audit ([#1322](https://github.com/opencb/opencga/issues/1322), [#1483](https://github.com/opencb/opencga/issues/1483))
* \[**FEATURE**] Allow queries based on permissions ([#1486](https://github.com/opencb/opencga/issues/1486))
* \[**FEATURE**] Enable possibility of creating asynchronous tasks support ([#1408](https://github.com/opencb/opencga/issues/1408), [#1509](https://github.com/opencb/opencga/issues/1509))
* \[**FEATURE**] Support ACID transactions in Catalog !! ([#1338](https://github.com/opencb/opencga/issues/1338))
* \[**FEATURE**] Implement an automatic client generator valid for R, Java, Python and JS OpenCGA libraries ([#1464](https://github.com/opencb/opencga/issues/1464))
* \[FEATURE] Add new permission to execute jobs ([#1445](https://github.com/opencb/opencga/issues/1445))
* \[FEATURE] Improve job data model ([#1447](https://github.com/opencb/opencga/issues/1447))
* \[FEATURE] Enable job dependencies ([#1475](https://github.com/opencb/opencga/issues/1475))
* \[FEATURE] Support multistudy jobs ([#1549](https://github.com/opencb/opencga/issues/1549))
* \[FEATURE] Add webhooks ([#1448](https://github.com/opencb/opencga/issues/1448))
* \[FEATURE] Create jobs top webservice and command line ([#1460](https://github.com/opencb/opencga/issues/1460))
* \[FEATURE] Create new **opencga** administrator user with a default project and study ([#1425](https://github.com/opencb/opencga/issues/1425), [#1491](https://github.com/opencb/opencga/issues/1491))
* \[FEATURE] Improve count functionality ([#1448](https://github.com/opencb/opencga/issues/1448))
* \[FEATURE] Add a new webservice to download files from external sources ([#1453](https://github.com/opencb/opencga/issues/1453))
* \[FEATURE] Add dynamic parameter types in VariableSets ([#1478](https://github.com/opencb/opencga/issues/1478))
* \[FEATURE] Add new *allowedKey* field to VariableSet ([#1554](https://github.com/opencb/opencga/issues/1554))
* \[FEATURE] Add new /head and /tail web services in file ([#1497](https://github.com/opencb/opencga/issues/1497))
* \[FEATURE] Add new jobs/log/head and job/logs/tail web services to see job logs ([#1495](https://github.com/opencb/opencga/issues/1495))
* \[FEATURE] Load annotations from TSV files ([#1488](https://github.com/opencb/opencga/issues/1488))
* \[FEATURE] Allow passing a map of sample ids when linking VCF files ([#1527](https://github.com/opencb/opencga/issues/1527))
* \[FEATURE] Give users the option to set their own statuses ([#1545](https://github.com/opencb/opencga/issues/1545))
* \[FEATURE] Add new permission to view variants ([#1559](https://github.com/opencb/opencga/issues/1559))
* \[FEATURE] Assign implicit permissions automatically ([#1561](https://github.com/opencb/opencga/issues/1561))
* \[FEATURE] Remove base64 codification from UUIDs ([#1569](https://github.com/opencb/opencga/issues/1569))
* \[FEATURE] Improve Solr queries by annotation ([#1484](https://github.com/opencb/opencga/issues/1484))
* \[FEATURE] Add new webservice to fetch individual relatives ([#1552](https://github.com/opencb/opencga/issues/1552))
* \[FEATURE] Add new webservice to fetch base64 content of images ([#1584](https://github.com/opencb/opencga/issues/1584))
* \[FEATURE] Keep individual references in samples ([#1346](https://github.com/opencb/opencga/issues/1346))
* \[CHANGE-FEATURE] Improve panel operations ([#1577](https://github.com/opencb/opencga/issues/1577))
* \[CHANGE] Configuration file changes ([#1415](https://github.com/opencb/opencga/issues/1415))
* \[CHANGE] DataResponse data model changes ([#1424](https://github.com/opencb/opencga/issues/1424))
* \[CHANGE] Rename variable type in VariableSet ([#1479](https://github.com/opencb/opencga/issues/1479))
* \[CHANGE] Remove group name ([#1513](https://github.com/opencb/opencga/issues/1513))
* \[CHANGE] Rename a few ACLs ([#1601](https://github.com/opencb/opencga/issues/1601))
* \[CHANGE] Data model changes ([#1538](https://github.com/opencb/opencga/issues/1538))
* \[CHANGE] Main login endpoint changes ([#1568](https://github.com/opencb/opencga/issues/1568))
* \[CHANGE] Change webservice to change user password ([#1586](https://github.com/opencb/opencga/issues/1586))
* \[CHANGE] Ensure all REST webservices return on OpenCGAResult ([#1569](https://github.com/opencb/opencga/issues/1569))
* \[BUGFIX] Sync users CLI didn't work for AD ([#1297](https://github.com/opencb/opencga/issues/1297))
* \[BUGFIX] Unable to update "relatedFiles" list ([#1451](https://github.com/opencb/opencga/issues/1451))
* \[PERFORMANCE] Move deleted documents to different collection ([#1369](https://github.com/opencb/opencga/issues/1369))
### Analysis
* \[CLINICAL] Tiering interpretation analysis for cancer ([#1300](https://github.com/opencb/opencga/issues/1300))
* \[VARIANT] Implement Fisher Test Analysis MapReduce ([#1361](https://github.com/opencb/opencga/issues/1361))
* \[VARIANT] Implement VariantStats OpenCGA Analysis ([#1376](https://github.com/opencb/opencga/issues/1376))
* \[VARIANT] Implement Gwas OpenCGA Analysis ([#1386](https://github.com/opencb/opencga/issues/1386))
* \[VARIANT] Complex sample query by variant to enable cohort creation for clinical trials ([#1474](https://github.com/opencb/opencga/issues/1474))
* \[CLINICAL] Implement mutational signature analysis ([#1490](https://github.com/opencb/opencga/issues/1490))
* \[CORE] Create a Dockerfile with the R packages used by OpenCGA analysis ([#1493](https://github.com/opencb/opencga/issues/1493))
* \[CLINICAL] Implement the inferred sex analysis ([#1544](https://github.com/opencb/opencga/issues/1544))
* \[ALIGNMENT] Implement statistics analysis for alignment coverage ([#1588](https://github.com/opencb/opencga/issues/1588))
* \[CLINICAL] Implement the relatedness analysis based on IBD/IBS ([#1521](https://github.com/opencb/opencga/issues/1521))
* \[CLINICAL] Implement genetic checks to compare with the reported results ([#1522](https://github.com/opencb/opencga/issues/1522))
### Variant Storage
* \[**FEATURE**] Support Hadoop3.x and HBase2.x ([#925](https://github.com/opencb/opencga/issues/925))
* \[**FEATURE**] Divide opencga-storage-hadoop-deps in submodules ([#1333](https://github.com/opencb/opencga/issues/1333))
* \[FEATURE] Store custom variant scores ([#708](https://github.com/opencb/opencga/issues/708))
* \[FEATURE] Allow load VCFs split by region in Hadoop ([#1471](https://github.com/opencb/opencga/issues/1471))
* \[FEATURE] Copy MapReduce jobs result submitted through an ssh connection ([#1432](https://github.com/opencb/opencga/issues/1432))
* \[FEATURE] Add FilterCount and MeanQuality to VariantStats ([#1502](https://github.com/opencb/opencga/issues/1502))
* \[FEATURE] Allow to configure the variant storage from REST ([#1518](https://github.com/opencb/opencga/issues/1518))
* \[FEATURE] Allow skip sample index when loading variant files ([#1530](https://github.com/opencb/opencga/issues/1530))
* \[FEATURE] Allow indexing multiple files per sample in StorageHadoop ([#1542](https://github.com/opencb/opencga/issues/1542))
* \[FEATURE] Return MendelianError code as an IssueEntry ([#1547](https://github.com/opencb/opencga/issues/1547))
* \[FEATURE] Extend sample filter functionality ([#1567](https://github.com/opencb/opencga/issues/1567))
* \[CHANGE] Add specific permissions to view variants ([#1559](https://github.com/opencb/opencga/issues/1559))
* \[CHANGE] Rename VariantQueryParams 'format', 'includeFormat' and 'info' ([#1556](https://github.com/opencb/opencga/issues/1556))
* \[CHANGE] Remove SAMPLE\_ID and FILE\_IDX from Format. Add INCLUDE\_SAMPLE\_ID ([#1555](https://github.com/opencb/opencga/issues/1555))
* \[CHANGE] Transform endpoint analysis/variants/sample/query into an Analysis ([#1435](https://github.com/opencb/opencga/issues/1435))
* \[CHANGE] Improve variant storage functionality for returning samples ([#1353](https://github.com/opencb/opencga/issues/1353))
* \[CHANGE] Change variant.id and variant.names content ([#1514](https://github.com/opencb/opencga/issues/1514))
* \[CHANGE] Make gene/id/xref query params more strict ([#1515](https://github.com/opencb/opencga/issues/1515))
* \[PERFORMANCE] Add extended clinical index to SampleIndex ([#1454](https://github.com/opencb/opencga/issues/1454))
* \[PERFORMANCE] Add Biotype+Ct combination to SampleInde ([#1364](https://github.com/opencb/opencga/issues/1364))
* \[PERFORMANCE] Count numTotalResults from covered SampleIndex queries asyncrhonously. ([#1352](https://github.com/opencb/opencga/issues/1352))
* \[PERFORMANCE] Improve SampleIndex File ([#1343](https://github.com/opencb/opencga/issues/1343))
* \[BUGFIX] Remove Jetty9.4 from Hadoop MapReduce classpath ([#1504](https://github.com/opencb/opencga/issues/1504))
## 1.4.2 (June 2019)
### Catalog
* \[CHANGE] Rename web services from */stats* to */aggregationStats* ([#1253](https://github.com/opencb/opencga/issues/1253))
* \[BUGFIX] Index fails when passing ":" instead of "/" ([#1241](https://github.com/opencb/opencga/issues/1241))
* \[BUGFIX] Alignment index daemon fails ([#1232](https://github.com/opencb/opencga/issues/1232))
* \[BUGFIX] Migration script issue ([#1226](https://github.com/opencb/opencga/issues/1226))
* \[BUGFIX] *Upload* web service fails when uploading to root folder ([#1276](https://github.com/opencb/opencga/issues/1276))
* \[BUGFIX] Missing *id* query parameter in *studies/{studies}/groups* web service ([#1275](https://github.com/opencb/opencga/issues/1275))
* \[BUGFIX] Versioning issues ([#1270](https://github.com/opencb/opencga/issues/1270))
* \[BUGFIX] Web service *studies/{studies}/groups/create* not working ([#1250](https://github.com/opencb/opencga/issues/1250))
* \[BUGFIX] Order not respected when querying lists of ids ([#1246](https://github.com/opencb/opencga/issues/1246))
* \[FEATURE] Support passing relatedFiles object to File during *link* ([#1295](https://github.com/opencb/opencga/issues/1295))
* \[FEATURE] Support *htsget* protocol for data streaming ([#1277](https://github.com/opencb/opencga/issues/1277))
* \[FEATURE] Add new user category "*application"* ([#1268](https://github.com/opencb/opencga/issues/1268))
* \[PERFORMANCE] Analysis queries take too much time ([#1245](https://github.com/opencb/opencga/issues/1245))
|
1.4.2. contains a small series of internal changes requiring running a small migration. To do so, please head to the main OpenCGA source folder and run the following:
mongo\_port: Typically 27017
database\_name: Typically opencga\_catalog
|
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
### Variant Storage
* \[CHANGE] Rename filter "`transcriptionFlag`" to "`transcriptFlag`" ([#1256](https://github.com/opencb/opencga/issues/1256))
* \[BUGFIX] HashMap$Node cannot be cast to java.util.HashMap$TreeNode ([#1323](https://github.com/opencb/opencga/issues/1323))
* \[BUGFIX] Wrong usage of SampleIndex on invalid queries ([#1274](https://github.com/opencb/opencga/issues/1274))
* \[BUGFIX] Fix RowKey generation for non symbolic structural variants #1259
* \[FEATURE] Add chromDensity query filtering by sample genotype ([#1249](https://github.com/opencb/opencga/issues/1249))
* \[FEATURE] Accept nested fields in chromDensity by sample genotype ([#1263](https://github.com/opencb/opencga/issues/1263))
* \[FEATURE] Use SamplingSize field at SampleIndex query executor ([#1339](https://github.com/opencb/opencga/issues/1339))
* \[FEATURE] Add new variant filters: "`cohortStatsAlt`", "`cohortStatsRef`" ([#1239](https://github.com/opencb/opencga/issues/1239))
* \[FEATURE] Query by compound heterozygous ([#1247](https://github.com/opencb/opencga/issues/1247))
* \[FEATURE] Export variants given a file of variant Ids ([#1254](https://github.com/opencb/opencga/issues/1254))
* \[FEATURE] Accept keyword "LoF" in consequence type filter ([#1262](https://github.com/opencb/opencga/issues/1262))
* \[FEATURE] Implement a MR to extract number of variants per file #1287
* \[PERFORMANCE] Remove unneeded FileMetadata reads from HBaseToStudyEntryConverter ([#1350](https://github.com/opencb/opencga/issues/1350))
* \[PERFORMANCE] Improve SampleIndex read performance ([#1319](https://github.com/opencb/opencga/issues/1319))
* \[PERFORMANCE] Add annotation counters to SampleIndex ([#1258](https://github.com/opencb/opencga/issues/1258))
* \[PERFORMANCE] Reduce size of SampleIndex table ([#1252](https://github.com/opencb/opencga/issues/1252))
* \[PERFORMANCE] Include parents genotype in the SampleIndex ([#1244](https://github.com/opencb/opencga/issues/1244))
* \[PERFORMANCE] Skip join SampleIndex with Variants table when possible #1242
* \[IMPROVEMENT] Improve approximate count of CompoundHeterozygous in StorageHadoop #1299
* \[IMPROVEMENT] Improve DeNovo variants definition ([#1340](https://github.com/opencb/opencga/issues/1340))
* \[IMPROVEMENT] Improve Genotype filter to include by default phased genotypes ([#1273](https://github.com/opencb/opencga/issues/1273))
* \[IMPROVEMENT] Make /sampleData support somatic studies ([#1329](https://github.com/opencb/opencga/issues/1329))
* \[IMPROVEMENT] Add VarinatStats to SampleData result ([#1286](https://github.com/opencb/opencga/issues/1286))
* \[IMPROVEMENT] Native implementation of sampleData endpoint in storage-hadoop ([#1285](https://github.com/opencb/opencga/issues/1285))
* \[IMPROVEMENT] Extend panel filter to include regions and single variants #1272
* \[IMPROVEMENT] Extract variant query executor from VariantStorageEngine #1240
* \[COMMAND LINE] Add missing SampleIndex command line operations ([#1282](https://github.com/opencb/opencga/issues/1282))
## 1.4.0-rc1 (August 2018)
### **Catalog**
* \[IMPROVEMENT] Recognise bigwig files automatically. ([#283](https://github.com/opencb/opencga/issues#283))
* \[IMPROVEMENT] **Major improvements in annotationSets** ([#635](https://github.com/opencb/opencga/issues#635), [#772](https://github.com/opencb/opencga/issues#772), [#849](https://github.com/opencb/opencga/issues#849)):
* Queries can be performed by any of the fields. ([#772](https://github.com/opencb/opencga/issues#772))
* All the fields are indexed in the database, supporting really fast queries. ([#772](https://github.com/opencb/opencga/issues#772))
* Projections can be added so only specific fields of an annotationSet are included. ([#635](https://github.com/opencb/opencga/issues#635), [#772](https://github.com/opencb/opencga/issues#772))
* Annotation set web services have been changed. ([#849](https://github.com/opencb/opencga/issues#849))
* See [AnnotationSets 1.4.0](http://docs.opencb.org/display/opencga/AnnotationSets+1.4.0) section from the OpenCGA documentation for the whole description of features.
* \[FEATURE] Implement new "Permission rules" feature ([#745](https://github.com/opencb/opencga/issues#745))
* \[FEATURE] Add new /admin web services to be able to perform administrative operations through REST ([#759](https://github.com/opencb/opencga/issues#759))
* \[IMPROVEMENT] Improve /user web services. Remove duplicated *configs* web services ([#661](https://github.com/opencb/opencga/issues#661))
* \[FEATURE] Implement *delete* operations for most of the entries ([#792](https://github.com/opencb/opencga/issues#792))
* \[IMPROVEMENT] Store *creationDate* field as an actual date object to improve queries ([#752](https://github.com/opencb/opencga/issues#752))
* \[CHANGE] Important *id* changes ([#819](https://github.com/opencb/opencga/issues#819)):
* Data model *ids* have been redefined to contain any user-defined *id* for every entry. Migrations produced in every entity are described in the ticket.
* Old numeric *ids* are now lost and only used for internal purposes. Users cannot query by those fields anymore.
* Update REST web services to support *id* changes ([#834](https://github.com/opencb/opencga/issues#834))
* \[IMPROVEMENT] Stop creating empty directories when linking *Files* ([#865](https://github.com/opencb/opencga/issues#865))
* \[FEATURE] Add new *{entry}Action* field to some update web services to support adding, setting or removing single entries to arrays of them. (Example: *samplesAction* to add, set or remove samples from an individual, file or cohort) ([#850](https://github.com/opencb/opencga/issues#850))
* \[IMPROVEMENT] Deprecate *VariableSet* web services and move them to *Study* web services. Improve and clean old *Study* web services ([#846](https://github.com/opencb/opencga/issues#846))
* \[IMPROVEMENT] General data model improvements described in [#823](https://github.com/opencb/opencga/issues#823):
* Add new inmutable ***uuid*** field to all entries.
* \[FEATURE] Create catalog solr sync mechanism and enable facet queries [#875](https://github.com/opencb/opencga/issues#875)
### Variant Storage
* \[FEATURE] Aggregate operation for all samples from a study in the Variant ([#757](https://github.com/opencb/opencga/issues/757))
* \[IMPROVEMENT] Move HBase write step to a separated MR for fill-missing operation ([#815](https://github.com/opencb/opencga/issues/815))
* \[IMPROVEMENT] Mark already processed variants in fill missing operation ([#803](https://github.com/opencb/opencga/issues/803))
* \[IMPROVEMENT] Do not iterate over all VcfRecords when filling gaps or missing variants ([#794](https://github.com/opencb/opencga/issues/794))
* \[IMPROVEMENT] Fill missing prepare step: copy variants to fill into archive table ([#793](https://github.com/opencb/opencga/issues/793))
* \[IMPROVEMENT] Split VcfSlice from archive table in REF and NON\_REF columns ([#778](https://github.com/opencb/opencga/issues/778))
* \[IMPROVEMENT] Split archive table in file batches ([#777](https://github.com/opencb/opencga/issues/777))
* \[CHANGE] Remove merge=advanced from storage-hadoop ([#796](https://github.com/opencb/opencga/issues/796))
* \[FEATURE] Index Sample Genotypes in HBase ([#838](https://github.com/opencb/opencga/issues/838))
* \[FEATURE] Genotype index intersect in HBase ([#862](https://github.com/opencb/opencga/issues/862))
* \[FEATURE] Integrate Variants export using MapReduce ([#867](https://github.com/opencb/opencga/issues/867))
* \[FEATURE] Support enriched genotypes when querying to SampleIndex ([#870](https://github.com/opencb/opencga/issues/870))
* \[IMPROVEMENT] Use variants SampleIndex when reading from MapReduce ([#868](https://github.com/opencb/opencga/issues/868))
* \[FEATURE] New Variant query filter "clinicalSignificance" ([#872](https://github.com/opencb/opencga/issues/872))
* \[FEATURE] Add new filters for INFO and FORMAT fields other than GT ([#863](https://github.com/opencb/opencga/issues/863))
* \[FEATURE] Add filter by QUAL in Storage Variants ([#809](https://github.com/opencb/opencga/issues/809))
* \[FEATURE] Accept enriched genotypes at GENOTYPE filter ([#750](https://github.com/opencb/opencga/issues/750))
* \[FEATURE] Variant filtering in a samples group ([#578](https://github.com/opencb/opencga/issues/578))
* \[CHANGE] Automatic INCLUDE\_STUDY if possible ([#878](https://github.com/opencb/opencga/issues/878))
* \[CHANGE] Automatic INCLUDE\_SAMPLE if GENOTYPE VariantQueryParam is present ([#814](https://github.com/opencb/opencga/issues/814))
* \[IMPROVEMENT] Match Consequence type with the specified Gene filter (if any) for Storage Hadoop ([#874](https://github.com/opencb/opencga/issues/874))
* \[CHANGE] Untie internal Catalog and Storage IDs ([#859](https://github.com/opencb/opencga/issues/859))
* \[FIX] Synchronize Catalog ID to UID changes in Storage Managers catalog ([#861](https://github.com/opencb/opencga/issues/861))
* \[FEATURE] Create a ProjectMetadata object for Variant Storage ([#832](https://github.com/opencb/opencga/issues/832))
* \[IMPROVEMENT] Move storage-hadoop metadata to a separated table ([#781](https://github.com/opencb/opencga/issues/781))
* \[FEATURE] Support and load BREAKENDS in Variant Storage ([#760](https://github.com/opencb/opencga/issues/760))
* \[FEATURE] Support of Symbolic and Structural variants in Hadoop Variant Storage ([#857](https://github.com/opencb/opencga/issues/857))
* \[IMPROVEMENT] Add direct loader for the first file in a chromosome to load directly to the variants collection ([#354](https://github.com/opencb/opencga/issues/354))
* \[FEATURE] Move removed variants into a trash bin ([#831](https://github.com/opencb/opencga/issues/831))
* \[CHANGE] By default, load all FORMAT fields in Variant Storage MongoDB ([#824](https://github.com/opencb/opencga/issues/824))
* \[CHANGE] Change storage-hadoop table naming policy ([#782](https://github.com/opencb/opencga/issues/782))
* \[IMPROVEMENT] Review and improve Variant Annotation in OpenCGA Storage ([#805](https://github.com/opencb/opencga/issues/805))
* \[IMPROVEMENT] Create one Stage Collection per study in Variants Storage MongoDB ([#801](https://github.com/opencb/opencga/issues/801))
* \[IMPROVEMENT] Filter out overlapping files in variants in storage-hadoop ([#779](https://github.com/opencb/opencga/issues/779))
* \[IMPROVEMENT] Improve remove file in variant-storage-hadoop ([#776](https://github.com/opencb/opencga/issues/776))
* \[IMPROVEMENT] Load annotation using BufferedMutator ([#775](https://github.com/opencb/opencga/issues/775))
* \[FIX] Do not store phoenix primary key columns in HBase as separated columns ([#802](https://github.com/opencb/opencga/issues/802))
* \[FIX] Adapt remove file to new archive schema ([#800](https://github.com/opencb/opencga/issues/800))
* \[FIX] Do not map namespace for VIEW tables under phoenix 4.12.0 ([#799](https://github.com/opencb/opencga/issues/799))
* \[FIX] Avoid PhoenixIOException timeout when dropping columns from phoenix ([#795](https://github.com/opencb/opencga/issues/795))
## 1.3.11 (August 2019)
### **Catalog**
* \[FEATURE] Support "application users" ([#1268](https://github.com/opencb/opencga/issues/1268)) (**migration required**) →
* \[FEATURE] Add CRAM support ([#1301](https://github.com/opencb/opencga/issues/1301))
* \[IMPROVEMENT] Change long fileId to File file in RelatedFile data model ([#1294](https://github.com/opencb/opencga/issues/1294)) (**migration required**) →
* \[IMPROVEMENT] RelatedFiles can be passed during file link ([#1295](https://github.com/opencb/opencga/issues/1295))
* \[FIX] Fix --sync-all option from opencga-admin.sh users sync command line ([#1297](https://github.com/opencb/opencga/issues/1297))
## 1.3.10 (February 2019)
### **Catalog**
* \[FIX] Remove base64 conversion of the secret key.
* \[FIX] Update pom dependencies to avoid conflicts.
## 1.3.9 (January 2019)
### **Catalog**
* \[FEATURE] Support Azure AD authentication.
* \[CHANGE] Add *id* to Group data model (**migration required**) →
## 1.3.8 (August 2018)
### **Catalog**
* \[FIX] Fix permission issue affecting users and groups with the "\_" symbol ([#881](https://github.com/opencb/opencga/issues#881))
## 1.3.7 (July 2018)
### **Catalog**
* \[ENHANCEMENT] Add new *tags* field to the File data model ([#855](https://github.com/opencb/opencga/issues#855))
* \[CHANGE] Configuration change. Add hooks to configuration file ([#856](https://github.com/opencb/opencga/issues#856))
## 1.3.6 (May 2018)
### **Catalog**
* \[FIX] Improve performance of sample queries filtering by individual ([#843](https://github.com/opencb/opencga/issues#843))
## 1.3.5 (May 2018)
### **Catalog**
* \[FIX] Fix issue when assigning permissions given the id(s) of different entities ([#836](https://github.com/opencb/opencga/issues#836))
## 1.3.4 (April 2018)
### **Catalog**
* \[ENHANCEMENT] Performance improvement when assigning permissions ([#829](https://github.com/opencb/opencga/issues#829))
## 1.3.3 (March 2018)
### **Catalog**
* \[ENHANCEMENT] Remove old deprecated fields from Family data model ([#810](https://github.com/opencb/opencga/issues#810))
* \[ENHANCEMENT] Allow looking for Individuals and Families by a new sample field ([#811](https://github.com/opencb/opencga/issues#811))
* \[FIX] Fix command line to support non-expiring tokens
* \[FIX] Support SKIP\_COUNT parameter
* \[CHANGE] Remove family completeness check
## 1.3.2 (February 2018)
### **Catalog**
* \[FIX] Add missing individual-sample indexes ([#790](https://github.com/opencb/opencga/issues#790))
* \[FIX] Fix smart name resolution ([#791](https://github.com/opencb/opencga/issues#791))
## 1.3.1 (February 2018)
### **Catalog**
* \[FIX] Propagation of permissions sample-individual not working ([#780](https://github.com/opencb/opencga/issues#780))
* \[ENHANCEMENT] Ask for admin password automatically when using admin command line ([#785](https://github.com/opencb/opencga/issues#785))
* \[FIX] Filtering by file size not working ([#786](https://github.com/opencb/opencga/issues#786))
## 1.3.0 (January 2018)
### General
* \[FEATURE] Admin migration command line ([#690](https://github.com/opencb/opencga/issues#690))
* \[FEATURE] Implement AutoComplete for CLI ([#714](https://github.com/opencb/opencga/issues#714))
### **Catalog**
* \[REMOVE] Remove ACL from data models. ([#666](https://github.com/opencb/opencga/issues/666))
* \[ENHANCEMENT] Clean old code and refactoring. ([#667](https://github.com/opencb/opencga/issues/667), [#668](https://github.com/opencb/opencga/issues/668), [#669](https://github.com/opencb/opencga/issues/669), [#670](https://github.com/opencb/opencga/issues/670))
* \[ENHANCEMENT] Remove some hidden and deprecated methods from webservices and command line ([#672](https://github.com/opencb/opencga/issues/672))
* \[CHANGE] Changes to family data model ([#677](https://github.com/opencb/opencga/issues/677))
* \[FEATURE] Add new analysis tool webservices ([#679](https://github.com/opencb/opencga/issues/679))
* **\[FEATURE]** Add version support for *Sample*, *Individual* and *Family* ([#684](https://github.com/opencb/opencga/issues/684))
* \[ENHANCEMENT] Clean and remove unnecessary dependencies for the client module ([#687](https://github.com/opencb/opencga/issues/687))
* \[ENHANCEMENT] Change some fields from *Clinical Analysis* (BETA) data model ([#688](https://github.com/opencb/opencga/issues/688), [#702](https://github.com/opencb/opencga/issues/702))
* \[ENHANCEMENT] Remove unnecessary *Relatives* data model ([#693](https://github.com/opencb/opencga/issues/693))
* \[ENHANCEMENT] Improve some individual webservices to better support the *Individual-Sample* relation ([#701](https://github.com/opencb/opencga/issues/701))
* \[CHANGE] Change *individual* parameter in the sample/create webservice ([#703](https://github.com/opencb/opencga/issues/703))
* \[CHANGE] Internal modification regarding the way the *Sample-Individual* relation was stored ([#706](https://github.com/opencb/opencga/issues/706))
* \[ENHANCEMENT] Add new *admins* group in studies ([#711](https://github.com/opencb/opencga/issues/711))
* \[ENHANCEMENT] Add new *stats* field to *Sample* data model ([#717](https://github.com/opencb/opencga/issues/717))
* \[CHANGE] Rename *ontologyTerms* field in *Sample, Individual* and *diseases* field in *Family* for *phenotypes* ([#718](https://github.com/opencb/opencga/issues/718))
* **\[FEATURE]** Add new option to export and import data from/to catalog ([#720](https://github.com/opencb/opencga/issues/720))
* \[ENHANCEMENT] Improve *groupby* webservices ([#721](https://github.com/opencb/opencga/issues/721))
* \[ENHANCEMENT] Support a list of ids in all GET webservices ([#727](https://github.com/opencb/opencga/issues/727))
* \[CHANGE] Internal modification: Change ACL delimiter used ([#740](https://github.com/opencb/opencga/issues/740))
### Variant Storage
* \[FEATURE] Make use of the new VariantMetadata model from Biodata ([#673](https://github.com/opencb/opencga/issues#673))
* \[FEATURE] Major support of Symbolic variants in Variants Storage ([#695](https://github.com/opencb/opencga/issues#695))
* \[FEATURE] Create profiles to select hadoop flavour ([s#707](https://github.com/opencb/opencga/issues#707))
* \[FEATURE] Store info fields on storage-hadoop improvement storage ([#704](https://github.com/opencb/opencga/issues#704))
* \[FEATURE] Allow loading multiple variant files from the same sample with non overlapping variants ([#696](https://github.com/opencb/opencga/issues#696))
* \[FEATURE] New optional pipeline step "fill-gaps" ([#713](https://github.com/opencb/opencga/issues#713))
* \[CHANGE] Rename some Variant REST query parameters ([#751](https://github.com/opencb/opencga/issues#751))
* \[CHANGE] GO and EXPRESSION filter must be combined as an AND with other region ([#694](https://github.com/opencb/opencga/issues#694))
* \[CHANGE] Update CellBase to v4.5.3 improvement ([#770](https://github.com/opencb/opencga/issues#770))
* \[ENHANCEMENT] Speed up GENOTYPE filter with FILES filter, when possible ([#675](https://github.com/opencb/opencga/issues#675))
* \[ENHANCEMENT] Add field "source" to VariantQueryResult ([#758](https://github.com/opencb/opencga/issues#758))
* \[ENHANCEMENT] Indicate if "numTotalResults" is an approximated count in VariantQueryResult ([#749](https://github.com/opencb/opencga/issues#749))
* \[FIX] Inconsistent configuration param to select variant annotator ([#747](https://github.com/opencb/opencga/issues#747))
* \[FIX] Duplicate Key Warn/Error in Stage Collection ([#766](https://github.com/opencb/opencga/issues#766))
* \[FIX] IllegalArgumentException when CellBaseRestVariantAnnotator skips a variant ([#746](https://github.com/opencb/opencga/issues#746))
* \[FIX] Concurrent table modification error when loading in Hadoop with merge=basic ([#709)](https://github.com/opencb/opencga/issues#709)
## 1.2.0 (September 2017)
### **Catalog**
* \[FIX] Fix job search by input and output files. ([#533](https://github.com/opencb/opencga/issues/533))
* \[ENHANCEMENT] Hide deprecated webservices. ([#599](https://github.com/opencb/opencga/issues/599))
* \[ENHANCEMENT] Set *ontologyTerms* array during Sample/Individual /create and /update webservices. ([#613](https://github.com/opencb/opencga/issues/613))
* \[FEATURE] Add the concept of release in OpenCGA. ([#616](https://github.com/opencb/opencga/issues/616))
* \[FIX] Fix behaviour where individual ids are not recognized when creating new family. ([#617](https://github.com/opencb/opencga/issues/617))
* \[FEATURE] Implement JWT based session management ([#618](https://github.com/opencb/opencga/issues/618))
* \[ENHANCEMENT] Change List\ for List\