Genomics England Research
Background
One of the goals of The 100,000 Genomes Project from Genomics England is to enable new medical research. Researchers will study how best to use genomics in healthcare and how best to interpret the data to help patients. The causes, diagnosis and treatment of disease will also be investigated. This is currently the largest national sequencing project of its kind in the world.
To achieve this goal Genomics England set up a Research environment for researchers and clinicians. OpenCGA, CellBase and IVA from OpenCB were installed as data platform. We loaded 64,078 whole genomes in OpenCGA, in total about 1 billion unique variants were loaded and indexed in OpenCGA Variant Storage, and all the metadata and clinical data for samples and patients were loaded in OpenCGA Catalog. OpenCGA was able to load and index about 6,000 samples a day, executing the variant annotation and computing different cohort stats for the all the data run in less than a week. In summary, all data was loaded, indexed, annotated and stats calculated in less than 2 weeks. Genomic variants were annotated using CellBase and the IVA front-end was installed for researchers and clinicians to analyse and visualise the data. In this document you can find a full report of about the loading and analysis of the 64,078 genomes.
Genomic and Clinical Data
Clinical data and genomic variants of 64,078 genomes were loaded and indexed in OpenCGA. In total we loaded more than 30,000 VCF files accounting for about 40TB of compressed disk space. Data was organised in four different datasets depending on the genome assembly (GRCh37 or GRCh38) and the type of study (germline or somatic), and this was organised in OpenCGA in three different Projects and four Studies:
Project
Study ID and Name
Samples
VCF Files
VCF File Type
Samples/File
Variants
GRCh37 Germline
RD37
Rare Disease GRCh37
12,142
5,329
Multi sample
2.28
298,763,059
GRCh38 Germline
RD38
Rare Disease GRCh38
33,180
16,591
Multi sample
2.00
437,740,498
GRCh38 Germline
CG38
Cancer Germline GRCh38
9,167
9,167
Single sample
1.00
286,136,051
GRCh38 Somatic
CS38
Cancer Somatic GRCh38
9,589
9,589
Somatic
1.00
398,402,166
Total
64,078
40,676
1,421,041,774
OpenCGA Catalog stores all the metadata and clinical data of files, samples, individuals and cohorts. Rare Disease studies also include pedigree metadata by defining families. Also, a Clinical Analysis were defined for each family. Several Variable Sets have been defined to store GEL custom data for all these entities.
Platform
For the Research environment we have used OpenCGA v1.4 using the new Hadoop Variant Storage that use Apache HBase as back-end because of the huge amount of data and analysis needed. We have also used CellBase v4.6 for the variant annotation. Finally we set up a IVA v1.0 web-based variant analysis tool.
The Hadoop cluster consists of about 30 nodes running Hortonworks HDP 2.6.5 (which comes with HBase 1.1.2) and a LSF queue for loading all the VCF files, see this table for more detail:
Node
Nodes
Cores
Memory (GB)
Storage (TB)
Hadoop Master
5
28
216
7.2 (6x1.2)
Hadoop Worker
30
28
216
7.2 (6x1.2)
LSF Loading Queue
10
12
364
Isilon storage
Genomic Data Load
In order to improve the loading performance, we set up a small LSF queue of ten computing nodes. This configuration allowed us to load multiple files at the same time. We configured LSF to load up to 6 VCF files per node resulting in 60 files being loaded in HBase in parallel without any incidence, by doing this we observed a 50x in loading throughput. This resulted in an average of 125 VCF files loaded per hour in studies RD37 and RD38, which is about 2 files per minute. In the study CG38 the performance was 240 VCF files per hour or about 4 files per minute.
Rare Disease Loading Performance
The files from Rare Disease studies (RD38 & RD37) contain 2 samples per file on average. This results in larger files, increasing the loading time compared with single-sample files. As mentioned above the loading performance was about 125 files per hour or 3,000 files per day. In terms of number of samples it is about 250 samples per hour or 6,000 samples a day.
The loading performance always depend on the number of variants and concurrent files being loaded, the performance was quite stable during the load and performance degradation was observed as can be seen here:
Concurrent files loaded
60
Average files loaded per hour
125.72
Load time per file
00:28:38
Saturation Study
As part of the data loading process we decided to study the number of unique variants added in each batch of 500 samples. We generated this saturation plot for RD38:
Cancer Loading Performance
The files from Cancer Germline studies (CG38) contain one sample per file. Compared with the Rare Disease, these files are smaller in size, therefore, as expected the file load was almost 2x faster. As mentioned above, the loading performance was about 240 genomes per hour or 5,800 files per day. In terms of number of samples it is about 5,800 samples a day, which is consistent with Rare Disease performance.
Concurrent files loaded
60
Average files loaded per hour
242.05
Load time per file
00:14:52
Analysis Benchmark
In this section you can find information about the performance of main variant storage operations and most common queries and clinical analysis. Please, for data loading performance information go to section Genomic Data Load above.
Variant Storage Operations
Variant Storage operations take care of preparing the data for executing queries and analysis. There are two main operations: Variant Annotation and Cohort Stats Calculation.
Variant Annotation
This operation uses the CellBase to annotate each unique variant in the database, this annotation include consequence types, population frequencies, conservation scores clinical info, ... and will be typically used for variant queries and clinical analysis. Variant annotation of the 585 million unique variants of project GRCh38 Germline took about 3 days, about 200 million variants were annotated per day.
Cohort Stats Calculation
Cohort Stats are used for filtering variants in a similar way as the population frequencies. A set of cohorts were defined in each study.
ALL with all samples in the study
PARENTS with all parents in the study (only for Rare Disease studies)
UNAFF_PARENTS with all unaffected parents in the study (only for Rare Disease studies)
Pre-computing stats for different cohort and ten of thousands of samples is a high-performance operation that run in less than 2 hours for each study.
Query and Aggregation Stats
To study the performance we used RD38 which the largest study with 438 million variants and 33,000 samples. We first run some queries to the aggregated data filtering by variant annotation and cohort stats. We were interested in the different index performance so we limit the results to be returned the first 10 variants excluding the genotypic data of the 33,000 samples, by doing this we remove the effect of reading from disk or transferring data through the network which is very variable across different clusters. For queries using patient data go to the next section. Here you can find some of the common queries executed.
Filters
Results
Total Results
Time (sec)
consequence type = LoF + missense_variant
10
3704626
0.189
consequence type = LoF + missense_variant
biotype = protein_coding
10
3576472
0.260
panel = with 200 genes
10
3882902
0.299
gene = BMPR2
10
37244
0.344
gene = BMPR2
consequence type = LoF
10
189
0.443
type = INDEL
10
79597426
0.802
type = INDEL
biotype = protein_coding
10
38799454
0.358
type = INDEL
biotype = protein_coding
consequence type = LoF + missense_variant
10
240443
0.556
consequence type = LoF + missense_variant
population frequency = 1000G ALL < 0.005
10
3157533
5.96
As can be observed most queries run below 1 second, you can combine as many filters as wanted.
Clinical Analysis
We also use here RD38 which is the largest study. Clinical queries, or sample queries, enforces queries to return variants of a specific set of samples. These queries can use all the filters from the general queries. The result here also includes a pathogenic prediction for each variant, which determines possible conditions associated to the variant.
Filters
Results
Total Results
Time (sec)
mode of inheritance = recessive
filter = PASS
10
211787
0.420
mode of inheritance = recessive
consequence type = LoF + missense_variant
10
710
1.95
mode of inheritance = recessive
consequence type = LoF + missense_variant
filter = PASS
10
656
1.92
mode of inheritance = recessive
consequence type = LoF + missense_variant
filter = PASS
panel = with 58 genes
10
7
2.11
de novo Analysis
filter = PASS
consequence type = LoF + missense_variant
24
24
0.680
Compound Heterozygous Analysis
filter = PASS
biotype = protein_coding
consequence type = LoF + missense_variant
417
417
10.930
As it can be observed most of the family clinical analysis run in less than 2 seconds in the largest study with 33,000 samples.
User Interfaces
Several user interfaces have been developed to query and analyse data from OpenCGA: IVA web-based tool, Python and R clients, and a command line.
IVA
IVA v1.0.3 was installed to provide a friendly web-based analysis tool to browse variants and execute clinical analysis.
Command line
You can also query variants efficiently using the command line built in. Performance depends on the number of samples fetched and the RPC used (REST or gRPC), in the best scenario you can fetch few thousands variants per second. You can see a simple example here producing a VCF file:
Support
OpenCB team is setting up Zetta Genomics, a start-up to offer support, consultancy and custom feature development. We have partnered with Microsoft Azure to ensure OpenCB Suite runs efficiently in Microsoft Azure cloud. We are running a proof-of-concept at the moment with GEL data to benchmark and test Azure.
Acknowledgements
We would like to thank Genomics England very much for their support and for trusting in OpenCGA and the rest of OpenCB Suite for this amazing release. In particular, we would like to thank Augusto Rendon, Anna Need, Carolyn Tregidgo, Frank Nankivell and Chris Odhams for their support, test and valuable feedback.
Last updated
