Release Notes
Last updated
Was this helpful?
Last updated
Was this helpful?
You can find more detailed information at .
[FEATURE] Improve audit (, )
[FEATURE] Allow queries based on permissions ()
[FEATURE] Enable possibility of creating asynchronous tasks support (, )
[FEATURE] Support ACID transactions in Catalog !! ()
[FEATURE] Implement an automatic client generator valid for R, Java, Python and JS OpenCGA libraries ()
[FEATURE] Add new permission to execute jobs ()
[FEATURE] Improve job data model ()
[FEATURE] Enable job dependencies ()
[FEATURE] Support multistudy jobs ()
[FEATURE] Add webhooks ()
[FEATURE] Create jobs top webservice and command line ()
[FEATURE] Create new opencga administrator user with a default project and study (, )
[FEATURE] Improve count functionality ()
[FEATURE] Add a new webservice to download files from external sources ()
[FEATURE] Add dynamic parameter types in VariableSets ()
[FEATURE] Add new allowedKey field to VariableSet ()
[FEATURE] Add new /head and /tail web services in file ()
[FEATURE] Add new jobs/log/head and job/logs/tail web services to see job logs ()
[FEATURE] Load annotations from TSV files ()
[FEATURE] Allow passing a map of sample ids when linking VCF files ()
[FEATURE] Give users the option to set their own statuses ()
[FEATURE] Add new permission to view variants ()
[FEATURE] Assign implicit permissions automatically ()
[FEATURE] Remove base64 codification from UUIDs ()
[FEATURE] Improve Solr queries by annotation ()
[FEATURE] Add new webservice to fetch individual relatives ()
[FEATURE] Add new webservice to fetch base64 content of images ()
[FEATURE] Keep individual references in samples ()
[CHANGE-FEATURE] Improve panel operations ()
[CHANGE] Configuration file changes ()
[CHANGE] DataResponse data model changes ()
[CHANGE] Rename variable type in VariableSet ()
[CHANGE] Remove group name ()
[CHANGE] Rename a few ACLs ()
[CHANGE] Data model changes ()
[CHANGE] Main login endpoint changes ()
[CHANGE] Change webservice to change user password ()
[CHANGE] Ensure all REST webservices return on OpenCGAResult ()
[BUGFIX] Sync users CLI didn't work for AD ()
[BUGFIX] Unable to update "relatedFiles" list ()
[PERFORMANCE] Move deleted documents to different collection ()
1.4.2. contains a small series of internal changes requiring running a small migration. To do so, please head to the main OpenCGA source folder and run the following:
mongo_port: Typically 27017
database_name: Typically opencga_catalog
[BUGFIX] Fix RowKey generation for non symbolic structural variants #1259
[FEATURE] Implement a MR to extract number of variants per file #1287
[PERFORMANCE] Skip join SampleIndex with Variants table when possible #1242
[IMPROVEMENT] Improve approximate count of CompoundHeterozygous in StorageHadoop #1299
[IMPROVEMENT] Extend panel filter to include regions and single variants #1272
[IMPROVEMENT] Extract variant query executor from VariantStorageEngine #1240
Data model ids have been redefined to contain any user-defined id for every entry. Migrations produced in every entity are described in the ticket.
Old numeric ids are now lost and only used for internal purposes. Users cannot query by those fields anymore.
Add new inmutable uuid field to all entries.
[FIX] Remove base64 conversion of the secret key.
[FIX] Update pom dependencies to avoid conflicts.
[FEATURE] Support Azure AD authentication.
[FIX] Fix command line to support non-expiring tokens
[FIX] Support SKIP_COUNT parameter
[CHANGE] Remove family completeness check
CLI install changed and need two additional parameters 1: secretKey, algorithm
Configuration file has changed and need to be adopted on all opencga installation
No logout
Field "exonNumber" replaced with "exonOverlap".
Added field "traitAssociation" that will replace "variantTraitAssociation" in next releases.
Add new permissions to admin user in the general configuration file. To add: VIEW_STUDY, UPDATE_STUDY and SHARE_STUDY
Added sampleIds parameter in each individual entry.
Java and Javascript client libs use always POST when available
[REST] Complete the implementation of exclude and include of fields for Variant queries
[GRPC] Complete the implementation of exclude and include of fields for Variant queries
Added support to fetch shared projects and studies.
Renamed diskUsage and diskQuota for size and quota respectively in all data models were present.
Create and update webservices have been all implemented via POST.
OpenCGA storage hadoop is not available in this version. To compile use this line:
mvn clean install -DskipTests -pl '!:opencga-storage-hadoop-core'
New command line interfaces (CLI) for users (opencga.sh), admins (opencga-admin.sh) and analysis (opencga-analysis.sh)
New catalog configuration file using YAML.
New authorisation method. A big list of permission have been defined.
New smart id resolver. Numerical ids are no longer mandatory and ids are now resolved internally given the alias.
New and improved java and python command lines.
New javascript, java and R client implementations.
Support for annotations in cohorts.
Many performance improvements and fixes in MongoBD storage engine, the most notable include a new load strategy to improve file merging scalability
New storage engine based on Apache HBase, this is completely functional but is tagged as beta until more tests are done.
New custom variant annotation implemented
An experimental gRPC server implemented
Third Beta
Second Beta
First Beta
[CLINICAL] Tiering interpretation analysis for cancer ()
[VARIANT] Implement Fisher Test Analysis MapReduce ()
[VARIANT] Implement VariantStats OpenCGA Analysis ()
[VARIANT] Implement Gwas OpenCGA Analysis ()
[VARIANT] Complex sample query by variant to enable cohort creation for clinical trials ()
[CLINICAL] Implement mutational signature analysis ()
[CORE] Create a Dockerfile with the R packages used by OpenCGA analysis ()
[CLINICAL] Implement the inferred sex analysis ()
[ALIGNMENT] Implement statistics analysis for alignment coverage ()
[CLINICAL] Implement the relatedness analysis based on IBD/IBS ()
[CLINICAL] Implement genetic checks to compare with the reported results ()
[FEATURE] Support Hadoop3.x and HBase2.x ()
[FEATURE] Divide opencga-storage-hadoop-deps in submodules ()
[FEATURE] Store custom variant scores ()
[FEATURE] Allow load VCFs split by region in Hadoop ()
[FEATURE] Copy MapReduce jobs result submitted through an ssh connection ()
[FEATURE] Add FilterCount and MeanQuality to VariantStats ()
[FEATURE] Allow to configure the variant storage from REST ()
[FEATURE] Allow skip sample index when loading variant files ()
[FEATURE] Allow indexing multiple files per sample in StorageHadoop ()
[FEATURE] Return MendelianError code as an IssueEntry ()
[FEATURE] Extend sample filter functionality ()
[CHANGE] Add specific permissions to view variants ()
[CHANGE] Rename VariantQueryParams 'format', 'includeFormat' and 'info' ()
[CHANGE] Remove SAMPLE_ID and FILE_IDX from Format. Add INCLUDE_SAMPLE_ID ()
[CHANGE] Transform endpoint analysis/variants/sample/query into an Analysis ()
[CHANGE] Improve variant storage functionality for returning samples ()
[CHANGE] Change variant.id and variant.names content ()
[CHANGE] Make gene/id/xref query params more strict ()
[PERFORMANCE] Add extended clinical index to SampleIndex ()
[PERFORMANCE] Add Biotype+Ct combination to SampleInde ()
[PERFORMANCE] Count numTotalResults from covered SampleIndex queries asyncrhonously. ()
[PERFORMANCE] Improve SampleIndex File ()
[BUGFIX] Remove Jetty9.4 from Hadoop MapReduce classpath ()
[CHANGE] Rename web services from /stats to /aggregationStats ()
[BUGFIX] Index fails when passing ":" instead of "/" ()
[BUGFIX] Alignment index daemon fails ()
[BUGFIX] Migration script issue ()
[BUGFIX] Upload web service fails when uploading to root folder ()
[BUGFIX] Missing id query parameter in studies/{studies}/groups web service ()
[BUGFIX] Versioning issues ()
[BUGFIX] Web service studies/{studies}/groups/create not working ()
[BUGFIX] Order not respected when querying lists of ids ()
[FEATURE] Support passing relatedFiles object to File during link ()
[FEATURE] Support htsget protocol for data streaming ()
[FEATURE] Add new user category "application" ()
[PERFORMANCE] Analysis queries take too much time ()
[CHANGE] Rename filter "transcriptionFlag" to "transcriptFlag" ()
[BUGFIX] HashMap$Node cannot be cast to java.util.HashMap$TreeNode ()
[BUGFIX] Wrong usage of SampleIndex on invalid queries ()
[FEATURE] Add chromDensity query filtering by sample genotype ()
[FEATURE] Accept nested fields in chromDensity by sample genotype ()
[FEATURE] Use SamplingSize field at SampleIndex query executor ()
[FEATURE] Add new variant filters: "cohortStatsAlt", "cohortStatsRef" ()
[FEATURE] Query by compound heterozygous ()
[FEATURE] Export variants given a file of variant Ids ()
[FEATURE] Accept keyword "LoF" in consequence type filter ()
[PERFORMANCE] Remove unneeded FileMetadata reads from HBaseToStudyEntryConverter ()
[PERFORMANCE] Improve SampleIndex read performance ()
[PERFORMANCE] Add annotation counters to SampleIndex ()
[PERFORMANCE] Reduce size of SampleIndex table ()
[PERFORMANCE] Include parents genotype in the SampleIndex ()
[IMPROVEMENT] Improve DeNovo variants definition ()
[IMPROVEMENT] Improve Genotype filter to include by default phased genotypes ()
[IMPROVEMENT] Make /sampleData support somatic studies ()
[IMPROVEMENT] Add VarinatStats to SampleData result ()
[IMPROVEMENT] Native implementation of sampleData endpoint in storage-hadoop ()
[COMMAND LINE] Add missing SampleIndex command line operations ()
[IMPROVEMENT] Recognise bigwig files automatically. ()
[IMPROVEMENT] Major improvements in annotationSets (, , ):
Queries can be performed by any of the fields. ()
All the fields are indexed in the database, supporting really fast queries. ()
Projections can be added so only specific fields of an annotationSet are included. (, )
Annotation set web services have been changed. ()
See section from the OpenCGA documentation for the whole description of features.
[FEATURE] Implement new "Permission rules" feature ()
[FEATURE] Add new /admin web services to be able to perform administrative operations through REST ()
[IMPROVEMENT] Improve /user web services. Remove duplicated configs web services ()
[FEATURE] Implement delete operations for most of the entries ()
[IMPROVEMENT] Store creationDate field as an actual date object to improve queries ()
[CHANGE] Important id changes ():
Update REST web services to support id changes ()
[IMPROVEMENT] Stop creating empty directories when linking Files ()
[FEATURE] Add new {entry}Action field to some update web services to support adding, setting or removing single entries to arrays of them. (Example: samplesAction to add, set or remove samples from an individual, file or cohort) ()
[IMPROVEMENT] Deprecate VariableSet web services and move them to Study web services. Improve and clean old Study web services ()
[IMPROVEMENT] General data model improvements described in :
[FEATURE] Create catalog solr sync mechanism and enable facet queries
[FEATURE] Aggregate operation for all samples from a study in the Variant ()
[IMPROVEMENT] Move HBase write step to a separated MR for fill-missing operation ()
[IMPROVEMENT] Mark already processed variants in fill missing operation ()
[IMPROVEMENT] Do not iterate over all VcfRecords when filling gaps or missing variants ()
[IMPROVEMENT] Fill missing prepare step: copy variants to fill into archive table ()
[IMPROVEMENT] Split VcfSlice from archive table in REF and NON_REF columns ()
[IMPROVEMENT] Split archive table in file batches ()
[CHANGE] Remove merge=advanced from storage-hadoop ()
[FEATURE] Index Sample Genotypes in HBase ()
[FEATURE] Genotype index intersect in HBase ()
[FEATURE] Integrate Variants export using MapReduce ()
[FEATURE] Support enriched genotypes when querying to SampleIndex ()
[IMPROVEMENT] Use variants SampleIndex when reading from MapReduce ()
[FEATURE] New Variant query filter "clinicalSignificance" ()
[FEATURE] Add new filters for INFO and FORMAT fields other than GT ()
[FEATURE] Add filter by QUAL in Storage Variants ()
[FEATURE] Accept enriched genotypes at GENOTYPE filter ()
[FEATURE] Variant filtering in a samples group ()
[CHANGE] Automatic INCLUDE_STUDY if possible ()
[CHANGE] Automatic INCLUDE_SAMPLE if GENOTYPE VariantQueryParam is present ()
[IMPROVEMENT] Match Consequence type with the specified Gene filter (if any) for Storage Hadoop ()
[CHANGE] Untie internal Catalog and Storage IDs ()
[FIX] Synchronize Catalog ID to UID changes in Storage Managers catalog ()
[FEATURE] Create a ProjectMetadata object for Variant Storage ()
[IMPROVEMENT] Move storage-hadoop metadata to a separated table ()
[FEATURE] Support and load BREAKENDS in Variant Storage ()
[FEATURE] Support of Symbolic and Structural variants in Hadoop Variant Storage ()
[IMPROVEMENT] Add direct loader for the first file in a chromosome to load directly to the variants collection ()
[FEATURE] Move removed variants into a trash bin ()
[CHANGE] By default, load all FORMAT fields in Variant Storage MongoDB ()
[CHANGE] Change storage-hadoop table naming policy ()
[IMPROVEMENT] Review and improve Variant Annotation in OpenCGA Storage ()
[IMPROVEMENT] Create one Stage Collection per study in Variants Storage MongoDB ()
[IMPROVEMENT] Filter out overlapping files in variants in storage-hadoop ()
[IMPROVEMENT] Improve remove file in variant-storage-hadoop ()
[IMPROVEMENT] Load annotation using BufferedMutator ()
[FIX] Do not store phoenix primary key columns in HBase as separated columns ()
[FIX] Adapt remove file to new archive schema ()
[FIX] Do not map namespace for VIEW tables under phoenix 4.12.0 ()
[FIX] Avoid PhoenixIOException timeout when dropping columns from phoenix ()
[FEATURE] Support "application users" () (migration required) →
[FEATURE] Add CRAM support ()
[IMPROVEMENT] Change long fileId to File file in RelatedFile data model () (migration required) →
[IMPROVEMENT] RelatedFiles can be passed during file link ()
[FIX] Fix --sync-all option from opencga-admin.sh users sync command line ()
[CHANGE] Add id to Group data model (migration required) →
[FIX] Fix permission issue affecting users and groups with the "_" symbol ()
[ENHANCEMENT] Add new tags field to the File data model ()
[CHANGE] Configuration change. Add hooks to configuration file ()
[FIX] Improve performance of sample queries filtering by individual ()
[FIX] Fix issue when assigning permissions given the id(s) of different entities ()
[ENHANCEMENT] Performance improvement when assigning permissions ()
[ENHANCEMENT] Remove old deprecated fields from Family data model ()
[ENHANCEMENT] Allow looking for Individuals and Families by a new sample field ()
[FIX] Add missing individual-sample indexes ()
[FIX] Fix smart name resolution ()
[FIX] Propagation of permissions sample-individual not working ()
[ENHANCEMENT] Ask for admin password automatically when using admin command line ()
[FIX] Filtering by file size not working ()
[FEATURE] Admin migration command line ()
[FEATURE] Implement AutoComplete for CLI ()
[REMOVE] Remove ACL from data models. ()
[ENHANCEMENT] Clean old code and refactoring. (, , , )
[ENHANCEMENT] Remove some hidden and deprecated methods from webservices and command line ()
[CHANGE] Changes to family data model ()
[FEATURE] Add new analysis tool webservices ()
[FEATURE] Add version support for Sample, Individual and Family ()
[ENHANCEMENT] Clean and remove unnecessary dependencies for the client module ()
[ENHANCEMENT] Change some fields from Clinical Analysis (BETA) data model (, )
[ENHANCEMENT] Remove unnecessary Relatives data model ()
[ENHANCEMENT] Improve some individual webservices to better support the Individual-Sample relation ()
[CHANGE] Change individual parameter in the sample/create webservice ()
[CHANGE] Internal modification regarding the way the Sample-Individual relation was stored ()
[ENHANCEMENT] Add new admins group in studies ()
[ENHANCEMENT] Add new stats field to Sample data model ()
[CHANGE] Rename ontologyTerms field in Sample, Individual and diseases field in Family for phenotypes ()
[FEATURE] Add new option to export and import data from/to catalog ()
[ENHANCEMENT] Improve groupby webservices ()
[ENHANCEMENT] Support a list of ids in all GET webservices ()
[CHANGE] Internal modification: Change ACL delimiter used ()
[FEATURE] Make use of the new VariantMetadata model from Biodata ()
[FEATURE] Major support of Symbolic variants in Variants Storage ()
[FEATURE] Create profiles to select hadoop flavour ()
[FEATURE] Store info fields on storage-hadoop improvement storage ()
[FEATURE] Allow loading multiple variant files from the same sample with non overlapping variants ()
[FEATURE] New optional pipeline step "fill-gaps" ()
[CHANGE] Rename some Variant REST query parameters ()
[CHANGE] GO and EXPRESSION filter must be combined as an AND with other region ()
[CHANGE] Update CellBase to v4.5.3 improvement ()
[ENHANCEMENT] Speed up GENOTYPE filter with FILES filter, when possible ()
[ENHANCEMENT] Add field "source" to VariantQueryResult ()
[ENHANCEMENT] Indicate if "numTotalResults" is an approximated count in VariantQueryResult ()
[FIX] Inconsistent configuration param to select variant annotator ()
[FIX] Duplicate Key Warn/Error in Stage Collection ()
[FIX] IllegalArgumentException when CellBaseRestVariantAnnotator skips a variant ()
[FIX] Concurrent table modification error when loading in Hadoop with merge=basic (
[FIX] Fix job search by input and output files. ()
[ENHANCEMENT] Hide deprecated webservices. ()
[ENHANCEMENT] Set ontologyTerms array during Sample/Individual /create and /update webservices. ()
[FEATURE] Add the concept of release in OpenCGA. ()
[FIX] Fix behaviour where individual ids are not recognized when creating new family. ()
[FEATURE] Implement JWT based session management ()
[ENHANCEMENT] Change List<Long> for List<Object> in corresponding data models ()
[ENHANCEMENT] Make ACL permissions part of the query ()
[FIX] Fix count parameter not working in search webservices and command line ()
[CHANGE] Change annotationsets REST webservices. ()
[FEATURE] Support multigroups ()
[ENHANCEMENT] Return HTTP 403 error code when user tries to access not granted data ()
[FEATURE] Add a new members group for every study ()
[FEATURE] Automatically sync OpenCGA groups with external LDAP groups during login ()
[FEATURE] Add new projects/search webservice ()
[FEATURE] Add new confidential parameter to VariableSet data model. It allows defining confidential variable sets and corresponding annotation sets, so a new special permission will be needed to access them. ()
[FEATURE] Propagate permissions from samples to individuals. ()
[ENHANCEMENT] Return HTTP 401 error code when user is not successfully logged in or the token is invalid. ()
[FEATURE] Improve Solr integration with VariantStorage when querying ()
[ENHANCEMENT] Improve Solr variant iterator by using Solr cursors ()
[ENHANCEMENT] Variant Solr Search manager improvements ()
[ENHANCEMENT] Configure VariantMerger and VariantNormalizer with VCFHeader ()
[ENHANCEMENT] Store DisplayConsequenceType from VariantAnnotation at MongoDB ()
[ENHANCEMENT] Return VariantTraitAssociation as TraitAssociation (EvidenceEntry) ()
[FEATURE] Remove files from variants storage ()
[FEATURE] Command line and rest endpoints for remove operations on variant storage ()
[FEATURE] Simple merge mode for loading variants in opencga-storage-hadoop ()
[FEATURE] VariantQueryParam INCLUDE_FORMATS. Select format fields to return ()
[FEATURE] Store other genotype fields on storage-hadoop ()
[FEATURE] Export variant statistics ()
[FIX] Sample filter not working when the sample is in multiple files ()
[FIX] Possible error loading overlaping multiallelic variants ()
[FIX] Do not delete files (or related entries) from catalog if still loaded in variants storage ()
[FIX] Avoid OutOfMemoryError updating storage metadata from catalog ()
[FIX] Error indexing vcf files containing "variants" in the file name()
Changes on , , , and require several migration scripts to be run over the Catalog database:
Changes on and require to execute a migration script on all Variants databases in MongoDB:
Changes in VariantAnnotation model. See
[ENHANCEMENT] Support integers and floats type for variables. ()
[BUG] Fix link race condition. ()
[FEATURE] Add new parameter propagate to Individual web service when setting permissions to propagate permissions to the related samples. ()
[FEATURE] Add support to give permissions using queries in sample, individual and file web services. ()
[CHANGE] Change Acl REST web services. ()
[FIX] Not return fields that are of no interest (using include/exclude). ()
[FEATURE] Add new admin command line to synchronise and add users from LDAP groups. ()
[FEATURE] Added new Family data model. ()
[FEATURE] Add list<Sample> to individual/create web service. ()
[FEATURE] Add new /meta/status, /meta/ping and /meta/about web services. ()
[ENHANCEMENT] Support creating an Individual when calling to the Sample create web service. ()
[DEPRECATE] Deprecate species field of Individual data model. ()
[ENHANCEMENT] Deprecate usage of variableSetId. Add field variableSet to corresponding web services that accepts an id or a name. ()
[ENHANCEMENT] Support passing an array of annotationSets when creating an Annotable entry (Sample, Cohort, Individual and Family). ()
[ENHANCEMENT] Add new type field to Sample model. ()
[FEATURE] Add father and mother information in attributes field of Individual. ()
[ENHANCEMENT] Add support to change public/private User registration ()
[DEPRECATE] Deprecate all xx/create and xx/update GET webservices. ()
[FEATURE] Make optional to merge non same overlapping variants in MongoDB ()
[FEATURE] Implement a benchmark framework for OpenCGA Storage ()
[ENHANCEMENT] Filter VcfRecord before converting into Variant object when possible ()
[BUGFIX] Not loading new overlapping variants in HBase ()
[ENHANCEMENT] Increment variant size threshold for CellBase annotation ()
[BUGFIX] Fix ArrayIndexOutOfBounds when loading variants data ()
[FEATURE] Add VCF export to gRPC command line ()
Catalog changes require this migration script:
Changes of require to execute a migration script on all Variants databases in MongoDB:
[BUGFIX] Fix VCF output format ()
[ENHANCEMENT] New dateOfBirth field added to Individual () and somatic field to Sample ()
[ENHANCEMENT] Performance improvement when annotating new variants ()
[FIX] Authenticated users can now see public data ()
Permissions assigned to individuals are directly propagated to samples ()
[CHANGED] CREATE and UPDATE permissions have been merged into WRITE ()
[CHANGED] Permissions given to folders are now propagated in the database (), this increase significantly the performance of the ACL resolution
[CHANGED] Changes in session data model ()
Complete the implementation of exclude and include of fields for Variant queries ()
[Python] new Python client implemented following the same architecture than Java and Javascript clients (). This improves the quality significantly and add many new features, thanks !
[REST] Add new files/create using POST to create new folders and files with some content ()
[REST] Rename parameters from acl/update ()
[REST] Hide from Swagger all create and update ACL web services using GET (), these should always use POST, this affects to studies, samples, files, ... (these will be removed in version 1.1)
[REST] Remove from users the GET methods to create and update filters ()
[REST] Add a warning to Swagger to all create and update methods using GET (). It is encourage that all create and update actions use always POST
Major changes in maven properties and configuration files. ()
catalog-configuration.yml and configuration.yml have been merged. ()
Added organism information to the project data model. ()
Closed sessions are now removed from the array of sessions (but they can still be found in the audit collection). ()
Improve error handling for storage operations (, , )
Add param --resume to opencga-analysis.sh and opencga-storage.sh command line ()
Import and export variants datasets (, )
Enable gRPC as top level feature ()
New top layer StorageManager connecting catalog with storage ()
Shade proto and guava dependencies for Hadoop ()
This release constitutes the first release candidate (RC1). This is the biggest release ever with more than 1,400 commits, special mention to and for their contribution in Catalog, Storage and Server components.
[IMPROVEMENT] Add direct loader for the first file in a chromosome to load directly to the variants collection ()