segment_liftover : a Python tool to convert segments between genome assemblies

Implementation

segment_liftover can convert both probe files and segment files at the same time or in separate runs. It starts from a structured directory or a list of files, then traverses and converts all files meeting the specified name pattern, and finally outputs to a designated directory. To convert a probe file, segment_liftover will first use the UCSC liftOver to convert the file, then apply an approximate conversion on probes that the UCSC liftOver failed to convert. To convert a segment file, segment_liftover will use the UCSC liftOver to convert the start and end positions of segments. A successful segment conversion needs to satisfy the following four criteria:

                                                                             position_{new_start} ≠ ∅                                                         (1)

                                                                             position_{new_end} ≠ ∅                                                         (2)

                                                                             chromosome_{new_start} = chromosome_{new_end}                         (3)

Where β controls the threshold of the length ratio and the default value is set to 2. If criteria (1) or (2) fails, segment_liftover will apply an approximate conversion; if the conversion still fails, it is reported as unconvertible. If criteria (3) or (4) fails, the conversion is reported as rejected (Figure 1c). The reason of failure is recorded in log files.

When a position cannot be converted by the UCSC liftOver, segment_liftover will attempt an approximate conversion and try to find a convertible position in the adjacency (Figure 1b). The range and the resolution of the search is defined by parameters –range and –step_size, respectively.

Operation

The segment_liftover tool is implemented in Python. The package is available for both Linux and OSX. It requires a minimum of 2G memory and the capacity of running Python 3. We recommend an installation using pip in a Python virtual environment. segment_liftover requires and depends on the UCSC liftOver program. A chain file, which provides alignments from source to target assembly, is also required. The chain files between common human assemblies (hg18, hg19 and hg38) are included in the program package. Chain files of other species and assemblies are available from the USCS Genome Browser. Figure 2 illustrates the work-flow of segment_liftover.

Figure 2. The workflow of segment_liftover:

(1) It can take either a folder or a file containing the list of files as the input. (2) It will try to convert by approximation when UCSC liftOver fails to convert a coordinate. (3) The directory structure will be kept in the output folder and detailed log files are also available.

Converting arrayMap data from hg19 to hg38

In this section, we provide two examples of using segment_liftover to convert probes and segments, respectively. The two examples are part of the pipeline which updates the arrayMap database, a reference resource of somatic genome copy number variations in cancer⁵, from human genome assembly hg19 to hg38. In the first example, we converted 44,632 probe files and 44,471 segment files from hg19 to hg38. The probe data were generated from nine Affymetrix genotyping array platforms, which currently only support annotations for hg19. Circular binary segmentation (CBS) analysis (DNAcopy R-package) was used to infer copy number segments from log2 values of probes. The final segment files contain a list of genomic regions separated by their copy number values⁶. We ran the segment_liftover tool on a 12-core, 128GB RAM machine with 8 parallel processes. It took 42 hours to convert 44,632 probe files with 5.5 billion probe positions and 40 minutes to convert 44,471 segment files with 4.8 million segments.

Overall, more than 99.99% of probes and more than 99% of segments could be directly converted from hg19 to hg38 (Figure 3). As conversion of segments is more complicated and involves a quality control procedure to ensure the meaningfulness of the segment, it is expected to have a higher number of unconvertible segments than probes. As shown in Figure 4, the unconvertible regions are mainly around telomeres, centromeres, or other gene-sparse locations. In total, 38 genetic elements were found to be affected by the conversion (description in Supplementary Table 1).

Figure 3. Compasion of conversion results in log 10.

Directly converted is the sum of successful conversion from UCSC liftOver; approximately converted is the sum of successful re-conversion in locus adjacency; converted but rejected is the sum of all rejections from quality control; unconvertible is the sum of everything that is not converted.

Figure 4. Genomic regions of unconvertible probe positions from human genome hg19.

Unconvertible positions are marked red on the karyogram, annotated with HGNC symbol or ENSEMBL gene ID (if HGNC not available), retrieved from biomaRt_2.30.0.

Comparison of different conversion strategies

In the second example, we compared the performance of different conversion strategies using 1,000 samples randomly drawn from the first example (Table 1). Copy number segments were generated using four different strategies (Figure 5): (1) segments in hg19 were generated from probes in hg19 using the aforementioned standard pipeline; (2) segments in hg38 were generated with probes converted from hg19 to hg38 using the standard pipeline; (3) segments converted from hg19 to hg38 with approximate conversion; (4) segments converted from hg19 to hg38 without approximate conversion.

Table 2 shows the comparison of segments conversion in average between (3), (4) and (2) (complete table in Supplementary Table 2). Exact segment matches are categorized as "perfect"; "minor difference" is defined the same as condition (3) of quality control; the rest of the segments are categorized as "significant difference"; "sum" is the total number of segments of a sample in average. By comparing the sum of reference hg19, reference hg38 and approximation, it shows that the conversion result is very close to the result of the standard pipeline. The difference between converted and generated sums is much smaller than the difference between two generated sums of different genome versions. On average, the approximate conversion could rescue one additional segment per file. Finally, we zoom into a specific example on chromosome 9 in GSM276858 (illustrated in Figure 5). Because of the removal of probes from hg19 to hg38, the second segment will be lost without approximate conversion.

Figure 5. Chromosome 9 from GSM276858 with probe and segment data from hg19 coordinate(left) and hg38(right).

Segments directly processed from hg38 probes are used as reference (top right). hg19 segments converted with approximation (middle right) and without approximation (bottom right) are used for comparison.

The two examples above demonstrated the efficiency and effectiveness of segment_liftover in processing large number of probe and segment files. It can provide conversion results that are similar to results generated from the standard pipeline. Moreover, with approximate conversion, the number of properly converted segments is slightly increased. In general, segment_liftover is able to provide reliable conversions and the ease of use.