orderchr - determine a chromosome order to minimize cross-over of links
orderchr -links linkfile.txt
-karyotype karyotype.txt
{ -shuffle_file chrs_to_shuffle.txt | -shuffle LIST | -shuffle_rx REGEX_LIST }
{-static LIST} {-static_rx REGEX_LIST}
{-init_order LIST} {-init_order_rx REGEX_LIST}
This ordering script is most suitable for dense link data (many-to-many) (i.e. significant fraction of all chromosome pairs have links). If your data is sparse (one-to-many) (i.e. many chromosomes receive links from only one chromosome), please try tools/orderchr-deterministic.
By examining the frequencies of chromosome-chromosome relationships defined in the link file, this script suggests a new order for chromosomes that results in fewer cross-overs between links. Simulated annealing is used to optimize the chromosome order (read below about parameters). Run the simulation a couple of times to check convergence (finding a global minimum is not guaranteed).
The set of chromosomes to shuffle is specified by either (a) link data, whereby all chromosomes that have links are subject to shuffling (b) -shuffle_file, whereby only those chromosomes listed in the file are shuffled (c) -shuffle, whereby the set of chromosomes to shuffle is specified by a list, and (d) -shuffle_rx, whereby only those chromosomes that match a regular expression are shuffled. Any chromosome that is not identified by one of these methods does not participate in the shuffling process. Specifically, any links to/from such chromosomes are not considered when minimizing link crossing.
> orderchr -links linkfile.txt
All chromosomes mentioned in the -links file will be subject to reordering. The initial order will be taken from order of appearance in the karyotype file.
> orderchr -links linkfile.txt -shuffle_file chrs_to_shuffle.txt
The set of chromosomes to shuffle is given in the -shuffle_file file, which contains one chromosome per line. For example
> cat chrs_to_shuffle.txt chr1 chr5 chr12 chr17 ...
The initial order will be taken from order of appearance in the file.
> orderchr -links linkfile.txt -shuffle_rx chr1
Same as MODE 1, except that chromosome list will be filtered using the regular expression and only those chromosomes that match the regular expression are shuffled.
In this example, chromosomes matching ``chr1'' will be shuffled (e.g. chr1, chr10, chr11, etc).
> orderchr -links linkfile.txt -shuffle chr1,chr2,chr6,chr7,chr10
Same as MODE 3, except that chromosomes are specified by a list.
In this example, chromosomes chr1,chr2,chr6,chr7, and chr10 will be shuffled.
You can combine -shuffle_file, -shuffle_rx and -shuffle to additively define the shuffle list.
The initial order of chromosomes can be defined in two ways. First, the method that is used to specify which chromosomes to shuffle will dictate the initial order. Modes 1 and 2 (see above) use the order of chromosomes as they appear in the karyotype file. Mode 3 (see above) uses the order from the shuffle file.
You can override the initial order using the -init_order parameter. The value of this parameter is expected to a comma-delimited list of chromosomes, which may be the full set or a subset of chromosomes.
For example, if the entire set of chromosomes to shuffle is chr1..chr5, then you can specify the initial order which explicitly orders each chromosome
-init_order chr2,chr5,chr1,chr3,chr4
or just a subset
-init_order chr2,chr5
In the latter case, the final order will be
{ chr2,chr5 } , { chr1,chr3,chr4 }
comprised of two order groups: leading group of chromosomes as ordered by -init_order and a group of remaining chromosomes, in order of appearance as set by parameters in the section DEFINING WHICH CHROMOSOMES TO SHUFFLE.
If a chromosome mentioned in -init_order is not a candidate for shuffling, its mention in the order string will be ignored.
The option -init_order_rx works just like -init_order, except that the list a list of regular expressions rather than chromsome names. For example,
-init_order chr1,chr2
is equivalent to
-init_order { chrs matching /chr1/ },{ chrs matching /chr2/ }
and for the canonical human genome with standard order this would be
-init_order chr1,chr10,chr11,chr2,chr20,chr21,chr22
Since this is a subset of chromosomes, the final initial order will be automatically completed by chromosomes from the karyotype file that were not explicitly ordered
chr1, chr10, chr11, chr2, chr20, chr21, chr22, chr3..chr9, chr12..chr19, chrX, chrY
If both -init_order_rx and -init_order are defined, order is initially defined by -init_order_rx and then refined using -init_order. Thus
-init_order chr10,chr20,chrx -init_order_rx chr1,chr2
will result in
chr10, chr20, chrx, chr1, chr11, chr2, chr21, chr22, chr3..chr9, chr12..chr19 ,chrY
After the set of chromosomes to shuffle has been defined, and the initial order has been set, you can define a subset of chromosomes to remain in the same order (static) throughout the shuffling process.
The difference between a chromosome (a) not being part of a shuffle set and (b) being part of a shuffle set, but remain static, is that in the former, links to chromosomes do not play a role in the ordering process whereas in the latter case links to these chromosomes contribute to the shuffle score. Thus, chromosomes which are static have all non-static chromosomes shuffled around them in order to minimize link crossover.
Defining static chromosomes is done by a comma-delimited list of regular expressions
> orderchr -links linkfile.txt -static_rx chr1
In this example, all chromosomes matching the regular expression chr1 will not have their order adjusted. Any links to/from these chromosomes will contribute to the total link crossing score, but the chromosomes themselves will not be moved. For example, if the original order of chromosomes is
chr1,chr2,chr3,chr10,chr11,chr20,chr21
then any shuffle solution will have the order
chr1,-,-,chr10,chr11,-,-
with chr1, chr10 and chr11 remaining fixed.
To define multiple regular expressions, use a list of regular expressions.
> orderchr -links linkfile.txt -static_rx chr1,x,y
Like with -init_order, you can use the chromosome names to define static entries using -static.
-static chr1,chr2,chr3
will keep chromosomes chr1, chr2 and chr3 always in the same position. You can combine -static_rx and -static
-static_rx chr1,chr2 -static chrx,chry
in which case all chromosomes that match either the regular expressions defined by -static_rx or the names defined by -static will be kept in the same position during shuffling.
The order optimization process comprises one or more rounds. Each round is defined by a <round> block in the <simulation> block
<simulation>
<round>
# settings for round 1
</round>
<round>
# settings for round 2
</round>
...
</simulation>
A round can be either a warmup (read below), or a full simulated annealing process (read below). The outcome of the warmup is deterministic, and thus the warmup should only be used as the first (optional) round.
During the warmup round, the initial order of the chromosomes is defined based on the degree of connectivity between chromosome pairs.
This warmup is most suited for data sets in which most relationships between chromosomes are many-to-one (e.g., chromosome A has links to many chromosomes B,C,D,... but each of B,C,D generally only links to A). Many-to-one data sets are common for alignments (e.g. chr A corresponds to the chromosome whereas and points to ctg A, B, C, ... all sequence contigs that map to disjoint regions on chr A).
The warmup algorithm is as follows. The chromosome (chrA) with the most links is selected first and used to initialize the new order. A list of all links to chrA is created, grouped by chromosome, and sorted based on the average position of the link on chrA. Chromosomes are added to the new order based on descending order of grouped link position. Once all chromosomes are placed, the next unplaced chromosome with the most links is selected and the process continues until all chromosomes are placed.
The warmup is deterministic - it will result in the same order each time. It is insensitive to the initial order, or values of -static and -static_rx.
<round> warmup = yes </round>
After the optional warmup round, all other rounds should be of stochastic type (this is the default round type, if warmup=yes is not set).
Parameters for the round are defined as follows
<round>
iterations = 1000
max_flips = 10
min_flips = 2
temp0 = 0.01
<round>
For the details of each parameter, read the section below. You can set parameter values to be relative to values of the previous round by prefixing the parameter with ``r''. For example,
<round>
iterations = 1000
...
temp0 = 0.01
<round>
<round>
iterations = r2
...
temp0 = r0.5
<round>
<round>
iterations = r2
...
temp0 = r0.5
<round>
defines three rounds. The first round has 1000 iterations with temp0=0.01. The second round has 2x iterations (2000) and a value of temp0 of 0.5*0.01=0.005. The third round has again 2x iterations (4000) and temp0 of 0.5*0.005=0.0025.
Relative parameter values are very useful for additional rounds when the transition probability is decreased (temp0 is lowered). You can decrease temp0 in relative steps, without needing to remember what the previous value was. This allows you to create a multi-round optimization schedule with all parameter defined in a single place (first round).
The solution at the end of a round is used as the initial order for the next round.
This method is an optimization method that encourages the discovery of a global minimum by traversing the space of solutions with a small (and decreasing as simulation runs) chance of visiting less desirable solutions.
There are three parameters that control the optimization.
The number of iterations to perform. At each iteration, the current solution is randomly modified and either accepted or rejected.
The optimization run is split into max_flips-min_flips+1 equal-sized intervals. During each iterval, the number of random chromosome pair swaps in the solution is given by
min_flips + (max_flips-min_flips)*(1-t)
where t is a relative round completion time t=0..1 at the current iteration.
For example, if max_flips is 5 and min_flips is 2 and iterations=1000. Then the number of random pair swaps is
iteration 1-249 5 iteration 250-499 4 iteration 500-749 3 iteration 750-1000 2
I suggest starting with a value that corresponds to 5% of the chromosomes. For example, if you have 100 chromosomes, use max_flips=5 to start. It's also a good idea to set min_flips=1 for the last round to avoid abandoning the solution (remember that in simulated annealing it is possible to discard a solution for a worse solution).
This parameter determines the probability of a transition to a less desirable solution. The transition probability is
p(dE) = temp0*exp( - dE/t )
where t=1..0 over the length of the simulation and dE is the relative change in the desirability of two solutions.
If temp0=1, then the probability of accepting a solution that is 10% worse (e.g. dE=0.1) is
p(0.1) = exp (-0.1/1) = 90% at start of simulation
= exp (-0.1/0.5) = 82% half way through simulation
= exp (-0.1/0.1) = 37% 90% of the way through simulation
By lowering temp0, you lower the probability of transition to a less desirable solution.
Do not adjust temp0 unless you feel that the simulation is (a) not traversing the solution space sufficiently - in which case make temp0 larger or (b) too many low-quality solutions are accepted - in which case make temp0 smaller.
Most of the time you'll want to adjust the chromosome order in a way to minimize the number of crossing links. However, you can set to maximize the number of crossing links by setting
optimize = maximize
<simulation> <round> iterations = 1000 max_flips = 10 # or set this to ~5% of your chromosomes min_flips = 1 temp0 = 0.01 </round> </simulation>
The purpose of rounds 2 and 3 is to successively decrease the transition probability to worse solutions and also decrease the degree to which successive candidate solutions vary from the current solution. In these rounds, a more careful search is carried out around the solution provided in round 1.
<simulation> <round> iterations = 1000 max_flips = 10 # or set this to ~5% of your chromosomes min_flips = 1 temp0 = 0.01 </round> </simulation> <simulation> <round> iterations = r2 # 2000 max_flips = 2 min_flips = 1 temp0 = r0.5 # 0.005 </round> <round> iterations = r2 # 4000 max_flips = 1 min_flips = 1 temp0 = r0.1 # 0.0005 </round> </simulation>
<simulation> <round> warmup = yes </round> <round> iterations = 1000 max_flips = 10 # or set this to ~5% of your chromosomes min_flips = 1 temp0 = 0.01 </round> </simulation> <simulation> <round> iterations = r2 # 2000 max_flips = 2 min_flips = 1 temp0 = r0.5 # 0.005 </round> <round> iterations = r2 # 4000 max_flips = 1 min_flips = 1 temp0 = r0.1 # 0.0005 </round> </simulation>
Expanded documentation and added _rx parameters.
Started and versioned.
Martin Krzywinski
Martin Krzywinski Genome Sciences Centre Vancouver BC Canada www.bcgsc.ca martink@bcgsc.ca