HOWTO PARALLELIZE MIGRATE
=========================
Peter Beerli
beerli@genetics.washington.edu
[updated July 21 2001]

Contents:
This text describes how you can improve the performance of migrate
when you have more than one locus and more than one computer at 
your fingertips. You can parallelize migrate runs 
(I) using a virtual parallel architecture with a message-passing 
interface (MPI) or (II) by hand.
The hand-version works but is cumbersome, 
the MPI-version is not tested yet by users, so there may
be glitches, but it works for me, but there are several things still 
untested: I did default runs where one estimates all parameters with plots 
and profiles, runs using sumfiles. Many specific option combination
most likley will work, except that those that involve file reading and writing,
these may fail. 


I. Message passing interface ====================================

(1) Secure as many computers for the analysis as you have loci
    in your dataset. Make sure that all computers can talk to each
    other. Currently my program will only work if they are a flavor of 
    UNIX (e.g. LINUX or MACOSX). Of course, you need an account on all 
    the machines.
    - Download LAM from  http://www.lam-mpi.org [I used version 6.5.2]
    - install LAM on all machines (if this goes over your head ask a
      sysadmin or other guru to help 
      (on MACOSX there is bug that prevents compilation: in the file
      lam-6.5.2/share/etc/registry.c: line 62: replace 
	#include <sys/sem.h> with #include <sys/semaphore.h>)
	I used the the following configure flags:
        ./configure --without-fc --without-romio --with-rsh=ssh
      [the with-rsh=ssh is optional if you are behind a firewall this may be overkill]	
    - prepare a file lamhosts according to the specs in the lam distribution, the master node
      needs to be the first machine mentioned.
      my lamhosts looks like this:
      phylo cpu=4
      darwin cpu=2
      evolution cpu=1
      buffon cpu=1
    
    - make sure that you can access all machines using ssh (I suggest NOT to use the standard rsh)
     without the need to specify a password, see man ssh-keygen and man ssh
     if you have firewalls installed on your individual systems then you would need to allow
     the individual machines to open/request ports on the other machines. 
     
    - change into the migrate-1.2/src/ directory
      configure and then use "make mpi-thread" to compile with the pthread library, or
      or with "make mpi-all", perhaps you want to rename the binary because it will NOT WORK
      as a standard program.
(2) If your machines has no cross-mounted file system,
    you need to make sure that the data and the program are all
    in the same path e.g. /home/beerli/migrate-test/infile [e.g. 
    a 120 loci dataset] and on EVERY machine, 
    ssh to your slowest machine [the master is not doing much except scheduling the workers] then try:
    
    - lamboot -v lamhost
    - mpirun -np 11 migrate-n
    [ watch in awe, that all your 10 loci gets analyzed at once,
     the log is not very comprehensive because all 11 processes
     write to the same console, 11 because there is one master-node
     who does only scheduling and maximization, 10 worker-nodes
     that do the actual tree rearrangements and the likelihood calculations.
     the number you specify has nothgin to do with the physical computers,
     LAM can run several nodes on a single CPU. Best performance will be 
     achieved if you specify the smaller of the number of available CPUs 
     or the number of loci+1.]
    - lamhalt -v 
      [on MACOSX you may need to rm the temporary directory by hand]	
    - send comments how it worked for you and improvements for my 
     [currently] too short and confusing guide.


II. BY HAND =========================================================
(1) Secure as many computers for the analysis as you have loci
    in your dataset.

(2) On one machine prepare a directory with

    - migrate-n

    run the program once, and adjust the run parameters using
    the menu. Use the sumfile option in the (I)nput menu
    and then save the parmfile with the 
    (W)rite parmfile option. Then (Q)uit.
    Edit the created parmfile and check if you can find 
    write-sumfile=YES
    then change menu=YES to menu=NO

(2) Copy this directory on each machine and name the directories
    e.g. locus1 locus2 .....
    If you use Appleshare be careful that you have also 
    directories for each locus.
     
(3) Prepare the infiles. One for each locus
    Copy the infiles into the directories.

(4) Start migrate-n on all machines

(5) Once all the migrate-n runs have finished, 
    copy all sumfiles onto a single machine
    it would be helpful if this is your fastest 
    with lots of RAM. Be careful not ot overwrite
    individual files (the have the same name" sumfile").
 
(6) Concatenate the sumfiles 

(7) The combined sumfile needs hand editing
    or you can use the PERL script 
    concat-sumfile
    if you cannot run the PERL script or 
    want to do it by hand, see the example below.

(8) make a save copy of the fixed combined sumfile

(9) run migrate-n
    and use option (D)atatype and there (g)enealogy
    and change other menu items if you want.
    
(10) voila, a multilocus outfile in a fraction of
     the time the program needs to run on a single machine.

=========================================================================
What to edit in a sumfile 

(1) the heading of a sumfile needs the two first comment lines
# begin genealogy-summary file of migrate 0.9.8 ------
#

(2) the third line needs editing, the first number is the number of loci
for single locus data it is 1, change it to the number of loci
1 3 9 0 1  [before]
4 3 9 0 1  [after, 4 loci]

(3) Search for ####### you will find  lines like the following
0 0 ####### locus 0, replicate 0 ################
the file start couting with 0, so the lines reads locus 1 and replicate 1
leave the first occurrence as it is. Goto the end of the file and 
remove
# end genealogy-summary file of migrate 0.9.8 ------

(4) Prepare the next sumfile.x to the master sumfile
    - Remove everything above 0 0 ######## locus 0, replicate 0 ..... 
    - change the number to    1 0 ######## locus 1, .....
      if you use replicates you need to change the replicates accordingly.
    - Remove the last line [except for the very last sumfile
(5) concatenate the above sufmile-fragment to the master sumfile

(6) Goto (4) until done 
   
======================================================================

Example of a sumfile

# begin genealogy-summary file of migrate 0.9.8 ------
#
1 3 9 0 1
0 0 ####### locus 0, replicate 0 ################    <<<<<<<<<<<<<<change this
1 0 0
                   0                    0                    0
101 0.01224715726902237366 0.24028906596661075978 68
0.01797109426997086853 0.46646854779862101381 88
0.00810026471633800565 0.36704951583234807222 98
0.010000 0.010000 0.010000 32.000000 23.000000 23.000000 29.000000 21.000000 27.000000 
3.53366272929252732415e-03 3.53366273324574875492e-03
5.30077896995009931885e-03 5.30077895638609870171e-03
3.74540337472894320145e-03 3.74540325539807769997e-03
2.61285099904684057037e+03 2.61285112767777718545e+03
1.87798679166376041394e+03 1.87798680725500389599e+03
1.27983305210177809386e+03 1.27983304107036315145e+03
1.61370252198881439654e+03 1.61370251775554697815e+03
2.59250790064689090286e+03 2.59250788575867818508e+03
3.33322424226003886361e+03 3.33322432150516533511e+03
39 1.71782306759650010393e-06
# end genealogy-summary file of migrate 0.9.8 ------





good luck

Peter
<beerli@genetics.washington.edu>


$Id: HOWTO-PARALLEL,v 1.2 2001/06/02 21:40:20 beerli Exp $