HOWTO PARALLELIZE MIGRATE ========================= Peter Beerli beerli@genetics.washington.edu [updated July 21 2001] Contents: This text describes how you can improve the performance of migrate when you have more than one locus and more than one computer at your fingertips. You can parallelize migrate runs (I) using a virtual parallel architecture with a message-passing interface (MPI) or (II) by hand. The hand-version works but is cumbersome, the MPI-version is not tested yet by users, so there may be glitches, but it works for me, but there are several things still untested: I did default runs where one estimates all parameters with plots and profiles, runs using sumfiles. Many specific option combination most likley will work, except that those that involve file reading and writing, these may fail. I. Message passing interface ==================================== (1) Secure as many computers for the analysis as you have loci in your dataset. Make sure that all computers can talk to each other. Currently my program will only work if they are a flavor of UNIX (e.g. LINUX or MACOSX). Of course, you need an account on all the machines. - Download LAM from http://www.lam-mpi.org [I used version 6.5.2] - install LAM on all machines (if this goes over your head ask a sysadmin or other guru to help (on MACOSX there is bug that prevents compilation: in the file lam-6.5.2/share/etc/registry.c: line 62: replace #include with #include ) I used the the following configure flags: ./configure --without-fc --without-romio --with-rsh=ssh [the with-rsh=ssh is optional if you are behind a firewall this may be overkill] - prepare a file lamhosts according to the specs in the lam distribution, the master node needs to be the first machine mentioned. my lamhosts looks like this: phylo cpu=4 darwin cpu=2 evolution cpu=1 buffon cpu=1 - make sure that you can access all machines using ssh (I suggest NOT to use the standard rsh) without the need to specify a password, see man ssh-keygen and man ssh if you have firewalls installed on your individual systems then you would need to allow the individual machines to open/request ports on the other machines. - change into the migrate-1.2/src/ directory configure and then use "make mpi-thread" to compile with the pthread library, or or with "make mpi-all", perhaps you want to rename the binary because it will NOT WORK as a standard program. (2) If your machines has no cross-mounted file system, you need to make sure that the data and the program are all in the same path e.g. /home/beerli/migrate-test/infile [e.g. a 120 loci dataset] and on EVERY machine, ssh to your slowest machine [the master is not doing much except scheduling the workers] then try: - lamboot -v lamhost - mpirun -np 11 migrate-n [ watch in awe, that all your 10 loci gets analyzed at once, the log is not very comprehensive because all 11 processes write to the same console, 11 because there is one master-node who does only scheduling and maximization, 10 worker-nodes that do the actual tree rearrangements and the likelihood calculations. the number you specify has nothgin to do with the physical computers, LAM can run several nodes on a single CPU. Best performance will be achieved if you specify the smaller of the number of available CPUs or the number of loci+1.] - lamhalt -v [on MACOSX you may need to rm the temporary directory by hand] - send comments how it worked for you and improvements for my [currently] too short and confusing guide. II. BY HAND ========================================================= (1) Secure as many computers for the analysis as you have loci in your dataset. (2) On one machine prepare a directory with - migrate-n run the program once, and adjust the run parameters using the menu. Use the sumfile option in the (I)nput menu and then save the parmfile with the (W)rite parmfile option. Then (Q)uit. Edit the created parmfile and check if you can find write-sumfile=YES then change menu=YES to menu=NO (2) Copy this directory on each machine and name the directories e.g. locus1 locus2 ..... If you use Appleshare be careful that you have also directories for each locus. (3) Prepare the infiles. One for each locus Copy the infiles into the directories. (4) Start migrate-n on all machines (5) Once all the migrate-n runs have finished, copy all sumfiles onto a single machine it would be helpful if this is your fastest with lots of RAM. Be careful not ot overwrite individual files (the have the same name" sumfile"). (6) Concatenate the sumfiles (7) The combined sumfile needs hand editing or you can use the PERL script concat-sumfile if you cannot run the PERL script or want to do it by hand, see the example below. (8) make a save copy of the fixed combined sumfile (9) run migrate-n and use option (D)atatype and there (g)enealogy and change other menu items if you want. (10) voila, a multilocus outfile in a fraction of the time the program needs to run on a single machine. ========================================================================= What to edit in a sumfile (1) the heading of a sumfile needs the two first comment lines # begin genealogy-summary file of migrate 0.9.8 ------ # (2) the third line needs editing, the first number is the number of loci for single locus data it is 1, change it to the number of loci 1 3 9 0 1 [before] 4 3 9 0 1 [after, 4 loci] (3) Search for ####### you will find lines like the following 0 0 ####### locus 0, replicate 0 ################ the file start couting with 0, so the lines reads locus 1 and replicate 1 leave the first occurrence as it is. Goto the end of the file and remove # end genealogy-summary file of migrate 0.9.8 ------ (4) Prepare the next sumfile.x to the master sumfile - Remove everything above 0 0 ######## locus 0, replicate 0 ..... - change the number to 1 0 ######## locus 1, ..... if you use replicates you need to change the replicates accordingly. - Remove the last line [except for the very last sumfile (5) concatenate the above sufmile-fragment to the master sumfile (6) Goto (4) until done ====================================================================== Example of a sumfile # begin genealogy-summary file of migrate 0.9.8 ------ # 1 3 9 0 1 0 0 ####### locus 0, replicate 0 ################ <<<<<<<<<<<<< $Id: HOWTO-PARALLEL,v 1.2 2001/06/02 21:40:20 beerli Exp $