1.. _workload_distribution: 2 3Workload distribution 4====================== 5 6Workload of thermal conductivity calculation can be distributed into 7computer nodes. The distribution over q-point grid points (:ref:`--gp 8option <gp_option>`), over phonon bands (:ref:`--bi option 9<bi_option>`), and over both of them are supported. Unless necessary, 10the distribution over bands is not recommended since it has some 11amount of overhead in the part of Fourier transformation of force 12constants. Therefore the distribution over grid-points is explained 13below. However since the distribution over bands works quite similarly as 14that over q-points, the usage can be easily guessed. 15 16On each computer node, pieces of lattice thermal conductivity 17calculation are executed. The resulting data for each grid point are 18stored in its ``kappa-mxxx-gx.hdf5`` file on each node by setting 19:ref:`--write_gamma option <write_gamma_option>`. Once all data are 20obtained, those data are collected by :ref:`--read_gamma option 21<read_gamma_option>` and the lattice thermal conductivity is obtained. 22 23.. contents:: 24 :depth: 2 25 :local: 26 27How to do it 28------------ 29 30The following example is executed in the ``Si-PBE`` example. 31 32To avoid re-calculating fc3 and fc2, ``fc3.hdf5`` and ``fc2.hdf5`` are 33created on a single node:: 34 35 % phono3py --dim="2 2 2" --sym-fc -c POSCAR-unitcell 36 37The indices of the irreducible grid-points neccesarry to specify 38``--ga`` option are found by :ref:`--wgp option <wgp_option>` 39 40:: 41 42 % phono3py --dim="2 2 2" --pa="0 1/2 1/2 1/2 0 1/2 1/2 1/2 0" -c POSCAR-unitcell --mesh="19 19 19" --fc3 --fc2 --br --wgp 43 44and they are stored in ``ir_grid_points.yaml``. 45 46:: 47 48 % egrep '^- grid_point:' ir_grid_points.yaml|awk '{printf("%d,",$3)}' 49 0,1,2,3,4,5,6,7,8,9,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,60,61,62,63,64,65,66,67,68,69,70,71,72,73,80,81,82,83,84,85,86,87,88,89,90,91,100,101,102,103,104,105,106,107,108,109,120,121,122,123,124,125,126,127,140,141,142,143,144,145,160,161,162,163,180,181,402,403,404,405,406,407,408,409,422,423,424,425,426,427,428,429,430,431,432,433,434,435,442,443,444,445,446,447,448,449,450,451,452,453,462,463,464,465,466,467,468,469,470,471,482,483,484,485,486,487,488,489,502,503,504,505,506,507,522,523,524,525,542,543,804,805,806,807,808,809,824,825,826,827,828,829,830,831,832,833,844,845,846,847,848,849,850,851,864,865,866,867,868,869,884,885,886,887,904,905,1206,1207,1208,1209,1226,1227,1228,1229,1230,1231,1246,1247,1248,1249,1266,1267,1608,1609,1628,1629, 50 51The calculated data on all the grid points shown above as indices are 52necessary to obtain lattice thermal conductivity. To distribute 53computational demands into computer nodes, a set of the grid-point 54indices are chosen and executed as follows:: 55 56 % phono3py --dim="2 2 2" --pa="0 1/2 1/2 1/2 0 1/2 1/2 1/2 0" -c POSCAR-unitcell --mesh="19 19 19" --fc3 --fc2 --br --gp="0,1,2,3,4,5,6,7,8,9,20,21,22,23,24,25" --write-gamma 57 58Then many ``kappa-m191919-gx.hdf5`` files are generated. These file 59names should not be altered because in reading the data by phono3py, 60those file names are supposed to be so, though there is a little 61freedom to arrange those file names, for which see :ref:`-o 62<output_filename_option>` and :ref:`-i <input_filename_option>` 63options. After completing calculations for all irreducible grid-point 64indices, the RTA thermal conductivity is computed by another run in a 65short time from the stored data: 66 67:: 68 69 % phono3py --dim="2 2 2" --pa="0 1/2 1/2 1/2 0 1/2 1/2 1/2 0" -c POSCAR-unitcell --mesh="19 19 19" --fc3 --fc2 --br --read-gamma 70 71A convenient script 72-------------------- 73 74The following short script may be useful to splitting all irreducible 75grid-point indices into of reasonable number of sets of grid-point 76indices for workload distribution. 77 78.. code-block:: python 79 80 #!/usr/bin/env python 81 82 import sys 83 import yaml 84 85 if len(sys.argv) > 1: 86 num = int(sys.argv[1]) 87 else: 88 num = 1 89 90 with open("ir_grid_points.yaml") as f: 91 data = yaml.load(f) 92 gps = [gp['grid_point'] for gp in data['ir_grid_points']] 93 gp_lists = [[] for i in range(num)] 94 for i, gp in enumerate(gps): 95 gp_lists[i % num].append(gp) 96 for gp_set in gp_lists: 97 print(",".join(["%d" % gp for gp in gp_set])) 98 99Supposed that this script is saved as ``divide_gps.py``, 100 101:: 102 103 % phono3py --dim="2 2 2" --pa="0 1/2 1/2 1/2 0 1/2 1/2 1/2 0" -c POSCAR-unitcell --mesh="19 19 19" --wgp 104 ... 105 % python divide_gps.py 20 106 0,30,52,82,120,402,434,468,524,844,1206 107 1,31,53,83,121,403,435,469,525,845,1207 108 2,32,54,84,122,404,442,470,542,846,1208 109 3,33,55,85,123,405,443,471,543,847,1209 110 4,34,60,86,124,406,444,482,804,848,1226 111 5,35,61,87,125,407,445,483,805,849,1227 112 6,36,62,88,126,408,446,484,806,850,1228 113 7,37,63,89,127,409,447,485,807,851,1229 114 8,40,64,90,140,422,448,486,808,864,1230 115 9,41,65,91,141,423,449,487,809,865,1231 116 20,42,66,100,142,424,450,488,824,866,1246 117 21,43,67,101,143,425,451,489,825,867,1247 118 22,44,68,102,144,426,452,502,826,868,1248 119 23,45,69,103,145,427,453,503,827,869,1249 120 24,46,70,104,160,428,462,504,828,884,1266 121 25,47,71,105,161,429,463,505,829,885,1267 122 26,48,72,106,162,430,464,506,830,886,1608 123 27,49,73,107,163,431,465,507,831,887,1609 124 28,50,80,108,180,432,466,522,832,904,1628 125 29,51,81,109,181,433,467,523,833,905,1629 126 127For example distributing into 20 computer nodes using a queueing 128system, 129 130.. code-block:: shell 131 132 % j=1; for i in `python divide_gps.py 20`;do echo $i; sed -e s/gps/$i/g -e s/num/$j/g job.sh|qsub; j=$((j+1)); done 133 134with ``job.sh`` (here for grid-engine): 135 136.. code-block:: shell 137 138 #$ -S /bin/zsh 139 #$ -cwd 140 #$ -N phono3py-num 141 #$ -pe mpi* 16 142 #$ -e err-phono3py-num.log 143 #$ -o std-phono3py-num.log 144 145 phono3py --dim="2 2 2" --pa="0 1/2 1/2 1/2 0 1/2 1/2 1/2 0" -c POSCAR-unitcell --mesh="19 19 19" --fc3 --fc2 --br --gp="gps" --write-gamma 146