1.. _workload_distribution:
2
3Workload distribution
4======================
5
6Workload of thermal conductivity calculation can be distributed into
7computer nodes. The distribution over q-point grid points (:ref:`--gp
8option <gp_option>`), over phonon bands (:ref:`--bi option
9<bi_option>`), and over both of them are supported. Unless necessary,
10the distribution over bands is not recommended since it has some
11amount of overhead in the part of Fourier transformation of force
12constants. Therefore the distribution over grid-points is explained
13below. However since the distribution over bands works quite similarly as
14that over q-points, the usage can be easily guessed.
15
16On each computer node, pieces of lattice thermal conductivity
17calculation are executed. The resulting data for each grid point are
18stored in its ``kappa-mxxx-gx.hdf5`` file on each node by setting
19:ref:`--write_gamma option <write_gamma_option>`. Once all data are
20obtained, those data are collected by :ref:`--read_gamma option
21<read_gamma_option>` and the lattice thermal conductivity is obtained.
22
23.. contents::
24   :depth: 2
25   :local:
26
27How to do it
28------------
29
30The following example is executed in the ``Si-PBE`` example.
31
32To avoid re-calculating fc3 and fc2, ``fc3.hdf5`` and ``fc2.hdf5`` are
33created on a single node::
34
35   % phono3py --dim="2 2 2" --sym-fc -c POSCAR-unitcell
36
37The indices of the irreducible grid-points neccesarry to specify
38``--ga`` option are found by :ref:`--wgp option <wgp_option>`
39
40::
41
42   % phono3py --dim="2 2 2" --pa="0 1/2 1/2 1/2 0 1/2 1/2 1/2 0" -c POSCAR-unitcell --mesh="19 19 19" --fc3 --fc2 --br --wgp
43
44and they are stored in ``ir_grid_points.yaml``.
45
46::
47
48   % egrep '^- grid_point:' ir_grid_points.yaml|awk '{printf("%d,",$3)}'
49   0,1,2,3,4,5,6,7,8,9,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,60,61,62,63,64,65,66,67,68,69,70,71,72,73,80,81,82,83,84,85,86,87,88,89,90,91,100,101,102,103,104,105,106,107,108,109,120,121,122,123,124,125,126,127,140,141,142,143,144,145,160,161,162,163,180,181,402,403,404,405,406,407,408,409,422,423,424,425,426,427,428,429,430,431,432,433,434,435,442,443,444,445,446,447,448,449,450,451,452,453,462,463,464,465,466,467,468,469,470,471,482,483,484,485,486,487,488,489,502,503,504,505,506,507,522,523,524,525,542,543,804,805,806,807,808,809,824,825,826,827,828,829,830,831,832,833,844,845,846,847,848,849,850,851,864,865,866,867,868,869,884,885,886,887,904,905,1206,1207,1208,1209,1226,1227,1228,1229,1230,1231,1246,1247,1248,1249,1266,1267,1608,1609,1628,1629,
50
51The calculated data on all the grid points shown above as indices are
52necessary to obtain lattice thermal conductivity. To distribute
53computational demands into computer nodes, a set of the grid-point
54indices are chosen and executed as follows::
55
56   % phono3py --dim="2 2 2" --pa="0 1/2 1/2 1/2 0 1/2 1/2 1/2 0" -c POSCAR-unitcell --mesh="19 19 19" --fc3 --fc2 --br --gp="0,1,2,3,4,5,6,7,8,9,20,21,22,23,24,25" --write-gamma
57
58Then many ``kappa-m191919-gx.hdf5`` files are generated. These file
59names should not be altered because in reading the data by phono3py,
60those file names are supposed to be so, though there is a little
61freedom to arrange those file names, for which see :ref:`-o
62<output_filename_option>` and :ref:`-i <input_filename_option>`
63options. After completing calculations for all irreducible grid-point
64indices, the RTA thermal conductivity is computed by another run in a
65short time from the stored data:
66
67::
68
69   % phono3py --dim="2 2 2" --pa="0 1/2 1/2 1/2 0 1/2 1/2 1/2 0" -c POSCAR-unitcell --mesh="19 19 19" --fc3 --fc2 --br --read-gamma
70
71A convenient script
72--------------------
73
74The following short script may be useful to splitting all irreducible
75grid-point indices into of reasonable number of sets of grid-point
76indices for workload distribution.
77
78.. code-block:: python
79
80   #!/usr/bin/env python
81
82   import sys
83   import yaml
84
85   if len(sys.argv) > 1:
86       num = int(sys.argv[1])
87   else:
88       num = 1
89
90   with open("ir_grid_points.yaml") as f:
91       data = yaml.load(f)
92       gps = [gp['grid_point'] for gp in data['ir_grid_points']]
93       gp_lists = [[] for i in range(num)]
94       for i, gp in enumerate(gps):
95           gp_lists[i % num].append(gp)
96       for gp_set in gp_lists:
97           print(",".join(["%d" % gp for gp in gp_set]))
98
99Supposed that this script is saved as ``divide_gps.py``,
100
101::
102
103   % phono3py --dim="2 2 2" --pa="0 1/2 1/2 1/2 0 1/2 1/2 1/2 0" -c POSCAR-unitcell --mesh="19 19 19" --wgp
104   ...
105   % python divide_gps.py 20
106   0,30,52,82,120,402,434,468,524,844,1206
107   1,31,53,83,121,403,435,469,525,845,1207
108   2,32,54,84,122,404,442,470,542,846,1208
109   3,33,55,85,123,405,443,471,543,847,1209
110   4,34,60,86,124,406,444,482,804,848,1226
111   5,35,61,87,125,407,445,483,805,849,1227
112   6,36,62,88,126,408,446,484,806,850,1228
113   7,37,63,89,127,409,447,485,807,851,1229
114   8,40,64,90,140,422,448,486,808,864,1230
115   9,41,65,91,141,423,449,487,809,865,1231
116   20,42,66,100,142,424,450,488,824,866,1246
117   21,43,67,101,143,425,451,489,825,867,1247
118   22,44,68,102,144,426,452,502,826,868,1248
119   23,45,69,103,145,427,453,503,827,869,1249
120   24,46,70,104,160,428,462,504,828,884,1266
121   25,47,71,105,161,429,463,505,829,885,1267
122   26,48,72,106,162,430,464,506,830,886,1608
123   27,49,73,107,163,431,465,507,831,887,1609
124   28,50,80,108,180,432,466,522,832,904,1628
125   29,51,81,109,181,433,467,523,833,905,1629
126
127For example distributing into 20 computer nodes using a queueing
128system,
129
130.. code-block:: shell
131
132   % j=1; for i in `python divide_gps.py 20`;do echo $i; sed -e s/gps/$i/g -e s/num/$j/g job.sh|qsub; j=$((j+1)); done
133
134with ``job.sh`` (here for grid-engine):
135
136.. code-block:: shell
137
138   #$ -S /bin/zsh
139   #$ -cwd
140   #$ -N phono3py-num
141   #$ -pe mpi* 16
142   #$ -e err-phono3py-num.log
143   #$ -o std-phono3py-num.log
144
145   phono3py --dim="2 2 2" --pa="0 1/2 1/2 1/2 0 1/2 1/2 1/2 0" -c POSCAR-unitcell --mesh="19 19 19" --fc3 --fc2 --br --gp="gps" --write-gamma
146