• Home
  • History
  • Annotate
Name Date Size #Lines LOC

..03-May-2022-

.github/H30-Apr-2021-159

process/H30-Apr-2021-2,7191,924

tests/H30-Apr-2021-13690

vendor/H03-May-2022-450,451375,607

.gitignoreH A D30-Apr-202166 119

.travis.ymlH A D30-Apr-20211.1 KiB3327

CHANGES.mdH A D30-Apr-20214.3 KiB8483

LICENSEH A D30-Apr-20211 KiB2217

README.mdH A D30-Apr-202123.5 KiB543417

benchmark.pyH A D30-Apr-20212 KiB6143

cobra_helper.goH A D30-Apr-20214.4 KiB167128

helper.goH A D30-Apr-20212.9 KiB11077

logging.goH A D30-Apr-20211.7 KiB4520

main.goH A D30-Apr-20211.4 KiB364

packaging.shH A D30-Apr-2021411 2115

root.goH A D30-Apr-202119.9 KiB639499

string_repl.goH A D30-Apr-20215 KiB187151

README.md

1# rush -- a cross-platform command-line tool for executing jobs in parallel
2
3[![Build Status](https://travis-ci.org/shenwei356/rush.svg?branch=master)](https://travis-ci.org/shenwei356/rush)
4[![Built with GoLang](https://img.shields.io/badge/powered_by-go-6362c2.svg?style=flat)](https://golang.org)
5[![Go Report Card](https://goreportcard.com/badge/github.com/shenwei356/rush)](https://goreportcard.com/report/github.com/shenwei356/rush)
6[![Cross-platform](https://img.shields.io/badge/platform-any-ec2eb4.svg?style=flat)](#download)
7[![Latest Version](https://img.shields.io/github/release/shenwei356/rush.svg?style=flat?maxAge=86400)](https://github.com/shenwei356/rush/releases)
8[![Github Releases](https://img.shields.io/github/downloads/shenwei356/rush/latest/total.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases)
9
10`rush` is a tool similar to [GNU parallel](https://www.gnu.org/software/parallel/)
11 and [gargs](https://github.com/brentp/gargs).
12 `rush` borrows some idea from them and has some unique features,
13  e.g.,
14  supporting custom defined variables,
15  resuming multi-line commands,
16  more advanced embeded replacement strings.
17
18These features make `rush` suitable for easily and flexibly parallelizing
19complex workflows in fields like Bioinformatics (see [examples](#examples) 18).
20
21
22## Table of Contents
23<!-- START doctoc generated TOC please keep comment here to allow auto update -->
24<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
25
26- [Features](#features)
27- [Performance](#performance)
28- [Installation](#installation)
29- [Usage](#usage)
30- [Examples](#examples)
31- [Special Cases](#special-cases)
32- [Contributors](#contributors)
33- [Acknowledgements](#acknowledgements)
34- [Contact](#contact)
35- [License](#license)
36
37<!-- END doctoc generated TOC please keep comment here to allow auto update -->
38
39
40## Features
41
42Major:
43
44- Supporting Linux, OS X and **Windows** (not CygWin)!
45- **Avoid mixed line from multiple processes without loss of performance**,
46  e.g. the first half of a line is from one process
47  and the last half of the line is from another process.
48  (`--line-buffer` in GNU parallel)
49- **Timeout** (`-t`). (`--timeout` in GNU parallel)
50- **Retry** (`-r`). (`--retry-failed --joblog` in GNU parallel)
51- **Safe exit after capturing Ctrl-C** (*not perfect*, you may stop it by typing ctrl-c or closing terminal)
52- **Continue** (`-c`). (`--resume --joblog` in GNU parallel,
53  ***<s/>sut it does not support multi-line commands, which are common in workflow</s>***)
54- **`awk -v` like custom defined variables** (`-v`). (***Using Shell variable in GNU parallel***)
55- **Keeping output in order of input** (`-k`). (Same `-k/--keep-order` in GNU parallel)
56- **Exit on first error(s)** (`-e`). (*not perfect*, you may stop it by typing ctrl-c or closing terminal) (`--halt 2` in GNU parallel)
57- **Settable record delimiter** (`-D`, default `\n`). (`--recstart` and `--recend` in GNU parallel)
58- **Settable records sending to every command** (`-n`, default `1`). (`-n/--max-args` in GNU parallel)
59- **Settable field delimiter** (`-d`, default `\s+`). (Same `-d/--delimiter` in GNU parallel)
60- **Practical replacement strings** (like GNU parallel):
61    - `{#}`, job ID. (Same in GNU parallel)
62    - `{}`, full data. (Same in GNU parallel)
63    - `{n}`, `n`th field in delimiter-delimited data. (Same in GNU parallel)
64    - Directory and file
65        - `{/}`, dirname. (`{//}` in GNU parallel)
66        - `{%}`, basename. (`{/}` in GNU parallel)
67        - `{.}`, remove the last extension. (Same in GNU parallel)
68        - `{:}`, remove any extension (***Not directly supported in GNU parallel***)
69        - `{^suffix}`, remove `suffix` (***Not directly supported in GNU parallel***)
70        - `{@regexp}`, capture submatch using regular expression (***Not directly supported in GNU parallel***)
71    - Combinations
72        - `{%.}`, `{%:}`, basename without extension
73        - `{2.}`, `{2/}`, `{2%.}`, manipulate `n`th field
74- **Preset variable (macro)**, e.g., `rush -v p={^suffix} 'echo {p}_new_suffix'`,
75where `{p}` is replaced with `{^suffix}`. (***Using Shell variable in GNU parallel***)
76
77Minor:
78
79- Dry run (`--dry-run`). (Same in GNU parallel)
80- Trim input data (`--trim`). (Same in GNU parallel)
81- Verbose output (`--verbose`). (Same in GNU parallel)
82
83[Differences between rush and GNU parallel](https://www.gnu.org/software/parallel/parallel_alternatives.html#DIFFERENCES-BETWEEN-Rush-AND-GNU-Parallel) on GNU parallel site.
84
85## Performance
86
87Performance of `rush` is similar to `gargs`, and they are both slightly faster than `parallel` (Perl) and both slower than `Rust parallel` ([discussion](https://github.com/shenwei356/rush/issues/1)).
88
89Note that speed is not the #.1 target, especially for processes that last long.
90
91
92## Installation
93
94`rush` is implemented in [Go](https://golang.org/) programming language,
95 executable binary files **for most popular operating systems** are freely available
96  in [release](https://github.com/shenwei356/rush/releases) page.
97
98#### Method 1: Download binaries
99
100[rush v0.4.2](https://github.com/shenwei356/rush/releases/tag/v0.4.2)
101[![Github Releases (by Release)](https://img.shields.io/github/downloads/shenwei356/rush/v0.4.2/total.svg)](https://github.com/shenwei356/rush/releases/tag/v0.4.2)
102
103***Tip: run `rush -V` to check update !!!***
104
105OS     |Arch      |File, (中国镜像)                                                                                                                                                                         |Download Count
106:------|:---------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
107Linux  |32-bit    |[rush_linux_386.tar.gz](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_linux_386.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_linux_386.tar.gz))                            |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_linux_386.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_linux_386.tar.gz)
108Linux  |**64-bit**|[**rush_linux_amd64.tar.gz**](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_linux_amd64.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_linux_amd64.tar.gz))                  |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_linux_amd64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_linux_amd64.tar.gz)
109OS X   |32-bit    |[rush_darwin_386.tar.gz](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_darwin_386.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_darwin_386.tar.gz))                         |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_darwin_386.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_darwin_386.tar.gz)
110OS X   |**64-bit**|[**rush_darwin_amd64.tar.gz**](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_darwin_amd64.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_darwin_amd64.tar.gz))               |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_darwin_amd64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_darwin_amd64.tar.gz)
111Windows|32-bit    |[rush_windows_386.exe.tar.gz](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_windows_386.exe.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_windows_386.exe.tar.gz))          |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_windows_386.exe.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_windows_386.exe.tar.gz)
112Windows|**64-bit**|[**rush_windows_amd64.exe.tar.gz**](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_windows_amd64.exe.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_windows_amd64.exe.tar.gz))|[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_windows_amd64.exe.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_windows_amd64.exe.tar.gz)
113
114
115Just [download](https://github.com/shenwei356/rush/releases) compressed
116executable file of your operating system,
117and decompress it with `tar -zxvf *.tar.gz` command or other tools.
118And then:
119
1201. **For Linux-like systems**
121    1. If you have root privilege simply copy it to `/usr/local/bin`:
122
123            sudo cp rush /usr/local/bin/
124
125    1. Or copy to anywhere in the environment variable `PATH`:
126
127            mkdir -p $HOME/bin/; cp rush $HOME/bin/
128
1291. **For windows**, just copy `rush.exe` to `C:\WINDOWS\system32`.
130
131#### Method 2: For Go developer
132
133    go get -u github.com/shenwei356/rush/
134
135
136## Usage
137
138```
139rush -- a cross-platform command-line tool for executing jobs in parallel
140
141Version: 0.4.2
142
143Author: Wei Shen <shenwei356@gmail.com>
144
145Homepage: https://github.com/shenwei356/rush
146
147Usage:
148  rush [flags] [command] [args of command...]
149
150Examples:
151  1. simple run, quoting is not necessary
152      $ seq 1 10 | rush echo {}
153  2. keep order
154      $ seq 1 10 | rush 'echo {}' -k
155  3. timeout
156      $ seq 1 | rush 'sleep 2; echo {}' -t 1
157  4. retry
158      $ seq 1 | rush 'python script.py' -r 3
159  5. dirname & basename & remove suffix
160      $ echo dir/file_1.txt.gz | rush 'echo {/} {%} {^_1.txt.gz}'
161      dir file.txt.gz dir/file
162  6. basename without last or any extension
163      $ echo dir.d/file.txt.gz | rush 'echo {.} {:} {%.} {%:}'
164      dir.d/file.txt dir.d/file file.txt file
165  7. job ID, combine fields and other replacement strings
166      $ echo 12 file.txt dir/s_1.fq.gz | rush 'echo job {#}: {2} {2.} {3%:^_1}'
167      job 1: file.txt file s
168  8. capture submatch using regular expression
169      $ echo read_1.fq.gz | rush 'echo {@(.+)_\d}'
170      read
171  9. custom field delimiter
172      $ echo a=b=c | rush 'echo {1} {2} {3}' -d =
173      a b c
174  10. custom record delimiter
175      $ echo a=b=c | rush -D "=" -k 'echo {}'
176      a
177      b
178      c
179      $ echo abc | rush -D "" -k 'echo {}'
180      a
181      b
182      c
183  11. assign value to variable, like "awk -v"
184      # seq 1 | rush 'echo Hello, {fname} {lname}!' -v fname=Wei,lname=Shen
185      $ seq 1 | rush 'echo Hello, {fname} {lname}!' -v fname=Wei -v lname=Shen
186      Hello, Wei Shen!
187  12. preset variable (Macro)
188      # equal to: echo read_1.fq.gz | rush 'echo {:^_1} {:^_1}_2.fq.gz'
189      $ echo read_1.fq.gz | rush -v p={:^_1} 'echo {p} {p}_2.fq.gz'
190      read read_2.fq.gz
191  13. save successful commands to continue in NEXT run
192      $ seq 1 3 | rush 'sleep {}; echo {}' -c -t 2
193      [INFO] ignore cmd #1: sleep 1; echo 1
194      [ERRO] run cmd #1: sleep 2; echo 2: time out
195      [ERRO] run cmd #2: sleep 3; echo 3: time out
196  14. escape special symbols
197      $ seq 1 | rush 'echo -e "a\tb" | awk "{print $1}"' -q
198      a
199
200  More examples: https://github.com/shenwei356/rush
201
202Flags:
203  -v, --assign strings            assign the value val to the variable var (format: var=val, val also supports replacement strings)
204      --cleanup-time int          time to allow child processes to clean up between stop / kill signals (unit: seconds, 0 for no time) (default 3) (default 3)
205  -c, --continue                  continue jobs. NOTES: 1) successful commands are saved in file (given by flag -C/--succ-cmd-file); 2) if the file does not exist, rush saves data so we can continue jobs next time; 3) if the file exists, rush ignores jobs in it and update the file
206      --dry-run                   print command but not run
207  -q, --escape                    escape special symbols like $ which you can customize by flag -Q/--escape-symbols
208  -Q, --escape-symbols string     symbols to escape (default "$#&`")
209  -d, --field-delimiter string    field delimiter in records, support regular expression (default "\\s+")
210  -h, --help                      help for rush
211      --immediate-output          print output immediately and interleaved, to aid debugging
212  -i, --infile strings            input data file, multi-values supported
213  -j, --jobs int                  run n jobs in parallel (default value depends on your device) (default 16)
214  -k, --keep-order                keep output in order of input
215      --no-kill-exes strings      exe names to exclude from kill signal, example: mspdbsrv.exe; or use all for all exes (default none)
216      --no-stop-exes strings      exe names to exclude from stop signal, example: mspdbsrv.exe; or use all for all exes (default none)
217  -n, --nrecords int              number of records sent to a command (default 1)
218  -o, --out-file string           out file ("-" for stdout) (default "-")
219      --print-retry-output        print output from retry commands (default true)
220      --propagate-exit-status     propagate child exit status up to the exit status of rush (default true)
221  -D, --record-delimiter string   record delimiter (default is "\n") (default "\n")
222  -J, --records-join-sep string   record separator for joining multi-records (default is "\n") (default "\n")
223  -r, --retries int               maximum retries (default 0)
224      --retry-interval int        retry interval (unit: second) (default 0)
225  -e, --stop-on-error             stop child processes on first error (not perfect, you may stop it by typing ctrl-c or closing terminal)
226  -C, --succ-cmd-file string      file for saving successful commands (default "successful_cmds.rush")
227  -t, --timeout int               timeout of a command (unit: seconds, 0 for no timeout) (default 0)
228  -T, --trim string               trim white space (" \t\r\n") in input (available values: "l" for left, "r" for right, "lr", "rl", "b" for both side)
229      --verbose                   print verbose information
230  -V, --version                   print version information and check for update
231```
232
233
234## Examples
235
2361. Simple run, quoting is not necessary
237
238        # seq 1 3 | rush 'echo {}'
239        $ seq 1 3 | rush echo {}
240        3
241        1
242        2
243
2441. Read data from file (`-i`)
245
246        $ rush echo {} -i data1.txt -i data2.txt
247
2481. Keep output order (`-k`)
249
250        $ seq 1 3 | rush 'echo {}' -k
251        1
252        2
253        3
254
2551. Timeout (`-t`)
256
257        $ time seq 1 | rush 'sleep 2; echo {}' -t 1
258        [ERRO] run command #1: sleep 2; echo 1: time out
259
260        real    0m1.010s
261        user    0m0.005s
262        sys     0m0.007s
263
2641. Retry (`-r`)
265
266        $ seq 1 | rush 'python unexisted_script.py' -r 1
267        python: can't open file 'unexisted_script.py': [Errno 2] No such file or directory
268        [WARN] wait command: python unexisted_script.py: exit status 2
269        python: can't open file 'unexisted_script.py': [Errno 2] No such file or directory
270        [ERRO] wait command: python unexisted_script.py: exit status 2
271
2721. Dirname (`{/}`) and basename (`{%}`) and remove custom suffix (`{^suffix}`)
273
274        $ echo dir/file_1.txt.gz | rush 'echo {/} {%} {^_1.txt.gz}'
275        dir file_1.txt.gz dir/file
276
2771. Get basename, and remove last (`{.}`) or any (`{:}`) extension
278
279        $ echo dir.d/file.txt.gz | rush 'echo {.} {:} {%.} {%:}'
280        dir.d/file.txt dir.d/file file.txt file
281
2821. Job ID, combine fields index and other replacement strings
283
284        $ echo 12 file.txt dir/s_1.fq.gz | rush 'echo job {#}: {2} {2.} {3%:^_1}'
285        job 1: file.txt file s
286
2871. Capture submatch using regular expression (`{@regexp}`)
288
289        $ echo read_1.fq.gz | rush 'echo {@(.+)_\d}'
290
2911. Custom field delimiter (`-d`)
292
293        $ echo a=b=c | rush 'echo {1} {2} {3}' -d =
294        a b c
295
2961. Send multi-lines to every command (`-n`)
297
298        $ seq 5 | rush -n 2 -k 'echo "{}"; echo'
299        1
300        2
301
302        3
303        4
304
305        5
306
307        # Multiple records are joined with separator `"\n"` (`-J/--records-join-sep`)
308        $ seq 5 | rush -n 2 -k 'echo "{}"; echo' -J ' '
309        1 2
310
311        3 4
312
313        5
314
315        $ seq 5 | rush -n 2 -k -j 3 'echo {1}'
316        1
317        3
318        5
319
3201. Custom record delimiter (`-D`), note that empty records are not used.
321
322        $ echo a b c d | rush -D " " -k 'echo {}'
323        a
324        b
325        c
326        d
327
328        $ echo abcd | rush -D "" -k 'echo {}'
329        a
330        b
331        c
332        d
333
334        # FASTA format
335        $ echo -ne ">seq1\nactg\n>seq2\nAAAA\n>seq3\nCCCC"
336        >seq1
337        actg
338        >seq2
339        AAAA
340        >seq3
341        CCCC
342
343        $ echo -ne ">seq1\nactg\n>seq2\nAAAA\n>seq3\nCCCC" | rush -D ">" 'echo FASTA record {#}: name: {1} sequence: {2}' -k -d "\n"
344        FASTA record 1: name: seq1 sequence: actg
345        FASTA record 2: name: seq2 sequence: AAAA
346        FASTA record 3: name: seq3 sequence: CCCC
347
3481. Assign value to variable, like `awk -v` (`-v`)
349
350        $ seq 1  | rush 'echo Hello, {fname} {lname}!' -v fname=Wei -v lname=Shen
351        Hello, Wei Shen!
352
353        $ seq 1  | rush 'echo Hello, {fname} {lname}!' -v fname=Wei,lname=Shen
354        Hello, Wei Shen!
355
356        $ for var in a b; do \
357        $   seq 1 3 | rush -k -v var=$var 'echo var: {var}, data: {}'; \
358        $ done
359        var: a, data: 1
360        var: a, data: 2
361        var: a, data: 3
362        var: b, data: 1
363        var: b, data: 2
364        var: b, data: 3
365
3661. **Preset variable** (`-v`), avoid repeatedly writing verbose replacement strings
367
368        # naive way
369        $ echo read_1.fq.gz | rush 'echo {:^_1} {:^_1}_2.fq.gz'
370        read read_2.fq.gz
371
372        # macro + removing suffix
373        $ echo read_1.fq.gz | rush -v p='{:^_1}' 'echo {p} {p}_2.fq.gz'
374
375        # macro + regular expression
376        $ echo read_1.fq.gz | rush -v p='{@(.+?)_\d}' 'echo {p} {p}_2.fq.gz'
377
3781. Escape special symbols
379
380        $ seq 1 | rush 'echo "I have $100"'
381        I have 00
382        $ seq 1 | rush 'echo "I have $100"' -q
383        I have $100
384        $ seq 1 | rush 'echo "I have $100"' -q --dry-run
385        echo "I have \$100"
386
387        $ seq 1 | rush 'echo -e "a\tb" | awk "{print $1}"'
388        a       b
389
390        $ seq 1 | rush 'echo -e "a\tb" | awk "{print $1}"' -q
391        a
392
3931. Interrupt jobs by `Ctrl-C`, rush will stop unfinished commands and exit.
394
395        $ seq 1 20 | rush 'sleep 1; echo {}'
396        ^C[CRIT] received an interrupt, stopping unfinished commands...
397        [ERRO] wait cmd #7: sleep 1; echo 7: signal: interrupt
398        [ERRO] wait cmd #5: sleep 1; echo 5: signal: killed
399        [ERRO] wait cmd #6: sleep 1; echo 6: signal: killed
400        [ERRO] wait cmd #8: sleep 1; echo 8: signal: killed
401        [ERRO] wait cmd #9: sleep 1; echo 9: signal: killed
402        1
403        3
404        4
405        2
406
4071. Continue/resume jobs (`-c`). When some jobs failed (by execution failure, timeout,
408    or cancelling by user with `Ctrl + C`),
409    please switch flag `-c/--continue` on and run again,
410    so that `rush` can save successful commands and ignore them in **NEXT** run.
411
412        $ seq 1 3 | rush 'sleep {}; echo {}' -t 3 -c
413        1
414        2
415        [ERRO] run cmd #3: sleep 3; echo 3: time out
416
417        # successful commands:
418        $ cat successful_cmds.rush
419        sleep 1; echo 1__CMD__
420        sleep 2; echo 2__CMD__
421
422        # run again
423        $ seq 1 3 | rush 'sleep {}; echo {}' -t 3 -c
424        [INFO] ignore cmd #1: sleep 1; echo 1
425        [INFO] ignore cmd #2: sleep 2; echo 2
426        [ERRO] run cmd #1: sleep 3; echo 3: time out
427
428    Commands of multi-lines (***Not supported in GNU parallel***)
429
430        $ seq 1 3 | rush 'sleep {}; echo {}; \
431        echo finish {}' -t 3 -c -C finished.rush
432        1
433        finish 1
434        2
435        finish 2
436        [ERRO] run cmd #3: sleep 3; echo 3; \
437        echo finish 3: time out
438
439        $ cat finished.rush
440        sleep 1; echo 1; \
441        echo finish 1__CMD__
442        sleep 2; echo 2; \
443        echo finish 2__CMD__
444
445        # run again
446        $ seq 1 3 | rush 'sleep {}; echo {}; \
447        echo finish {}' -t 3 -c -C finished.rush
448        [INFO] ignore cmd #1: sleep 1; echo 1; \
449        echo finish 1
450        [INFO] ignore cmd #2: sleep 2; echo 2; \
451        echo finish 2
452        [ERRO] run cmd #1: sleep 3; echo 3; \
453        echo finish 3: time out
454
455    Commands are saved to file (`-C`) right after it finished, so we can view
456    the check finished jobs:
457
458        grep -c __CMD__ successful_cmds.rush
459
4601. A comprehensive example: downloading 1K+ pages given by three URL list files
461   using `phantomjs save_page.js` (some page contents are dynamicly generated by Javascript,
462   so `wget` does not work). Here I set max jobs number (`-j`) as `20`,
463   each job has a max running time (`-t`) of `60` seconds and `3` retry changes
464   (`-r`). Continue flag `-c` is also switched on, so we can continue unfinished
465   jobs. Luckily, it's accomplished in one run :smile:
466
467        $ for f in $(seq 2014 2016); do \
468        $    /bin/rm -rf $f; mkdir -p $f; \
469        $    cat $f.html.txt | rush -v d=$f -d = 'phantomjs save_page.js "{}" > {d}/{3}.html' -j 20 -t 60 -r 3 -c; \
470        $ done
471
4721. A bioinformatics example: mapping with `bwa`, and processing result with `samtools`:
473
474        $ tree raw.cluster.clean.mapping
475        raw.cluster.clean.mapping
476        ├── M1
477        │   ├── M1_1.fq.gz -> ../../raw.cluster.clean/M1/M1_1.fq.gz
478        │   ├── M1_2.fq.gz -> ../../raw.cluster.clean/M1/M1_2.fq.gz
479        ...
480
481        $ ref=ref/xxx.fa
482        $ threads=25
483        $ ls -d raw.cluster.clean.mapping/* \
484            | rush -v ref=$ref -v j=$threads \
485                'bwa mem -t {j} -M -a {ref} {}/{%}_1.fq.gz {}/{%}_2.fq.gz > {}/{%}.sam; \
486                samtools view -bS {}/{%}.sam > {}/{%}.bam; \
487                samtools sort -T {}/{%}.tmp -@ {j} {}/{%}.bam -o {}/{%}.sorted.bam; \
488                samtools index {}/{%}.sorted.bam; \
489                samtools flagstat {}/{%}.sorted.bam > {}/{%}.sorted.bam.flagstat; \
490                /bin/rm {}/{%}.bam {}/{%}.sam;' \
491                -j 2 --verbose -c -C mapping.rush
492
493    Since `{}/{%}` appears many times, we can use preset variable (macro) to
494    simplify it:
495
496        $ ls -d raw.cluster.clean.mapping/* \
497            | rush -v ref=$ref -v j=$threads -v p='{}/{%}' \
498                'bwa mem -t {j} -M -a {ref} {p}_1.fq.gz {p}_2.fq.gz > {p}.sam; \
499                samtools view -bS {p}.sam > {p}.bam; \
500                samtools sort -T {p}.tmp -@ {j} {p}.bam -o {p}.sorted.bam; \
501                samtools index {p}.sorted.bam; \
502                samtools flagstat {p}.sorted.bam > {p}.sorted.bam.flagstat; \
503                /bin/rm {p}.bam {p}.sam;' \
504                -j 2 --verbose -c -C mapping.rush
505
506
507## Special Cases
508
509- Shell `grep` returns exit code `1` when no matches found.
510`rush` thinks it failed to run.
511 Please use `grep foo bar || true` instead of `grep foo bar`.
512
513        $ seq 1 | rush 'echo abc | grep 123'
514        [ERRO] wait cmd #1: echo abc | grep 123: exit status 1
515        $ seq 1 | rush 'echo abc | grep 123 || true'
516
517## Contributors
518
519- [Wei Shen](https://github.com/shenwei356)
520- [Brian Burgin](https://github.com/bburgin)
521
522## Acknowledgements
523
524Specially thank [@brentp](https://github.com/brentp)
525and his [gargs](https://github.com/brentp/gargs), from which `rush` borrows
526some ideas.
527
528Thank [@bburgin](https://github.com/bburgin) for his contribution on improvement
529of child process management.
530
531## Contact
532
533[Create an issue](https://github.com/shenwei356/rush/issues) to report bugs,
534propose new functions or ask for help.
535
536## License
537
538[MIT License](https://github.com/shenwei356/rush/blob/master/LICENSE)
539
540## Starchart
541
542<img src="https://starchart.cc/shenwei356/rush.svg" alt="Stargazers over time" style="max-width: 100%">
543