README.md
1# rush -- a cross-platform command-line tool for executing jobs in parallel
2
3[![Build Status](https://travis-ci.org/shenwei356/rush.svg?branch=master)](https://travis-ci.org/shenwei356/rush)
4[![Built with GoLang](https://img.shields.io/badge/powered_by-go-6362c2.svg?style=flat)](https://golang.org)
5[![Go Report Card](https://goreportcard.com/badge/github.com/shenwei356/rush)](https://goreportcard.com/report/github.com/shenwei356/rush)
6[![Cross-platform](https://img.shields.io/badge/platform-any-ec2eb4.svg?style=flat)](#download)
7[![Latest Version](https://img.shields.io/github/release/shenwei356/rush.svg?style=flat?maxAge=86400)](https://github.com/shenwei356/rush/releases)
8[![Github Releases](https://img.shields.io/github/downloads/shenwei356/rush/latest/total.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases)
9
10`rush` is a tool similar to [GNU parallel](https://www.gnu.org/software/parallel/)
11 and [gargs](https://github.com/brentp/gargs).
12 `rush` borrows some idea from them and has some unique features,
13 e.g.,
14 supporting custom defined variables,
15 resuming multi-line commands,
16 more advanced embeded replacement strings.
17
18These features make `rush` suitable for easily and flexibly parallelizing
19complex workflows in fields like Bioinformatics (see [examples](#examples) 18).
20
21
22## Table of Contents
23<!-- START doctoc generated TOC please keep comment here to allow auto update -->
24<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
25
26- [Features](#features)
27- [Performance](#performance)
28- [Installation](#installation)
29- [Usage](#usage)
30- [Examples](#examples)
31- [Special Cases](#special-cases)
32- [Contributors](#contributors)
33- [Acknowledgements](#acknowledgements)
34- [Contact](#contact)
35- [License](#license)
36
37<!-- END doctoc generated TOC please keep comment here to allow auto update -->
38
39
40## Features
41
42Major:
43
44- Supporting Linux, OS X and **Windows** (not CygWin)!
45- **Avoid mixed line from multiple processes without loss of performance**,
46 e.g. the first half of a line is from one process
47 and the last half of the line is from another process.
48 (`--line-buffer` in GNU parallel)
49- **Timeout** (`-t`). (`--timeout` in GNU parallel)
50- **Retry** (`-r`). (`--retry-failed --joblog` in GNU parallel)
51- **Safe exit after capturing Ctrl-C** (*not perfect*, you may stop it by typing ctrl-c or closing terminal)
52- **Continue** (`-c`). (`--resume --joblog` in GNU parallel,
53 ***<s/>sut it does not support multi-line commands, which are common in workflow</s>***)
54- **`awk -v` like custom defined variables** (`-v`). (***Using Shell variable in GNU parallel***)
55- **Keeping output in order of input** (`-k`). (Same `-k/--keep-order` in GNU parallel)
56- **Exit on first error(s)** (`-e`). (*not perfect*, you may stop it by typing ctrl-c or closing terminal) (`--halt 2` in GNU parallel)
57- **Settable record delimiter** (`-D`, default `\n`). (`--recstart` and `--recend` in GNU parallel)
58- **Settable records sending to every command** (`-n`, default `1`). (`-n/--max-args` in GNU parallel)
59- **Settable field delimiter** (`-d`, default `\s+`). (Same `-d/--delimiter` in GNU parallel)
60- **Practical replacement strings** (like GNU parallel):
61 - `{#}`, job ID. (Same in GNU parallel)
62 - `{}`, full data. (Same in GNU parallel)
63 - `{n}`, `n`th field in delimiter-delimited data. (Same in GNU parallel)
64 - Directory and file
65 - `{/}`, dirname. (`{//}` in GNU parallel)
66 - `{%}`, basename. (`{/}` in GNU parallel)
67 - `{.}`, remove the last extension. (Same in GNU parallel)
68 - `{:}`, remove any extension (***Not directly supported in GNU parallel***)
69 - `{^suffix}`, remove `suffix` (***Not directly supported in GNU parallel***)
70 - `{@regexp}`, capture submatch using regular expression (***Not directly supported in GNU parallel***)
71 - Combinations
72 - `{%.}`, `{%:}`, basename without extension
73 - `{2.}`, `{2/}`, `{2%.}`, manipulate `n`th field
74- **Preset variable (macro)**, e.g., `rush -v p={^suffix} 'echo {p}_new_suffix'`,
75where `{p}` is replaced with `{^suffix}`. (***Using Shell variable in GNU parallel***)
76
77Minor:
78
79- Dry run (`--dry-run`). (Same in GNU parallel)
80- Trim input data (`--trim`). (Same in GNU parallel)
81- Verbose output (`--verbose`). (Same in GNU parallel)
82
83[Differences between rush and GNU parallel](https://www.gnu.org/software/parallel/parallel_alternatives.html#DIFFERENCES-BETWEEN-Rush-AND-GNU-Parallel) on GNU parallel site.
84
85## Performance
86
87Performance of `rush` is similar to `gargs`, and they are both slightly faster than `parallel` (Perl) and both slower than `Rust parallel` ([discussion](https://github.com/shenwei356/rush/issues/1)).
88
89Note that speed is not the #.1 target, especially for processes that last long.
90
91
92## Installation
93
94`rush` is implemented in [Go](https://golang.org/) programming language,
95 executable binary files **for most popular operating systems** are freely available
96 in [release](https://github.com/shenwei356/rush/releases) page.
97
98#### Method 1: Download binaries
99
100[rush v0.4.2](https://github.com/shenwei356/rush/releases/tag/v0.4.2)
101[![Github Releases (by Release)](https://img.shields.io/github/downloads/shenwei356/rush/v0.4.2/total.svg)](https://github.com/shenwei356/rush/releases/tag/v0.4.2)
102
103***Tip: run `rush -V` to check update !!!***
104
105OS |Arch |File, (中国镜像) |Download Count
106:------|:---------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
107Linux |32-bit |[rush_linux_386.tar.gz](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_linux_386.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_linux_386.tar.gz)) |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_linux_386.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_linux_386.tar.gz)
108Linux |**64-bit**|[**rush_linux_amd64.tar.gz**](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_linux_amd64.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_linux_amd64.tar.gz)) |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_linux_amd64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_linux_amd64.tar.gz)
109OS X |32-bit |[rush_darwin_386.tar.gz](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_darwin_386.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_darwin_386.tar.gz)) |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_darwin_386.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_darwin_386.tar.gz)
110OS X |**64-bit**|[**rush_darwin_amd64.tar.gz**](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_darwin_amd64.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_darwin_amd64.tar.gz)) |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_darwin_amd64.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_darwin_amd64.tar.gz)
111Windows|32-bit |[rush_windows_386.exe.tar.gz](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_windows_386.exe.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_windows_386.exe.tar.gz)) |[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_windows_386.exe.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_windows_386.exe.tar.gz)
112Windows|**64-bit**|[**rush_windows_amd64.exe.tar.gz**](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_windows_amd64.exe.tar.gz), ([mirror](http://app.shenwei.me/data/rush/rush_windows_amd64.exe.tar.gz))|[![Github Releases (by Asset)](https://img.shields.io/github/downloads/shenwei356/rush/latest/rush_windows_amd64.exe.tar.gz.svg?maxAge=3600)](https://github.com/shenwei356/rush/releases/download/v0.4.2/rush_windows_amd64.exe.tar.gz)
113
114
115Just [download](https://github.com/shenwei356/rush/releases) compressed
116executable file of your operating system,
117and decompress it with `tar -zxvf *.tar.gz` command or other tools.
118And then:
119
1201. **For Linux-like systems**
121 1. If you have root privilege simply copy it to `/usr/local/bin`:
122
123 sudo cp rush /usr/local/bin/
124
125 1. Or copy to anywhere in the environment variable `PATH`:
126
127 mkdir -p $HOME/bin/; cp rush $HOME/bin/
128
1291. **For windows**, just copy `rush.exe` to `C:\WINDOWS\system32`.
130
131#### Method 2: For Go developer
132
133 go get -u github.com/shenwei356/rush/
134
135
136## Usage
137
138```
139rush -- a cross-platform command-line tool for executing jobs in parallel
140
141Version: 0.4.2
142
143Author: Wei Shen <shenwei356@gmail.com>
144
145Homepage: https://github.com/shenwei356/rush
146
147Usage:
148 rush [flags] [command] [args of command...]
149
150Examples:
151 1. simple run, quoting is not necessary
152 $ seq 1 10 | rush echo {}
153 2. keep order
154 $ seq 1 10 | rush 'echo {}' -k
155 3. timeout
156 $ seq 1 | rush 'sleep 2; echo {}' -t 1
157 4. retry
158 $ seq 1 | rush 'python script.py' -r 3
159 5. dirname & basename & remove suffix
160 $ echo dir/file_1.txt.gz | rush 'echo {/} {%} {^_1.txt.gz}'
161 dir file.txt.gz dir/file
162 6. basename without last or any extension
163 $ echo dir.d/file.txt.gz | rush 'echo {.} {:} {%.} {%:}'
164 dir.d/file.txt dir.d/file file.txt file
165 7. job ID, combine fields and other replacement strings
166 $ echo 12 file.txt dir/s_1.fq.gz | rush 'echo job {#}: {2} {2.} {3%:^_1}'
167 job 1: file.txt file s
168 8. capture submatch using regular expression
169 $ echo read_1.fq.gz | rush 'echo {@(.+)_\d}'
170 read
171 9. custom field delimiter
172 $ echo a=b=c | rush 'echo {1} {2} {3}' -d =
173 a b c
174 10. custom record delimiter
175 $ echo a=b=c | rush -D "=" -k 'echo {}'
176 a
177 b
178 c
179 $ echo abc | rush -D "" -k 'echo {}'
180 a
181 b
182 c
183 11. assign value to variable, like "awk -v"
184 # seq 1 | rush 'echo Hello, {fname} {lname}!' -v fname=Wei,lname=Shen
185 $ seq 1 | rush 'echo Hello, {fname} {lname}!' -v fname=Wei -v lname=Shen
186 Hello, Wei Shen!
187 12. preset variable (Macro)
188 # equal to: echo read_1.fq.gz | rush 'echo {:^_1} {:^_1}_2.fq.gz'
189 $ echo read_1.fq.gz | rush -v p={:^_1} 'echo {p} {p}_2.fq.gz'
190 read read_2.fq.gz
191 13. save successful commands to continue in NEXT run
192 $ seq 1 3 | rush 'sleep {}; echo {}' -c -t 2
193 [INFO] ignore cmd #1: sleep 1; echo 1
194 [ERRO] run cmd #1: sleep 2; echo 2: time out
195 [ERRO] run cmd #2: sleep 3; echo 3: time out
196 14. escape special symbols
197 $ seq 1 | rush 'echo -e "a\tb" | awk "{print $1}"' -q
198 a
199
200 More examples: https://github.com/shenwei356/rush
201
202Flags:
203 -v, --assign strings assign the value val to the variable var (format: var=val, val also supports replacement strings)
204 --cleanup-time int time to allow child processes to clean up between stop / kill signals (unit: seconds, 0 for no time) (default 3) (default 3)
205 -c, --continue continue jobs. NOTES: 1) successful commands are saved in file (given by flag -C/--succ-cmd-file); 2) if the file does not exist, rush saves data so we can continue jobs next time; 3) if the file exists, rush ignores jobs in it and update the file
206 --dry-run print command but not run
207 -q, --escape escape special symbols like $ which you can customize by flag -Q/--escape-symbols
208 -Q, --escape-symbols string symbols to escape (default "$#&`")
209 -d, --field-delimiter string field delimiter in records, support regular expression (default "\\s+")
210 -h, --help help for rush
211 --immediate-output print output immediately and interleaved, to aid debugging
212 -i, --infile strings input data file, multi-values supported
213 -j, --jobs int run n jobs in parallel (default value depends on your device) (default 16)
214 -k, --keep-order keep output in order of input
215 --no-kill-exes strings exe names to exclude from kill signal, example: mspdbsrv.exe; or use all for all exes (default none)
216 --no-stop-exes strings exe names to exclude from stop signal, example: mspdbsrv.exe; or use all for all exes (default none)
217 -n, --nrecords int number of records sent to a command (default 1)
218 -o, --out-file string out file ("-" for stdout) (default "-")
219 --print-retry-output print output from retry commands (default true)
220 --propagate-exit-status propagate child exit status up to the exit status of rush (default true)
221 -D, --record-delimiter string record delimiter (default is "\n") (default "\n")
222 -J, --records-join-sep string record separator for joining multi-records (default is "\n") (default "\n")
223 -r, --retries int maximum retries (default 0)
224 --retry-interval int retry interval (unit: second) (default 0)
225 -e, --stop-on-error stop child processes on first error (not perfect, you may stop it by typing ctrl-c or closing terminal)
226 -C, --succ-cmd-file string file for saving successful commands (default "successful_cmds.rush")
227 -t, --timeout int timeout of a command (unit: seconds, 0 for no timeout) (default 0)
228 -T, --trim string trim white space (" \t\r\n") in input (available values: "l" for left, "r" for right, "lr", "rl", "b" for both side)
229 --verbose print verbose information
230 -V, --version print version information and check for update
231```
232
233
234## Examples
235
2361. Simple run, quoting is not necessary
237
238 # seq 1 3 | rush 'echo {}'
239 $ seq 1 3 | rush echo {}
240 3
241 1
242 2
243
2441. Read data from file (`-i`)
245
246 $ rush echo {} -i data1.txt -i data2.txt
247
2481. Keep output order (`-k`)
249
250 $ seq 1 3 | rush 'echo {}' -k
251 1
252 2
253 3
254
2551. Timeout (`-t`)
256
257 $ time seq 1 | rush 'sleep 2; echo {}' -t 1
258 [ERRO] run command #1: sleep 2; echo 1: time out
259
260 real 0m1.010s
261 user 0m0.005s
262 sys 0m0.007s
263
2641. Retry (`-r`)
265
266 $ seq 1 | rush 'python unexisted_script.py' -r 1
267 python: can't open file 'unexisted_script.py': [Errno 2] No such file or directory
268 [WARN] wait command: python unexisted_script.py: exit status 2
269 python: can't open file 'unexisted_script.py': [Errno 2] No such file or directory
270 [ERRO] wait command: python unexisted_script.py: exit status 2
271
2721. Dirname (`{/}`) and basename (`{%}`) and remove custom suffix (`{^suffix}`)
273
274 $ echo dir/file_1.txt.gz | rush 'echo {/} {%} {^_1.txt.gz}'
275 dir file_1.txt.gz dir/file
276
2771. Get basename, and remove last (`{.}`) or any (`{:}`) extension
278
279 $ echo dir.d/file.txt.gz | rush 'echo {.} {:} {%.} {%:}'
280 dir.d/file.txt dir.d/file file.txt file
281
2821. Job ID, combine fields index and other replacement strings
283
284 $ echo 12 file.txt dir/s_1.fq.gz | rush 'echo job {#}: {2} {2.} {3%:^_1}'
285 job 1: file.txt file s
286
2871. Capture submatch using regular expression (`{@regexp}`)
288
289 $ echo read_1.fq.gz | rush 'echo {@(.+)_\d}'
290
2911. Custom field delimiter (`-d`)
292
293 $ echo a=b=c | rush 'echo {1} {2} {3}' -d =
294 a b c
295
2961. Send multi-lines to every command (`-n`)
297
298 $ seq 5 | rush -n 2 -k 'echo "{}"; echo'
299 1
300 2
301
302 3
303 4
304
305 5
306
307 # Multiple records are joined with separator `"\n"` (`-J/--records-join-sep`)
308 $ seq 5 | rush -n 2 -k 'echo "{}"; echo' -J ' '
309 1 2
310
311 3 4
312
313 5
314
315 $ seq 5 | rush -n 2 -k -j 3 'echo {1}'
316 1
317 3
318 5
319
3201. Custom record delimiter (`-D`), note that empty records are not used.
321
322 $ echo a b c d | rush -D " " -k 'echo {}'
323 a
324 b
325 c
326 d
327
328 $ echo abcd | rush -D "" -k 'echo {}'
329 a
330 b
331 c
332 d
333
334 # FASTA format
335 $ echo -ne ">seq1\nactg\n>seq2\nAAAA\n>seq3\nCCCC"
336 >seq1
337 actg
338 >seq2
339 AAAA
340 >seq3
341 CCCC
342
343 $ echo -ne ">seq1\nactg\n>seq2\nAAAA\n>seq3\nCCCC" | rush -D ">" 'echo FASTA record {#}: name: {1} sequence: {2}' -k -d "\n"
344 FASTA record 1: name: seq1 sequence: actg
345 FASTA record 2: name: seq2 sequence: AAAA
346 FASTA record 3: name: seq3 sequence: CCCC
347
3481. Assign value to variable, like `awk -v` (`-v`)
349
350 $ seq 1 | rush 'echo Hello, {fname} {lname}!' -v fname=Wei -v lname=Shen
351 Hello, Wei Shen!
352
353 $ seq 1 | rush 'echo Hello, {fname} {lname}!' -v fname=Wei,lname=Shen
354 Hello, Wei Shen!
355
356 $ for var in a b; do \
357 $ seq 1 3 | rush -k -v var=$var 'echo var: {var}, data: {}'; \
358 $ done
359 var: a, data: 1
360 var: a, data: 2
361 var: a, data: 3
362 var: b, data: 1
363 var: b, data: 2
364 var: b, data: 3
365
3661. **Preset variable** (`-v`), avoid repeatedly writing verbose replacement strings
367
368 # naive way
369 $ echo read_1.fq.gz | rush 'echo {:^_1} {:^_1}_2.fq.gz'
370 read read_2.fq.gz
371
372 # macro + removing suffix
373 $ echo read_1.fq.gz | rush -v p='{:^_1}' 'echo {p} {p}_2.fq.gz'
374
375 # macro + regular expression
376 $ echo read_1.fq.gz | rush -v p='{@(.+?)_\d}' 'echo {p} {p}_2.fq.gz'
377
3781. Escape special symbols
379
380 $ seq 1 | rush 'echo "I have $100"'
381 I have 00
382 $ seq 1 | rush 'echo "I have $100"' -q
383 I have $100
384 $ seq 1 | rush 'echo "I have $100"' -q --dry-run
385 echo "I have \$100"
386
387 $ seq 1 | rush 'echo -e "a\tb" | awk "{print $1}"'
388 a b
389
390 $ seq 1 | rush 'echo -e "a\tb" | awk "{print $1}"' -q
391 a
392
3931. Interrupt jobs by `Ctrl-C`, rush will stop unfinished commands and exit.
394
395 $ seq 1 20 | rush 'sleep 1; echo {}'
396 ^C[CRIT] received an interrupt, stopping unfinished commands...
397 [ERRO] wait cmd #7: sleep 1; echo 7: signal: interrupt
398 [ERRO] wait cmd #5: sleep 1; echo 5: signal: killed
399 [ERRO] wait cmd #6: sleep 1; echo 6: signal: killed
400 [ERRO] wait cmd #8: sleep 1; echo 8: signal: killed
401 [ERRO] wait cmd #9: sleep 1; echo 9: signal: killed
402 1
403 3
404 4
405 2
406
4071. Continue/resume jobs (`-c`). When some jobs failed (by execution failure, timeout,
408 or cancelling by user with `Ctrl + C`),
409 please switch flag `-c/--continue` on and run again,
410 so that `rush` can save successful commands and ignore them in **NEXT** run.
411
412 $ seq 1 3 | rush 'sleep {}; echo {}' -t 3 -c
413 1
414 2
415 [ERRO] run cmd #3: sleep 3; echo 3: time out
416
417 # successful commands:
418 $ cat successful_cmds.rush
419 sleep 1; echo 1__CMD__
420 sleep 2; echo 2__CMD__
421
422 # run again
423 $ seq 1 3 | rush 'sleep {}; echo {}' -t 3 -c
424 [INFO] ignore cmd #1: sleep 1; echo 1
425 [INFO] ignore cmd #2: sleep 2; echo 2
426 [ERRO] run cmd #1: sleep 3; echo 3: time out
427
428 Commands of multi-lines (***Not supported in GNU parallel***)
429
430 $ seq 1 3 | rush 'sleep {}; echo {}; \
431 echo finish {}' -t 3 -c -C finished.rush
432 1
433 finish 1
434 2
435 finish 2
436 [ERRO] run cmd #3: sleep 3; echo 3; \
437 echo finish 3: time out
438
439 $ cat finished.rush
440 sleep 1; echo 1; \
441 echo finish 1__CMD__
442 sleep 2; echo 2; \
443 echo finish 2__CMD__
444
445 # run again
446 $ seq 1 3 | rush 'sleep {}; echo {}; \
447 echo finish {}' -t 3 -c -C finished.rush
448 [INFO] ignore cmd #1: sleep 1; echo 1; \
449 echo finish 1
450 [INFO] ignore cmd #2: sleep 2; echo 2; \
451 echo finish 2
452 [ERRO] run cmd #1: sleep 3; echo 3; \
453 echo finish 3: time out
454
455 Commands are saved to file (`-C`) right after it finished, so we can view
456 the check finished jobs:
457
458 grep -c __CMD__ successful_cmds.rush
459
4601. A comprehensive example: downloading 1K+ pages given by three URL list files
461 using `phantomjs save_page.js` (some page contents are dynamicly generated by Javascript,
462 so `wget` does not work). Here I set max jobs number (`-j`) as `20`,
463 each job has a max running time (`-t`) of `60` seconds and `3` retry changes
464 (`-r`). Continue flag `-c` is also switched on, so we can continue unfinished
465 jobs. Luckily, it's accomplished in one run :smile:
466
467 $ for f in $(seq 2014 2016); do \
468 $ /bin/rm -rf $f; mkdir -p $f; \
469 $ cat $f.html.txt | rush -v d=$f -d = 'phantomjs save_page.js "{}" > {d}/{3}.html' -j 20 -t 60 -r 3 -c; \
470 $ done
471
4721. A bioinformatics example: mapping with `bwa`, and processing result with `samtools`:
473
474 $ tree raw.cluster.clean.mapping
475 raw.cluster.clean.mapping
476 ├── M1
477 │ ├── M1_1.fq.gz -> ../../raw.cluster.clean/M1/M1_1.fq.gz
478 │ ├── M1_2.fq.gz -> ../../raw.cluster.clean/M1/M1_2.fq.gz
479 ...
480
481 $ ref=ref/xxx.fa
482 $ threads=25
483 $ ls -d raw.cluster.clean.mapping/* \
484 | rush -v ref=$ref -v j=$threads \
485 'bwa mem -t {j} -M -a {ref} {}/{%}_1.fq.gz {}/{%}_2.fq.gz > {}/{%}.sam; \
486 samtools view -bS {}/{%}.sam > {}/{%}.bam; \
487 samtools sort -T {}/{%}.tmp -@ {j} {}/{%}.bam -o {}/{%}.sorted.bam; \
488 samtools index {}/{%}.sorted.bam; \
489 samtools flagstat {}/{%}.sorted.bam > {}/{%}.sorted.bam.flagstat; \
490 /bin/rm {}/{%}.bam {}/{%}.sam;' \
491 -j 2 --verbose -c -C mapping.rush
492
493 Since `{}/{%}` appears many times, we can use preset variable (macro) to
494 simplify it:
495
496 $ ls -d raw.cluster.clean.mapping/* \
497 | rush -v ref=$ref -v j=$threads -v p='{}/{%}' \
498 'bwa mem -t {j} -M -a {ref} {p}_1.fq.gz {p}_2.fq.gz > {p}.sam; \
499 samtools view -bS {p}.sam > {p}.bam; \
500 samtools sort -T {p}.tmp -@ {j} {p}.bam -o {p}.sorted.bam; \
501 samtools index {p}.sorted.bam; \
502 samtools flagstat {p}.sorted.bam > {p}.sorted.bam.flagstat; \
503 /bin/rm {p}.bam {p}.sam;' \
504 -j 2 --verbose -c -C mapping.rush
505
506
507## Special Cases
508
509- Shell `grep` returns exit code `1` when no matches found.
510`rush` thinks it failed to run.
511 Please use `grep foo bar || true` instead of `grep foo bar`.
512
513 $ seq 1 | rush 'echo abc | grep 123'
514 [ERRO] wait cmd #1: echo abc | grep 123: exit status 1
515 $ seq 1 | rush 'echo abc | grep 123 || true'
516
517## Contributors
518
519- [Wei Shen](https://github.com/shenwei356)
520- [Brian Burgin](https://github.com/bburgin)
521
522## Acknowledgements
523
524Specially thank [@brentp](https://github.com/brentp)
525and his [gargs](https://github.com/brentp/gargs), from which `rush` borrows
526some ideas.
527
528Thank [@bburgin](https://github.com/bburgin) for his contribution on improvement
529of child process management.
530
531## Contact
532
533[Create an issue](https://github.com/shenwei356/rush/issues) to report bugs,
534propose new functions or ask for help.
535
536## License
537
538[MIT License](https://github.com/shenwei356/rush/blob/master/LICENSE)
539
540## Starchart
541
542<img src="https://starchart.cc/shenwei356/rush.svg" alt="Stargazers over time" style="max-width: 100%">
543