1{ 2 "cells": [ 3 { 4 "cell_type": "markdown", 5 "metadata": {}, 6 "source": [ 7 "# Running Scripts from IPython" 8 ] 9 }, 10 { 11 "cell_type": "markdown", 12 "metadata": {}, 13 "source": [ 14 "IPython has a `%%script` cell magic, which lets you run a cell in\n", 15 "a subprocess of any interpreter on your system, such as: bash, ruby, perl, zsh, R, etc.\n", 16 "\n", 17 "It can even be a script of your own, which expects input on stdin." 18 ] 19 }, 20 { 21 "cell_type": "code", 22 "execution_count": 1, 23 "metadata": { 24 "collapsed": false 25 }, 26 "outputs": [], 27 "source": [ 28 "import sys" 29 ] 30 }, 31 { 32 "cell_type": "markdown", 33 "metadata": {}, 34 "source": [ 35 "## Basic usage" 36 ] 37 }, 38 { 39 "cell_type": "markdown", 40 "metadata": {}, 41 "source": [ 42 "To use it, simply pass a path or shell command to the program you want to run on the `%%script` line,\n", 43 "and the rest of the cell will be run by that script, and stdout/err from the subprocess are captured and displayed." 44 ] 45 }, 46 { 47 "cell_type": "code", 48 "execution_count": 2, 49 "metadata": { 50 "collapsed": false 51 }, 52 "outputs": [ 53 { 54 "name": "stdout", 55 "output_type": "stream", 56 "text": [ 57 "hello from Python 2.7.9 (default, Jan 29 2015, 06:27:40) \n", 58 "[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)]\n" 59 ] 60 } 61 ], 62 "source": [ 63 "%%script python2\n", 64 "import sys\n", 65 "print 'hello from Python %s' % sys.version" 66 ] 67 }, 68 { 69 "cell_type": "code", 70 "execution_count": 3, 71 "metadata": { 72 "collapsed": false 73 }, 74 "outputs": [ 75 { 76 "name": "stdout", 77 "output_type": "stream", 78 "text": [ 79 "hello from Python: 3.4.2 |Continuum Analytics, Inc.| (default, Oct 21 2014, 17:42:20) \n", 80 "[GCC 4.2.1 (Apple Inc. build 5577)]\n" 81 ] 82 } 83 ], 84 "source": [ 85 "%%script python3\n", 86 "import sys\n", 87 "print('hello from Python: %s' % sys.version)" 88 ] 89 }, 90 { 91 "cell_type": "markdown", 92 "metadata": {}, 93 "source": [ 94 "IPython also creates aliases for a few common interpreters, such as bash, ruby, perl, etc.\n", 95 "\n", 96 "These are all equivalent to `%%script <name>`" 97 ] 98 }, 99 { 100 "cell_type": "code", 101 "execution_count": 4, 102 "metadata": { 103 "collapsed": false 104 }, 105 "outputs": [ 106 { 107 "name": "stdout", 108 "output_type": "stream", 109 "text": [ 110 "Hello from Ruby 2.0.0\n" 111 ] 112 } 113 ], 114 "source": [ 115 "%%ruby\n", 116 "puts \"Hello from Ruby #{RUBY_VERSION}\"" 117 ] 118 }, 119 { 120 "cell_type": "code", 121 "execution_count": 5, 122 "metadata": { 123 "collapsed": false 124 }, 125 "outputs": [ 126 { 127 "name": "stdout", 128 "output_type": "stream", 129 "text": [ 130 "hello from /usr/local/bin/bash\n" 131 ] 132 } 133 ], 134 "source": [ 135 "%%bash\n", 136 "echo \"hello from $BASH\"" 137 ] 138 }, 139 { 140 "cell_type": "markdown", 141 "metadata": {}, 142 "source": [ 143 "## Capturing output" 144 ] 145 }, 146 { 147 "cell_type": "markdown", 148 "metadata": {}, 149 "source": [ 150 "You can also capture stdout/err from these subprocesses into Python variables, instead of letting them go directly to stdout/err" 151 ] 152 }, 153 { 154 "cell_type": "code", 155 "execution_count": 6, 156 "metadata": { 157 "collapsed": false 158 }, 159 "outputs": [ 160 { 161 "name": "stdout", 162 "output_type": "stream", 163 "text": [ 164 "hi, stdout\n" 165 ] 166 }, 167 { 168 "name": "stderr", 169 "output_type": "stream", 170 "text": [ 171 "hello, stderr\n" 172 ] 173 } 174 ], 175 "source": [ 176 "%%bash\n", 177 "echo \"hi, stdout\"\n", 178 "echo \"hello, stderr\" >&2\n" 179 ] 180 }, 181 { 182 "cell_type": "code", 183 "execution_count": 7, 184 "metadata": { 185 "collapsed": false 186 }, 187 "outputs": [], 188 "source": [ 189 "%%bash --out output --err error\n", 190 "echo \"hi, stdout\"\n", 191 "echo \"hello, stderr\" >&2" 192 ] 193 }, 194 { 195 "cell_type": "code", 196 "execution_count": 8, 197 "metadata": { 198 "collapsed": false 199 }, 200 "outputs": [ 201 { 202 "name": "stdout", 203 "output_type": "stream", 204 "text": [ 205 "hello, stderr\n", 206 "\n", 207 "hi, stdout\n", 208 "\n" 209 ] 210 } 211 ], 212 "source": [ 213 "print(error)\n", 214 "print(output)" 215 ] 216 }, 217 { 218 "cell_type": "markdown", 219 "metadata": {}, 220 "source": [ 221 "## Background Scripts" 222 ] 223 }, 224 { 225 "cell_type": "markdown", 226 "metadata": {}, 227 "source": [ 228 "These scripts can be run in the background, by adding the `--bg` flag.\n", 229 "\n", 230 "When you do this, output is discarded unless you use the `--out/err`\n", 231 "flags to store output as above." 232 ] 233 }, 234 { 235 "cell_type": "code", 236 "execution_count": 9, 237 "metadata": { 238 "collapsed": false 239 }, 240 "outputs": [ 241 { 242 "name": "stdout", 243 "output_type": "stream", 244 "text": [ 245 "Starting job # 0 in a separate thread.\n" 246 ] 247 } 248 ], 249 "source": [ 250 "%%ruby --bg --out ruby_lines\n", 251 "for n in 1...10\n", 252 " sleep 1\n", 253 " puts \"line #{n}\"\n", 254 " STDOUT.flush\n", 255 "end" 256 ] 257 }, 258 { 259 "cell_type": "markdown", 260 "metadata": {}, 261 "source": [ 262 "When you do store output of a background thread, these are the stdout/err *pipes*,\n", 263 "rather than the text of the output." 264 ] 265 }, 266 { 267 "cell_type": "code", 268 "execution_count": 10, 269 "metadata": { 270 "collapsed": false 271 }, 272 "outputs": [ 273 { 274 "data": { 275 "text/plain": [ 276 "<open file '<fdopen>', mode 'rb' at 0x10a4be660>" 277 ] 278 }, 279 "execution_count": 10, 280 "metadata": {}, 281 "output_type": "execute_result" 282 } 283 ], 284 "source": [ 285 "ruby_lines" 286 ] 287 }, 288 { 289 "cell_type": "code", 290 "execution_count": 11, 291 "metadata": { 292 "collapsed": false 293 }, 294 "outputs": [ 295 { 296 "name": "stdout", 297 "output_type": "stream", 298 "text": [ 299 "line 1\n", 300 "line 2\n", 301 "line 3\n", 302 "line 4\n", 303 "line 5\n", 304 "line 6\n", 305 "line 7\n", 306 "line 8\n", 307 "line 9\n", 308 "\n" 309 ] 310 } 311 ], 312 "source": [ 313 "print(ruby_lines.read())" 314 ] 315 }, 316 { 317 "cell_type": "markdown", 318 "metadata": {}, 319 "source": [ 320 "## Arguments to subcommand" 321 ] 322 }, 323 { 324 "cell_type": "markdown", 325 "metadata": {}, 326 "source": [ 327 "You can pass arguments the subcommand as well,\n", 328 "such as this example instructing Python to use integer division from Python 3:" 329 ] 330 }, 331 { 332 "cell_type": "code", 333 "execution_count": 12, 334 "metadata": { 335 "collapsed": false 336 }, 337 "outputs": [ 338 { 339 "name": "stdout", 340 "output_type": "stream", 341 "text": [ 342 "0.333333333333\n" 343 ] 344 } 345 ], 346 "source": [ 347 "%%script python2 -Qnew\n", 348 "print 1/3" 349 ] 350 }, 351 { 352 "cell_type": "markdown", 353 "metadata": {}, 354 "source": [ 355 "You can really specify *any* program for `%%script`,\n", 356 "for instance here is a 'program' that echos the lines of stdin, with delays between each line." 357 ] 358 }, 359 { 360 "cell_type": "code", 361 "execution_count": 13, 362 "metadata": { 363 "collapsed": false 364 }, 365 "outputs": [ 366 { 367 "name": "stdout", 368 "output_type": "stream", 369 "text": [ 370 "Starting job # 2 in a separate thread.\n" 371 ] 372 } 373 ], 374 "source": [ 375 "%%script --bg --out bashout bash -c \"while read line; do echo $line; sleep 1; done\"\n", 376 "line 1\n", 377 "line 2\n", 378 "line 3\n", 379 "line 4\n", 380 "line 5\n" 381 ] 382 }, 383 { 384 "cell_type": "markdown", 385 "metadata": {}, 386 "source": [ 387 "Remember, since the output of a background script is just the stdout pipe,\n", 388 "you can read it as lines become available:" 389 ] 390 }, 391 { 392 "cell_type": "code", 393 "execution_count": 14, 394 "metadata": { 395 "collapsed": false 396 }, 397 "outputs": [ 398 { 399 "name": "stdout", 400 "output_type": "stream", 401 "text": [ 402 "0.0s: line 1\n", 403 "1.0s: line 2\n", 404 "2.0s: line 3\n", 405 "3.0s: line 4\n", 406 "4.0s: line 5\n" 407 ] 408 } 409 ], 410 "source": [ 411 "import time\n", 412 "tic = time.time()\n", 413 "line = True\n", 414 "while True:\n", 415 " line = bashout.readline()\n", 416 " if not line:\n", 417 " break\n", 418 " sys.stdout.write(\"%.1fs: %s\" %(time.time()-tic, line))\n", 419 " sys.stdout.flush()\n" 420 ] 421 }, 422 { 423 "cell_type": "markdown", 424 "metadata": {}, 425 "source": [ 426 "## Configuring the default ScriptMagics" 427 ] 428 }, 429 { 430 "cell_type": "markdown", 431 "metadata": {}, 432 "source": [ 433 "The list of aliased script magics is configurable.\n", 434 "\n", 435 "The default is to pick from a few common interpreters, and use them if found, but you can specify your own in ipython_config.py:\n", 436 "\n", 437 " c.ScriptMagics.scripts = ['R', 'pypy', 'myprogram']\n", 438 "\n", 439 "And if any of these programs do not apear on your default PATH, then you would also need to specify their location with:\n", 440 "\n", 441 " c.ScriptMagics.script_paths = {'myprogram': '/opt/path/to/myprogram'}" 442 ] 443 } 444 ], 445 "metadata": { 446 "kernelspec": { 447 "display_name": "Python 3", 448 "language": "python", 449 "name": "python3" 450 }, 451 "language_info": { 452 "codemirror_mode": { 453 "name": "ipython", 454 "version": 3 455 }, 456 "file_extension": ".py", 457 "mimetype": "text/x-python", 458 "name": "python", 459 "nbconvert_exporter": "python", 460 "pygments_lexer": "ipython3", 461 "version": "3.4.2" 462 } 463 }, 464 "nbformat": 4, 465 "nbformat_minor": 0 466} 467