1This is gnuastro.info, produced by makeinfo version 6.8 from 2gnuastro.texi. 3 4This book documents version 0.16 of the GNU Astronomy Utilities 5(Gnuastro). Gnuastro provides various programs and libraries for 6astronomical data manipulation and analysis. 7 8 Copyright © 2015-2021, Free Software Foundation, Inc. 9 10 Permission is granted to copy, distribute and/or modify this 11 document under the terms of the GNU Free Documentation License, 12 Version 1.3 or any later version published by the Free Software 13 Foundation; with no Invariant Sections, no Front-Cover Texts, and 14 no Back-Cover Texts. A copy of the license is included in the 15 section entitled “GNU Free Documentation License”. 16INFO-DIR-SECTION Astronomy 17START-INFO-DIR-ENTRY 18* Gnuastro: (gnuastro). GNU Astronomy Utilities. 19* libgnuastro: (gnuastro)Gnuastro library. Full Gnuastro library doc. 20 21* help-gnuastro: (gnuastro)help-gnuastro mailing list. Getting help. 22 23* bug-gnuastro: (gnuastro)Report a bug. How to report bugs 24 25* Arithmetic: (gnuastro)Arithmetic. Arithmetic operations on pixels. 26* astarithmetic: (gnuastro)Invoking astarithmetic. Options to Arithmetic. 27 28* BuildProgram: (gnuastro)BuildProgram. Compile and run programs using Gnuastro’s library. 29* astbuildprog: (gnuastro)Invoking astbuildprog. Options to BuildProgram. 30 31* ConvertType: (gnuastro)ConvertType. Convert different file types. 32* astconvertt: (gnuastro)Invoking astconvertt. Options to ConvertType. 33 34* Convolve: (gnuastro)Convolve. Convolve an input file with kernel. 35* astconvolve: (gnuastro)Invoking astconvolve. Options to Convolve. 36 37* CosmicCalculator: (gnuastro)CosmicCalculator. For cosmological params. 38* astcosmiccal: (gnuastro)Invoking astcosmiccal. Options to CosmicCalculator. 39 40* Crop: (gnuastro)Crop. Crop region(s) from image(s). 41* astcrop: (gnuastro)Invoking astcrop. Options to Crop. 42 43* Fits: (gnuastro)Fits. View and manipulate FITS extensions and keywords. 44* astfits: (gnuastro)Invoking astfits. Options to Fits. 45 46* MakeCatalog: (gnuastro)MakeCatalog. Make a catalog from labeled image. 47* astmkcatalog: (gnuastro)Invoking astmkcatalog. Options to MakeCatalog. 48 49* MakeNoise: (gnuastro)MakeNoise. Make (add) noise to an image. 50* astmknoise: (gnuastro)Invoking astmknoise. Options to MakeNoise. 51 52* MakeProfiles: (gnuastro)MakeProfiles. Make mock profiles. 53* astmkprof: (gnuastro)Invoking astmkprof. Options to MakeProfiles. 54 55* Match: (gnuastro)Match. Match two separate catalogs. 56* astmatch: (gnuastro)Invoking astmatch. Options to Match. 57 58* NoiseChisel: (gnuastro)NoiseChisel. Detect signal in noise. 59* astnoisechisel: (gnuastro)Invoking astnoisechisel. Options to NoiseChisel. 60 61* Segment: (gnuastro)Segment. Segment detections based on signal structure. 62* astsegment: (gnuastro)Invoking astsegment. Options to Segment. 63 64* Query: (gnuastro)Query. Access remote databases for downloading data. 65* astquery: (gnuastro)Invoking astquery. Options to Query. 66 67* Statistics: (gnuastro)Statistics. Get image Statistics. 68* aststatistics: (gnuastro)Invoking aststatistics. Options to Statistics. 69 70* Table: (gnuastro)Table. Read and write FITS binary or ASCII tables. 71* asttable: (gnuastro)Invoking asttable. Options to Table. 72 73* Warp: (gnuastro)Warp. Warp a dataset to a new grid. 74* astwarp: (gnuastro)Invoking astwarp. Options to Warp. 75 76* astscript: (gnuastro)Installed scripts. Gnuastro’s installed scripts. 77* astscript-sort-by-night: (gnuastro)Invoking astscript-sort-by-night. Options to this script 78* astscript-radial-profile: (gnuastro)Invoking astscript-radial-profile. Options to this script 79* astscript-ds9-region: (gnuastro)Invoking astscript-ds9-region. Options to this script 80 81END-INFO-DIR-ENTRY 82 83 84File: gnuastro.info, Node: ConvertType, Next: Table, Prev: Fits, Up: Data containers 85 865.2 ConvertType 87=============== 88 89The FITS format used in astronomy was defined mainly for archiving, 90transmission, and processing. In other situations, the data might be 91useful in other formats. For example, when you are writing a paper or 92report, or if you are making slides for a talk, you can’t use a FITS 93image. Other image formats should be used. In other cases you might 94want your pixel values in a table format as plain text for input to 95other programs that don’t recognize FITS. ConvertType is created for 96such situations. The various types will increase with future updates 97and based on need. 98 99 The conversion is not only one way (from FITS to other formats), but 100two ways (except the EPS and PDF formats(1)). So you can also convert a 101JPEG image or text file into a FITS image. Basically, other than 102EPS/PDF, you can use any of the recognized formats as different color 103channel inputs to get any of the recognized outputs. So before 104explaining the options and arguments (in *note Invoking astconvertt::), 105we’ll start with a short description of the recognized files types in 106*note Recognized file formats::, followed a short introduction to 107digital color in *note Color::. 108 109* Menu: 110 111* Recognized file formats:: Recognized file formats 112* Color:: Some explanations on color. 113* Aligning images with small WCS offsets:: When the WCS slightly differs. 114* Annotations for figure in paper:: Adding coordinates or physical scale. 115* Invoking astconvertt:: Options and arguments to ConvertType. 116 117 ---------- Footnotes ---------- 118 119 (1) Because EPS and PDF are vector, not raster/pixelated formats 120 121 122File: gnuastro.info, Node: Recognized file formats, Next: Color, Prev: ConvertType, Up: ConvertType 123 1245.2.1 Recognized file formats 125----------------------------- 126 127The various standards and the file name extensions recognized by 128ConvertType are listed below. Currently Gnuastro uses the file name’s 129suffix to identify the format. 130 131FITS or IMH 132 Astronomical data are commonly stored in the FITS format (or the 133 older data IRAF ‘.imh’ format), a list of file name suffixes which 134 indicate that the file is in this format is given in *note 135 Arguments::. 136 137 Each image extension of a FITS file only has one value per 138 pixel/element. Therefore, when used as input, each input FITS 139 image contributes as one color channel. If you want multiple 140 extensions in one FITS file for different color channels, you have 141 to repeat the file name multiple times and use the ‘--hdu’, 142 ‘--hdu2’, ‘--hdu3’ or ‘--hdu4’ options to specify the different 143 extensions. 144 145JPEG 146 The JPEG standard was created by the Joint photographic experts 147 group. It is currently one of the most commonly used image 148 formats. Its major advantage is the compression algorithm that is 149 defined by the standard. Like the FITS standard, this is a raster 150 graphics format, which means that it is pixelated. 151 152 A JPEG file can have 1 (for gray-scale), 3 (for RGB) and 4 (for 153 CMYK) color channels. If you only want to convert one JPEG image 154 into other formats, there is no problem, however, if you want to 155 use it in combination with other input files, make sure that the 156 final number of color channels does not exceed four. If it does, 157 then ConvertType will abort and notify you. 158 159 The file name endings that are recognized as a JPEG file for input 160 are: ‘.jpg’, ‘.JPG’, ‘.jpeg’, ‘.JPEG’, ‘.jpe’, ‘.jif’, ‘.jfif’ and 161 ‘.jfi’. 162 163TIFF 164 TIFF (or Tagged Image File Format) was originally designed as a 165 common format for scanners in the early 90s and since then it has 166 grown to become very general. In many aspects, the TIFF standard 167 is similar to the FITS image standard: it can allow data of many 168 types (see *note Numeric data types::), and also allows multiple 169 images to be stored in a single file (each image in the file is 170 called a ‘directory’ in the TIFF standard). However, unlike FITS, 171 it can only store images, it has no constructs for tables. Another 172 (inconvenient) difference with the FITS standard is that keyword 173 names are stored as numbers, not human-readable text. 174 175 However, outside of astronomy, because of its support of different 176 numeric data types, many fields use TIFF images for accurate (for 177 example 16-bit integer or floating point for example) imaging data. 178 179 Currently ConvertType can only read TIFF images, if you are 180 interested in writing TIFF images, please get in touch with us. 181 182EPS 183 The Encapsulated PostScript (EPS) format is essentially a one page 184 PostScript file which has a specified size. PostScript also 185 includes non-image data, for example lines and texts. It is a 186 fully functional programming language to describe a document. 187 Therefore in ConvertType, EPS is only an output format and cannot 188 be used as input. Contrary to the FITS or JPEG formats, PostScript 189 is not a raster format, but is categorized as vector graphics. 190 191 The Portable Document Format (PDF) is currently the most common 192 format for documents. Some believe that PDF has replaced 193 PostScript and that PostScript is now obsolete. This view is 194 wrong, a PostScript file is an actual plain text file that can be 195 edited like any program source with any text editor. To be able to 196 display its programmed content or print, it needs to pass through a 197 processor or compiler. A PDF file can be thought of as the 198 processed output of the compiler on an input PostScript file. 199 PostScript, EPS and PDF were created and are registered by Adobe 200 Systems. 201 202 With these features in mind, you can see that when you are 203 compiling a document with TeX or LaTeX, using an EPS file is much 204 more low level than a JPEG and thus you have much greater control 205 and therefore quality. Since it also includes vector graphic lines 206 we also use such lines to make a thin border around the image to 207 make its appearance in the document much better. No matter the 208 resolution of the display or printer, these lines will always be 209 clear and not pixelated. In the future, addition of text might be 210 included (for example labels or object IDs) on the EPS output. 211 However, this can be done better with tools within TeX or LaTeX 212 such as PGF/Tikz(1). 213 214 If the final input image (possibly after all operations on the flux 215 explained below) is a binary image or only has two colors of black 216 and white (in segmentation maps for example), then PostScript has 217 another great advantage compared to other formats. It allows for 1 218 bit pixels (pixels with a value of 0 or 1), this can decrease the 219 output file size by 8 times. So if a gray-scale image is binary, 220 ConvertType will exploit this property in the EPS and PDF (see 221 below) outputs. 222 223 The standard formats for an EPS file are ‘.eps’, ‘.EPS’, ‘.epsf’ 224 and ‘.epsi’. The EPS outputs of ConvertType have the ‘.eps’ 225 suffix. 226 227PDF 228 As explained above, a PDF document is a static document description 229 format, viewing its result is therefore much faster and more 230 efficient than PostScript. To create a PDF output, ConvertType 231 will make a PostScript page description and convert that to PDF 232 using GPL Ghostscript. The suffixes recognized for a PDF file are: 233 ‘.pdf’, ‘.PDF’. If GPL Ghostscript cannot be run on the PostScript 234 file, it will remain and a warning will be printed. 235 236‘blank’ 237 This is not actually a file type! But can be used to fill one 238 color channel with a blank value. If this argument is given for 239 any color channel, that channel will not be used in the output. 240 241Plain text 242 Plain text files have the advantage that they can be viewed with 243 any text editor or on the command-line. Most programs also support 244 input as plain text files. As input, each plain text file is 245 considered to contain one color channel. 246 247 In ConvertType, the recognized extensions for plain text files are 248 ‘.txt’ and ‘.dat’. As described in *note Invoking astconvertt::, 249 if you just give these extensions, (and not a full filename) as 250 output, then automatic output will be preformed to determine the 251 final output name (see *note Automatic output::). Besides these, 252 when the format of a file cannot be recognized from its name, 253 ConvertType will fall back to plain text mode. So you can use any 254 name (even without an extension) for a plain text input or output. 255 Just note that when the suffix is not recognized, automatic output 256 will not be preformed. 257 258 The basic input/output on plain text images is very similar to how 259 tables are read/written as described in *note Gnuastro text table 260 format::. Simply put, the restrictions are very loose, and there 261 is a convention to define a name, units, data type (see *note 262 Numeric data types::), and comments for the data in a commented 263 line. The only difference is that as a table, a text file can 264 contain many datasets (columns), but as a 2D image, it can only 265 contain one dataset. As a result, only one information comment 266 line is necessary for a 2D image, and instead of the starting ‘‘# 267 Column N’’ (‘N’ is the column number), the information line for a 268 2D image must start with ‘‘# Image 1’’. When ConvertType is asked 269 to output to plain text file, this information comment line is 270 written before the image pixel values. 271 272 When converting an image to plain text, consider the fact that if 273 the image is large, the number of columns in each line will become 274 very large, possibly making it very hard to open in some text 275 editors. 276 277Standard output (command-line) 278 This is very similar to the plain text output, but instead of 279 creating a file to keep the printed values, they are printed on the 280 command line. This can be very useful when you want to redirect 281 the results directly to another program in one command with no 282 intermediate file. The only difference is that only the pixel 283 values are printed (with no information comment line). To print to 284 the standard output, set the output name to ‘‘stdout’’. 285 286 ---------- Footnotes ---------- 287 288 (1) <http://sourceforge.net/projects/pgf/> 289 290 291File: gnuastro.info, Node: Color, Next: Aligning images with small WCS offsets, Prev: Recognized file formats, Up: ConvertType 292 2935.2.2 Color 294----------- 295 296Color is defined by mixing various measurements/filters. In digital 297monitors or common digital cameras, colors are displayed/stored by 298mixing the three basic colors of red, green and blue (RGB) with various 299proportions. When printing on paper, standard printers use the cyan, 300magenta, yellow and key (CMYK, key=black) color space. In other words, 301for each displayed/printed pixel of a color image, the dataset/image has 302three or four values. 303 304 To store/show the three values for each pixel, cameras and monitors 305allocate a certain fraction of each pixel’s area to red, green and blue 306filters. These three filters are thus built into the hardware at the 307pixel level. However, because measurement accuracy is very important in 308scientific instruments, and we want to do measurements (take images) 309with various/custom filters (without having to order a new expensive 310detector!), scientific detectors use the full area of the pixel to store 311one value for it in a single/mono channel dataset. To make measurements 312in different filters, we just place a filter in the light path before 313the detector. Therefore, the FITS format that is used to store 314astronomical datasets is inherently a mono-channel format (see *note 315Recognized file formats:: or *note Fits::). 316 317 When a subject has been imaged in multiple filters, you can feed each 318different filter into the red, green and blue channels and obtain a 319colored visualization. In ConvertType, you can do this by giving each 320separate single-channel dataset (for example in the FITS image format) 321as an argument (in the proper order), then asking for the output in a 322format that supports multi-channel datasets (for example JPEG or PDF, 323see the examples in *note Invoking astconvertt::). 324 325 As discussed above, color is not defined when a dataset/image 326contains a single value for each pixel. However, we interact with 327scientific datasets through monitors or printers (which allow multiple 328values per pixel and produce color with them). As a result, there is a 329lot of freedom in visualizing a single-channel dataset. The most basic 330is to use shades of black (because of its strong contrast with white). 331This scheme is called grayscale. To help in visualization, more complex 332mappings can be defined. For example, the values can be scaled to a 333range of 0 to 360 and used as the “Hue” term of the Hue-Saturation-Value 334(https://en.wikipedia.org/wiki/HSL_and_HSV) (HSV) color space (while 335fixing the “Saturation” and “Value” terms). In ConvertType, you can use 336the ‘--colormap’ option to choose between different mappings of 337mono-channel inputs, see *note Invoking astconvertt::. 338 339 Since grayscale is a commonly used mapping of single-valued datasets, 340we’ll continue with a closer look at how it is stored. One way to 341represent a gray-scale image in different color spaces is to use the 342same proportions of the primary colors in each pixel. This is the 343common way most FITS image viewers work: for each pixel, they fill all 344the channels with the single value. While this is necessary for 345displaying a dataset, there are downsides when storing/saving this type 346of grayscale visualization (for example in a paper). 347 348 • Three (for RGB) or four (for CMYK) values have to be stored for 349 every pixel, this makes the output file very heavy (in terms of 350 bytes). 351 352 • If printing, the printing errors of each color channel can make the 353 printed image slightly more blurred than it actually is. 354 355 To solve both these problems when storing grayscale visualization, 356the best way is to save a single-channel dataset into the black channel 357of the CMYK color space. The JPEG standard is the only common standard 358that accepts CMYK color space. 359 360 The JPEG and EPS standards set two sizes for the number of bits in 361each channel: 8-bit and 12-bit. The former is by far the most common 362and is what is used in ConvertType. Therefore, each channel should have 363values between 0 to 2^8-1=255. From this we see how each pixel in a 364gray-scale image is one byte (8 bits) long, in an RGB image, it is 3 365bytes long and in CMYK it is 4 bytes long. But thanks to the JPEG 366compression algorithms, when all the pixels of one channel have the same 367value, that channel is compressed to one pixel. Therefore a Grayscale 368image and a CMYK image that has only the K-channel filled are 369approximately the same file size. 370 371 372File: gnuastro.info, Node: Aligning images with small WCS offsets, Next: Annotations for figure in paper, Prev: Color, Up: ConvertType 373 3745.2.3 Aligning images with small WCS offsets 375-------------------------------------------- 376 377In order to have nice color images, it is important that the images be 378properly aligned. This is usually the case in many scenarios, but it 379some times happens that the images have a small WCS offset, even though 380they have the same size. In such cases you can use the script below to 381align the images into approximately the same pixel grid (to within about 3820.5 pixels which is sufficient in many color-image usage scenarios). 383 384 The script below does the job using Gnuastro’s *note Warp:: and *note 385Crop:: programs. Simply copy the lines below into a plain-text file 386with your favorite text editor and save it as ‘my-align.sh’. Don’t 387forget to set the variables of the first three lines to specify the file 388names (without the ‘.fits’ suffix) and the HDUs of your inputs. These 389four lines are all you need to edit, leave the rest unchanged. Also, if 390you are copy/pasting the script from a PDF, be careful that the 391single-quotes used in AWK may need to be corrected. 392 393 #!/bin/sh 394 395 # Set the input names (without the '.fits' suffix), 396 # and their HDUs. 397 r=RED_IMAGE_NO_SUFFIX; rhdu=1 398 g=GREEN_IMAGE_NO_SUFFIX; ghdu=1 399 b=BLUE_IMAGE_NO_SUFFIX; bhdu=1 400 401 # To stop the script if there is a crash 402 set -e 403 404 # Align all the images to the celestial poles. 405 astwarp $r.fits --align -h$rhdu -o $r-aligned.fits 406 astwarp $g.fits --align -h$ghdu -o $g-aligned.fits 407 astwarp $b.fits --align -h$bhdu -o $b-aligned.fits 408 409 # Calculate the final WCS-based center and image-based width based on 410 # the G-band (in RGB) image. 411 centerwcs=$(astfits $g-aligned.fits --skycoverage --quiet \ 412 | awk 'NR==1{printf "%g %g", $1,$2}') 413 widthpix=$(astfits $g-aligned.fits -h1 --quiet \ 414 --keyvalue=NAXIS1,NAXIS2 \ 415 | awk '{printf "%d,%d", $1, $2}') 416 417 # Crop all the images around the desired center and width. 418 for f in $r $g $b; do 419 centerpix=$(echo $centerwcs \ 420 | asttable -c'arith $1 $2 wcs-to-img' \ 421 --wcsfile=$f-aligned.fits \ 422 | awk '{printf "%g,%g", $1, $2}') 423 astcrop $f-aligned.fits --mode=img --width=$widthpix \ 424 --center=$centerpix -o$f-use.fits 425 rm $f-aligned.fits 426 done 427 428 Once you have have saved the file and come back to your command-line 429you can run the script like this: 430 431 $ chmod +x my-align.sh 432 $ ./my-align.sh 433 434Of course, feel free to hack it and modify it to fit your datasets, like 435the rest of Gnuastro, this script is released under GNU GPLv.3 and 436above, see *note Your rights::. 437 438 439File: gnuastro.info, Node: Annotations for figure in paper, Next: Invoking astconvertt, Prev: Aligning images with small WCS offsets, Up: ConvertType 440 4415.2.4 Annotations for figure in paper 442------------------------------------- 443 444To make a nice figure from your FITS images, it is important to show 445more than merely the raw image (converted to a printer friendly format 446like PDF or JPEG). Annotations (or visual metadata) over the raw image 447greatly help the readers clearly see your argument and put the 448image/result in a larger context. Examples include: 449 • Coordinates (Right Ascension and Declination) on the edges of the 450 image, so viewers of your paper or presentation slides can get a 451 physical feeling of the field’s sky coverage. 452 • Thick line that has a fixed tangential size (for example in kilo 453 parsecs) at the redshift/distance of interest. 454 • Contours over the image to show radio/X-ray emission, over an 455 optical image for example. 456 • Text or arrows or etc, over certain parts of the image. 457 458 Because of the modular philosophy of Gnuastro, ConvertType is only 459focused on converting your FITS images to printer friendly formats like 460JPEG or PDF. But to present your results in a slide or paper, you will 461often need to annotate the raw JPEG or PDF with some of the features 462above. The good news is that there are many powerful plotting programs 463that you can use to add such annotations. As a result, there is no 464point in making a new one, specific to Gnuastro. In this section, we’ll 465demonstrate this using the very powerful PGFPlots(1) package of LaTeX. 466 467*Single script for easy running:* In this section we are reviewing the 468reason and details of every step which is good for educational purposes. 469But when you know the steps already, these separate code blocks can be 470annoying. Therefore the full script (except for the data download step) 471is available in *note Full script of annotations on figure::. 472 473 PGFPlots uses the same LaTeX graphic engine that typesets your 474paper/slide. Therefore when you build your plots and figures using 475PGFPlots (and its underlying package PGF/TiKZ(2)) your plots will blend 476beautifully within your text: same fonts, same colors, same line 477properties and etc. Since most papers (and presentation slides(3)) are 478made with LaTeX, PGFPlots is therefore the best tool for those who use 479LaTeX to create documents. PGFPlots also doesn’t need any extra 480dependencies beyond a basic/minimal TeX-live installation, so it is much 481more reliable than tools like Matplotlib in Python that have hundreds of 482fast-evolving dependencies(4). 483 484 To demonstrate this, we’ll create a surface brightness image of a 485galaxy in the F160W filter of the ABYSS survey(5). In the code-block 486below, let’s make a “build” directory to keep intermediate files and 487avoid populating the source. Afterwards, we’ll download the full image 488and crop out a 20 arcmin wide image around the galaxy with the commands 489below. You can run these commands in an empty directory. 490 491 $ mkdir build 492 $ wget http://cdsarc.u-strasbg.fr/ftp/J/A+A/621/A133/fits/ah_f160w.fits 493 $ astcrop ah_f160w.fits --center=53.1616278,-27.7802446 --mode=wcs \ 494 --width=20/3600 --output=build/crop.fits 495 496 To better show the low surface brightness (LSB) outskirts, we’ll warp 497the image, then convert the pixel units to surface brightness with the 498commands below. It is very important that the warping is done _before_ 499the conversion to surface brightness (in units of mag/arcsec$^2$), 500because the definition of surface brightness is non-linear. For more, 501see the Surface brightness topic of *note Brightness flux magnitude::. 502 503 $ zeropoint=25.94 504 $ astwarp build/crop.fits --centeroncorner --scale=1/3 \ 505 --output=build/scaled.fits 506 $ pixarea=$(astfits build/scaled.fits --pixelscale --quiet \ 507 | awk '{print $1*3600*$2*3600}') 508 $ astarithmetic build/scaled.fits abs $zeropoint counts-to-mag \ 509 $pixarea log10 2.5 x + --output=build/sb.fits 510 511 We are now ready to convert the surface brightness image into a PDF. 512To better show the LSB features, we’ll also limit the color range with 513the ‘--fluxlow’ and ‘--fluxhigh’ options: all pixels with a surface 514brightness brighter than 22 mag/arcsec$^2$ will be shown as black, and 515all pixels with a surface brightness fainter than 30 mag/arcsec$^2$ will 516be white. These thresholds are being defined as variables, because we 517will also need them below (to pass into PGFPlots). We will also set 518‘--borderwidth=0’, because the coordinate system we will add over the 519image will effectively be a border for the image (separating it from the 520background). 521 522 $ sblow=22 523 $ sbhigh=30 524 $ astconvertt build/sb.fits --colormap=gray --borderwidth=0 \ 525 --fluxhigh=$sbhigh --fluxlow=$sblow --output=build/sb.pdf 526 527 Please open ‘sb.pdf’ and have a look. Also, please open ‘sb.fits’ in 528DS9 (or any other FITS viewer) and play with the color range. Can the 529surface brightness limits be changed to better show the LSB structure? 530If so, you are free to change the limits above. 531 532 We now have the printable PDF representation of the image, but as 533discussed above, its not enough for a paper. We’ll add 1) a thick line 534showing the size of 20 kpc (kilo parsecs) at the redshift of the central 535galaxy, 2) coordinates and 3) a color bar, showing the surface 536brightness level of each grayscale level. 537 538 To get the first job done, we first need to know the redshift of the 539central galaxy. To do this, we can use Gnuastro’s Query program to look 540into all the objects in NED within this image (only asking for the RA, 541Dec and redshift columns). We will then use the Match program to find 542the NED entry that corresponds to our galaxy. 543 544 $ astquery ned --dataset=objdir --overlapwith=build/sb.fits \ 545 --column=ra,dec,z --output=ned.fits 546 $ astmatch ned.fits -h1 --coord=53.1616278,-27.7802446 \ 547 --ccol1=RA,Dec --aperture=1/3600 548 $ redshift=$(asttable ned_matched.fits -cz) 549 $ echo $redshift 550 551 Now that we know the redshift of the central object, we can define 552the coordinates of the thick line that will show the length of 20 kpc at 553that redshift. It will be a horizontal line (fixed Declination) across 554a range of RA. The start of this thick line will be located at the top 555edge of the image (at the 95-percent of the width and height of the 556image). With the commands below we’ll find the three necessary 557parameters (one declination and two RAs). Just note that in 558astronomical images, RA increases to the left/east, which is the reason 559we are using the minimum and ‘+’ to find the RA starting point. 560 561 $ scalelineinkpc=20 562 $ coverage=$(astfits build/sb.fits --skycoverage --quiet | awk 'NR==2') 563 $ scalelinedec=$(echo $coverage | awk '{print $4-($4-$3)*0.05}') 564 $ scalelinerastart=$(echo $coverage | awk '{print $1+($2-$1)*0.05}') 565 $ scalelineraend=$(astcosmiccal --redshift=$redshift --arcsectandist \ 566 | awk '{start='$scalelinerastart'; \ 567 width='$scalelineinkpc'/$1/3600; \ 568 print start+width}') 569 570 To draw coordinates over the image, we need to feed these values into 571PGFPlots. But manually entering numbers into the PGFPlots source will 572be very frustrating and prone to many errors! Fortunately there is an 573easy way to do this: LaTeX macros. New macros are defined by this LaTeX 574command: 575 \newcommand{\macroname}{value} 576Anywhere that LaTeX confronts ‘\macroname’, it will replace ‘value’ when 577building the output. We will have one file called ‘macros.tex’ in the 578build directory and define macros based on those values. We will use 579the shell’s ‘printf’ command to write these macro definition lines into 580the macro file. We just have to use double backslashes in the ‘printf’ 581command, because backslash is a meaningful character for ‘printf’, but 582we want to keep one of them. Also, we put a ‘\n’ at the end of each 583line, otherwise, all the commands will go into a single line of the 584macro file. We will also place the random ‘‘ma’’ string at the start of 585all our LaTeX macros to help identify the macros for this plot. 586 587 $ macros=build/macros.tex 588 $ printf '\\newcommand{\\maScaleDec}'"{$scalelinedec}\n" > $macros 589 $ printf '\\newcommand{\\maScaleRAa}'"{$scalelinerastart}\n" >> $macros 590 $ printf '\\newcommand{\\maScaleRAb}'"{$scalelineraend}\n" >> $macros 591 $ printf '\\newcommand{\\maScaleKpc}'"{$scalelineinkpc}\n" >> $macros 592 $ printf '\\newcommand{\\maCenterZ}'"{$redshift}\n" >> $macros 593 594 Please open the macros file after these commands and have a look to 595see if they do conform to the expected format above. Another set of 596macros we will need to feed into PGFPlots is the coordinates of the 597image corners. Fortunately the ‘coverage’ variable found above is also 598useful here. We just need to extract each item before feeding it into 599the macros. To do this, we’ll use AWK and keep each value with the 600temporary shell variable ‘‘v’’. 601 602 $ v=$(echo $coverage | awk '{print $1}') 603 $ printf '\\newcommand{\\maCropRAMin}'"{$v}\n" >> $macros 604 $ v=$(echo $coverage | awk '{print $2}') 605 $ printf '\\newcommand{\\maCropRAMax}'"{$v}\n" >> $macros 606 $ v=$(echo $coverage | awk '{print $3}') 607 $ printf '\\newcommand{\\maCropDecMin}'"{$v}\n" >> $macros 608 $ v=$(echo $coverage | awk '{print $4}') 609 $ printf '\\newcommand{\\maCropDecMax}'"{$v}\n" >> $macros 610 611 Finally, we also need to pass some other numbers to PGFPlots: 1) the 612major tick distance (in the coordinate axes that will be printed on the 613edge of the image). We’ll assume 7 ticks for this image. 2) The 614minimum and maximum surface brightness values that we gave to 615ConvertType when making the PDF; PGFPlots will define its color-bar 616based on these two values. 617 618 $ v=$(echo $coverage | awk '{print ($2-$1)/7}') 619 $ printf '\\newcommand{\\maTickDist}'"{$v}\n" >> $macros 620 $ printf '\\newcommand{\\maSBlow}'"{$sblow}\n" >> $macros 621 $ printf '\\newcommand{\\maSBhigh}'"{$sbhigh}\n" >> $macros 622 623 All the necessary numbers are now ready. Please copy the contents 624below into a file called ‘my-figure.tex’. This is the PGFPlots source 625for this particular plot. Besides the coordinates and scale-line, we 626will also add some text over the image and an orange arrow pointing to 627the central object with its redshift printed over it. The parameters 628are generally human-readable, so you should be able to get a good 629feeling of every line. There are also comments which will show up as a 630different color when you copy this into a plain-text editor. 631 632\begin{tikzpicture} 633 634 %% Define the coordinates and colorbar 635 \begin{axis}[ 636 at={(0,0)}, 637 axis on top, 638 x dir=reverse, 639 scale only axis, 640 width=\linewidth, 641 height=\linewidth, 642 minor tick num=10, 643 xmin=\maCropRAMin, 644 xmax=\maCropRAMax, 645 ymin=\maCropDecMin, 646 ymax=\maCropDecMax, 647 enlargelimits=false, 648 every tick/.style={black}, 649 xtick distance=\maTickDist, 650 ytick distance=\maTickDist, 651 yticklabel style={rotate=90}, 652 ylabel={Declination (degrees)}, 653 xlabel={Right Ascension (degrees)}, 654 ticklabel style={font=\small, 655 /pgf/number format/.cd, precision=4,/tikz/.cd}, 656 x label style={at={(axis description cs:0.5,0.02)}, 657 anchor=north,font=\small}, 658 y label style={at={(axis description cs:0.07,0.5)}, 659 anchor=south,font=\small}, 660 colorbar, 661 colormap name=gray, 662 point meta min=\maSBlow, 663 point meta max=\maSBhigh, 664 colorbar style={ 665 at={(1.01,1)}, 666 ylabel={Surface brightness (mag/arcsec$^2$)}, 667 yticklabel style={ 668 /pgf/number format/.cd, precision=1, /tikz/.cd}, 669 y label style={at={(axis description cs:5.3,0.5)}, 670 anchor=south,font=\small}, 671 }, 672 ] 673 674 %% Put the image in the proper positions of the plot. 675 \addplot graphics[ xmin=\maCropRAMin, xmax=\maCropRAMax, 676 ymin=\maCropDecMin, ymax=\maCropDecMax] 677 {sb.pdf}; 678 679 %% Draw the scale factor. 680 \addplot[black, line width=5, name=scaleline] coordinates 681 {(\maScaleRAa,\maScaleDec) (\maScaleRAb,\maScaleDec)} 682 node [anchor=north west] {\large $\maScaleKpc$ kpc}; 683 \end{axis} 684 685 %% Add some text anywhere over the plot. The text is added two 686 %% times: the first time with a white background (that with a 687 %% certain opacity), the second time just the text with opacity. 688 \node[anchor=south west, fill=white, opacity=0.5] 689 at (0.01\linewidth,0.01\linewidth) 690 {(a) Text can be added here}; 691 \node[anchor=south west] 692 at (0.01\linewidth,0.01\linewidth) 693 {(a) Text can be added here}; 694 695 %% Add an arrow to highlight certain structures. 696 \draw [->, red!70!yellow, line width=5] 697 (0.35\linewidth,0.35\linewidth) 698 -- node [anchor=south, rotate=45]{$z=\maCenterZ$} 699 (0.45\linewidth,0.45\linewidth); 700\end{tikzpicture} 701 702 Finally, we need another simple LaTeX source for the main PDF 703“report” that will host this figure. This can actually be your paper or 704slides for example. Here, we’ll suffice to the minimal working example. 705 706\documentclass{article} 707 708%% Import the TiKZ package and activate its "external" feature. 709\usepackage{tikz} 710\usetikzlibrary{external} 711\tikzexternalize 712 713%% PGFPlots (which uses TiKZ). 714\usepackage{pgfplots} 715\pgfplotsset{axis line style={thick}} 716\pgfplotsset{ 717 /pgfplots/colormap={gray}{rgb255=(0,0,0) rgb255=(255,255,255)} 718} 719 720%% Import the macros. 721\input{macros.tex} 722 723%% Start document. 724\begin{document} 725You can write anything here. 726 727%% Add the figure and its caption. 728\begin{figure} 729 \input{my-figure.tex} 730 \caption{A demo image.} 731\end{figure} 732 733%% Finish the document. 734\end{document} 735 736 You are now ready to create the PDF. But LaTeX creates many temporary 737files, so to avoid populating our top-level directory, we’ll copy the 738two ‘.tex’ files into the build directory, go there and run LaTeX. 739Before running it though, we’ll first delete all the files that have the 740name pattern ‘*-figure0*’, these are “external” files created by 741TiKZ+PGFPlots, including the actual PDF of the figure. 742 743 $ cp report.tex my-figure.tex build 744 $ cd build 745 $ rm -f *-figure0* 746 $ pdflatex -shell-escape -halt-on-error report.tex 747 748 You now have the full “report” in ‘report.pdf’. Try adding some 749extra text on top of the figure, or in the caption and re-running the 750last four commands. Also try changing the 20kpc scale line length to 75150kpc, or try changing the redshift, to see how the length and text of 752the thick scale-line will automatically change. But the good news is 753that you also have the raw PDF of the figure that you can use in other 754places. You can see that file in ‘report-figure0.pdf’. 755 756 In a larger paper, you can add multiple such figures (with different 757‘.tex’ files that are placed in different ‘figure’ environments with 758different captions). Each figure will get a number in the build 759directory. TiKZ also allows setting a file name for each “external” 760figure (to avoid such numbers that can be annoying if the image orders 761are changed). PGFPlots is also highly customizable, you can make a lot 762of changes and customizations. Both TiKZ(6) and PGFPLots(7) have 763wonderful manuals, so have a look trough them. 764 765* Menu: 766 767* Full script of annotations on figure:: All the steps in one script 768 769 ---------- Footnotes ---------- 770 771 (1) 772<http://mirrors.ctan.org/graphics/pgf/contrib/pgfplots/doc/pgfplots.pdf> 773 774 (2) <http://mirrors.ctan.org/graphics/pgf/base/doc/pgfmanual.pdf> 775 776 (3) To build slides, LaTeX has packages like Beamer, see 777<http://mirrors.ctan.org/macros/latex/contrib/beamer/doc/beameruserguide.pdf> 778 779 (4) See Figure 1 of Alliez et al. 2020 at 780<https://arxiv.org/pdf/1905.11123.pdf> 781 782 (5) <http://research.iac.es/proyecto/abyss> 783 784 (6) <http://mirrors.ctan.org/graphics/pgf/base/doc/pgfmanual.pdf> 785 786 (7) 787<http://mirrors.ctan.org/graphics/pgf/contrib/pgfplots/doc/pgfplots.pdf> 788 789 790File: gnuastro.info, Node: Full script of annotations on figure, Prev: Annotations for figure in paper, Up: Annotations for figure in paper 791 7925.2.4.1 Full script of annotations on figure 793............................................ 794 795In *note Annotations for figure in paper::, we each one of the steps to 796add annotations over an image were described in detail. So if you have 797understood the steps, but need to add annotations over an image, 798repeating those steps individually will be annoying. Therefore in this 799section, we’ll summarize all the steps in a single script that you can 800simply copy-paste into a text editor, configure, and run. 801 802*Necessary files:* To run this script, you will need an image to crop 803your object from (here assuming its called ‘ah_f160w.fits’ with a 804certain zeropoint) and two ‘my-figure.tex’ and ‘report.tex’ files that 805were fully included in *note Annotations for figure in paper::. Also, 806we have brought the redshift as a parameter here. But if the center of 807your image always points to your main object, you can also include the 808Query command to automatically find the object’s redshift from NED. 809Alternatively, your image may already be cropped, in this case, you can 810remove the cropping step and 811 812# Parameters. 813sblow=22 # Minimum surface brightness. 814sbhigh=30 # Maximum surface brightness. 815bdir=build # Build directory location on filesystem. 816numticks=7 # Number of major ticks in each axis. 817redshift=0.619 # Redshift of object of interest. 818zeropoint=25.94 # Zeropoint of input image. 819scalelineinkpc=20 # Length of scale-line (in kilo parsecs). 820input=ah_f160w.fits # Name of input (to crop). 821 822# Stop the script in case of a crash. 823set -e 824 825# Build directory 826if ! [ -d $bdir ]; then mkdir $bdir; fi 827 828# Crop out the desired region. 829crop=$bdir/crop.fits 830astcrop $input --center=53.1616278,-27.7802446 --mode=wcs \ 831 --width=20/3600 --output=$crop 832 833# Warp the image to larger pixels to show surface brightness better. 834scaled=$bdir/scaled.fits 835astwarp $crop --centeroncorner --scale=1/3 --output=$scaled 836 837# Calculate the pixel area and convert image to Surface brightness. 838sb=$bdir/sb.fits 839pixarea=$(astfits $scaled --pixelscale --quiet \ 840 | awk '{print $1*3600*$2*3600}') 841astarithmetic $scaled abs $zeropoint counts-to-mag \ 842 $pixarea log10 2.5 x + --output=$sb 843 844# Convert the Surface brightness image into PDF. 845sbpdf=$bdir/sb.pdf 846astconvertt $sb --colormap=gray --borderwidth=0 \ 847 --fluxhigh=$sbhigh --fluxlow=$sblow --output=$sbpdf 848 849# Specify the coordinates of the scale line (specifying a certain 850# width in kpc). We will put it on the top-right side of the image (5% 851# of the full width of the image away from the edge). 852coverage=$(astfits $sb --skycoverage --quiet | awk 'NR==2') 853scalelinedec=$(echo $coverage | awk '{print $4-($4-$3)*0.05}') 854scalelinerastart=$(echo $coverage | awk '{print $1+($2-$1)*0.05}') 855scalelineraend=$(astcosmiccal --redshift=$redshift --arcsectandist \ 856 | awk '{start='$scalelinerastart'; \ 857 width='$scalelineinkpc'/$1/3600; \ 858 print start+width}') 859 860# Write the LaTeX macros to use in plot. Start with the thick line 861# showing tangential distance. 862macros=$bdir/macros.tex 863printf '\\newcommand{\\maScaleDec}'"{$scalelinedec}\n" > $macros 864printf '\\newcommand{\\maScaleRAa}'"{$scalelinerastart}\n" >> $macros 865printf '\\newcommand{\\maScaleRAb}'"{$scalelineraend}\n" >> $macros 866printf '\\newcommand{\\maScaleKpc}'"{$scalelineinkpc}\n" >> $macros 867printf '\\newcommand{\\maCenterZ}'"{$redshift}\n" >> $macros 868 869# Add image extrema for the coordinates. 870v=$(echo $coverage | awk '{print $1}') 871printf '\\newcommand{\maCropRAMin}'"{$v}\n" >> $macros 872v=$(echo $coverage | awk '{print $2}') 873printf '\\newcommand{\maCropRAMax}'"{$v}\n" >> $macros 874v=$(echo $coverage | awk '{print $3}') 875printf '\\newcommand{\maCropDecMin}'"{$v}\n" >> $macros 876v=$(echo $coverage | awk '{print $4}') 877printf '\\newcommand{\maCropDecMax}'"{$v}\n" >> $macros 878 879# Distance between each tick value. 880v=$(echo $coverage | awk '{print ($2-$1)/'$numticks'}') 881printf '\\newcommand{\maTickDist}'"{$v}\n" >> $macros 882printf '\\newcommand{\maSBlow}'"{$sblow}\n" >> $macros 883printf '\\newcommand{\maSBhigh}'"{$sbhigh}\n" >> $macros 884 885# Copy the LaTeX source into the build directory and go there to run 886# it and have all the temporary LaTeX files there. 887cp report.tex my-figure.tex $bdir 888cd $bdir 889rm -f *-figure0* 890pdflatex -shell-escape -halt-on-error report.tex 891 892 893File: gnuastro.info, Node: Invoking astconvertt, Prev: Annotations for figure in paper, Up: ConvertType 894 8955.2.5 Invoking ConvertType 896-------------------------- 897 898ConvertType will convert any recognized input file type to any specified 899output type. The executable name is ‘astconvertt’ with the following 900general template 901 902 $ astconvertt [OPTION...] InputFile [InputFile2] ... [InputFile4] 903 904One line examples: 905 906 ## Convert an image in FITS to PDF: 907 $ astconvertt image.fits --output=pdf 908 909 ## Similar to before, but use the Viridis color map: 910 $ astconvertt image.fits --colormap=viridis --output=pdf 911 912 ## Convert an image in JPEG to FITS (with multiple extensions 913 ## if its color): 914 $ astconvertt image.jpg -oimage.fits 915 916 ## Use three plain text 2D arrays to create an RGB JPEG output: 917 $ astconvertt f1.txt f2.txt f3.fits -o.jpg 918 919 ## Use two images and one blank for an RGB EPS output: 920 $ astconvertt M31_r.fits M31_g.fits blank -oeps 921 922 ## Directly pass input from output of another program through Standard 923 ## input (not a file). 924 $ cat 2darray.txt | astconvertt -oimg.fits 925 926The output’s file format will be interpreted from the value given to the 927‘--output’ option. It can either be given on the command-line or in any 928of the configuration files (see *note Configuration files::). Note that 929if the output suffix is not recognized, it will default to plain text 930format, see *note Recognized file formats::. 931 932 At most four input files (one for each color channel for formats that 933allow it) are allowed in ConvertType. The first input dataset can 934either be a file or come from Standard input (see *note Standard 935input::). The order of multiple input files is important. After 936reading the input file(s) the number of color channels in all the inputs 937will be used to define which color space to use for the outputs and how 938each color channel is interpreted. 939 940 Some formats can allow more than one color channel (for example in 941the JPEG format, see *note Recognized file formats::). If there is one 942input dataset (color channel) the output will be gray-scale, if three 943input datasets (color channels) are given, they are respectively 944considered to be the red, green and blue color channels. Finally, if 945there are four color channels they will be be cyan, magenta, yellow and 946black (CMYK colors). 947 948 The value to ‘--output’ (or ‘-o’) can be either a full file name or 949just the suffix of the desired output format. In the former case, it 950will used for the output. In the latter case, the name of the output 951file will be set based on the automatic output guidelines, see *note 952Automatic output::. Note that the suffix name can optionally start a 953‘.’ (dot), so for example ‘--output=.jpg’ and ‘--output=jpg’ are 954equivalent. See *note Recognized file formats::. 955 956 Besides the common set of options explained in *note Common 957options::, the options to ConvertType can be classified into input, 958output and flux related options. The majority of the options are to do 959with the flux range. Astronomical data usually have a very large 960dynamic range (difference between maximum and minimum value) and 961different subjects might be better demonstrated with a limited flux 962range. 963 964Input: 965‘-h STR/INT’ 966‘--hdu=STR/INT’ 967 Input HDU name or counter (counting from 0) for each input FITS 968 file. If the same HDU should be used from all the FITS files, you 969 can use the ‘--globalhdu’ option described below. In ConvertType, 970 it is possible to call the HDU option multiple times for the 971 different input FITS or TIFF files in the same order that they are 972 called on the command-line. Note that in the TIFF standard, one 973 ‘directory’ (similar to a FITS HDU) may contain multiple color 974 channels (for example when the image is in RGB). 975 976 Except for the fact that multiple calls are possible, this option 977 is identical to the common ‘--hdu’ in *note Input output options::. 978 The number of calls to this option cannot be less than the number 979 of input FITS or TIFF files, but if there are more, the extra HDUs 980 will be ignored, note that they will be read in the order described 981 in *note Configuration file precedence::. 982 983 Unlike CFITSIO, libtiff (which is used to read TIFF files) only 984 recognizes numbers (counting from zero, similar to CFITSIO) for 985 ‘directory’ identification. Hence the concept of names is not 986 defined for the directories and the values to this option for TIFF 987 files must be numbers. 988 989‘-g STR/INT’ 990‘--globalhdu=STR/INT’ 991 Use the value given to this option (a HDU name or a counter, 992 starting from 0) for the HDU identifier of all the input FITS 993 files. This is useful when all the inputs are distributed in 994 different files, but have the same HDU in those files. 995 996Output: 997 998‘-w FLT’ 999‘--widthincm=FLT’ 1000 The width of the output in centimeters. This is only relevant for 1001 those formats that accept such a width (not plain text for 1002 example). For most digital purposes, the number of pixels is far 1003 more important than the value to this parameter because you can 1004 adjust the absolute width (in inches or centimeters) in your 1005 document preparation program. 1006 1007‘-b INT’ 1008‘--borderwidth=INT’ 1009 The width of the border to be put around the EPS and PDF outputs in 1010 units of PostScript points. There are 72 or 28.35 PostScript 1011 points in an inch or centimeter respectively. In other words, 1012 there are roughly 3 PostScript points in every millimeter. If you 1013 are planning on adding a border, its significance is highly 1014 correlated with the value you give to the ‘--widthincm’ parameter. 1015 1016 Unfortunately in the document structuring convention of the 1017 PostScript language, the “bounding box” has to be in units of 1018 PostScript points with no fractions allowed. So the border values 1019 only have to be specified in integers. To have a final border that 1020 is thinner than one PostScript point in your document, you can ask 1021 for a larger width in ConvertType and then scale down the output 1022 EPS or PDF file in your document preparation program. For example 1023 by setting ‘width’ in your ‘includegraphics’ command in TeX or 1024 LaTeX. Since it is vector graphics, the changes of size have no 1025 effect on the quality of your output quality (pixels don’t get 1026 different values). 1027 1028‘-x’ 1029‘--hex’ 1030 Use Hexadecimal encoding in creating EPS output. By default the 1031 ASCII85 encoding is used which provides a much better compression 1032 ratio. When converted to PDF (or included in TeX or LaTeX which is 1033 finally saved as a PDF file), an efficient binary encoding is used 1034 which is far more efficient than both of them. The choice of EPS 1035 encoding will thus have no effect on the final PDF. 1036 1037 So if you want to transfer your EPS files (for example if you want 1038 to submit your paper to arXiv or journals in PostScript), their 1039 storage might become important if you have large images or lots of 1040 small ones. By default ASCII85 encoding is used which offers a 1041 much better compression ratio (nearly 40 percent) compared to 1042 Hexadecimal encoding. 1043 1044‘-u INT’ 1045‘--quality=INT’ 1046 The quality (compression) of the output JPEG file with values from 1047 0 to 100 (inclusive). For other formats the value to this option 1048 is ignored. Note that only in gray-scale (when one input color 1049 channel is given) will this actually be the exact quality (each 1050 pixel will correspond to one input value). If it is in color mode, 1051 some degradation will occur. While the JPEG standard does support 1052 loss-less graphics, it is not commonly supported. 1053 1054‘--colormap=STR[,FLT,...]’ 1055 The color map to visualize a single channel. The first value given 1056 to this option is the name of the color map, which is shown below. 1057 Some color maps can be configured. In this case, the configuration 1058 parameters are optionally given as numbers following the name of 1059 the color map for example see ‘hsv’. The table below contains the 1060 usable names of the color maps that are currently supported: 1061 1062 ‘gray’ 1063 ‘grey’ 1064 Grayscale color map. This color map doesn’t have any 1065 parameters. The full dataset range will be scaled to 0 and 1066 $2^8-1=255$ to be stored in the requested format. 1067 1068 ‘hsv’ 1069 Hue, Saturation, Value(1) color map. If no values are given 1070 after the name (‘--colormap=hsv’), the dataset will be scaled 1071 to 0 and 360 for hue covering the full spectrum of colors. 1072 However, you can limit the range of hue (to show only a 1073 special color range) by explicitly requesting them after the 1074 name (for example ‘--colormap=hsv,20,240’). 1075 1076 The mapping of a single-channel dataset to HSV is done through 1077 the Hue and Value elements: Lower dataset elements have lower 1078 “value” _and_ lower “hue”. This creates darker colors for 1079 fainter parts, while also respecting the range of colors. 1080 1081 ‘viridis’ 1082 Viridis is the default colormap of the popular Matplotlib 1083 module of Python and available in many other visualization 1084 tools like PGFPlots. 1085 1086 ‘sls’ 1087 The SLS color range, taken from the commonly used SAO DS9 1088 (http://ds9.si.edu). The advantage of this color range is 1089 that it starts with black, going into dark blue and finishes 1090 with the brighter colors of red and white. So unlike the HSV 1091 color range, it includes black and white and brighter colors 1092 (like yellow, red) show the larger values. 1093 1094 ‘sls-inverse’ 1095 The inverse of the SLS color map (see above), where the lowest 1096 value corresponds to white and the highest value is black. 1097 While SLS is good for visualizing on the monitor, SLS-inverse 1098 is good for printing. 1099 1100‘--rgbtohsv’ 1101 When there are three input channels and the output is in the FITS 1102 format, interpret the three input channels as red, green and blue 1103 channels (RGB) and convert them to the hue, saturation, value (HSV) 1104 color space. 1105 1106 The currently supported output formats of ConvertType don’t have 1107 native support for HSV. Therefore this option is only supported 1108 when the output is in FITS format and each of the hue, saturation 1109 and value arrays can be saved as one FITS extension in the output 1110 for further analysis (for example to select a certain color). 1111 1112Flux range: 1113 1114‘-c STR’ 1115‘--change=STR’ 1116 (‘=STR’) Change pixel values with the following format ‘"from1:to1, 1117 from2:to2,..."’. This option is very useful in displaying labeled 1118 pixels (not actual data images which have noise) like segmentation 1119 maps. In labeled images, usually a group of pixels have a fixed 1120 integer value. With this option, you can manipulate the labels 1121 before the image is displayed to get a better output for print or 1122 to emphasize on a particular set of labels and ignore the rest. 1123 The labels in the images will be changed in the same order given. 1124 By default first the pixel values will be converted then the pixel 1125 values will be truncated (see ‘--fluxlow’ and ‘--fluxhigh’). 1126 1127 You can use any number for the values irrespective of your final 1128 output, your given values are stored and used in the double 1129 precision floating point format. So for example if your input 1130 image has labels from 1 to 20000 and you only want to display those 1131 with labels 957 and 11342 then you can run ConvertType with these 1132 options: 1133 1134 $ astconvertt --change=957:50000,11342:50001 --fluxlow=5e4 \ 1135 --fluxhigh=1e5 segmentationmap.fits --output=jpg 1136 1137 While the output JPEG format is only 8 bit, this operation is done 1138 in an intermediate step which is stored in double precision 1139 floating point. The pixel values are converted to 8-bit after all 1140 operations on the input fluxes have been complete. By placing the 1141 value in double quotes you can use as many spaces as you like for 1142 better readability. 1143 1144‘-C’ 1145‘--changeaftertrunc’ 1146 Change pixel values (with ‘--change’) after truncation of the flux 1147 values, by default it is the opposite. 1148 1149‘-L FLT’ 1150‘--fluxlow=FLT’ 1151 The minimum flux (pixel value) to display in the output image, any 1152 pixel value below this value will be set to this value in the 1153 output. If the value to this option is the same as ‘--fluxhigh’, 1154 then no flux truncation will be applied. Note that when multiple 1155 channels are given, this value is used for all the color channels. 1156 1157‘-H FLT’ 1158‘--fluxhigh=FLT’ 1159 The maximum flux (pixel value) to display in the output image, see 1160 ‘--fluxlow’. 1161 1162‘-m INT’ 1163‘--maxbyte=INT’ 1164 This is only used for the JPEG and EPS output formats which have an 1165 8-bit space for each channel of each pixel. The maximum value in 1166 each pixel can therefore be $2^8-1=255$. With this option you can 1167 change (decrease) the maximum value. By doing so you will decrease 1168 the dynamic range. It can be useful if you plan to use those 1169 values for other purposes. 1170 1171‘-A INT’ 1172‘--forcemin=INT’ 1173 Enforce the value of ‘--fluxlow’ (when its given), even if its 1174 smaller than the minimum of the dataset and the output is format 1175 supporting color. This is particularly useful when you are 1176 converting a number of images to a common image format like JPEG or 1177 PDF with a single command and want them all to have the same range 1178 of colors, independent of the contents of the dataset. Note that 1179 if the minimum value is smaller than ‘--fluxlow’, then this option 1180 is redundant. 1181 1182 By default, when the dataset only has two values, _and_ the output 1183 format is PDF or EPS, ConvertType will use the PostScript 1184 optimization that allows setting the pixel values per bit, not byte 1185 (*note Recognized file formats::). This can greatly help reduce 1186 the file size. However, when ‘--fluxlow’ or ‘--fluxhigh’ are 1187 called, this optimization is disabled: even though there are only 1188 two values (is binary), the difference between them does not 1189 correspond to the full contrast of black and white. 1190 1191‘-B INT’ 1192‘--forcemax=INT’ 1193 Similar to ‘--forcemin’, but for the maximum. 1194 1195‘-i’ 1196‘--invert’ 1197 For 8-bit output types (JPEG, EPS, and PDF for example) the final 1198 value that is stored is inverted so white becomes black and vice 1199 versa. The reason for this is that astronomical images usually 1200 have a very large area of blank sky in them. The result will be 1201 that a large are of the image will be black. Note that this 1202 behavior is ideal for gray-scale images, if you want a color image, 1203 the colors are going to be mixed up. 1204 1205 ---------- Footnotes ---------- 1206 1207 (1) <https://en.wikipedia.org/wiki/HSL_and_HSV> 1208 1209 1210File: gnuastro.info, Node: Table, Next: Query, Prev: ConvertType, Up: Data containers 1211 12125.3 Table 1213========= 1214 1215Tables are the high-level products of processing on low-leveler data 1216like images or spectra. For example in Gnuastro, MakeCatalog will 1217process the pixels over an object and produce a catalog (or table) with 1218the properties of each object like magnitudes, positions and etc (see 1219*note MakeCatalog::). Each one of these properties is a column in its 1220output catalog (or table) and for each input object, we have a row. 1221 1222 When there are only a small number of objects (rows) and not too many 1223properties (columns), then a simple plain text file is mainly enough to 1224store, transfer, or even use the produced data. However, to be more 1225efficient, astronomers have defined the FITS binary table standard to 1226store data in a binary format (which cannot be seen in a text editor 1227text). This can offer major advantages: the file size will be greatly 1228reduced and the reading and writing will also be faster (because the RAM 1229and CPU also work in binary). The acceptable table formats are fully 1230described in *note Tables::. 1231 1232 Binary tables are not easily readable with basic plain-text editors. 1233There is no fixed/unified standard on how the zero and ones should be 1234interpreted. Unix-like operating systems have flourished because of a 1235simple fact: communication between the various tools is based on human 1236readable characters(1). So while the FITS table standards are very 1237beneficial for the tools that recognize them, they are hard to use in 1238the vast majority of available software. This creates limitations for 1239their generic use. 1240 1241 Table is Gnuastro’s solution to this problem. Table has a large set 1242of operations that you can directly do on any recognized table (like 1243selecting certain rows, doing arithmetic on the columns and etc). For 1244operations that Table doesn’t do internally, FITS tables (ASCII or 1245binary) are directly accessible to the users of Unix-like operating 1246systems (in particular those working the command-line or shell, see 1247*note Command-line interface::). With Table, a FITS table (in binary or 1248ASCII formats) is only one command away from AWK (or any other tool you 1249want to use). Just like a plain text file that you read with the ‘cat’ 1250command. You can pipe the output of Table into any other tool for 1251higher-level processing, see the examples in *note Invoking asttable:: 1252for some simple examples. 1253 1254 In the sections below we describe how to effectively use the Table 1255program. We start with *note Column arithmetic::, where the basic 1256concept and methods of applying arithmetic operations on one or more 1257columns are discussed. Afterwards, in *note Operation precedence in 1258Table::, we review the various types of operations available and their 1259precedence in an instance of calling Table. This is a good place to get 1260a general feeling of all the things you can do with Table. Finally, in 1261*note Invoking asttable::, we give some examples and describe each 1262option in Table. 1263 1264* Menu: 1265 1266* Column arithmetic:: How to do operations on table columns. 1267* Operation precedence in Table:: Order of running options in Table. 1268* Invoking asttable:: Options and arguments to Table. 1269 1270 ---------- Footnotes ---------- 1271 1272 (1) In “The art of Unix programming”, Eric Raymond makes this 1273suggestion to programmers: “When you feel the urge to design a complex 1274binary file format, or a complex binary application protocol, it is 1275generally wise to lie down until the feeling passes.”. This is a great 1276book and strongly recommended, give it a look if you want to truly enjoy 1277your work/life in this environment. 1278 1279 1280File: gnuastro.info, Node: Column arithmetic, Next: Operation precedence in Table, Prev: Table, Up: Table 1281 12825.3.1 Column arithmetic 1283----------------------- 1284 1285In many scenarios, you want to apply some kind of operation on the 1286columns and save them in another table or feed them into another 1287program. With Table you can do a rich set of operations on the contents 1288of one or more columns in a table, and save the resulting values as new 1289column(s) in the output table. For seeing the precedence of Column 1290arithmetic in relation to other Table operators, see *note Operation 1291precedence in Table::. 1292 1293 To enable column arithmetic, the first 6 characters of the value to 1294‘--column’ (‘-c’) should be the activation word ‘‘arith ’’ (note the 1295space character in the end, after ‘‘arith’’). After the activation 1296word, you can use reverse polish notation to identify the operators and 1297their operands, see *note Reverse polish notation::. Just note that 1298white-space characters are used between the tokens of the arithmetic 1299expression and that they are meaningful to the command-line environment. 1300Therefore the whole expression (including the activation word) has to be 1301quoted on the command-line or in a shell script (see the examples 1302below). 1303 1304 To identify a column you can directly use its name, or specify its 1305number (counting from one, see *note Selecting table columns::). When 1306you are giving a column number, it is necessary to prefix the number 1307with a ‘$’, similar to AWK. Otherwise the number is not distinguishable 1308from a constant number to use in the arithmetic operation. 1309 1310 For example with the command below, the first two columns of 1311‘table.fits’ will be printed along with a third column that is the 1312result of multiplying the first column with $10^{10}$ (for example to 1313convert wavelength from Meters to Angstroms). Note that without the 1314‘<$>’, it is not possible to distinguish between “1” as a 1315column-counter, or “1” as a constant number to use in the arithmetic 1316operation. Also note that because of the significance of <$> for the 1317command-line environment, the single-quotes are the recommended quoting 1318method (as in an AWK expression), not double-quotes (for the 1319significance of using single quotes see the box below). 1320 1321 $ asttable table.fits -c1,2 -c'arith $1 1e10 x' 1322 1323*Single quotes when string contains <$>*: On the command-line, or in 1324shell-scripts, <$> is used to expand variables, for example ‘echo $PATH’ 1325prints the value (a string of characters) in the variable ‘PATH’, it 1326will not simply print ‘$PATH’. This operation is also permitted within 1327double quotes, so ‘echo "$PATH"’ will produce the same output. This is 1328good when printing values, for example in the command below, ‘$PATH’ 1329will expand to the value within it. 1330 1331 $ echo "My path is: $PATH" 1332 1333 If you actually want to return the literal string ‘$PATH’, not the 1334value in the ‘PATH’ variable (like the scenario here in column 1335arithmetic), you should put it in single quotes like below. The printed 1336value here will include the ‘$’, please try it to see for your self and 1337compare to above. 1338 1339 $ echo 'My path is: $PATH' 1340 1341 Therefore, when your column arithmetic involves the <$> sign (to 1342specify columns by number), quote your ‘arith ’ string with a single 1343quotation mark. Otherwise you can use both single or double quotes. 1344 1345 Alternatively, if the columns have meta-data and the first two are 1346respectively called ‘AWAV’ and ‘SPECTRUM’, the command above is 1347equivalent to the command below. Note that the character ‘<$>’ is no 1348longer necessary in this scenario (because names will not be confused 1349with numbers): 1350 1351 $ asttable table.fits -cAWAV,SPECTRUM -c'arith AWAV 1e10 x' 1352 1353 Comparison of the two commands above clearly shows why it is 1354recommended to use column names instead of numbers. When the columns 1355have descriptive names, the command/script actually becomes much more 1356readable, describing the intent of the operation. It is also 1357independent of the low-level table structure: for the second command, 1358the column numbers of the ‘AWAV’ and ‘SPECTRUM’ columns in ‘table.fits’ 1359is irrelevant. 1360 1361 Column arithmetic changes the values of the data within the column. 1362So the old column meta data can’t be used any more. By default the 1363output column of the arithmetic operation will be given a generic 1364metadata (for example its name will be ‘ARITH_1’, which is hardly 1365useful!). But meta data are critically important and it is good 1366practice to always have short, but descriptive, names for each columns, 1367units and also some comments for more explanation. To add metadata to a 1368column, you can use the ‘--colmetadata’ option that is described in 1369*note Invoking asttable:: and *note Operation precedence in Table::. 1370 1371 Finally, since the arithmetic expressions are a value to ‘--column’, 1372it doesn’t necessarily have to be a separate option, so the commands 1373above are also identical to the command below (note that this only has 1374one ‘-c’ option). Just be very careful with the quoting! 1375 1376 $ asttable table.fits -cAWAV,SPECTRUM,'arith AWAV 1e10 x' 1377 1378 Almost all the arithmetic operators of *note Arithmetic operators:: 1379are also supported for column arithmetic in Table. In particular, the 1380few that are not present in the Gnuastro library(1) aren’t yet supported 1381for column arithmetic. Besides the operators in *note Arithmetic 1382operators::, several operators are only available in Table to use on 1383table columns. 1384 1385‘wcs-to-img’ 1386 Convert the given WCS positions to image/dataset coordinates based 1387 on the number of dimensions in the WCS structure of ‘--wcshdu’ 1388 extension/HDU in ‘--wcsfile’. It will output the same number of 1389 columns. The first popped operand is the last FITS dimension. 1390 1391 For example the two commands below (which have the same output) 1392 will produce 5 columns. The first three columns are the input 1393 table’s ID, RA and Dec columns. The fourth and fifth columns will 1394 be the pixel positions in ‘image.fits’ that correspond to each RA 1395 and Dec. 1396 1397 $ asttable table.fits -cID,RA,DEC,'arith RA DEC wcs-to-img' \ 1398 --wcsfile=image.fits 1399 $ asttable table.fits -cID,RA -cDEC \ 1400 -c'arith RA DEC wcs-to-img' --wcsfile=image.fits 1401 1402‘img-to-wcs’ 1403 Similar to ‘wcs-to-img’, except that image/dataset coordinates are 1404 converted to WCS coordinates. 1405 1406‘distance-flat’ 1407 Return the distance between two points assuming they are on a flat 1408 surface. Note that each point needs two coordinates, so this 1409 operator needs four operands (currently it only works for 2D 1410 spaces). The first and second popped operands are considered to 1411 belong to one point and the third and fourth popped operands to the 1412 second point. 1413 1414 Each of the input points can be a single coordinate or a full table 1415 column (containing many points). In other words, the following 1416 commands are all valid: 1417 1418 $ asttable table.fits \ 1419 -c'arith X1 Y1 X2 Y2 distance-flat' 1420 $ asttable table.fits \ 1421 -c'arith X Y 12.345 6.789 distance-flat' 1422 $ asttable table.fits \ 1423 -c'arith 12.345 6.789 X Y distance-flat' 1424 1425 In the first case we are assuming that ‘table.fits’ has the 1426 following four columns ‘X1’, ‘Y1’, ‘X2’, ‘Y2’. The returned column 1427 by this operator will be the difference between two points in each 1428 row with coordinates like the following (‘X1’, ‘Y1’) and (‘X2’, 1429 ‘Y2’). In other words, for each row, the distance between 1430 different points is calculated. In the second and third cases 1431 (which are identical), it is assumed that ‘table.fits’ has the two 1432 columns ‘X’ and ‘Y’. The returned column by this operator will be 1433 the difference of each row with the fixed point at (12.345, 6.789). 1434 1435‘distance-on-sphere’ 1436 Return the spherical angular distance (along a great circle, in 1437 degrees) between the given two points. Note that each point needs 1438 two coordinates (in degrees), so this operator needs four operands. 1439 The first and second popped operands are considered to belong to 1440 one point and the third and fourth popped operands to the second 1441 point. 1442 1443 Each of the input points can be a single coordinate or a full table 1444 column (containing many points). In other words, the following 1445 commands are all valid: 1446 1447 $ asttable table.fits \ 1448 -c'arith RA1 DEC1 RA2 DEC2 distance-on-sphere' 1449 $ asttable table.fits \ 1450 -c'arith RA DEC 9.876 5.432 distance-on-sphere' 1451 $ asttable table.fits \ 1452 -c'arith 9.876 5.432 RA DEC distance-on-sphere' 1453 1454 In the first case we are assuming that ‘table.fits’ has the 1455 following four columns ‘RA1’, ‘DEC1’, ‘RA2’, ‘DEC2’. The returned 1456 column by this operator will be the difference between two points 1457 in each row with coordinates like the following (‘RA1’, ‘DEC1’) and 1458 (‘RA2’, ‘DEC2’). In other words, for each row, the angular 1459 distance between different points is calculated. In the second and 1460 third cases (which are identical), it is assumed that ‘table.fits’ 1461 has the two columns ‘RA’ and ‘DEC’. The returned column by this 1462 operator will be the difference of each row with the fixed point at 1463 (9.876, 5.432). 1464 1465 The distance (along a great circle) on a sphere between two points 1466 is calculated with the equation below, where $r_1$, $r_2$, $d_1$ 1467 and $d_2$ are the right ascensions and declinations of points 1 and 1468 2. 1469 1470 $$\cos(d)=\sin(d_1)\sin(d_2)+\cos(d_1)\cos(d_2)\cos(r_1-r_2)$$ 1471 1472‘ra-to-degree’ 1473 Convert the hour-wise Right Ascension (RA) string, in the 1474 sexagesimal format of ‘_h_m_s’ or ‘_:_:_’, to degrees. Note that 1475 the input column has to have a string format. In FITS tables, 1476 string columns are well-defined. For plain-text tables, please 1477 follow the standards defined in *note Gnuastro text table format::, 1478 otherwise the string column won’t be read. 1479 $ asttable catalog.fits -c'arith RA ra-to-degree' 1480 $ asttable catalog.fits -c'arith $5 ra-to-degree' 1481 1482‘dec-to-degree’ 1483 Convert the sexagesimal Declination (Dec) string, in the format of 1484 ‘_d_m_s’ or ‘_:_:_’, to degrees (a single floating point number). 1485 For more details please see the ‘ra-to-degree’ operator. 1486 1487‘degree-to-ra’ 1488 Convert degrees (a column with a single floating point number) to 1489 the Right Ascension, RA, string (in the sexagesimal format hours, 1490 minutes and seconds, written as ‘_h_m_s’). The output will be a 1491 string column so no further mathematical operations can be done on 1492 it. The output file can be in any format (for example FITS or 1493 plain-text). If it is plain-text, the string column will be 1494 written following the standards described in *note Gnuastro text 1495 table format::. 1496 1497‘degree-to-dec’ 1498 Convert degrees (a column with a single floating point number) to 1499 the Declination, Dec, string (in the format of ‘_d_m_s’). See the 1500 ‘degree-to-ra’ for more on the format of the output. 1501 1502‘date-to-sec’ 1503 The popped operand should be a string column in the FITS date 1504 format (most generally: ‘YYYY-MM-DDThh:mm:ss.ddd...’). This 1505 operator will return the corresponding Unix epoch time (number of 1506 seconds that have passed since 00:00:00 Thursday, January 1st, 1507 1970). The returned operand will be named ‘UNIXSEC’ (short for 1508 Unix-seconds). If any of the times have sub-second precision, the 1509 numeric datatype of the column will be 64-bit floating point type. 1510 Otherwise it will be a 64-bit, signed integer, see *note Numeric 1511 data types::. 1512 1513 For example, in the example below we are using this operator, in 1514 combination with the ‘--keyvalue’ option of the Fits program, to 1515 sort your desired FITS files by observation date (value in the 1516 ‘DATE-OBS’ keyword in example below): 1517 1518 $ astfits *.fits --keyvalue=DATE-OBS --colinfoinstdout \ 1519 | asttable -cFILENAME,'arith DATE-OBS date-to-sec' \ 1520 --colinfoinstdout \ 1521 | asttable --sort=UNIXSEC 1522 1523 If you don’t need to see the Unix-seconds any more, you can add a 1524 ‘-cFILENAME’ (short for ‘--column=FILENAME’) at the end. For more 1525 on ‘--keyvalue’, see *note Keyword inspection and manipulation::. 1526 1527 ---------- Footnotes ---------- 1528 1529 (1) For a list of the Gnuastro library arithmetic operators, please 1530see the macros starting with ‘GAL_ARITHMETIC_OP’ and ending with the 1531operator name in *note Arithmetic on datasets::. 1532 1533 1534File: gnuastro.info, Node: Operation precedence in Table, Next: Invoking asttable, Prev: Column arithmetic, Up: Table 1535 15365.3.2 Operation precedence in Table 1537----------------------------------- 1538 1539The Table program can do many operations on the rows and columns of the 1540input tables and they aren’t always applied in the order you call the 1541operation on the command-line. In this section we will describe which 1542operation is done before/after which operation. Knowing this precedence 1543table is important to avoid confusion when you ask for more than one 1544operation. For a description of each option, please see *note Invoking 1545asttable::. 1546 1547Column information (‘--information’ or ‘-i’) 1548 When given this option, the column data are not read at all. Table 1549 simply reads the column metadata (name, units, numeric data type 1550 and comments), and the number of rows and prints them. Table then 1551 terminates and no other operation is done. This can therefore be 1552 called at the end of an arbitrarily long Table command only to 1553 remember the column metadata, then deleted to continue writing the 1554 command (using the shell’s history to retrieve the previous command 1555 with an up-arrow key). 1556 1557Columns from other files (‘--catcolumns’ and ‘--catcolumnfile’) 1558 With this feature, you can import columns from other tables (in 1559 other files). The rest of the operations below are done on the 1560 rows, therefore you can merge the columns of various tables into 1561 one table, then start limiting the rows to have in the output. 1562 1563 If any of the row-based operations below are requested in the same 1564 Table command, they will also be applied to the rows of these added 1565 columns. However, the conditions to keep/reject rows can only be 1566 applied to the rows of the main input table. 1567 1568Row selection by value in a column 1569 • ‘--range’: only keep rows within a certain interval in given 1570 column. 1571 • ‘--inpolygon’: only keep rows within the polygon of 1572 ‘--polygon’. 1573 • ‘--outpolygon’: only keep rows outside the polygon of 1574 ‘--polygon’. 1575 • ‘--equal’: only keep rows with specified value in given 1576 column. 1577 • ‘--notequal’: only keep rows without specified value in given 1578 column. 1579 These options take certain column(s) as input and remove some rows 1580 from the full table (all columns), based on the given limitations. 1581 They can be called any number of times (to limit the final rows 1582 based on values in different columns for example). Since these are 1583 row-rejection operations, their internal order is irrelevant. In 1584 other words, it makes no difference if ‘--equal’ is called before 1585 or after ‘--range’ for example. 1586 1587 As a side-effect, because NaN/blank values are defined to fail on 1588 any condition, these operations will also remove rows with 1589 NaN/blank values in the specified column they are checking. Also, 1590 the columns that are used for these operations don’t necessarily 1591 have to be in the final output table (you may not need the column 1592 after doing the selection based on it). 1593 1594 Even though these options are applied after merging columns from 1595 other tables, currently their condition-columns can only come from 1596 the main input table. In other words, even though the rows of the 1597 added columns (from another file) will also be selected with these 1598 options, the condition to keep/reject rows cannot be taken from the 1599 newly added columns. 1600 1601 These options are applied first because the speed of later 1602 operations can be greatly affected by the number of rows. For 1603 example, if you also call the ‘--sort’ option, and your row 1604 selection will result in 50 rows (from an input of 1000 rows), 1605 limiting the number of rows can greatly speed up the sorting in 1606 your final output. 1607 1608Sorting (‘--sort’) 1609 Sort of the rows based on values in a certain column. The column 1610 to sort by can only come from the main input table columns (not 1611 columns that may have been added with ‘--catcolumnfile’). 1612 1613Row selection (by position) 1614 • ‘--head’: keep only requested number of top rows. 1615 • ‘--tail’: keep only requested number of bottom rows. 1616 • ‘--rowrandom’: keep only a random number of rows. 1617 • ‘--rowlimit’: keep only rows within a certain positional 1618 interval. 1619 1620 These options limit/select rows based on their position within the 1621 table (not their value in any certain column). 1622 1623Column arithmetic 1624 Once the final rows are selected in the requested order, column 1625 arithmetic is done (if requested). For more on column arithmetic, 1626 see *note Column arithmetic::. 1627 1628Column metadata (‘--colmetadata’) 1629 Changing column metadata is necessary after column arithmetic or 1630 adding new columns from other tables (that were done above). 1631 1632Row selection (‘--noblank’) 1633 Theoretically, this method of row selection by value should have 1634 been done with the other logically similar options like ‘--range’ 1635 or ‘--equal’. However, those options are applied to the raw 1636 (input) column value. In some scenarios, you need to apply 1637 arithmetic operations on the columns (through *note Column 1638 arithmetic::) before rejecting the undesired rows. After the 1639 arithmetic operation is done, you can use the ‘where’ operator to 1640 set the non-desired columns to NaN/blank and use ‘--noblank’ to 1641 remove them at the end. See the example below for applying any 1642 generic value-based row selection based on ‘--noblank’. 1643 1644 As an example, let’s review how Table interprets the command below. 1645We are assuming that ‘table.fits’ contains at least three columns: ‘RA’, 1646‘DEC’ and ‘PARAM’ and you only want the RA and Dec of the rows where 1647$p\times 2<5$ ($p$ is the value of each row in the ‘PARAM’ column). 1648 1649 asttable table.fits -cRA,DEC --noblank=MULTIP \ 1650 -c'arith PARAM 2 x set-i i i 5 gt nan where' \ 1651 --colmetadata=3,MULTIP,unit,"Description of column" 1652 1653Due to the precedence described in this section, Table does these 1654operations (which are independent of the order of the operations written 1655on the command-line): 1656 1657 1. At the start (with ‘-cRA,DEC’), Table reads the ‘RA’ and ‘DEC’ 1658 columns. 1659 2. In between all the operations in the command above, Column 1660 arithmetic (with ‘-c'arith ...'’) has the highest precedence. So 1661 the arithmetic operation is done and stored as a new (third) 1662 column. In this arithmetic operation, we multiply all the values 1663 of the ‘PARAM’ column by 2, then set all those with a value larger 1664 than 5 to NaN (for more on understanding this operation, see the 1665 ‘‘set-’’ and ‘‘where’’ operators in *note Arithmetic operators::). 1666 3. Updating column metadata (with ‘--colmetadata’) is then done to 1667 give a name (‘MULTIP’) to the newly calculated (third) column. 1668 During the process, besides a name, we also set a unit and 1669 description for the new column. These metadata entries are _very 1670 important_, so always be sure to add metadata after doing column 1671 arithmetic. 1672 4. The lowest precedence operation is ‘--noblank=MULTIP’. So only 1673 rows that aren’t blank/NaN in the ‘MULTIP’ column are kept. 1674 5. Finally, the output table (with three columns) is written to the 1675 command-line. If you also want to print the column metadata, you 1676 can use the ‘--colinfoinstdout’ option. Alternatively, if you want 1677 the output in a file, you can use the ‘--output’ option to save the 1678 table in FITS or plain-text format. 1679 1680*Out of precedence:* It may happen that your desired operation needs a 1681separate precedence. In this case you can pipe the output of Table into 1682another call of Table and use the ‘--colinfoinstdout’ option to preserve 1683the metadata between the two calls. 1684 1685 For example, let’s assume that you want to sort the output table from 1686the example command above based on the new ‘MULTIP’ column. Since 1687sorting is done prior to column arithmetic, you can’t do it in one 1688command, but you can circumvent this limitation by simply piping the 1689output (including metadata) to another call to Table: 1690 1691 asttable table.fits -cRA,DEC --noblank=MULTIP --colinfoinstdout \ 1692 -c'arith PARAM 2 x set-i i i 5 gt nan where' \ 1693 --colmetadata=3,MULTIP,unit,"Description of column" \ 1694 | asttable --sort=MULTIP --output=selected.fits 1695 1696 1697File: gnuastro.info, Node: Invoking asttable, Prev: Operation precedence in Table, Up: Table 1698 16995.3.3 Invoking Table 1700-------------------- 1701 1702Table will read/write, select, modify, or show the information of the 1703rows and columns in recognized Table formats (including FITS binary, 1704FITS ASCII, and plain text table files, see *note Tables::). Output 1705columns can also be determined by number or regular expression matching 1706of column names, units, or comments. The executable name is ‘asttable’ 1707with the following general template 1708 1709 $ asttable [OPTION...] InputFile 1710 1711One line examples: 1712 1713 ## Get the table column information (name, data type, or units): 1714 $ asttable bintab.fits --information 1715 1716 ## Print columns named RA and DEC, followed by all the columns where 1717 ## the name starts with "MAG_": 1718 $ asttable bintab.fits --column=RA --column=DEC --column=/^MAG_/ 1719 1720 ## Similar to the above, but with one call to `--column' (or `-c'), 1721 ## also sort the rows by the input's photometric redshift (`Z_PHOT') 1722 ## column. To confirm the sort, you can add `Z_PHOT' to the columns 1723 ## to print. 1724 $ asttable bintab.fits -cRA,DEC,/^MAG_/ --sort=Z_PHOT 1725 1726 ## Similar to the above, but only print rows that have a photometric 1727 ## redshift between 2 and 3. 1728 $ asttable bintab.fits -cRA,DEC,/^MAG_/ --range=Z_PHOT,2:3 1729 1730 ## Only print rows with a value in the 10th column above 100000: 1731 $ asttable bintab.fits --range=10,10e5,inf 1732 1733 ## Only print the 2nd column, and the third column multiplied by 5, 1734 ## Save the resulting two columns in `table.txt' 1735 $ asttable bintab.fits -c2,'arith $2 5 x' -otable.fits 1736 1737 ## Sort the output columns by the third column, save output: 1738 $ asttable bintab.fits --sort=3 -ooutput.txt 1739 1740 ## Subtract the first column from the second in `cat.fits' (can also 1741 ## be a text table) and keep the third and fourth columns. 1742 $ asttable cat.txt -c'arith $2 $1 -',3,4 -ocat.fits 1743 1744 ## Convert sexagesimal coordinates to degrees (same can be done in a 1745 ## large table given as argument). 1746 $ echo "7h34m35.5498 31d53m14.352s" | asttable 1747 1748 ## Convert RA and Dec in degrees to sexagesimal (same can be done in a 1749 ## large table given as argument). 1750 echo "113.64812416667 31.88732" \ 1751 | asttable -c'arith $1 degree-to-ra $2 degree-to-dec' 1752 1753 Table’s input dataset can be given either as a file or from Standard 1754input (piped from another program, see *note Standard input::). In the 1755absence of selected columns, all the input’s columns and rows will be 1756written to the output. The full set of operations Table can do are 1757described in detail below, but for a more high-level introduction to the 1758various operations, and their precedence, see *note Operation precedence 1759in Table::. 1760 1761 If any output file is explicitly requested (with ‘--output’) the 1762output table will be written in it. When no output file is explicitly 1763requested the output table will be written to the standard output. If 1764the specified output is a FITS file, the type of FITS table (binary or 1765ASCII) will be determined from the ‘--tabletype’ option. If the output 1766is not a FITS file, it will be printed as a plain text table (with space 1767characters between the columns). When the columns are accompanied by 1768meta-data (like column name, units, or comments), this information will 1769also printed in the plain text file before the table, as described in 1770*note Gnuastro text table format::. 1771 1772 For the full list of options common to all Gnuastro programs please 1773see *note Common options::. Options can also be stored in directory, 1774user or system-wide configuration files to avoid repeating on the 1775command-line, see *note Configuration files::. Table does not follow 1776Automatic output that is common in most Gnuastro programs, see *note 1777Automatic output::. Thus, in the absence of an output file, the 1778selected columns will be printed on the command-line with no column 1779information, ready for redirecting to other tools like ‘awk’. 1780 1781*Sexagesimal coordinates as floats in plain-text tables:* When a column 1782is determined to be a floating point type (32-bit or 64-bit) in a 1783plain-text table, it can contain sexagesimal values in the format of 1784‘‘_h_m_s’’ (for RA) and ‘‘_d_m_s’’ (for Dec), where the ‘‘_’’s are 1785place-holders for numbers. In this case, the string will be immediately 1786converted to a single floating point number (in units of degrees) and 1787stored in memory with the rest of the column or table. Besides being 1788useful in large tables, with this feature, conversion to sexagesimal 1789coordinates to degrees becomes very easy, for example: 1790 echo "7h34m35.5498 31d53m14.352s" | asttable 1791 1792The inverse can also be done with the more general column arithmetic 1793operators: 1794 echo "113.64812416667 31.88732" \ 1795 | asttable -c'arith $1 degree-to-ra $2 degree-to-dec' 1796 1797If you want to preserve the sexagesimal contents of a column, you should 1798store that column as a string, see *note Gnuastro text table format::. 1799 1800‘-i’ 1801‘--information’ 1802 Only print the column information in the specified table on the 1803 command-line and exit. Each column’s information (number, name, 1804 units, data type, and comments) will be printed as a row on the 1805 command-line. Note that the FITS standard only requires the data 1806 type (see *note Numeric data types::), and in plain text tables, no 1807 meta-data/information is mandatory. Gnuastro has its own 1808 convention in the comments of a plain text table to store and 1809 transfer this information as described in *note Gnuastro text table 1810 format::. 1811 1812 This option will take precedence over all other operations in 1813 Table, so when it is called along with other operations, they will 1814 be ignored, see *note Operation precedence in Table::. This can be 1815 useful if you forget the identifier of a column after you have 1816 already typed some on the command-line. You can simply add a ‘-i’ 1817 to your already-written command (without changing anything) and run 1818 Table, to see the whole list of column names and information. Then 1819 you can use the shell history (with the up arrow key on the 1820 keyboard), and retrieve the last command with all the previously 1821 typed columns present, delete ‘-i’ and add the identifier you had 1822 forgot. 1823 1824‘-c STR/INT’ 1825‘--column=STR/INT’ 1826 Set the output columns either by specifying the column number, or 1827 name. For more on selecting columns, see *note Selecting table 1828 columns::. If a value of this option starts with ‘‘arith ’’, 1829 column arithmetic will be activated, allowing you to 1830 edit/manipulate column contents. For more on column arithmetic see 1831 *note Column arithmetic::. 1832 1833 To ask for multiple columns this option can be used in two ways: 1) 1834 multiple calls to this option, 2) using a comma between each column 1835 specifier in one call to this option. These different solutions 1836 may be mixed in one call to Table: for example, ‘‘-cRA,DEC,MAG’’, 1837 or ‘‘-cRA,DEC -cMAG’’ are both equivalent to ‘‘-cRA -cDEC -cMAG’’. 1838 The order of the output columns will be the same order given to the 1839 option or in the configuration files (see *note Configuration file 1840 precedence::). 1841 1842 This option is not mandatory, if no specific columns are requested, 1843 all the input table columns are output. When this option is called 1844 multiple times, it is possible to output one column more than once. 1845 1846‘-w FITS’ 1847‘--wcsfile=FITS’ 1848 FITS file that contains the WCS to be used in the ‘wcs-to-img’ and 1849 ‘img-to-wcs’ operators of *note Column arithmetic::. The extension 1850 name/number within the FITS file can be specified with ‘--wcshdu’. 1851 1852 If the value to this option is ‘‘none’’, no WCS will be written in 1853 the output. 1854 1855‘-W STR’ 1856‘--wcshdu=STR’ 1857 FITS extension/HDU in the FITS file given to ‘--wcsfile’ (see the 1858 description of ‘--wcsfile’ for more). 1859 1860‘-L FITS/TXT’ 1861‘--catcolumnfile=FITS/TXT’ 1862 Concatenate (or add, or append) the columns of this option’s value 1863 (a filename) to the output columns. This option may be called 1864 multiple times (to add columns from more than one file into the 1865 final output), the columns from each file will be added in the same 1866 order that this option is called. The number of rows in the 1867 file(s) given to this option has to be the same as the input table 1868 (before any type of row-selection), see *note Operation precedence 1869 in Table::. 1870 1871 By default all the columns of the given file will be appended, if 1872 you only want certain columns to be appended, use the 1873 ‘--catcolumns’ option to specify their name or number (see *note 1874 Selecting table columns::). Note that the columns given to 1875 ‘--catcolumns’ must be present in all the given files (if this 1876 option is called more than once with more than one file). 1877 1878 If the file given to this option is a FITS file, its necessary to 1879 also define the corresponding HDU/extension with ‘--catcolumnhdu’. 1880 Also note that no operation (for example row selection, arithmetic 1881 or etc) is applied to the table given to this option. 1882 1883 If the appended columns have a name, the column names of each file 1884 will be appended with a ‘-N’, where ‘N’ is a counter starting from 1885 1 for each appended file. This is done because when concatenating 1886 columns from multiple tables (more than two) into one, they may 1887 have the same name, and its not good practice to have multiple 1888 columns with the same name. You can disable this feature with 1889 ‘--catcolumnrawname’. Generally, you can use the ‘--colmetadata’ 1890 option to update column metadata. 1891 1892 For example, let’s assume you have two catalogs of the same objects 1893 (same number of rows) in different filters. Such that 1894 ‘f160w-cat.fits’ has a ‘MAGNITUDE’ column that has the magnitude of 1895 each object in the ‘F160W’ filter and similarly ‘f105w-cat.fits’, 1896 also has a ‘MAGNITUDE’ column, but for the ‘F105W’ filter. You can 1897 use column concatenation like below to import the ‘MAGNITUDE’ 1898 column from the ‘F105W’ catalog into the ‘F160W’ catalog, while 1899 giving each magnitude column a different name: 1900 1901 asttable f160w-cat.fits --output=both.fits \ 1902 --catcolumnfile=f105w-cat.fits --catcolumns=MAGNITUDE \ 1903 --colmetadata=MAGNITUDE,MAG-F160W,log,"Magnitude in F160W" \ 1904 --colmetadata=MAGNITUDE-1,MAG-F105W,log,"Magnitude in F105W" 1905 1906 For a more complete example, see *note Working with catalogs 1907 estimating colors::. 1908 1909‘-u STR/INT’ 1910‘--catcolumnhdu=STR/INT’ 1911 The HDU/extension of the FITS file(s) that should be concatenated, 1912 or appended, with ‘--catcolumnfile’. If ‘--catcolumn’ is called 1913 more than once with more than one FITS file, its necessary to call 1914 this option more than once. The HDUs will be loaded in the same 1915 order as the FITS files given to ‘--catcolumnfile’. 1916 1917‘-C STR/INT’ 1918‘--catcolumns=STR/INT’ 1919 The column(s) in the file(s) given to ‘--catcolumnfile’ to append. 1920 When this option is not given, all the columns will be 1921 concatenated. See ‘--catcolumnfile’ for more. 1922 1923‘--catcolumnrawname’ 1924 Don’t modify the names of the concatenated (appended) columns, see 1925 description in ‘--catcolumnfile’. 1926 1927‘-O’ 1928‘--colinfoinstdout’ 1929 Add column metadata when the output is printed in the standard 1930 output. Usually the standard output is used for a fast visual 1931 check, or to pipe into other metadata-agnostic programs (like AWK) 1932 for further processing. So by default meta-data aren’t included. 1933 But when piping to other Gnuastro programs (where metadata can be 1934 interpreted and used) it is recommended to use this option and use 1935 column names in the next program. 1936 1937‘-r STR,FLT:FLT’ 1938‘--range=STR,FLT:FLT’ 1939 Only output rows that have a value within the given range in the 1940 ‘STR’ column (can be a name or counter). Note that the range is 1941 only inclusive in the lower-limit. For example with 1942 ‘--range=sn,5:20’ the output’s columns will only contain rows that 1943 have a value in the ‘sn’ column (not case-sensitive) that is 1944 greater or equal to 5, and less than 20. For the precedence of 1945 this operation in relation to others, see *note Operation 1946 precedence in Table::. 1947 1948 This option can be called multiple times (different ranges for 1949 different columns) in one run of the Table program. This is very 1950 useful for selecting the final rows from multiple criteria/columns. 1951 1952 The chosen column doesn’t have to be in the output columns. This 1953 is good when you just want to select using one column’s values, but 1954 don’t need that column anymore afterwards. 1955 1956 For one example of using this option, see the example under 1957 ‘--sigclip-median’ in *note Invoking aststatistics::. 1958 1959‘--inpolygon=STR1,STR2’ 1960 Only return rows where the given coordinates are inside the polygon 1961 specified by the ‘--polygon’ option. The coordinate columns are 1962 the given ‘STR1’ and ‘STR2’ columns, they can be a column name or 1963 counter (see *note Selecting table columns::). For the precedence 1964 of this operation in relation to others, see *note Operation 1965 precedence in Table::. 1966 1967 Note that the chosen columns doesn’t have to be in the output 1968 columns (which are specified by the ‘--column’ option). For 1969 example if we want to select rows in the polygon specified in *note 1970 Dataset inspection and cropping::, this option can be used like 1971 this (you can remove the double quotations and write them all in 1972 one line if you remove the white-spaces around the colon separating 1973 the column vertices): 1974 1975 asttable table.fits --inpolygon=RA,DEC \ 1976 --polygon="53.187414,-27.779152 \ 1977 : 53.159507,-27.759633 \ 1978 : 53.134517,-27.787144 \ 1979 : 53.161906,-27.807208" \ 1980 1981 *Flat/Euclidean space: * The ‘--inpolygon’ option assumes a 1982 flat/Euclidean space so it is only correct for RA and Dec when the 1983 polygon size is very small like the example above. If your polygon 1984 is a degree or larger, it may not return correct results. Please 1985 get in touch if you need such a feature (see *note Suggest new 1986 feature::). 1987 1988‘--outpolygon=STR1,STR2’ 1989 Only return rows where the given coordinates are outside the 1990 polygon specified by the ‘--polygon’ option. This option is very 1991 similar to the ‘--inpolygon’ option, so see the description there 1992 for more. 1993 1994‘--polygon=STR’ 1995‘--polygon=FLT,FLT:FLT,FLT:...’ 1996 The polygon to use for the ‘--inpolygon’ and ‘--outpolygon’ 1997 options. This option behaves identically to the same option in the 1998 Crop program, so for more information on how to use it, see *note 1999 Crop options::. 2000 2001‘-e STR,INT/FLT,...’ 2002‘--equal=STR,INT/FLT,...’ 2003 Only output rows that are equal to the given number(s) in the given 2004 column. The first argument is the column identifier (name or 2005 number, see *note Selecting table columns::), after that you can 2006 specify any number of values. For the precedence of this operation 2007 in relation to others, see *note Operation precedence in Table::. 2008 2009 For example ‘--equal=ID,5,6,8’ will only print the rows that have a 2010 value of 5, 6, or 8 in the ‘ID’ column. This option can also be 2011 called multiple times, so ‘--equal=ID,4,5 --equal=ID,6,7’ has the 2012 same effect as ‘--equal=4,5,6,7’. 2013 2014 The ‘--equal’ and ‘--notequal’ options also work when the given 2015 column has a string type. In this case the given value to the 2016 option will also be parsed as a string, not as a number. When 2017 dealing with string columns, be careful with trailing white space 2018 characters (the actual value maybe adjusted to the right, left, or 2019 center of the column’s width). If you need to account for such 2020 white spaces, you can use shell quoting. For example 2021 ‘--equal=NAME," myname "’. 2022 2023 *Equality and floating point numbers:* Floating point numbers are 2024 only approximate values (see *note Numeric data types::). In this 2025 context, their equality depends on how the the input table was 2026 originally stored (as a plain text table or as an ASCII/binary FITS 2027 table). If you want to select floating point numbers, it is 2028 strongly recommended to use the ‘--range’ option and set a very 2029 small interval around your desired number, don’t use ‘--equal’ or 2030 ‘--notequal’. 2031 2032‘-n STR,INT/FLT,...’ 2033‘--notequal=STR,INT/FLT,...’ 2034 Only output rows that are _not_ equal to the given number(s) in the 2035 given column. The first argument is the column identifier (name or 2036 number, see *note Selecting table columns::), after that you can 2037 specify any number of values. For example ‘--notequal=ID,5,6,8’ 2038 will only print the rows where the ‘ID’ column doesn’t have value 2039 of 5, 6, or 8. This option can also be called multiple times, so 2040 ‘--notequal=ID,4,5 --notequal=ID,6,7’ has the same effect as 2041 ‘--notequal=4,5,6,7’. 2042 2043 Be very careful if you want to use the non-equality with floating 2044 point numbers, see the special note under ‘--equal’ for more. This 2045 option also works when the given column has a string type, see the 2046 description under ‘--equal’ (above) for more. 2047 2048‘-s STR’ 2049‘--sort=STR’ 2050 Sort the output rows based on the values in the ‘STR’ column (can 2051 be a column name or number). By default the sort is done in 2052 ascending/increasing order, to sort in a descending order, use 2053 ‘--descending’. For the precedence of this operation in relation 2054 to others, see *note Operation precedence in Table::. 2055 2056 The chosen column doesn’t have to be in the output columns. This 2057 is good when you just want to sort using one column’s values, but 2058 don’t need that column anymore afterwards. 2059 2060‘-d’ 2061‘--descending’ 2062 When called with ‘--sort’, rows will be sorted in descending order. 2063 2064‘-H INT’ 2065‘--head=INT’ 2066 Only print the given number of rows from the _top_ of the final 2067 table. Note that this option only affects the _output_ table. For 2068 example if you use ‘--sort’, or ‘--range’, the printed rows are the 2069 first _after_ applying the sort sorting, or selecting a range of 2070 the full input. This option cannot be called with ‘--tail’, 2071 ‘--rowlimit’ or ‘--rowrandom’. For the precedence of this 2072 operation in relation to others, see *note Operation precedence in 2073 Table::. 2074 2075 If the given value to ‘--head’ is 0, the output columns won’t have 2076 any rows and if its larger than the number of rows in the input 2077 table, all the rows are printed (this option is effectively 2078 ignored). This behavior is taken from the ‘head’ program in GNU 2079 Coreutils. 2080 2081‘-t INT’ 2082‘--tail=INT’ 2083 Only print the given number of rows from the _bottom_ of the final 2084 table. See ‘--head’ for more. This option cannot be called with 2085 ‘--head’, ‘--rowlimit’ or ‘--rowrandom’. 2086 2087‘--rowlimit=INT,INT’ 2088 Only return the rows within the requested positional range 2089 (inclusive on both sides). Therefore, ‘--rowlimit=5,7’ will return 2090 3 of the input rows, row 5, 6 and 7. This option will abort if any 2091 of the given values is larger than the total number of rows in the 2092 table. For the precedence of this operation in relation to others, 2093 see *note Operation precedence in Table::. 2094 2095 With the ‘--head’ or ‘--tail’ options you can only see the top or 2096 bottom few rows. However, with this option, you can limit the 2097 returned rows to a contiguous set of rows in the middle of the 2098 table. Therefore this option cannot be called with ‘--head’, 2099 ‘--tail’, or ‘--rowrandom’. 2100 2101‘--rowrandom=INT’ 2102 Select ‘INT’ rows from the input table by random (assuming a 2103 uniform distribution). This option is applied _after_ the 2104 value-based selection options (like ‘--sort’, ‘--range’, 2105 ‘--polygon’ and etc). On the other hand, only the row counters are 2106 randomly selected, this option doesn’t change the order. 2107 Therefore, if ‘--rowrandom’ is called together with ‘--sort’, the 2108 returned rows are still sorted. This option cannot be called with 2109 ‘--head’, ‘--tail’, or ‘--rowlimit’. For the precedence of this 2110 operation in relation to others, see *note Operation precedence in 2111 Table::. 2112 2113 This option will only have an effect if ‘INT’ is larger than the 2114 number of rows when it is activated (after the value-based 2115 selection options have been applied). When there are fewer rows, a 2116 warning is printed, saying that this option has no effect. The 2117 warning can be disabled with the ‘--quiet’ option. 2118 2119 Due to its nature (to be random), the output of this option differs 2120 in each run. Therefore 5 calls to Table with ‘--rowrandom’ on the 2121 same input table will generate 5 different outputs. If you want a 2122 reproducible random selection, set the ‘GSL_RNG_SEED’ environment 2123 variable and also use the ‘--envseed’ option, for more see *note 2124 Generating random numbers::. 2125 2126‘--envseed’ 2127 Read the random number generator seed from the ‘GSL_RNG_SEED’ 2128 environment variable for ‘--rowrandom’ (instead of generating a 2129 different seed internally on every run). This is useful if you 2130 want a reproducible random selection of the input rows. For more, 2131 see *note Generating random numbers::. 2132 2133‘-b STR[,STR[,STR]]’ 2134‘--noblank=STR[,STR[,STR]]’ 2135 Remove all rows in the given _output_ columns that have a blank 2136 value. Like above, the columns can be specified by their name or 2137 number (counting from 1). 2138 2139 For example if ‘table.fits’ has blank values (NaN in floating point 2140 types) in the ‘magnitude’ and ‘sn’ columns, with 2141 ‘--noblank=magnitude,sn’, the output will not contain any rows with 2142 blank values in these columns. 2143 2144 If you want _all_ columns to be checked, simply set the value to 2145 ‘_all’ (in other words: ‘--noblank=_all’). This mode is useful 2146 when there are many columns in the table and you want a “clean” 2147 output table (with no blank values in any column): entering their 2148 name or number one-by-one can be buggy and frustrating. In this 2149 mode, no other column name should be given. For example if you 2150 give ‘--noblank=_all,magnitude’, then Table will assume that your 2151 table actually has a column named ‘_all’ and ‘magnitude’, and if it 2152 doesn’t, it will abort with an error. 2153 2154 This option is applied just before writing the final table (after 2155 ‘--colmetadata’ has finished). So in case you changed the column 2156 metadata, or added new columns, you can use the new names, or the 2157 newly defined column numbers. For the precedence of this operation 2158 in relation to others, see *note Operation precedence in Table::. 2159 2160‘-m STR/INT,STR[,STR[,STR]]’ 2161‘--colmetadata=STR/INT,STR[,STR[,STR]]’ 2162 Update the specified column metadata in the output table. This 2163 option is applied after all other column-related operations are 2164 complete, for example column arithmetic, or column concatenation. 2165 For the precedence of this operation in relation to others, see 2166 *note Operation precedence in Table::. 2167 2168 The first value (before the first comma) given to this option is 2169 the column’s identifier. It can either be a counter (positive 2170 integer, counting from 1), or a name (the column’s name in the 2171 output if this option wasn’t called). 2172 2173 After the to-be-updated column is identified, at least one other 2174 string should be given, with a maximum of three strings. The first 2175 string after the original name will the the selected column’s new 2176 name. The next (optional) string will be the selected column’s 2177 unit and the third (optional) will be its comments. If the two 2178 optional strings aren’t given, the original column’s units or 2179 comments will remain unchanged. Some examples of this option are 2180 available in the tutorials, in particular *note Working with 2181 catalogs estimating colors::. Here are some more specific examples 2182 2183 ‘--colmetadata=MAGNITUDE,MAG_F160W’ 2184 This will convert name of the original ‘MAGNITUDE’ column to 2185 ‘MAG_F160W’, leaving the unit and comments unchanged. 2186 2187 ‘--colmetadata=3,MAG_F160W,mag’ 2188 This will convert name of the third column of the final output 2189 to ‘MAG_F160W’ and the units to ‘mag’, while leaving the 2190 comments untouched. 2191 2192 ‘--colmetadata=MAGNITUDE,MAG_F160W,mag,"Magnitude in F160W filter"’ 2193 This will convert name of the original ‘MAGNITUDE’ column to 2194 ‘MAG_F160W’, and the units to ‘mag’ and the comments to 2195 ‘Magnitude in F160W filter’. Note the double quotations 2196 around the comment string, they are necessary to preserve the 2197 white-space characters within the column comment from the 2198 command-line, into the program (otherwise, upon reaching a 2199 white-space character, the shell will consider this option to 2200 be finished and cause un-expected behavior). 2201 2202 If your table is large and generated by a script, you can first do 2203 all your operations on your table’s data and write it into a 2204 temporary file (maybe called ‘temp.fits’). Then, look into that 2205 file’s metadata (with ‘asttable temp.fits -i’) to see the exact 2206 column positions and possible names, then add the necessary calls 2207 to this option to your previous call to ‘asttable’, so it writes 2208 proper metadata in the same run (for example in a script or 2209 Makefile). Recall that when a name is given, this option will 2210 update the metadata of the first column that matches, so if you 2211 have multiple columns with the same name, you can call this options 2212 multiple times with the same first argument to change them all to 2213 different names. 2214 2215 Finally, if you already have a FITS table by other means (for 2216 example by downloading) and you merely want to update the column 2217 metadata and leave the data intact, it is much more efficient to 2218 directly modify the respective FITS header keywords with ‘astfits’, 2219 using the keyword manipulation features described in *note Keyword 2220 inspection and manipulation::. ‘--colmetadata’ is mainly intended 2221 for scenarios where you want to edit the data so it will always 2222 load the full/partial dataset into memory, then write out the 2223 resulting datasets with updated/corrected metadata. 2224 2225 2226File: gnuastro.info, Node: Query, Prev: Table, Up: Data containers 2227 22285.4 Query 2229========= 2230 2231There are many astronomical databases available for downloading 2232astronomical data. Most follow the International Virtual Observatory 2233Alliance (IVOA, <https://ivoa.net>) standards (and in particular the 2234Table Access Protocol, or TAP(1)). With TAP, it is possible to submit 2235your queries via a command-line downloader (for example ‘curl’) to only 2236get specific tables, targets (rows in a table) or measurements (columns 2237in a table): you don’t have to download the full table (which can be 2238very large in some cases)! These customizations are done through the 2239Astronomical Data Query Language (ADQL(2)). 2240 2241 Therefore, if you are sufficiently familiar with TAP and ADQL, you 2242can easily custom-download any part of an online dataset. However, you 2243also need to keep a record of the URLs of each database and in many 2244cases, the commands will become long and hard/buggy to type on the 2245command-line. On the other hand, most astronomers don’t know TAP or 2246ADQL at all, and are forced to go to the database’s web page which is 2247slow (it needs to download so many images, and has too much annoying 2248information), requires manual interaction (further making it slow and 2249buggy), and can’t be automated. 2250 2251 Gnuastro’s Query program is designed to be the middle-man in this 2252process: it provides a simple high-level interface to let you specify 2253your constraints on what you want to download. It then internally 2254constructs the command to download the data based on your inputs and 2255runs it to download your desired data. Query also prints the full 2256command before it executes it (if not called with ‘--quiet’). Also, if 2257you ask for a FITS output table, the full command is written into its 22580-th extension along with other input parameters to query (all Gnuastro 2259programs generally keep their input configuration parameters as FITS 2260keywords in the zero-th output). You can see it with Gnuastro’s Fits 2261program, like below: 2262 2263 $ astfits query-output.fits -h0 2264 2265 With the full command used to download the dataset, you only need a 2266minimal knowledge of ADQL to do lower-level customizations on your 2267downloaded dataset. You can simply copy that command and change the 2268parts of the query string you want: ADQL is very powerful! For example 2269you can ask the server to do mathematical operations on the columns and 2270apply selections after those operations, or combine/match multiple 2271datasets and etc. We will try to add high-level interfaces for such 2272capabilities, but generally, don’t limit yourself to the high-level 2273operations (that can’t cover everything!). 2274 2275* Menu: 2276 2277* Available databases:: List of available databases to Query. 2278* Invoking astquery:: Inputs, outputs and configuration of Query. 2279 2280 ---------- Footnotes ---------- 2281 2282 (1) <https://ivoa.net/documents/TAP> 2283 2284 (2) <https://ivoa.net/documents/ADQL> 2285 2286 2287File: gnuastro.info, Node: Available databases, Next: Invoking astquery, Prev: Query, Up: Query 2288 22895.4.1 Available databases 2290------------------------- 2291 2292The current list of databases supported by Query are listed at the end 2293of this section. To get the list of available datasets within each 2294database, you can use the ‘--information’ option. For example with the 2295command below you can get a list of the roughly 100 datasets that are 2296available within the ESA Gaia server with their description: 2297 2298 $ astquery gaia --information 2299 2300However, other databases like VizieR host many more datasets (tens of 2301thousands!). Therefore it is very inconvenient to get the _full_ 2302information every time you want to find your dataset of interest (the 2303full metadata file VizieR is more than 20Mb). In such cases, you can 2304limit the downloaded and displayed information with the ‘--limitinfo’ 2305option. For example with the first command below, you can get all 2306datasets relating to the MUSE (an instrument on the Very Large 2307Telescope), and those that include Roland Bacon (Principle Investigator 2308of MUSE) as an author (‘Bacon, R.’). Recall that ‘-i’ is the short 2309format of ‘--information’. 2310 2311 $ astquery vizier -i --limitinfo=MUSE 2312 $ astquery vizier -i --limitinfo="Bacon R." 2313 2314 Once you find the recognized name of your desired dataset, you can 2315see the column information of that dataset with adding the dataset name. 2316For example, with the command below you can see the column metadata in 2317the ‘J/A+A/608/A2/udf10’ dataset (one of the datasets in the search 2318above) using this command: 2319 2320 $ astquery vizier --dataset=J/A+A/608/A2/udf10 -i 2321 2322 For very popular datasets of a database, Query provides an 2323easier-to-remember short name that you can feed to ‘--dataset’. This 2324short name will map to the officially recognized name of the dataset on 2325the server. In this mode, Query will also set positional columns 2326accordingly. For example most VizieR datasets have an ‘RAJ2000’ column 2327(the RA and the epoch of 2000) so it is the default RA column name for 2328coordinate search (using ‘--center’ or ‘--overlapwith’). However, some 2329datasets don’t have this column (for example SDSS DR12). So when you 2330use the short name and Query knows about this dataset, it will 2331internally set the coordinate columns that SDSS DR12 has: ‘RA_ICRS’ and 2332‘DEC_ICRS’. Recall that you can always change the coordinate columns 2333with ‘--ccol’. 2334 2335 For example in the VizieR and Gaia databases, the recognized name for 2336the early data release 3 data is respectively ‘I/350/gaiaedr3’ and 2337‘gaiaedr3.gaia_source’. These technical names can be hard to remember. 2338Therefore Query provides ‘gaiaedr3’ (for VizieR) and ‘edr3’ (for ESA’s 2339Gaia) shortcuts which you can give to ‘--dataset’ instead. They will be 2340directly mapped to the fully recognized name by Query. In the list 2341below that describes the available databases, the available short names 2342are also listed. 2343 2344*Not all datasets support TAP:* Large databases like VizieR have TAP 2345access for all their datasets. However, smaller databases haven’t 2346implemented TAP for all their tables. Therefore some datasets that are 2347searchable in their web interface may not be available for a TAP search. 2348To see the full list of TAP-ed datasets in a database, use the 2349‘--information’ (or ‘-i’) option with the dataset name like the command 2350below. 2351 2352 $ astquery astron -i 2353 2354If your desired dataset isn’t in this list, but has web-access, contact 2355the database maintainers and ask them to add TAP access for it. After 2356they do it, you should see the name added to the output list of the 2357command above. 2358 2359 The list of databases recognized by Query (and their names in Query) 2360is described below. Since Query is a new member of the Gnuastro family 2361(first available in Gnuastro 0.14), this list will hopefully grow 2362significantly in the next releases. If you have any particular datasets 2363in mind, please let us know by sending an email to 2364‘bug-gnuastro@gnu.org’. If the dataset supports IVOA’s TAP (Table 2365Access Protocol), it should be very easy to add. 2366 2367‘astron’ 2368 The ASTRON Virtual Observatory service (<https://vo.astron.nl>) is 2369 a database focused on radio astronomy data and images, primarily 2370 those collected by ASTRON itself. A query to ‘astron’ is submitted 2371 to ‘https://vo.astron.nl/__system__/tap/run/tap/sync’. 2372 2373 Here is the list of short names for dataset(s) in ASTRON’s VO 2374 service: 2375 • ‘tgssadr --> tgssadr.main’ 2376 2377‘gaia’ 2378 The Gaia project (<https://www.cosmos.esa.int/web/gaia>) database 2379 which is a large collection of star positions on the celestial 2380 sphere, as well as peculiar velocities, parallaxes and magnitudes 2381 in some bands among many others. Besides scientific studies (like 2382 studying resolved stellar populations in the Galaxy and its halo), 2383 Gaia is also invaluable for raw data calibrations, like astrometry. 2384 A query to ‘gaia’ is submitted to 2385 ‘https://gea.esac.esa.int/tap-server/tap/sync’. 2386 2387 Here is the list of short names for popular datasets within Gaia: 2388 • ‘edr3 --> gaiaedr3.gaia_source’ 2389 • ‘dr2 --> gaiadr2.gaia_source’ 2390 • ‘dr1 --> gaiadr1.gaia_source’ 2391 • ‘tycho2 --> public.tycho2’ 2392 • ‘hipparcos --> public.hipparcos’ 2393 2394‘ned’ 2395 The NASA/IPAC Extragalactic Database (NED, 2396 <http://ned.ipac.caltech.edu>) is a fusion database, integrating 2397 the information about extra-galactic sources from many large sky 2398 surveys into a single catalog. It covers the full spectrum, from 2399 Gamma rays to radio frequencies and is updated when new data 2400 arrives. A TAP query to ‘ned’ is submitted to 2401 ‘https://ned.ipac.caltech.edu/tap/sync’. 2402 2403 • ‘objdir --> NEDTAP.objdir’: default TAP-based dataset in NED. 2404 2405 • ‘extinction’: A command-line interface to the NED Extinction 2406 Calculator 2407 (https://ned.ipac.caltech.edu/extinction_calculator). It only 2408 takes a central coordinate and returns a VOTable of the 2409 calculated extinction in many commonly used filters at that 2410 point. As a result, options like ‘--width’ or ‘--radius’ are 2411 not supported. However, Gnuastro doesn’t yet support the 2412 VOTable format. Therefore, if you specify an ‘--output’ file, 2413 it should have an ‘.xml’ suffix and the downloaded file will 2414 not be checked. 2415 2416 Until VOTable support is added to Gnuastro, you can use GREP, 2417 AWK and SED to convert the VOTable data into a FITS table with 2418 a command like below (assuming the queried VOTable is called 2419 ‘ned-extinction.xml’): 2420 2421 grep '^<TR><TD>' ned-extinction.xml \ 2422 | sed -e's|<TR><TD>||' \ 2423 -e's|</TD></TR>||' \ 2424 -e's|</TD><TD>|@|g' \ 2425 | awk 'BEGIN{FS="@"; \ 2426 print "# Column 1: FILTER [name,str15] Filter name"; \ 2427 print "# Column 2: CENTRAL [um,f32] Central Wavelength"; \ 2428 print "# Column 3: EXTINCTION [mag,f32] Galactic Ext."; \ 2429 print "# Column 4: ADS_REF [ref,str50] ADS reference"} \ 2430 {printf "%-15s %g %g %s\n", $1, $2, $3, $4}' \ 2431 | asttable -oned-extinction.fits 2432 2433 Once the table is in FITS, you can easily get the extinction 2434 for a certain filter (for example the ‘SDSS r’ filter) like 2435 the command below: 2436 2437 asttable ned-extinction.fits --equal=FILTER,"SDSS r" \ 2438 -cEXTINCTION 2439 2440‘vizier’ 2441 Vizier (<https://vizier.u-strasbg.fr>) is arguably the largest 2442 catalog database in astronomy: containing more than 20500 catalogs 2443 as of mid January 2021. Almost all published catalogs in major 2444 projects, and even the tables in many papers are archived and 2445 accessible here. For example VizieR also has a full copy of the 2446 Gaia database mentioned below, with some additional standardized 2447 columns (like RA and Dec in J2000). 2448 2449 The current implementation of ‘--limitinfo’ only looks into the 2450 description of the datasets, but since VizieR is so large, there is 2451 still a lot of room for improvement. Until then, if ‘--limitinfo’ 2452 isn’t sufficient, you can use VizieR’s own web-based search for 2453 your desired dataset: <http://cdsarc.u-strasbg.fr/viz-bin/cat> 2454 2455 Because VizieR curates such a diverse set of data from tens of 2456 thousands of projects and aims for interoperability between them, 2457 the column names in VizieR may not be identical to the column names 2458 in the surveys’ own databases (Gaia in the example above). A query 2459 to ‘vizier’ is submitted to 2460 ‘http://tapvizier.u-strasbg.fr/TAPVizieR/tap/sync’. 2461 2462 Here is the list of short names for popular datasets within VizieR 2463 (sorted alphabetically by their short name). Please feel free to 2464 suggest other major catalogs (covering a wide area or commonly used 2465 in your field).. For details on each dataset with necessary 2466 citations, and links to web pages, look into their details with 2467 their ViziR names in <https://vizier.u-strasbg.fr/viz-bin/VizieR>. 2468 • ‘2mass --> II/246/out’ (2MASS All-Sky Catalog) 2469 • ‘akarifis --> II/298/fis’ (AKARI/FIS All-Sky Survey) 2470 • ‘allwise --> II/328/allwise’ (AllWISE Data Release) 2471 • ‘apass9 --> II/336/apass9’ (AAVSO Photometric All Sky Survey, 2472 DR9) 2473 • ‘catwise --> II/365/catwise’ (CatWISE 2020 catalog) 2474 • ‘des1 --> II/357/des_dr1’ (Dark Energy Survey data release 1) 2475 • ‘gaiadr2 --> I/345/gaia2’ (GAIA Data Release 2) 2476 • ‘gaiaedr3 --> I/350/gaiaedr3’ (GAIA early Data Release 3) 2477 • ‘galex5 --> II/312/ais’ (All-sky Survey of GALEX DR5) 2478 • ‘nomad --> I/297/out’ (Naval Observatory Merged Astrometric 2479 Dataset) 2480 • ‘panstarrs1 --> II/349/ps1’ (Pan-STARRS Data Release 1). 2481 • ‘ppmxl --> I/317/sample’ (Positions and proper motions on the 2482 ICRS) 2483 • ‘sdss12 --> V/147/sdss12’ (SDSS Photometric Catalogue, Release 2484 12) 2485 • ‘usnob1 --> I/284/out’ (Whole-Sky USNO-B1.0 Catalog) 2486 • ‘ucac5 --> I/340/ucac5’ (5th U.S. Naval Obs. CCD Astrograph 2487 Catalog) 2488 • ‘unwise --> II/363/unwise’ (Band-merged unWISE Catalog) 2489 • ‘wise --> II/311/wise’ (WISE All-Sky data Release) 2490 2491 2492File: gnuastro.info, Node: Invoking astquery, Prev: Available databases, Up: Query 2493 24945.4.2 Invoking Query 2495-------------------- 2496 2497Query provides a high-level interface to downloading subsets of data 2498from databases. The executable name is ‘astquery’ with the following 2499general template 2500 2501 $ astquery DATABASE-NAME [OPTION...] ... 2502 2503One line examples: 2504 2505 2506 ## Information about all datasets in ESA's GAIA database: 2507 $ astquery gaia --information 2508 2509 ## Only show catalogs in VizieR that have 'MUSE' in their 2510 ## description. The '-i' is short for '--information'. 2511 $ astquery vizier -i --limitinfo=MUSE 2512 2513 ## List of columns in 'J/A+A/608/A2/udf10' (one of the above). 2514 $ astquery vizier --dataset=J/A+A/608/A2/udf10 -i 2515 2516 ## ID, RA and Dec of all Gaia sources within an image. 2517 $ astquery gaia --dataset=edr3 --overlapwith=image.fits \ 2518 -csource_id,ra,dec 2519 2520 ## RA, Dec and Spectroscopic redshifts of objects in SDSS DR12 2521 ## spectroscopic redshift that overlap with 'image.fits'. 2522 $ astquery vizier --dataset=sdss12 --overlapwith=image.fits \ 2523 -cRA_ICRS,DE_ICRS,zsp --range=zsp,1e-10,inf 2524 2525 ## All columns of all entries in the Gaia eDR3 catalog (hosted at 2526 ## VizieR) within 1 arc-minute of the given coordinate. 2527 $ astquery vizier --dataset=I/350/gaiaedr3 --output=my-gaia.fits \ 2528 --center=113.8729761,31.9027152 --radius=1/60 \ 2529 2530 ## Similar to above, but only ID, RA and Dec columns for objects with 2531 ## magnitude range 10 to 15. In VizieR, this column is called 'Gmag'. 2532 $ astquery vizier --dataset=I/350/gaiaedr3 --output=my-gaia.fits \ 2533 --center=113.8729761,31.9027152 --radius=1/60 \ 2534 --range=Gmag,10:15 -cEDR3Name,RAJ2000,DEJ2000 2535 2536 Query takes a single argument which is the name of the database. For 2537the full list of available databases and accessing them, see *note 2538Available databases::. There are two methods to query the databases, 2539each is more fully discussed in its option’s description below. 2540 • *Low-level:* With ‘--query’ you can directly give a raw query 2541 statement that is recognized by the database. This is very low 2542 level and will require a good knowledge of the database’s query 2543 language, but of course, it is much more powerful. If this option 2544 is given, the raw string is directly passed to the server and all 2545 other constraints/options (for Query’s high-level interface) are 2546 ignored. 2547 • *High-level:* With the high-level options (like ‘--column’, 2548 ‘--center’, ‘--radius’, ‘--range’ and other constraining options 2549 below), the low-level query will be constructed automatically for 2550 the particular database. This method is only limited to the 2551 generic capabilities that Query provides for all servers. So 2552 ‘--query’ is more powerful, however, in this mode, you don’t need 2553 any knowledge of the database’s query language. You can see the 2554 internally generated query on the terminal (if ‘--quiet’ is not 2555 used) or in the 0-th extension of the output (if it is a FITS 2556 file). This full command contains the internally generated query. 2557 2558 The name of the downloaded output file can be set with ‘--output’. 2559The requested output format can have any of the *note Recognized table 2560formats:: (currently ‘.txt’ or ‘.fits’). Like all Gnuastro programs, if 2561the output is a FITS file, the zero-th/first HDU of the output will 2562contain all the command-line options given to Query as well as the full 2563command used to access the server. When ‘--output’ is not set, the 2564output name will be in the format of ‘NAME-STRING.fits’, where ‘NAME’ is 2565the name of the database and ‘STRING’ is a randomly selected 6-character 2566set of numbers and alphabetic characters. With this feature, a second 2567run of ‘astquery’ that isn’t called with ‘--output’ will not over-write 2568an already downloaded one. Generally, when calling Query more than 2569once, it is recommended to set an output name for each call based on 2570your project’s context. 2571 2572 The outputs of Query will have a common output format, irrespective 2573of the used database. To achieve this, Query will ask the databases to 2574provide a FITS table output (for larger tables, FITS can consume much 2575less download volume). After downloading is complete, the raw 2576downloaded file will be read into memory once by Query, and written into 2577the file given to ‘--output’. The raw downloaded file will be deleted 2578by default, but can be preserved with the ‘--keeprawdownload’ option. 2579This strategy avoids unnecessary surprises depending on database. For 2580example some databases can download a compressed FITS table, even though 2581we ask for FITS. But with the strategy above, the final output will be 2582an uncompressed FITS file. The metadata that is added by Query 2583(including the full download command) is also very useful for future 2584usage of the downloaded data. Unfortunately many databases don’t write 2585the input queries into their generated tables. 2586 2587‘--dry-run’ 2588 Only print the final download command to contact the server, don’t 2589 actually run it. This option is good when you want to check the 2590 finally constructed query or download options given to the download 2591 program. You may also want to use the constructed command as a 2592 base to do further customizations on it and run it yourself. 2593 2594‘-k’ 2595‘--keeprawdownload’ 2596 Don’t delete the raw downloaded file from the database. The name 2597 of the raw download will have a ‘OUTPUT-raw-download.fits’ format. 2598 Where ‘OUTPUT’ is either the base-name of the final output file 2599 (without a suffix). 2600 2601‘-i’ 2602‘--information’ 2603 Print the information of all datasets (tables) within a database or 2604 all columns within a database. When ‘--dataset’ is specified, the 2605 latter mode (all column information) is downloaded and printed and 2606 when its not defined, all dataset information (within the database) 2607 is printed. 2608 2609 Some databases (like VizieR) contain tens of thousands of datasets, 2610 so you can limit the downloaded and printed information for 2611 available databases with the ‘--limitinfo’ option (described 2612 below). Dataset descriptions are often large and contain a lot of 2613 text (unlike column descriptions). Therefore when printing the 2614 information of all datasets within a database, the information 2615 (e.g., database name) will be printed on separate lines before the 2616 description. However, when printing column information, the output 2617 has the same format as a similar option in Table (see *note 2618 Invoking asttable::). 2619 2620 Important note to consider: the printed order of the datasets or 2621 columns is just for displaying in the printed output. You cannot 2622 ask for datasets or columns based on the printed order, you need to 2623 use dataset or column names. 2624 2625‘-L STR’ 2626‘--limitinfo=STR’ 2627 Limit the information that is downloaded and displayed (with 2628 ‘--information’) to those that have the string given to this option 2629 in their description. Note that _this is case-sensitive_. This 2630 option is only relevant when ‘--information’ is also called. 2631 2632 Databases may have thousands (or tens of thousands) of datasets. 2633 Therefore just the metadata (information) to show with 2634 ‘--information’ can be tens of megabytes (for example the full 2635 VizieR metadata file is about 23Mb as of January 2021). Once 2636 downloaded, it can also be hard to parse manually. With 2637 ‘--limitinfo’, only the metadata of datasets that contain this 2638 string _in their description_ will be downloaded and displayed, 2639 greatly improving the speed of finding your desired dataset. 2640 2641‘-Q "STR"’ 2642‘--query="STR"’ 2643 Directly specify the query to be passed onto the database. The 2644 queries will generally contain space and other meta-characters, so 2645 we recommend placing the query within quotations. 2646 2647‘-s STR’ 2648‘--dataset=STR’ 2649 The dataset to query within the database (not compatible with 2650 ‘--query’). This option is mandatory when ‘--query’ or 2651 ‘--information’ aren’t provided. You can see the list of available 2652 datasets within a database using ‘--information’ (possibly 2653 supplemented by ‘--limitinfo’). The output of ‘--information’ will 2654 contain the recognized name of the datasets within that database. 2655 You can pass the recognized name directly to this option. For more 2656 on finding and using your desired database, see *note Available 2657 databases::. 2658 2659‘-c STR’ 2660‘--column=STR[,STR[,...]]’ 2661 The column name(s) to retrieve from the dataset in the given order 2662 (not compatible with ‘--query’). If not given, all the dataset’s 2663 columns for the selected rows will be queried (which can be 2664 large!). This option can take multiple values in one instance (for 2665 example ‘--column=ra,dec,mag’), or in multiple instances (for 2666 example ‘-cra -cdec -cmag’), or mixed (for example ‘-cra,dec 2667 -cmag’). 2668 2669 In case, you don’t know the full list of the dataset’s column names 2670 a-priori, and you don’t want to download all the columns (which can 2671 greatly decrease your download speed), you can use the 2672 ‘--information’ option combined with the ‘--dataset’ option, see 2673 *note Available databases::. 2674 2675‘-H INT’ 2676‘--head=INT’ 2677 Only ask for the first ‘INT’ rows of the finally selected columns, 2678 not all the rows. This can be good when your search can result a 2679 large dataset, but before downloading the full volume, you want to 2680 see the top rows and get a feeling of what the whole dataset looks 2681 like. 2682 2683‘-v FITS’ 2684‘--overlapwith=FITS’ 2685 File name of FITS file containing an image (in the HDU given by 2686 ‘--hdu’) to use for identifying the region to query in the give 2687 database and dataset. Based on the image’s WCS and pixel size, the 2688 sky coverage of the image is estimated and values to the 2689 ‘--center’, ‘--width’ will be calculated internally. Hence this 2690 option cannot be used with ‘--center’, ‘--width’ or ‘--radius’. 2691 Also, since it internally generates the query, it can’t be used 2692 with ‘--query’. 2693 2694 Note that if the image has WCS distortions and the reference point 2695 for the WCS is not within the image, the WCS will not be 2696 well-defined. Therefore the resulting catalog may not overlap, or 2697 correspond to a larger/small area in the sky. 2698 2699‘-C FLT,FLT’ 2700‘--center=FLT,FLT’ 2701 The spatial center position (mostly RA and Dec) to use for the 2702 automatically generated query (not compatible with ‘--query’). The 2703 given values will be compared to two columns in the database to 2704 find/return rows within a certain region around this center 2705 position will be requested and downloaded. Pre-defined RA and Dec 2706 column names are defined in Query for every database, however you 2707 can use ‘--ccol’ to select other columns to use instead. The 2708 region can either be a circle and the point (configured with 2709 ‘--radius’) or a box/rectangle around the point (configured with 2710 ‘--width’). 2711 2712‘--ccol=STR,STR’ 2713 The name of the coordinate-columns in the dataset to compare with 2714 the values given to ‘--center’. Query will use its internal 2715 defaults for each dataset (for example ‘RAJ2000’ and ‘DEJ2000’ for 2716 VizieR data). But each dataset is treated separately and it isn’t 2717 guaranteed that these columns exist in all datasets. Also, more 2718 than one coordinate system/epoch may be present in a dataset and 2719 you can use this option to construct your spatial constraint based 2720 on the others coordinate systems/epochs. 2721 2722‘-r FLT’ 2723‘--radius=FLT’ 2724 The radius about the requested center to use for the automatically 2725 generated query (not compatible with ‘--query’). The radius is in 2726 units of degrees, but you can use simple division with this option 2727 directly on the command-line. For example if you want a radius of 2728 20 arc-minutes or 20 arc-seconds, you can use ‘--radius=20/60’ or 2729 ‘--radius=20/3600’ respectively (which is much more human-friendly 2730 than ‘0.3333’ or ‘0.005556’). 2731 2732‘-w FLT[,FLT]’ 2733‘--width=FLT[,FLT]’ 2734 The square (or rectangle) side length (width) about the requested 2735 center to use for the automatically generated query (not compatible 2736 with ‘--query’). If only one value is given to ‘--width’ the 2737 region will be a square, but if two values are given, the widths of 2738 the query box along each dimension will be different. The value(s) 2739 is (are) in the same units as the coordinate column (see ‘--ccol’, 2740 usually RA and Dec which are degrees). You can use simple division 2741 for each value directly on the command-line if you want relatively 2742 small (and more human-friendly) sizes. For example if you want 2743 your box to be 1 arc-minutes along the RA and 2 arc-minutes along 2744 Dec, you can use ‘--width=1/60,2/60’. 2745 2746‘-g STR,FLT,FLT’ 2747‘--range=STR,FLT,FLT’ 2748 The column name and numerical range (inclusive) of acceptable 2749 values in that column (not compatible with ‘--query’). This option 2750 can be called multiple times for applying range limits on many 2751 columns in one call (thus greatly reducing the download size). For 2752 example when used on the ESA gaia database, you can use 2753 ‘--range=phot_g_mean_mag,10:15’ to only get rows that have a value 2754 between 10 and 15 (inclusive on both sides) in the 2755 ‘phot_g_mean_mag’ column. 2756 2757 If you want all rows larger, or smaller, than a certain number, you 2758 can use ‘inf’, or ‘-inf’ as the first or second values 2759 respectively. For example, if you want objects with SDSS 2760 spectroscopic redshifts larger than 2 (from the VizieR ‘sdss12’ 2761 database), you can use ‘--range=zsp,2,inf’ 2762 2763 If you want the interval to not be inclusive on both sides, you can 2764 run ‘astquery’ once and get the command that it executes. Then you 2765 can edit it to be non-inclusive on your desired side. 2766 2767‘-b STR[,STR]’ 2768‘--noblank=STR[,STR]’ 2769 Only ask for rows that don’t have a blank value in the ‘STR’ 2770 column. This option can be called many times, and each call can 2771 have multiple column names (separated by a comma or <,>). For 2772 example if you want the retrieved rows to not have a blank value in 2773 columns ‘A’, ‘B’, ‘C’ and ‘D’, you can use ‘--noblank=A -bB,C,D’. 2774 2775‘--sort=STR[,STR]’ 2776 Ask for the server to sort the downloaded data based on the given 2777 columns. For example let’s assume your desired catalog has column 2778 ‘Z’ for redshift and column ‘MAG_R’ for magnitude in the R band. 2779 When you call ‘--sort=Z,MAG_R’, it will primarily sort the columns 2780 based on the redshift, but if two objects have the same redshift, 2781 they will be sorted by magnitude. You can add as many columns as 2782 you like for higher-level sorting. 2783 2784 2785File: gnuastro.info, Node: Data manipulation, Next: Data analysis, Prev: Data containers, Up: Top 2786 27876 Data manipulation 2788******************* 2789 2790Images are one of the major formats of data that is used in astronomy. 2791The functions in this chapter explain the GNU Astronomy Utilities which 2792are provided for their manipulation. For example cropping out a part of 2793a larger image or convolving the image with a given kernel or applying a 2794transformation to it. 2795 2796* Menu: 2797 2798* Crop:: Crop region(s) from a dataset. 2799* Arithmetic:: Arithmetic on input data. 2800* Convolve:: Convolve an image with a kernel. 2801* Warp:: Warp/Transform an image to a different grid. 2802 2803 2804File: gnuastro.info, Node: Crop, Next: Arithmetic, Prev: Data manipulation, Up: Data manipulation 2805 28066.1 Crop 2807======== 2808 2809Astronomical images are often very large, filled with thousands of 2810galaxies. It often happens that you only want a section of the image, 2811or you have a catalog of sources and you want to visually analyze them 2812in small postage stamps. Crop is made to do all these things. When 2813more than one crop is required, Crop will divide the crops between 2814multiple threads to significantly reduce the run time. 2815 2816 Astronomical surveys are usually extremely large. So large in fact, 2817that the whole survey will not fit into a reasonably sized file. 2818Because of this, surveys usually cut the final image into separate tiles 2819and store each tile in a file. For example the COSMOS survey’s Hubble 2820space telescope, ACS F814W image consists of 81 separate FITS images, 2821with each one having a volume of 1.7 Giga bytes. 2822 2823 Even though the tile sizes are chosen to be large enough that too 2824many galaxies/targets don’t fall on the edges of the tiles, inevitably 2825some do. So when you simply crop the image of such targets from one 2826tile, you will miss a large area of the surrounding sky (which is 2827essential in estimating the noise). Therefore in its WCS mode, Crop 2828will stitch parts of the tiles that are relevant for a target (with the 2829given width) from all the input images that cover that region into the 2830output. Of course, the tiles have to be present in the list of input 2831files. 2832 2833 Besides cropping postage stamps around certain coordinates, Crop can 2834also crop arbitrary polygons from an image (or a set of tiles by 2835stitching the relevant parts of different tiles within the polygon), see 2836‘--polygon’ in *note Invoking astcrop::. Alternatively, it can crop out 2837rectangular regions through the ‘--section’ option from one image, see 2838*note Crop section syntax::. 2839 2840* Menu: 2841 2842* Crop modes:: Basic modes to define crop region. 2843* Crop section syntax:: How to define a section to crop. 2844* Blank pixels:: Pixels with no value. 2845* Invoking astcrop:: Calling Crop on the command-line 2846 2847 2848File: gnuastro.info, Node: Crop modes, Next: Crop section syntax, Prev: Crop, Up: Crop 2849 28506.1.1 Crop modes 2851---------------- 2852 2853In order to be comprehensive, intuitive, and easy to use, there are two 2854ways to define the crop: 2855 2856 1. From its center and side length. For example if you already know 2857 the coordinates of an object and want to inspect it in an image or 2858 to generate postage stamps of a catalog containing many such 2859 coordinates. 2860 2861 2. The vertices of the crop region, this can be useful for larger 2862 crops over many targets, for example to crop out a uniformly deep, 2863 or contiguous, region of a large survey. 2864 2865 Irrespective of how the crop region is defined, the coordinates to 2866define the crop can be in Image (pixel) or World Coordinate System (WCS) 2867standards. All coordinates are read as floating point numbers (not 2868integers, except for the ‘--section’ option, see below). By setting the 2869_mode_ in Crop, you define the standard that the given coordinates must 2870be interpreted. Here, the different ways to specify the crop region are 2871discussed within each standard. For the full list options, please see 2872*note Invoking astcrop::. 2873 2874 When the crop is defined by its center, the respective (integer) 2875central pixel position will be found internally according to the FITS 2876standard. To have this pixel positioned in the center of the cropped 2877region, the final cropped region will have an add number of pixels (even 2878if you give an even number to ‘--width’ in image mode). 2879 2880 Furthermore, when the crop is defined as by its center, Crop allows 2881you to only keep crops what don’t have any blank pixels in the vicinity 2882of their center (your primary target). This can be very convenient when 2883your input catalog/coordinates originated from another survey/filter 2884which is not fully covered by your input image, to learn more about this 2885feature, please see the description of the ‘--checkcenter’ option in 2886*note Invoking astcrop::. 2887 2888Image coordinates 2889 In image mode (‘--mode=img’), Crop interprets the pixel coordinates 2890 and widths in units of the input data-elements (for example pixels 2891 in an image, not world coordinates). In image mode, only one image 2892 may be input. The output crop(s) can be defined in multiple ways 2893 as listed below. 2894 2895 Center of multiple crops (in a catalog) 2896 The center of (possibly multiple) crops are read from a text 2897 file. In this mode, the columns identified with the 2898 ‘--coordcol’ option are interpreted as the center of a crop 2899 with a width of ‘--width’ pixels along each dimension. The 2900 columns can contain any floating point value. The value to 2901 ‘--output’ option is seen as a directory which will host (the 2902 possibly multiple) separate crop files, see *note Crop 2903 output:: for more. For a tutorial using this feature, please 2904 see *note Finding reddest clumps and visual inspection::. 2905 2906 Center of a single crop (on the command-line) 2907 The center of the crop is given on the command-line with the 2908 ‘--center’ option. The crop width is specified by the 2909 ‘--width’ option along each dimension. The given coordinates 2910 and width can be any floating point number. 2911 2912 Vertices of a single crop 2913 In Image mode there are two options to define the vertices of 2914 a region to crop: ‘--section’ and ‘--polygon’. The former is 2915 lower-level (doesn’t accept floating point vertices, and only 2916 a rectangular region can be defined), it is also only 2917 available in Image mode. Please see *note Crop section 2918 syntax:: for a full description of this method. 2919 2920 The latter option (‘--polygon’) is a higher-level method to 2921 define any polygon (with any number of vertices) with floating 2922 point values. Please see the description of this option in 2923 *note Invoking astcrop:: for its syntax. 2924 2925WCS coordinates 2926 In WCS mode (‘--mode=wcs’), the coordinates and widths are 2927 interpreted using the World Coordinate System (WCS, that must 2928 accompany the dataset), not pixel coordinates. In WCS mode, Crop 2929 accepts multiple datasets as input. When the cropped region 2930 (defined by its center or vertices) overlaps with multiple of the 2931 input images/tiles, the overlapping regions will be taken from the 2932 respective input (they will be stitched when necessary for each 2933 output crop). 2934 2935 In this mode, the input images do not necessarily have to be the 2936 same size, they just need to have the same orientation and pixel 2937 resolution. Currently only orientation along the celestial 2938 coordinates is accepted, if your input has a different orientation 2939 you can use Warp’s ‘--align’ option to align the image before 2940 cropping it (see *note Warp::). 2941 2942 Each individual input image/tile can even be smaller than the final 2943 crop. In any case, any part of any of the input images which 2944 overlaps with the desired region will be used in the crop. Note 2945 that if there is an overlap in the input images/tiles, the pixels 2946 from the last input image read are going to be used for the 2947 overlap. Crop will not change pixel values, so it assumes your 2948 overlapping tiles were cutout from the same original image. There 2949 are multiple ways to define your cropped region as listed below. 2950 2951 Center of multiple crops (in a catalog) 2952 Similar to catalog inputs in Image mode (above), except that 2953 the values along each dimension are assumed to have the same 2954 units as the dataset’s WCS information. For example, the 2955 central RA and Dec value for each crop will be read from the 2956 first and second calls to the ‘--coordcol’ option. The width 2957 of the cropped box (in units of the WCS, or degrees in RA and 2958 Dec mode) must be specified with the ‘--width’ option. 2959 2960 Center of a single crop (on the command-line) 2961 You can specify the center of only one crop box with the 2962 ‘--center’ option. If it exists in the input images, it will 2963 be cropped similar to the catalog mode, see above also for 2964 ‘--width’. 2965 2966 Vertices of a single crop 2967 The ‘--polygon’ option is a high-level method to define any 2968 convex polygon (with any number of vertices). Please see the 2969 description of this option in *note Invoking astcrop:: for its 2970 syntax. 2971 2972 *CAUTION:* In WCS mode, the image has to be aligned with the 2973 celestial coordinates, such that the first FITS axis is parallel 2974 (opposite direction) to the Right Ascension (RA) and the second 2975 FITS axis is parallel to the declination. If these conditions 2976 aren’t met for an image, Crop will warn you and abort. You can use 2977 Warp’s ‘--align’ option to align the input image with these 2978 coordinates, see *note Warp::. 2979 2980 As a summary, if you don’t specify a catalog, you have to define the 2981cropped region manually on the command-line. In any case the mode is 2982mandatory for Crop to be able to interpret the values given as 2983coordinates or widths. 2984 2985 2986File: gnuastro.info, Node: Crop section syntax, Next: Blank pixels, Prev: Crop modes, Up: Crop 2987 29886.1.2 Crop section syntax 2989------------------------- 2990 2991When in image mode, one of the methods to crop only one rectangular 2992section from the input image is to use the ‘--section’ option. Crop has 2993a powerful syntax to read the box parameters from a string of 2994characters. If you leave certain parts of the string to be empty, Crop 2995can fill them for you based on the input image sizes. 2996 2997 To define a box, you need the coordinates of two points: the first 2998(‘X1’, ‘Y1’) and the last pixel (‘X2’, ‘Y2’) pixel positions in the 2999image, or four integer numbers in total. The four coordinates can be 3000specified with one string in this format: ‘‘X1:X2,Y1:Y2’’. This string 3001is given to the ‘--section’ option. Therefore, the pixels along the 3002first axis that are $\geq$‘X1’ and $\leq$‘X2’ will be included in the 3003cropped image. The same goes for the second axis. Note that each 3004different term will be read as an integer, not a float. 3005 3006 The reason it only accepts integers is that ‘--section’ is a 3007low-level option (which is also very fast!). For a higher-level way to 3008specify region (any polygon, not just a box), please see the ‘--polygon’ 3009option in *note Crop options::. Also note that in the FITS standard, 3010pixel indexes along each axis start from unity(1) not zero(0). 3011 3012 You can omit any of the values and they will be filled automatically. 3013The left hand side of the colon (‘:’) will be filled with ‘1’, and the 3014right side with the image size. So, ‘2:,:’ will include the full range 3015of pixels along the second axis and only those with a first axis index 3016larger than ‘2’ in the first axis. If the colon is omitted for a 3017dimension, then the full range is automatically used. So the same 3018string is also equal to ‘2:,’ or ‘2:’ or even ‘2’. If you want such a 3019case for the second axis, you should set it to: ‘,2’. 3020 3021 If you specify a negative value, it will be seen as before the 3022indexes of the image which are outside the image along the bottom or 3023left sides when viewed in SAO DS9. In case you want to count from the 3024top or right sides of the image, you can use an asterisk (‘*’). When 3025confronted with a ‘*’, Crop will replace it with the maximum length of 3026the image in that dimension. So ‘*-10:*+10,*-20:*+20’ will mean that 3027the crop box will be 20\times40 pixels in size and only include the top 3028corner of the input image with 3/4 of the image being covered by blank 3029pixels, see *note Blank pixels::. 3030 3031 If you feel more comfortable with space characters between the 3032values, you can use as many space characters as you wish, just be 3033careful to put your value in double quotes, for example 3034‘--section="5:200, 123:854"’. If you forget the quotes, anything after 3035the first space will not be seen by ‘--section’ and you will most 3036probably get an error because the rest of your string will be read as a 3037filename (which most probably doesn’t exist). See *note Command-line:: 3038for a description of how the command-line works. 3039 3040 3041File: gnuastro.info, Node: Blank pixels, Next: Invoking astcrop, Prev: Crop section syntax, Up: Crop 3042 30436.1.3 Blank pixels 3044------------------ 3045 3046The cropped box can potentially include pixels that are beyond the image 3047range. For example when a target in the input catalog was very near the 3048edge of the input image. The parts of the cropped image that were not 3049in the input image will be filled with the following two values 3050depending on the data type of the image. In both cases, SAO DS9 will 3051not color code those pixels. 3052 • If the data type of the image is a floating point type (float or 3053 double), IEEE NaN (Not a number) will be used. 3054 • For integer types, pixels out of the image will be filled with the 3055 value of the ‘BLANK’ keyword in the cropped image header. The 3056 value assigned to it is the lowest value possible for that type, so 3057 you will probably never need it any way. Only for the unsigned 3058 character type (‘BITPIX=8’ in the FITS header), the maximum value 3059 is used because it is unsigned, the smallest value is zero which is 3060 often meaningful. 3061 You can ask for such blank regions to not be included in the output 3062crop image using the ‘--noblank’ option. In such cases, there is no 3063guarantee that the image size of your outputs are what you asked for. 3064 3065 In some survey images, unfortunately they do not use the ‘BLANK’ FITS 3066keyword. Instead they just give all pixels outside of the survey area a 3067value of zero. So by default, when dealing with float or double image 3068types, any values that are 0.0 are also regarded as blank regions. This 3069can be turned off with the ‘--zeroisnotblank’ option. 3070 3071 3072File: gnuastro.info, Node: Invoking astcrop, Prev: Blank pixels, Up: Crop 3073 30746.1.4 Invoking Crop 3075------------------- 3076 3077Crop will crop a region from an image. If in WCS mode, it will also 3078stitch parts from separate images in the input files. The executable 3079name is ‘astcrop’ with the following general template 3080 3081 $ astcrop [OPTION...] [ASCIIcatalog] ASTRdata ... 3082 3083One line examples: 3084 3085 ## Crop all objects in cat.txt from image.fits: 3086 $ astcrop --catalog=cat.txt image.fits 3087 3088 ## Crop all options in catalog (with RA,DEC) from all the files 3089 ## ending in `_drz.fits' in `/mnt/data/COSMOS/': 3090 $ astcrop --mode=wcs --catalog=cat.txt /mnt/data/COSMOS/*_drz.fits 3091 3092 ## Crop the outer 10 border pixels of the input image: 3093 $ astcrop --section=10:*-10,10:*-10 --hdu=2 image.fits 3094 3095 ## Crop region around RA and Dec of (189.16704, 62.218203): 3096 $ astcrop --mode=wcs --center=189.16704,62.218203 goodsnorth.fits 3097 3098 ## Crop region around pixel coordinate (568.342, 2091.719): 3099 $ astcrop --mode=img --center=568.342,2091.719 --width=201 image.fits 3100 3101Crop has one mandatory argument which is the input image name(s), shown 3102above with ‘ASTRdata ...’. You can use shell expansions, for example 3103‘*’ for this if you have lots of images in WCS mode. If the crop box 3104centers are in a catalog, you can use the ‘--catalog’ option. In other 3105cases, you have to provide the single cropped output parameters must be 3106given with command-line options. See *note Crop output:: for how the 3107output file name(s) can be specified. For the full list of general 3108options to all Gnuastro programs (including Crop), please see *note 3109Common options::. 3110 3111 Floating point numbers can be used to specify the crop region (except 3112the ‘--section’ option, see *note Crop section syntax::). In such 3113cases, the floating point values will be used to find the desired 3114integer pixel indices based on the FITS standard. Hence, Crop 3115ultimately doesn’t do any sub-pixel cropping (in other words, it doesn’t 3116change pixel values). If you need such crops, you can use *note Warp:: 3117to first warp the image to the a new pixel grid, then crop from that. 3118For example, let’s assume you want a crop from pixels 12.982 to 80.982 3119along the first dimension. You should first translate the image by 3120$-0.482$ (note that the edge of a pixel is at integer multiples of 3121$0.5$). So you should run Warp with ‘--translate=-0.482,0’ and then 3122crop the warped image with ‘--section=13:81’. 3123 3124 There are two ways to define the cropped region: with its center or 3125its vertices. See *note Crop modes:: for a full description. In the 3126former case, Crop can check if the central region of the cropped image 3127is indeed filled with data or is blank (see *note Blank pixels::), and 3128not produce any output when the center is blank, see the description 3129under ‘--checkcenter’ for more. 3130 3131 When in catalog mode, Crop will run in parallel unless you set 3132‘--numthreads=1’, see *note Multi-threaded operations::. Note that when 3133multiple outputs are created with threads, the outputs will not be 3134created in the same order. This is because the threads are asynchronous 3135and thus not started in order. This has no effect on each output, see 3136*note Finding reddest clumps and visual inspection:: for a tutorial on 3137effectively using this feature. 3138 3139* Menu: 3140 3141* Crop options:: A list of all the options with explanation. 3142* Crop output:: The outputs of Crop. 3143* Crop known issues:: Known issues in running Crop. 3144 3145 3146File: gnuastro.info, Node: Crop options, Next: Crop output, Prev: Invoking astcrop, Up: Invoking astcrop 3147 31486.1.4.1 Crop options 3149.................... 3150 3151The options can be classified into the following contexts: Input, Output 3152and operating mode options. Options that are common to all Gnuastro 3153program are listed in *note Common options:: and will not be repeated 3154here. 3155 3156 When you are specifying the crop vertices your self (through 3157‘--section’, or ‘--polygon’) on relatively small regions (depending on 3158the resolution of your images) the outputs from image and WCS mode can 3159be approximately equivalent. However, as the crop sizes get large, the 3160curved nature of the WCS coordinates have to be considered. For 3161example, when using ‘--section’, the right ascension of the bottom left 3162and top left corners will not be equal. If you only want regions within 3163a given right ascension, use ‘--polygon’ in WCS mode. 3164 3165Input image parameters: 3166 3167‘--hstartwcs=INT’ 3168 Specify the first keyword card (line number) to start finding the 3169 input image world coordinate system information. This is useful 3170 when certain header keywords of the input may cause bad conflicts 3171 with your crop (see an example described below). To get line 3172 numbers of the header keywords, you can pipe the fully printed 3173 header into ‘cat -n’ like below: 3174 3175 $ astfits image.fits -h1 | cat -n 3176 3177 For example, distortions have only been present in WCSLIB from 3178 version 5.15 (released in mid 2016). Therefore some pipelines 3179 still apply their own specific set of WCS keywords for distortions 3180 and put them into the image header along with those that WCSLIB 3181 does recognize. So now that WCSLIB recognizes most of the standard 3182 distortion parameters, they will get confused with the old ones and 3183 give wrong results. For example in the CANDELS-GOODS South images 3184 that were created before WCSLIB 5.15(1). 3185 3186 The two ‘--hstartwcs’ and ‘--hendwcs’ are thus provided so when 3187 using older datasets, you can specify what region in the FITS 3188 headers you want to use to read the WCS keywords. Note that this 3189 is only relevant for reading the WCS information, basic data 3190 information like the image size are read separately. These two 3191 options will only be considered when the value to ‘--hendwcs’ is 3192 larger than that of ‘--hstartwcs’. So if they are equal or 3193 ‘--hstartwcs’ is larger than ‘--hendwcs’, then all the input 3194 keywords will be parsed to get the WCS information of the image. 3195 3196‘--hendwcs=INT’ 3197 Specify the last keyword card to read for specifying the image 3198 world coordinate system on the input images. See ‘--hstartwcs’ 3199 3200Crop box parameters: 3201 3202‘-c FLT[,FLT[,...]]’ 3203‘--center=FLT[,FLT[,...]]’ 3204 The central position of the crop in the input image. The positions 3205 along each dimension must be separated by a comma (<,>) and 3206 fractions are also acceptable. The number of values given to this 3207 option must be the same as the dimensions of the input dataset. 3208 The width of the crop should be set with ‘--width’. The units of 3209 the coordinates are read based on the value to the ‘--mode’ option, 3210 see below. 3211 3212‘-w FLT[,FLT[,...]]’ 3213‘--width=FLT[,FLT[,...]]’ 3214 Width of the cropped region about coordinate given to ‘--center’. 3215 If in WCS mode, value(s) given to this option will be read in the 3216 same units as the dataset’s WCS information along this dimension. 3217 This option may take either a single value (to be used for all 3218 dimensions: ‘--width=10’ in image-mode will crop a $10\times10$ 3219 pixel image) or multiple values (a specific value for each 3220 dimension: ‘--width=10,20’ in image-mode will crop a $10\times20$ 3221 pixel image). 3222 3223 The ‘--width’ option also accepts fractions. For example if you 3224 want the width of your crop to be 3 by 5 arcseconds along RA and 3225 Dec respectively and you are in wcs-mode, you can use: 3226 ‘--width=3/3600,5/3600’. 3227 3228 The final output will have an odd number of pixels to allow easy 3229 identification of the pixel which keeps your requested coordinate 3230 (from ‘--center’ or ‘--catalog’). If you want an even sided crop, 3231 you can run Crop afterwards with ‘--section=":*-1,:*-1"’ or 3232 ‘--section=2:,2:’ (depending on which side you don’t need), see 3233 *note Crop section syntax::. 3234 3235 The basic reason for making an odd-sided crop is that your given 3236 central coordinate will ultimately fall within a discrete pixel in 3237 the image (defined by the FITS standard). When the crop has an odd 3238 number of pixels in each dimension, that pixel can be very well 3239 defined as the “central” pixel of the crop, making it unambiguously 3240 easy to identify. However, for an even-sided crop, it will be very 3241 hard to identify the central pixel (it can be on any of the four 3242 pixels adjacent to the central point of the image!). 3243 3244‘-l STR’ 3245‘-l FLT:FLT,...’ 3246‘--polygon=STR’ 3247‘--polygon=FLT,FLT:FLT,FLT:...’ 3248 Polygon vertice coordinates (when value is in ‘FLT,FLT:FLT,FLT:...’ 3249 format) or the filename of a SAO DS9 region file (when the value 3250 has no ‘,’ or ‘:’ characters). The vertices are used to define the 3251 polygon: in the same order given to this option. When the vertices 3252 are not necessarily ordered in the proper order (for example one 3253 vertice in a square comes after its diagonal opposite), you can add 3254 the ‘--polygonsort’ option which will attempt to sort the vertices 3255 before cropping. Note that for concave polygons, sorting is not 3256 recommended because there is no unique solution, for more, see the 3257 description under ‘--polygonsort’. 3258 3259 This option can be used both in the image and WCS modes, see *note 3260 Crop modes::. If a SAO DS9 region file is used, the coordinate 3261 mode of Crop will be determined by the contents of the file and any 3262 value given to ‘--mode’ is ignored. The cropped image will be the 3263 size of the rectangular region that completely encompasses the 3264 polygon. By default all the pixels that are outside of the polygon 3265 will be set as blank values (see *note Blank pixels::). However, 3266 if ‘--polygonout’ is called all pixels internal to the vertices 3267 will be set to blank. In WCS-mode, you may provide many FITS 3268 images/tiles: Crop will stitch them to produce this cropped region, 3269 then apply the polygon. 3270 3271 The syntax for the polygon vertices is similar to, and simpler 3272 than, that for ‘--section’. In short, the dimensions of each 3273 coordinate are separated by a comma (<,>) and each vertex is 3274 separated by a colon (<:>). You can define as many vertices as you 3275 like. If you would like to use space characters between the 3276 dimensions and vertices to make them more human-readable, then you 3277 have to put the value to this option in double quotation marks. 3278 3279 For example, let’s assume you want to work on the deepest part of 3280 the WFC3/IR images of Hubble Space Telescope eXtreme Deep Field 3281 (HST-XDF). According to the web page 3282 (https://archive.stsci.edu/prepds/xdf/)(2) the deepest part is 3283 contained within the coordinates: 3284 3285 [ (53.187414,-27.779152), (53.159507,-27.759633), 3286 (53.134517,-27.787144), (53.161906,-27.807208) ] 3287 3288 They have provided mask images with only these pixels in the 3289 WFC3/IR images, but what if you also need to work on the same 3290 region in the full resolution ACS images? Also what if you want to 3291 use the CANDELS data for the shallow region? Running Crop with 3292 ‘--polygon’ will easily pull out this region of the image for you, 3293 irrespective of the resolution. If you have set the operating mode 3294 to WCS mode in your nearest configuration file (see *note 3295 Configuration files::), there is no need to call ‘--mode=wcs’ on 3296 the command line. 3297 3298 $ astcrop --mode=wcs desired-filter-image(s).fits \ 3299 --polygon="53.187414,-27.779152 : 53.159507,-27.759633 : \ 3300 53.134517,-27.787144 : 53.161906,-27.807208" 3301 3302 More generally, you have an image and want to define the polygon 3303 yourself (it isn’t already published like the example above). As 3304 the number of vertices increases, checking the vertex coordinates 3305 on a FITS viewer (for example SAO DS9) and typing them in, one by 3306 one, can be very tedious and prone to typo errors. In such cases, 3307 you can make a polygon “region” in DS9 and using your mouse, easily 3308 define (and visually see) it. Given that SAO DS9 has a graphic 3309 user interface (GUI), if you don’t have the polygon vertices 3310 before-hand, it is much more easier build your polygon there and 3311 pass it onto Crop through the region file. 3312 3313 You can take the following steps to make an SAO DS9 region file 3314 containing your polygon. Open your desired FITS image with SAO DS9 3315 and activate its “region” mode with Edit→Region. Then define the 3316 region as a polygon with Region→Shape→Polygon. Click on the 3317 approximate center of the region you want and a small square will 3318 appear. By clicking on the vertices of the square you can shrink 3319 or expand it, clicking and dragging anywhere on the edges will 3320 enable you to define a new vertex. After the region has been 3321 nicely defined, save it as a file with Region→“Save Regions”. You 3322 can then select the name and address of the output file, keep the 3323 format as ‘REG (*.reg)’ and press the “OK” button. In the next 3324 window, keep format as “ds9” and “Coordinate System” as “fk5” for 3325 RA and Dec (or “Image” for pixel coordinates). A plain text file 3326 is now created (let’s call it ‘ds9.reg’) which you can pass onto 3327 Crop with ‘--polygon=ds9.reg’. 3328 3329 For the expected format of the region file, see the description of 3330 ‘gal_ds9_reg_read_polygon’ in *note SAO DS9 library::. However, 3331 since SAO DS9 makes this file for you, you don’t usually need to 3332 worry about its internal format unless something un-expected 3333 happens and you find a bug. 3334 3335‘--polygonout’ 3336 Keep all the regions outside the polygon and mask the inner ones 3337 with blank pixels (see *note Blank pixels::). This is practically 3338 the inverse of the default mode of treating polygons. Note that 3339 this option only works when you have only provided one input image. 3340 If multiple images are given (in WCS mode), then the full area 3341 covered by all the images has to be shown and the polygon excluded. 3342 This can lead to a very large area if large surveys like COSMOS are 3343 used. So Crop will abort and notify you. In such cases, it is 3344 best to crop out the larger region you want, then mask the smaller 3345 region with this option. 3346 3347‘--polygonsort’ 3348 Sort the given set of vertices to the ‘--polygon’ option. For a 3349 concave polygon it will sort the vertices correctly, however for a 3350 convex polygon it there is no unique sorting, so be careful because 3351 the crop may not be what you expected. 3352 3353 Polygons come in two classes: convex and concave (or generally, 3354 non-convex!), see below for a demonstration. Convex polygons are 3355 those where all inner angles are less than 180 degrees. By 3356 contrast, a convex polygon is one where an inner angle may be more 3357 than 180 degrees. 3358 3359 Concave Polygon Convex Polygon 3360 3361 D --------C D------------- C 3362 \ | E / | 3363 \E | \ | 3364 / | \ | 3365 A--------B A ----------B 3366 3367‘-s STR’ 3368‘--section=STR’ 3369 Section of the input image which you want to be cropped. See *note 3370 Crop section syntax:: for a complete explanation on the syntax 3371 required for this input. 3372 3373‘-C FITS/TXT’ 3374‘--catalog=FITS/TXT’ 3375 File name of catalog for making multiple crops from the input 3376 images/cubes. The catalog can be in any of Gnuastro’s recognized 3377 *note Recognized table formats::. The columns containing the 3378 coordinates for the crop centers can be specified with the 3379 ‘--coordcol’ option (using column names or numbers, see *note 3380 Selecting table columns::). The catalog can also contain the name 3381 of each crop, you can specify the column containing the name with 3382 the ‘--namecol’. 3383 3384‘--cathdu=STR/INT’ 3385 The HDU (extension) containing the catalog (if the file given to 3386 ‘--catalog’ is a FITS file). This can either be the HDU name (if 3387 it has one) or number (counting from 0). By default (if this 3388 option is not given), the second HDU will be used (equivalent to 3389 ‘--cathdu=1’. For more on how to specify the HDU, see the 3390 explanation of the ‘--hdu’ option in *note Input output options::. 3391 3392‘-x STR/INT’ 3393‘--coordcol=STR/INT’ 3394 The column in a catalog to read as a coordinate. The value can be 3395 either the column number (starting from 1), or a match/search in 3396 the table meta-data, see *note Selecting table columns::. This 3397 option must be called multiple times, depending on the number of 3398 dimensions in the input dataset. If it is called more than 3399 necessary, the extra columns (later calls to this option on the 3400 command-line or configuration files) will be ignored, see *note 3401 Configuration file precedence::. 3402 3403‘-n STR/INT’ 3404‘--namecol=STR/INT’ 3405 Column selection of crop file name. The value can be either the 3406 column number (starting from 1), or a match/search in the table 3407 meta-data, see *note Selecting table columns::. This option can be 3408 used both in Image and WCS modes, and not a mandatory. When a 3409 column is given to this option, the final crop base file name will 3410 be taken from the contents of this column. The directory will be 3411 determined by the ‘--output’ option (current directory if not 3412 given) and the value to ‘--suffix’ will be appended. When this 3413 column isn’t given, the row number will be used instead. 3414 3415Output options: 3416 3417‘-c FLT/INT’ 3418‘--checkcenter=FLT/INT’ 3419 Square box width of region in the center of the image to check for 3420 blank values. If any of the pixels in this central region of a 3421 crop (defined by its center) are blank, then it will not be stored 3422 in an output file. If the value to this option is zero, no 3423 checking is done. This check is only applied when the cropped 3424 region(s) are defined by their center (not by the vertices, see 3425 *note Crop modes::). 3426 3427 The units of the value are interpreted based on the ‘--mode’ value 3428 (in WCS or pixel units). The ultimate checked region size (in 3429 pixels) will be an odd integer around the center (converted from 3430 WCS, or when an even number of pixels are given to this option). 3431 In WCS mode, the value can be given as fractions, for example if 3432 the WCS units are in degrees, ‘0.1/3600’ will correspond to a check 3433 size of 0.1 arcseconds. 3434 3435 Because survey regions don’t often have a clean square or rectangle 3436 shape, some of the pixels on the sides of the survey FITS image 3437 don’t commonly have any data and are blank (see *note Blank 3438 pixels::). So when the catalog was not generated from the input 3439 image, it often happens that the image does not have data over some 3440 of the points. 3441 3442 When the given center of a crop falls in such regions or outside 3443 the dataset, and this option has a non-zero value, no crop will be 3444 created. Therefore with this option, you can specify a width of a 3445 small box (3 pixels is often good enough) around the central pixel 3446 of the cropped image. You can check which crops were created and 3447 which weren’t from the command-line (if ‘--quiet’ was not called, 3448 see *note Operating mode options::), or in Crop’s log file (see 3449 *note Crop output::). 3450 3451‘-p STR’ 3452‘--suffix=STR’ 3453 The suffix (or post-fix) of the output files for when you want all 3454 the cropped images to have a special ending. One case where this 3455 might be helpful is when besides the science images, you want the 3456 weight images (or exposure maps, which are also distributed with 3457 survey images) of the cropped regions too. So in one run, you can 3458 set the input images to the science images and ‘--suffix=_s.fits’. 3459 In the next run you can set the weight images as input and 3460 ‘--suffix=_w.fits’. 3461 3462‘--primaryimghdu’ 3463 Write the output into the primary (0-th) HDU/extension of the 3464 output. By default, like all Gnuastro’s default outputs, no data 3465 is written in the primary extension because the FITS standard 3466 suggests keeping that extension free of data and only for meta 3467 data. 3468 3469‘-b’ 3470‘--noblank’ 3471 Pixels outside of the input image that are in the crop box will not 3472 be used. By default they are filled with blank values (depending 3473 on type), see *note Blank pixels::. This option only applies only 3474 in Image mode, see *note Crop modes::. 3475 3476‘-z’ 3477‘--zeroisnotblank’ 3478 In float or double images, it is common to give the value of zero 3479 to blank pixels. If the input image type is one of these two 3480 types, such pixels will also be considered as blank. You can 3481 disable this behavior with this option, see *note Blank pixels::. 3482 3483Operating mode options: 3484 3485‘-O STR’ 3486‘--mode=STR’ 3487 Operate in Image mode or WCS mode when the input coordinates can be 3488 both image or WCS. The value must either be ‘img’ or ‘wcs’, see 3489 *note Crop modes:: for a full description. 3490 3491 ---------- Footnotes ---------- 3492 3493 (1) <https://archive.stsci.edu/pub/hlsp/candels/goods-s/gs-tot/v1.0/> 3494 3495 (2) <https://archive.stsci.edu/prepds/xdf/> 3496 3497 3498File: gnuastro.info, Node: Crop output, Next: Crop known issues, Prev: Crop options, Up: Invoking astcrop 3499 35006.1.4.2 Crop output 3501................... 3502 3503The string given to ‘--output’ option will be interpreted depending on 3504how many crops were requested, see *note Crop modes::: 3505 3506 • When a catalog is given, the value of the ‘--output’ (see *note 3507 Common options::) will be read as the directory to store the output 3508 cropped images. Hence if it doesn’t already exist, Crop will abort 3509 with an “No such file or directory” error. 3510 3511 The crop file names will consist of two parts: a variable part (the 3512 row number of each target starting from 1) along with a fixed 3513 string which you can set with the ‘--suffix’ option. Optionally, 3514 you may also use the ‘--namecol’ option to define a column in the 3515 input catalog to use as the file name instead of numbers. 3516 3517 • When only one crop is desired, the value to ‘--output’ will be read 3518 as a file name. If no output is specified or if it is a directory, 3519 the output file name will follow the automatic output names of 3520 Gnuastro, see *note Automatic output::: The string given to 3521 ‘--suffix’ will be replaced with the ‘.fits’ suffix of the input. 3522 3523 By default, as suggested by the FITS standard and implemented in all 3524Gnuastro programs, the first/primary extension of the output files will 3525only contain meta data. The cropped images/cubes will be written into 3526the 2nd HDU of their respective FITS file (which is actually counted as 3527‘1’ because HDU counting starts from ‘0’). However, if you want the 3528cropped data to be written into the primary (0-th) HDU, run Crop with 3529the ‘--primaryimghdu’ option. 3530 3531 The header of each output cropped image will contain the names of the 3532input image(s) it was cut from. If a name is longer than the 70 3533character space that the FITS standard allows for header keyword values, 3534the name will be cut into several keywords from the nearest slash (</>). 3535The keywords have the following format: ‘ICFn_m’ (for Crop File). Where 3536‘n’ is the number of the image used in this crop and ‘m’ is the part of 3537the name (it can be broken into multiple keywords). Following the name 3538is another keyword named ‘ICFnPIX’ which shows the pixel range from that 3539input image in the same syntax as *note Crop section syntax::. So this 3540string can be directly given to the ‘--section’ option later. 3541 3542 Once done, a log file can be created in the current directory with 3543the ‘--log’ option. This file will have three columns and the same 3544number of rows as the number of cropped images. There are also comments 3545on the top of the log file explaining basic information about the run 3546and descriptions for the columns. A short description of the columns is 3547also given below: 3548 3549 1. The cropped image file name for that row. 3550 2. The number of input images that were used to create that image. 3551 3. A ‘0’ if the central few pixels (value to the ‘--checkcenter’ 3552 option) are blank and ‘1’ if they aren’t. When the crop was not 3553 defined by its center (see *note Crop modes::), or ‘--checkcenter’ 3554 was given a value of 0 (see *note Invoking astcrop::), the center 3555 will not be checked and this column will be given a value of ‘-1’. 3556 3557 3558File: gnuastro.info, Node: Crop known issues, Prev: Crop output, Up: Invoking astcrop 3559 35606.1.4.3 Crop known issues 3561......................... 3562 3563When running Crop, you may encounter strange errors and bugs. In these 3564cases, please report a bug and we will try to fix it as soon as 3565possible, see *note Report a bug::. However, some things are beyond our 3566control, or may take too long to fix directly. In this section we list 3567such known issues that may occur in known cases and suggest the hack (or 3568work-around) to fix the problem: 3569 3570Crash with ‘Killed’ when cropping catalog from ‘.fits.gz’ 3571 This happens because CFISTIO (that reads and writes FITS files) 3572 will internally decompress the file in a temporary place (possibly 3573 in the RAM), then start reading from it. On the other hand, by 3574 default when given a catalog (with many crops) and not specifying 3575 ‘--numthreads’, Crop will use the maximum number of threads 3576 available on your system to do each crop faster. On an normal (not 3577 compressed) file, parallel access will not cause a problem, 3578 however, when attempting parallel access with the maximum number of 3579 threads on a compressed file, CFITSIO crashes with ‘Killed’. 3580 Therefore the following solutions can be used to fix this crash: 3581 3582 • Decrease the number of threads (at the minimum, set 3583 ‘--numthreads=1’). Since this solution doesn’t attempt to 3584 change any of your previous Crop command components or doesn’t 3585 change your local file structure, it is the preferred way. 3586 3587 • Decompress the file (with the command below) and feed the 3588 ‘.fits’ file into Crop without changing the number of threads. 3589 3590 $ gunzip -k image.fits.gz 3591 3592 3593File: gnuastro.info, Node: Arithmetic, Next: Convolve, Prev: Crop, Up: Data manipulation 3594 35956.2 Arithmetic 3596============== 3597 3598It is commonly necessary to do operations on some or all of the elements 3599of a dataset independently (pixels in an image). For example, in the 3600reduction of raw data it is necessary to subtract the Sky value (*note 3601Sky value::) from each image image. Later (once the images as warped 3602into a single grid using Warp for example, see *note Warp::), the images 3603are co-added (the output pixel grid is the average of the pixels of the 3604individual input images). Arithmetic is Gnuastro’s program for such 3605operations on your datasets directly from the command-line. It 3606currently uses the reverse polish or post-fix notation, see *note 3607Reverse polish notation:: and will work on the native data types of the 3608input images/data to reduce CPU and RAM resources, see *note Numeric 3609data types::. For more information on how to run Arithmetic, please see 3610*note Invoking astarithmetic::. 3611 3612* Menu: 3613 3614* Reverse polish notation:: The current notation style for Arithmetic 3615* Arithmetic operators:: List of operators known to Arithmetic 3616* Invoking astarithmetic:: How to run Arithmetic: options and output 3617 3618 3619File: gnuastro.info, Node: Reverse polish notation, Next: Arithmetic operators, Prev: Arithmetic, Up: Arithmetic 3620 36216.2.1 Reverse polish notation 3622----------------------------- 3623 3624The most common notation for arithmetic operations is the infix notation 3625(https://en.wikipedia.org/wiki/Infix_notation) where the operator goes 3626between the two operands, for example $4+5$. The infix notation is the 3627preferred way in most programming languages which come with scripting 3628features for large programs. This is because the infix notation 3629requires a way to define precedence when more than one operator is 3630involved. 3631 3632 For example consider the statement ‘5 + 6 / 2’. Should 6 first be 3633divided by 2, then added by 5? Or should 5 first be added with 6, then 3634divided by 2? Therefore we need parenthesis to show precedence: 3635‘5+(6/2)’ or ‘(5+6)/2’. Furthermore, if you need to leave a value for 3636later processing, you will need to define a variable for it; for example 3637‘a=(5+6)/2’. 3638 3639 Gnuastro provides libraries where you can also use infix notation in 3640C or C++ programs. However, Gnuastro’s programs are primarily designed 3641to be run on the command-line and the level of complexity that infix 3642notation requires can be annoying/confusing to write on the command-line 3643(where they can get confused with the shell’s parenthesis or variable 3644definitions). Therefore Gnuastro’s Arithmetic and Table (when doing 3645column arithmetic) programs use the the post-fix notation, also known as 3646reverse polish notation 3647(https://en.wikipedia.org/wiki/Reverse_Polish_notation). For example, 3648instead of writing ‘5+6’, we write ‘5 6 +’. 3649 3650 The Wikipedia article on the reverse polish notation provides some 3651excellent explanation on this notation but here we will give a short 3652summary here for self-sufficiency. In short, in the reverse polish 3653notation, the operator is placed after the operands. As we will see 3654below this removes the need to define parenthesis and lets you use 3655previous values without needing to define a variable. In the future(1) 3656we do plan to also optionally allow infix notation when arithmetic 3657operations on datasets are desired, but due to time constraints on the 3658developers we can’t do it immediately. 3659 3660 To easily understand how the reverse polish notation works, you can 3661think of each operand (‘5’ and ‘6’ in the example above) as a node in a 3662“last-in-first-out” stack. One such stack in daily life is a stack of 3663dishes in the kitchen: you put a clean dish, on the top of a stack of 3664dishes when its ready for later usage. Later, when you need a dish, you 3665pick the top one (hence the “last” dish placed “in” the stack is the 3666“first” dish that comes “out” when necessary). 3667 3668 Each operator will need a certain number of operands (in the example 3669above, the ‘+’ operator needs two operands: ‘5’ and ‘6’). In the 3670kitchen metaphor, an operator can be an oven. Every time an operator is 3671confronted, the operator takes (or “pops”) the number of operands it 3672needs from the top of the stack (so they don’t exist in the stack any 3673more), does its operation, and places (or “pushes”) the result back on 3674top of the stack. So if you want the average of 5 and 6, you would 3675write: ‘5 6 + 2 /’. The operations that are done are: 3676 3677 1. ‘5’ is an operand, so Arithmetic pushes it to the top of the stack 3678 (which is initially empty). In the kitchen metaphor, you can 3679 visualize this as taking a new dish from the cabinet, putting the 3680 number 5 inside of the dish, and putting the dish on top of the 3681 (empty) cooking table in front of you. You now have a stack of one 3682 dish on the table in front of you. 3683 2. ‘6’ is also an operand, so it is pushed to the top of the stack. 3684 Like before, you can visualize this as taking a new dish from the 3685 cabinet, putting the number 6 in it and placing it on top of the 3686 previous dish. You now have a stack of two dishes on the table in 3687 front of you. 3688 3. ‘+’ is a _binary_ operator, so it will pop the top two elements of 3689 the stack out of it, and perform addition on them (the order is 3690 $5+6$ in the example above). The result is ‘11’ which is pushed to 3691 the top of the stack. 3692 3693 To visualize this, you can think of the ‘+’ operator as an oven 3694 with a place for two dishes. You pick up the top-most dish (that 3695 has the number 6 in it) and put it in the oven. The top dish is 3696 now the one that has has the number 5. You also pick it up and put 3697 it in the oven, and close the oven door. When the oven has 3698 finished its cooking, it produces a single output (in one dish, 3699 with the number 11 inside of it). You take that output dish and 3700 put it back on the table. You now have a stack of one dish on the 3701 table in front of you. 3702 4. ‘2’ is an operand so push it onto the top of the stack. In the 3703 kitchen metaphor, you again go to the cabinet, pick up a dish and 3704 put the number 2 inside of it and put the dish over the previous 3705 dish (that has the number 11). You now have a stack of two dishes 3706 on the table in front of you. 3707 5. ‘/’ (division) is a binary operator, so pull out the top two 3708 elements of the stack (top-most is ‘2’, then ‘11’) and divide the 3709 second one by the first. In the kitchen metaphor, the ‘/’ operator 3710 can be visualized as a microwave that takes two dishes. But unlike 3711 the oven (‘+’ operator) before, the order of inputs matters (they 3712 are on top of each other: with the top dish holder being the 3713 nominator and the bottom one being the denominator). Again, you 3714 look to your stack of dishes on the table. 3715 3716 You pick up the top one (with value 2 inside of it) and put it in 3717 the microwave’s bottom (denominator) dish holder. Then you go back 3718 to your stack of dishes on the table and pick up the top dish (with 3719 value 11 inside of it) and put that in the top (nominator) dish 3720 holder. The microwave will do its work and when its finished, 3721 returns a new dish with the single value 5.5 inside of it. You 3722 pick up the dish from the microwave and place it back on the table. 3723 3724 6. There are no more operands or operators, so simply return the 3725 remaining operand in the output. In the kitchen metaphor, you see 3726 that your recipe has no more steps, so you just pick up the 3727 remaining dish and take it to the dining room to enjoy a good 3728 dinner. 3729 3730 In the Arithmetic program, the operands can be FITS images of any 3731dimensionality, or numbers (see *note Invoking astarithmetic::). In 3732Table’s column arithmetic, they can be any column in the table (a series 3733of numbers in an array) or a single number (see *note Column 3734arithmetic::). 3735 3736 With this notation, very complicated procedures can be created 3737without the need for parenthesis or worrying about precedence. Even 3738functions which take an arbitrary number of arguments can be defined in 3739this notation. This is a very powerful notation and is used in 3740languages like Postscript (2) which produces PDF files when compiled. 3741 3742 ---------- Footnotes ---------- 3743 3744 (1) <https://savannah.gnu.org/task/index.php?13867> 3745 3746 (2) See the EPS and PDF part of *note Recognized file formats:: for a 3747little more on the Postscript language. 3748 3749 3750File: gnuastro.info, Node: Arithmetic operators, Next: Invoking astarithmetic, Prev: Reverse polish notation, Up: Arithmetic 3751 37526.2.2 Arithmetic operators 3753-------------------------- 3754 3755In this section, list of recognized operators in Arithmetic (and the 3756Table program’s *note Column arithmetic::) and discussed in detail with 3757examples. As mentioned before, to be able to easily do complex 3758operations on the command-line, the Reverse Polish Notation is used 3759(where you write ‘$4\quad5\quad+$’ instead of ‘$4 + 5$’), if you aren’t 3760already familiar with it, before continuing, please see *note Reverse 3761polish notation::. 3762 3763 The operands to all operators can be a data array (for example a FITS 3764image or data cube) or a number, the output will be an array or number 3765according to the inputs. For example a number multiplied by an array 3766will produce an array. The numerical data type of the output of each 3767operator is described within it. 3768 3769*Blank pixels in Arithmetic:* Blank pixels in the image (see *note Blank 3770pixels::) will be stored based on the data type. When the input is 3771floating point type, blank values are NaN. One aspect of NaN values is 3772that by definition they will fail on _any_ comparison. Hence both equal 3773and not-equal operators will fail when both their operands are NaN! 3774Therefore, the only way to guarantee selection of blank pixels is 3775through the ‘isblank’ operator explained above. 3776 3777 One way you can exploit this property of the NaN value to your 3778advantage is when you want a fully zero-valued image (even over the 3779blank pixels) based on an already existing image (with same size and 3780world coordinate system settings). The following command will produce 3781this for you: 3782 3783 $ astarithmetic input.fits nan eq --output=all-zeros.fits 3784 3785Note that on the command-line you can write NaN in any case (for example 3786‘NaN’, or ‘NAN’ are also acceptable). Reading NaN as a floating point 3787number in Gnuastro isn’t case-sensitive. 3788 3789* Menu: 3790 3791* Basic mathematical operators:: For example +, -, /, log, pow, and etc. 3792* Trigonometric and hyperbolic operators:: sin, cos, atan, asinh, and etc 3793* Unit conversion operators:: magnitudes to counts, or parsecs to AUs, and etc. 3794* Statistical operators:: Statistics of a single dataset (for example mean). 3795* Stacking operators:: Coadding or combining multiple datasets into one. 3796* Filtering operators:: Smoothing a dataset through mixing pixel with neighbors. 3797* Interpolation operators:: Giving blank pixels a value. 3798* Dimensionality changing operators:: Collapse or expand a dataset. 3799* Conditional operators:: Select certain pixels within the dataset. 3800* Mathematical morphology operators:: Work on binary images, for example erode. 3801* Bitwise operators:: Work on bits within one pixel. 3802* Numerical type conversion operators:: Convert the numeric datatype of a dataset. 3803* Adding noise operators:: Add noise to a dataset. 3804* Elliptical shape operators:: Operations that are focused on an ellipse. 3805* Building new dataset:: How to construct an empty dataset from scratch. 3806* Operand storage in memory or a file:: Tools for complex operations in one command. 3807 3808 3809File: gnuastro.info, Node: Basic mathematical operators, Next: Trigonometric and hyperbolic operators, Prev: Arithmetic operators, Up: Arithmetic operators 3810 38116.2.2.1 Basic mathematical operators 3812.................................... 3813 3814These are some of the most common operations you will be doing on your 3815data and include, so no further explanation is necessary. If you are 3816new to Gnuastro, just read the description of each carefully. 3817 3818‘+’ 3819 Addition, so “‘4 5 +’” is equivalent to $4+5$. For example in the 3820 command below, the value 20000 is added to each pixel’s value in 3821 ‘image.fits’: 3822 $ astarithmetic 20000 image.fits + 3823 You can also use this operator is to sum the values of one pixel in 3824 two images (which have to be the same size). For example in the 3825 commands below (which are identical, see paragraph after the 3826 commands), each pixel of ‘sum.fits’ is the sum of the same pixel’s 3827 values in ‘a.fits’ and ‘b.fits’. 3828 $ astarithmetic a.fits b.fits + -h1 -h1 --output=sum.fits 3829 $ astarithmetic a.fits b.fits + -g1 --output=sum.fits 3830 The HDU/extension has to be specified for each image with ‘-h’. 3831 However, if the HDUs are the same in all inputs, you can use ‘-g’ 3832 to only specify the HDU once 3833 3834 If you need to add more than one dataset, one way is to use this 3835 operator multiple times, for example see the two commands below 3836 that are identical in the Reverse Polish Notation (*note Reverse 3837 polish notation::): 3838 $ astarithmetic a.fits b.fits + c.fits + -osum.fits 3839 $ astarithmetic a.fits b.fits c.fits + + -osum.fits 3840 3841 However, this can get annoying/buggy if you have more than three or 3842 four images, in that case, a better way to sum data is to use the 3843 ‘sum’ operator (which also ignores blank pixels), that is discussed 3844 below. 3845 3846‘-’ 3847 Subtraction, so “‘4 5 -’” is equivalent to $4-5$. Usage of this 3848 operator is similar to ‘+’ operator, for example: 3849 $ astarithmetic 20000 image.fits - 3850 $ astarithmetic a.fits b.fits - -g1 --output=sub.fits 3851 3852‘x’ 3853 Multiplication, so “‘4 5 x’” is equivalent to $4\times5$. For 3854 example in the command below, the value of each output pixel is 5 3855 times its value in ‘image.fits’: 3856 $ astarithmetic image.fits 5 x 3857 And you can multiply the value of each pixel in two images, like 3858 this: 3859 $ astarithmetic a.fits a.fits x -g1 –output=multip.fits 3860 3861‘/’ 3862 Division, so “‘4 5 /’” is equivalent to $4/5$. Like the 3863 multiplication, for example 3864 $ astarithmetic image.fits 5 -h1 / 3865 $ astarithmetic a.fits b.fits / -g1 –output=div.fits 3866 3867‘%’ 3868 Modulo (remainder), so “‘3 2 %’” will return $1$. Note that the 3869 modulo operator only works on integer types (see *note Numeric data 3870 types::). This operator is therefore not defined for most 3871 processed astronomical astronomical images that have floating-point 3872 value. However it is useful in labeled images, for example *note 3873 Segment output::). In such cases, each pixel is the integer label 3874 of the object it is associated with hence with the example command 3875 below, we can change the labels to only be between 1 and 4 and 3876 decrease all objects on the image to 4/5th (all objects with a 3877 label that is a multiple of 5 will be set to 0). 3878 $ astarithmetic label.fits 5 1 % 3879 3880‘abs’ 3881 Absolute value of first operand, so “‘4 abs’” is equivalent to 3882 $|4|$. For example the output of the command bellow will not have 3883 any negative pixels (all negative pixels will be multiplied by $-1$ 3884 to become positive) 3885 $ astarithmetic image.fits abs 3886 3887‘pow’ 3888 First operand to the power of the second, so “‘4.3 5 pow’” is 3889 equivalent to $4.3^{5}$. For example with the command below all 3890 pixels will be squared 3891 $ astarithmetic image.fits 2 pow 3892 3893‘sqrt’ 3894 The square root of the first operand, so “‘5 sqrt’” is equivalent 3895 to $\sqrt{5}$. Since the square root is only defined for positive 3896 values, any negative-valued pixel will become NaN (blank). The 3897 output will have a floating point type, but its precision is 3898 determined from the input: if the input is a 64-bit floating point, 3899 the output will also be 64-bit. Otherwise, the output will be 3900 32-bit floating point (see *note Numeric data types:: for the 3901 respective precision). Therefore if you require 64-bit precision 3902 in estimating the square root, convert the input to 64-bit floating 3903 point first, for example with ‘5 float64 sqrt’. For example each 3904 pixel of the output of the command below will be the square root of 3905 that pixel in the input. 3906 $ astarithmetic image.fits sqrt 3907 3908 If you just want to scale an image with negative values using this 3909 operator (for better visual inspection, and the actual values don’t 3910 matter for you), you can subtract the image from its minimum value, 3911 then take its square root: 3912 3913 $ astarithmetic image.fits image.fits minvalue - sqrt -g1 3914 3915 Alternatively, to avoid reading the image into memory two times, 3916 you can use the ‘set-’ operator to read it into the variable ‘i’ 3917 and use ‘i’ two times to speed up the operation (described below): 3918 3919 $ astarithmetic image.fits set-i i i minvalue - sqrt 3920 3921‘log’ 3922 Natural logarithm of first operand, so “‘4 log’” is equivalent to 3923 $ln(4)$. Negative pixels will become NaN, and the output type is 3924 determined from the input, see the explanation under ‘sqrt’ for 3925 more on these features. For example the command below will take 3926 the natural logarithm of every pixel in the input. 3927 $ astarithmetic image.fits log --output=log.fits 3928 3929‘log10’ 3930 Base-10 logarithm of first popped operand, so “‘4 log’” is 3931 equivalent to $log_{10}(4)$. Negative pixels will become NaN, and 3932 the output type is determined from the input, see the explanation 3933 under ‘sqrt’ for more on these features. For example the command 3934 below will take the base-10 logarithm of every pixel in the input. 3935 $ astarithmetic image.fits log10 3936 3937 3938File: gnuastro.info, Node: Trigonometric and hyperbolic operators, Next: Unit conversion operators, Prev: Basic mathematical operators, Up: Arithmetic operators 3939 39406.2.2.2 Trigonometric and hyperbolic operators 3941.............................................. 3942 3943All the trigonometric and hyperbolic functions are described here. One 3944good thing with these operators is that they take inputs and outputs in 3945degrees (which we usually need as input or output), not radians (like 3946most other programs/libraries). 3947 3948‘sin’ 3949‘cos’ 3950‘tan’ 3951 Basic trigonometric functions. They take one operand, in units of 3952 degrees. 3953 3954‘asin’ 3955‘acos’ 3956‘atan’ 3957 Inverse trigonometric functions. They take one operand and the 3958 returned values are in units of degrees. 3959 3960‘atan2’ 3961 Inverse tangent (output in units of degrees) that uses the signs of 3962 the input coordinates to distinguish between the quadrants. This 3963 operator therefore needs two operands: the first popped operand is 3964 assumed to be the X axis position of the point, and the second 3965 popped operand is its Y axis coordinate. 3966 3967 For example see the commands below. To be more clear, we are using 3968 Table’s *note Column arithmetic:: which uses exactly the same 3969 internal library function as the Arithmetic program for images. We 3970 are showing the results for four points in the four quadrants of 3971 the 2D space (if you want to try running them, you don’t need to 3972 type/copy the parts after <#>). The first point (2,2) is in the 3973 first quadrant, therefore the returned angle is 45 degrees. But 3974 the second, third and fourth points are in the quadrants of the 3975 same order, and the returned angles reflect the quadrant. 3976 3977 $ echo " 2 2" | asttable -c'arith $2 $1 atan2' # --> 45 3978 $ echo " 2 -2" | asttable -c'arith $2 $1 atan2' # --> -45 3979 $ echo "-2 -2" | asttable -c'arith $2 $1 atan2' # --> -135 3980 $ echo "-2 2" | asttable -c'arith $2 $1 atan2' # --> 135 3981 3982 However, if you simply use the classic arc-tangent operator 3983 (‘atan’) for the same points, the result will only be in two 3984 quadrants as you see below: 3985 3986 $ echo " 2 2" | asttable -c'arith $2 $1 / atan' # --> 45 3987 $ echo " 2 -2" | asttable -c'arith $2 $1 / atan' # --> -45 3988 $ echo "-2 -2" | asttable -c'arith $2 $1 / atan' # --> 45 3989 $ echo "-2 2" | asttable -c'arith $2 $1 / atan' # --> -45 3990 3991‘sinh’ 3992‘cosh’ 3993‘tanh’ 3994 Hyperbolic sine, cosine, and tangent. These operators take a 3995 single operand. 3996 3997‘asinh’ 3998‘acosh’ 3999‘atanh’ 4000 Inverse Hyperbolic sine, cosine, and tangent. These operators take 4001 a single operand. 4002 4003 4004File: gnuastro.info, Node: Unit conversion operators, Next: Statistical operators, Prev: Trigonometric and hyperbolic operators, Up: Arithmetic operators 4005 40066.2.2.3 Unit conversion operators 4007................................. 4008 4009It often happens that you have data in one unit (for example magnitudes 4010to measure the brightness of a galaxy), but would like to convert it 4011into another (for example electron counts on your CCD). While the 4012equations for the unit conversions can be easily found on the internet, 4013the operators in this section are designed to simplify the process and 4014let you do it easily. 4015 4016‘counts-to-mag’ 4017 Convert counts (usually CCD outputs) to magnitudes using the given 4018 zeropoint. The zero point is the first popped operand and the 4019 count image or value is the second popped operand. 4020 4021 For example assume you have measured the standard deviation of the 4022 noise in an image to be ‘0.1’ counts, and the image’s zero point is 4023 ‘22.5’ and you want to measure the _per-pixel_ surface brightness 4024 limit of the dataset(1). To apply this operator on an image, 4025 simply replace ‘0.1’ with the image name, as described below. 4026 4027 $ astarithmetic 0.1 22.5 counts-to-mag --quiet 4028 4029 Of course, you can also convert every pixel in an image (or table 4030 column in Table’s *note Column arithmetic::) with this operator if 4031 you replace the second popped operand with an image/column name. 4032 For an example of applying this operator on an image, see the 4033 description of surface brightness in *note Brightness flux 4034 magnitude::, where we’ll convert an image’s pixel values to surface 4035 brightness. 4036 4037‘mag-to-counts’ 4038 Convert magnitudes to counts (usually CCD outputs) using the given 4039 zeropoint. The zero point is the first popped operand and the 4040 magnitude value is the second. For example if an object has a 4041 magnitude of 20, you can estimate the counts corresponding to it 4042 (when the image has a zeropoint of 24.8) with this command: Note 4043 that because the output is a single number, we are using ‘--quiet’ 4044 to avoid printing extra information. 4045 4046 $ astarithmetic 20 24.8 mag-to-counts --quiet 4047 4048‘counts-to-jy’ 4049 Convert counts (usually CCD outputs) to Janskys through an 4050 AB-magnitude based zeropoint. The top-popped operand is assumed to 4051 be the AB-magnitude zero point and the second-popped operand is 4052 assumed to be a dataset in units of counts (an image in Arithmetic, 4053 and a column in Table’s *note Column arithmetic::). For the full 4054 equation and basic definitions, see *note Brightness flux 4055 magnitude::. 4056 4057 For example SDSS images are calibrated in units of nanomaggies, 4058 with a fixed zero point magnitude of 22.5. Therefore you can 4059 convert the units of SDSS image pixels to Janskys with the command 4060 below: 4061 4062 $ astarithmetic sdss-image.fits 22.5 counts-to-jy 4063 4064‘au-to-pc’ 4065 Convert Astronomical Units (AUs) to Parsecs (PCs). This operator 4066 takes a single argument which is interpreted to be the input AUs. 4067 The conversion is based on the definition of Parsecs: $1 \rm{PC} = 4068 1/tan(1^{\prime\prime}) \rm{AU}$, where $1^{\prime\prime}$ is one 4069 arcseconds. In other words, $1 (\rm{PC}) = 648000/\pi (\rm{AU})$. 4070 For example, if we take Pluto’s average distance to the Sun to be 4071 40 AUs, we can obtain its distance in Parsecs using this command: 4072 4073 echo 40 | asttable -c'arith $1 au-to-pc' 4074 4075‘pc-to-au’ 4076 Convert Parsecs (PCs) to Astronomical Units (AUs). This operator 4077 takes a single argument which is interpreted to be the input PCs. 4078 For more on the conversion equation, see description of ‘au-to-pc’. 4079 For example, Proxima Centauri (the nearest star to the Solar 4080 system) is 1.3020 Parsecs from the Sun, we can calculate this 4081 distance in units of AUs with the command below: 4082 4083 echo 1.3020 | asttable -c'arith $1 pc-to-au' 4084 4085‘ly-to-pc’ 4086 Convert Light-years (LY) to Parsecs (PCs). This operator takes a 4087 single argument which is interpreted to be the input LYs. The 4088 conversion is done from IAU’s definition of the light-year 4089 (9460730472580800 m $\approx$ 63241.077 AU = 0.306601 PC, for the 4090 conversion of AU to PC, see the description of ‘au-to-pc’). 4091 4092 For example the distance of Andromeda galaxy to our galaxy is 2.5 4093 million light-years, so its distance in kilo-Parsecs can be 4094 calculated with the command below (note that we want the output in 4095 kilo-parsecs, so we are dividing the output of this operator by 4096 1000): 4097 4098 echo 2.5e6 | asttable -c'arith $1 ly-to-pc 1000 /' 4099 4100‘pc-to-ly’ 4101 Convert Parsecs (PCs) to Light-years (LY). This operator takes a 4102 single argument which is interpreted to be the input PCs. For the 4103 conversion and an example of the inverse of this operator, see the 4104 description of ‘ly-to-pc’. 4105 4106‘ly-to-au’ 4107 Convert Light-years (LY) to Astronomical Units (AUs). This 4108 operator takes a single argument which is interpreted to be the 4109 input LYs. For the conversion and a similar example, see the 4110 description of ‘ly-to-pc’. 4111 4112‘au-to-ly’ 4113 Convert Astronomical Units (AUs) to Light-years (LY). This operator 4114 takes a single argument which is interpreted to be the input AUs. 4115 For the conversion and a similar example, see the description of 4116 ‘ly-to-pc’. 4117 4118 ---------- Footnotes ---------- 4119 4120 (1) The _per-pixel_ surface brightness limit is the magnitude of the 4121noise standard deviation. For more on surface brightness see *note 4122Brightness flux magnitude::. In the example command, because the output 4123is a single number, we are using ‘--quiet’ to avoid printing extra 4124information. 4125 4126 4127File: gnuastro.info, Node: Statistical operators, Next: Stacking operators, Prev: Unit conversion operators, Up: Arithmetic operators 4128 41296.2.2.4 Statistical operators 4130............................. 4131 4132The operators in this section take a single dataset as input, and will 4133return the desired statistic as a single value. 4134 4135‘minvalue’ 4136 Minimum value in the first popped operand, so “‘a.fits minvalue’” 4137 will push the minimum pixel value in this image onto the stack. 4138 When this operator acts on a single image, the output (operand that 4139 is put back on the stack) will no longer be an image, but a number. 4140 The output of this operand is in the same type as the input. This 4141 operator is mainly intended for multi-element datasets (for example 4142 images or data cubes), if the popped operand is a number, it will 4143 just return it without any change. 4144 4145 Note that when the final remaining/output operand is a single 4146 number, it is printed onto the standard output. For example with 4147 the command below the minimum pixel value in ‘image.fits’ will be 4148 printed in the terminal: 4149 $ astarithmetic image.fits minvalue 4150 4151 However, the output above also includes a lot of extra information 4152 that are not relevant in this context. If you just want the final 4153 number, run Arithmetic in quiet mode: 4154 $ astarithmetic image.fits minvalue -q 4155 4156 Also see the description of ‘sqrt’ for other example usages of this 4157 operator. 4158 4159‘maxvalue’ 4160 Maximum value of first operand in the same type, similar to 4161 ‘minvalue’, see the description there for more. For example 4162 $ astarithmetic image.fits maxvalue -q 4163 4164‘numbervalue’ 4165 Number of non-blank elements in first operand in the ‘uint64’ type 4166 (since it is always a positive integer, see *note Numeric data 4167 types::). Its usage is similar to ‘minvalue’, for example 4168 $ astarithmetic image.fits numbervalue -q 4169 4170‘sumvalue’ 4171 Sum of non-blank elements in first operand in the ‘float32’ type. 4172 Its usage is similar to ‘minvalue’, for example 4173 $ astarithmetic image.fits sumvalue -q 4174 4175‘meanvalue’ 4176 Mean value of non-blank elements in first operand in the ‘float32’ 4177 type. Its usage is similar to ‘minvalue’, for example 4178 $ astarithmetic image.fits meanvalue -q 4179 4180‘stdvalue’ 4181 Standard deviation of non-blank elements in first operand in the 4182 ‘float32’ type. Its usage is similar to ‘minvalue’, for example 4183 $ astarithmetic image.fits stdvalue -q 4184 4185‘medianvalue’ 4186 Median of non-blank elements in first operand with the same type. 4187 Its usage is similar to ‘minvalue’, for example 4188 $ astarithmetic image.fits medianvalue -q 4189 4190‘unique’ 4191 Remove all duplicate (and blank) elements from the first popped 4192 operand. The unique elements of the dataset will be stored in a 4193 single-dimensional dataset. 4194 4195 Recall that by default, single-dimensional datasets are stored as a 4196 table column in the output. But you can use ‘--onedasimage’ or 4197 ‘--onedonstdout’ to respectively store them as a single-dimensional 4198 FITS array/image, or to print them on the standard output. 4199 4200 Although you can use this operator on the floating point dataset, 4201 due to floating-point errors it may give non-reasonable values: 4202 because the tenth digit of the decimal point is also considered 4203 although it may be statistically meaningless, see *note Numeric 4204 data types::. It is therefore better/recommended to use it on the 4205 integer dataset like the labeled images of *note Segment output:: 4206 where each pixel has the integer label of the object/clump it is 4207 associated with. For example let’s assume you have cropped a 4208 region of a larger labeled image and want to find the 4209 labels/objects that are within the crop. With this operator, this 4210 job is trivial: 4211 $ astarithmetic seg-crop.fits unique 4212 4213‘size’ 4214 Size of the dataset along a given FITS (or FORTRAN) dimension 4215 (counting from 1). The desired dimension should be the first 4216 popped operand and the dataset must be the second popped operand. 4217 The output will be a single unsigned integer (dimensions cannot be 4218 negative). For example, the following command will produce the 4219 size of the first extension/HDU (the default HDU) of ‘a.fits’ along 4220 the second FITS axis. 4221 4222 $ astarithmetic a.fits 2 size 4223 4224 4225File: gnuastro.info, Node: Stacking operators, Next: Filtering operators, Prev: Statistical operators, Up: Arithmetic operators 4226 42276.2.2.5 Stacking operators 4228.......................... 4229 4230The operators in this section are used when you have multiple datasets 4231that you would like to merge into one, commonly known as “stacking” or 4232“coaddition”. For example, you have taken ten exposures of your 4233scientific target, and you would like to combine them all into one deep 4234stacked image that is deeper. 4235 4236 When calling these operators you should determine how many operands 4237they should take in (unlike the rest of the operators that have a fixed 4238number of input operands). As described in the first operand below, you 4239do this through their first popped operand (which should be a single 4240integer number that is larger than one). 4241 4242‘min’ 4243 For each pixel, find the minimum value in all given datasets. The 4244 output will have the same type as the input. 4245 4246 The first popped operand to this operator must be a positive 4247 integer number which specifies how many further operands should be 4248 popped from the stack. All the subsequently popped operands must 4249 have the same type and size. This operator (and all the 4250 variable-operand operators similar to it that are discussed below) 4251 will work in multi-threaded mode unless Arithmetic is called with 4252 the ‘--numthreads=1’ option, see *note Multi-threaded operations::. 4253 4254 Each pixel of the output of the ‘min’ operator will be given the 4255 minimum value of the same pixel from all the popped 4256 operands/images. For example the following command will produce an 4257 image with the same size and type as the three inputs, but each 4258 output pixel value will be the minimum of the same pixel’s values 4259 in all three input images. 4260 4261 $ astarithmetic a.fits b.fits c.fits 3 min 4262 4263 Important notes: 4264 4265 • NaN/blank pixels will be ignored, see *note Blank pixels::. 4266 4267 • The output will have the same type as the inputs. This is 4268 natural for the ‘min’ and ‘max’ operators, but for other 4269 similar operators (for example ‘sum’, or ‘average’) the 4270 per-pixel operations will be done in double precision floating 4271 point and then stored back in the input type. Therefore, if 4272 the input was an integer, C’s internal type conversion will be 4273 used. 4274 4275 • The operation will be multi-threaded, greatly speeding up the 4276 process if you have large and numerous data to stack. You can 4277 disable multi-threaded operations with the ‘--numthreads=1’ 4278 option (see *note Multi-threaded operations::). 4279 4280‘max’ 4281 For each pixel, find the maximum value in all given datasets. The 4282 output will have the same type as the input. This operator is 4283 called similar to the ‘min’ operator, please see there for more. 4284 For example 4285 $ astarithmetic a.fits b.fits c.fits 3 min -omax.fits 4286 4287‘number’ 4288 For each pixel count the number of non-blank pixels in all given 4289 datasets. The output will be an unsigned 32-bit integer datatype 4290 (see *note Numeric data types::). This operator is called similar 4291 to the ‘min’ operator, please see there for more. For example 4292 $ astarithmetic a.fits b.fits c.fits 3 number -onum.fits 4293 4294 Some datasets may have blank values (which are also ignored in all 4295 similar operators like ‘min’, ‘sum’, ‘mean’ or ‘median’). Hence, 4296 the final pixel values of this operator will not, in general, be 4297 equal to the number of inputs. This operator is therefore mostly 4298 called in parallel with those operators to know the “weight” of 4299 each pixel (in case you want to only keep pixels that had the full 4300 exposure for example). 4301 4302‘sum’ 4303 For each pixel, calculate the sum in all given datasets. The 4304 output will have the a single-precision (32-bit) floating point 4305 type. This operator is called similar to the ‘min’ operator, 4306 please see there for more. For example 4307 $ astarithmetic a.fits b.fits c.fits 3 sum -ostack-sum.fits 4308 4309‘mean’ 4310 For each pixel, calculate the mean in all given datasets. The 4311 output will have the a single-precision (32-bit) floating point 4312 type. This operator is called similar to the ‘min’ operator, 4313 please see there for more. For example 4314 $ astarithmetic a.fits b.fits c.fits 3 mean -ocoadd-mean.fits 4315 4316‘std’ 4317 For each pixel, find the standard deviation in all given datasets. 4318 The output will have the a single-precision (32-bit) floating point 4319 type. This operator is called similar to the ‘min’ operator, 4320 please see there for more. For example 4321 $ astarithmetic a.fits b.fits c.fits 3 std -ostd.fits 4322 4323‘median’ 4324 For each pixel, find the median in all given datasets. The output 4325 will have the a single-precision (32-bit) floating point type. 4326 This operator is called similar to the ‘min’ operator, please see 4327 there for more. For example 4328 $ astarithmetic a.fits b.fits c.fits 3 mean \ 4329 --output=stack-median.fits 4330 4331‘quantile’ 4332 For each pixel, find the quantile from all given datasets. The 4333 output will have the same numeric data type and size as the input 4334 datasets. Besides the input datasets, the quantile operator also 4335 needs a single parameter (the requested quantile). The parameter 4336 should be the first popped operand, with a value between (and 4337 including) 0 and 1. The second popped operand must be the number 4338 of datasets to use. 4339 4340 In the example below, the first-popped operand (‘0.7’) is the 4341 quantile, the second-popped operand (‘3’) is the number of datasets 4342 to pop. 4343 4344 astarithmetic a.fits b.fits c.fits 3 0.7 quantile 4345 4346‘sigclip-number’ 4347 For each pixel, find the sigma-clipped number (after removing 4348 outliers) in all given datasets. The output will have the an 4349 unsigned 32-bit integer type (see *note Numeric data types::). 4350 4351 This operator will combine the specified number of inputs into a 4352 single output that contains the number of remaining elements after 4353 $\sigma$-clipping on each element/pixel (for more on 4354 $\sigma$-clipping, see *note Sigma clipping::). This operator is 4355 very similar to ‘min’, with the exception that it expects two 4356 operands (parameters for sigma-clipping) before the total number of 4357 inputs. The first popped operand is the termination criteria and 4358 the second is the multiple of $\sigma$. 4359 4360 For example in the command below, the first popped operand (‘0.2’) 4361 is the sigma clipping termination criteria. If the termination 4362 criteria is larger than, or equal to, 1 it is interpreted as the 4363 number of clips to do. But if it is between 0 and 1, then it is 4364 the tolerance level on the standard deviation (see *note Sigma 4365 clipping::). The second popped operand (‘5’) is the multiple of 4366 sigma to use in sigma-clipping. The third popped operand (‘10’) is 4367 number of datasets that will be used (similar to the first popped 4368 operand to ‘min’). 4369 4370 astarithmetic a.fits b.fits c.fits 3 5 0.2 sigclip-number 4371 4372‘sigclip-median’ 4373 For each pixel, find the sigma-clipped median in all given 4374 datasets. The output will have the a single-precision (32-bit) 4375 floating point type. This operator is called similar to the 4376 ‘sigclip-number’ operator, please see there for more. For example 4377 astarithmetic a.fits b.fits c.fits 3 5 0.2 sigclip-median 4378 4379‘sigclip-mean’ 4380 For each pixel, find the sigma-clipped mean in all given datasets. 4381 The output will have the a single-precision (32-bit) floating point 4382 type. This operator is called similar to the ‘sigclip-number’ 4383 operator, please see there for more. For example 4384 astarithmetic a.fits b.fits c.fits 3 5 0.2 sigclip-mean 4385 4386‘sigclip-std’ 4387 For each pixel, find the sigma-clipped standard deviation in all 4388 given datasets. The output will have the a single-precision 4389 (32-bit) floating point type. This operator is called similar to 4390 the ‘sigclip-number’ operator, please see there for more. For 4391 example 4392 astarithmetic a.fits b.fits c.fits 3 5 0.2 sigclip-std 4393 4394 4395File: gnuastro.info, Node: Filtering operators, Next: Interpolation operators, Prev: Stacking operators, Up: Arithmetic operators 4396 43976.2.2.6 Filtering (smoothing) operators 4398....................................... 4399 4400Image filtering is commonly used for smoothing: every pixel value in the 4401output image is created by applying a certain statistic to the pixels in 4402its vicinity. 4403 4404‘filter-mean’ 4405 Apply mean filtering (or moving average 4406 (https://en.wikipedia.org/wiki/Moving_average)) on the input 4407 dataset. During mean filtering, each pixel (data element) is 4408 replaced by the mean value of all its surrounding pixels (excluding 4409 blank values). The number of surrounding pixels in each dimension 4410 (to calculate the mean) is determined through the earlier operands 4411 that have been pushed onto the stack prior to the input dataset. 4412 The number of necessary operands is determined by the dimensions of 4413 the input dataset (first popped operand). The order of the 4414 dimensions on the command-line is the order in FITS format. Here 4415 is one example: 4416 4417 $ astarithmetic 5 4 image.fits filter-mean 4418 4419 In this example, each pixel is replaced by the mean of a 5 by 4 box 4420 around it. The box is 5 pixels along the first FITS dimension 4421 (horizontal when viewed in ds9) and 4 pixels along the second FITS 4422 dimension (vertical). 4423 4424 Each pixel will be placed in the center of the box that the mean is 4425 calculated on. If the given width along a dimension is even, then 4426 the center is assumed to be between the pixels (not in the center 4427 of a pixel). When the pixel is close to the edge, the pixels of 4428 the box that fall outside the image are ignored. Therefore, on the 4429 edge, less points will be used in calculating the mean. 4430 4431 The final effect of mean filtering is to smooth the input image, it 4432 is essentially a convolution with a kernel that has identical 4433 values for all its pixels (is flat), see *note Convolution 4434 process::. 4435 4436 Note that blank pixels will also be affected by this operator: if 4437 there are any non-blank elements in the box surrounding a blank 4438 pixel, in the filtered image, it will have the mean of the 4439 non-blank elements, therefore it won’t be blank any more. If blank 4440 elements are important for your analysis, you can use the ‘isblank’ 4441 with the ‘where’ operator to set them back to blank after 4442 filtering. 4443 4444‘filter-median’ 4445 Apply median filtering 4446 (https://en.wikipedia.org/wiki/Median_filter) on the input dataset. 4447 This is very similar to ‘filter-mean’, except that instead of the 4448 mean value of the box pixels, the median value is used to replace a 4449 pixel value. For more on how to use this operator, please see 4450 ‘filter-mean’. 4451 4452 The median is less susceptible to outliers compared to the mean. 4453 As a result, after median filtering, the pixel values will be more 4454 discontinuous than mean filtering. 4455 4456‘filter-sigclip-mean’ 4457 Apply a $\sigma$-clipped mean filtering onto the input dataset. 4458 This is very similar to ‘filter-mean’, except that all outliers 4459 (identified by the $\sigma$-clipping algorithm) have been removed, 4460 see *note Sigma clipping:: for more on the basics of this 4461 algorithm. As described there, two extra input parameters are 4462 necessary for $\sigma$-clipping: the multiple of $\sigma$ and the 4463 termination criteria. ‘filter-sigclip-mean’ therefore needs to pop 4464 two other operands from the stack after the dimensions of the box. 4465 4466 For example the line below uses the same box size as the example of 4467 ‘filter-mean’. However, all elements in the box that are 4468 iteratively beyond $3\sigma$ of the distribution’s median are 4469 removed from the final calculation of the mean until the change in 4470 $\sigma$ is less than $0.2$. 4471 4472 $ astarithmetic 3 0.2 5 4 image.fits filter-sigclip-mean 4473 4474 The median (which needs a sorted dataset) is necessary for 4475 $\sigma$-clipping, therefore ‘filter-sigclip-mean’ can be 4476 significantly slower than ‘filter-mean’. However, if there are 4477 strong outliers in the dataset that you want to ignore (for example 4478 emission lines on a spectrum when finding the continuum), this is a 4479 much better solution. 4480 4481‘filter-sigclip-median’ 4482 Apply a $\sigma$-clipped median filtering onto the input dataset. 4483 This operator and its necessary operands are almost identical to 4484 ‘filter-sigclip-mean’, except that after $\sigma$-clipping, the 4485 median value (which is less affected by outliers than the mean) is 4486 added back to the stack. 4487 4488 4489File: gnuastro.info, Node: Interpolation operators, Next: Dimensionality changing operators, Prev: Filtering operators, Up: Arithmetic operators 4490 44916.2.2.7 Interpolation operators 4492............................... 4493 4494Interpolation is the process of removing blank pixels from a dataset (by 4495giving them a value based on the non-blank neighbors). 4496 4497‘interpolate-medianngb’ 4498 Interpolate the blank elements of the second popped operand with 4499 the median of nearest non-blank neighbors to each. The number of 4500 the nearest non-blank neighbors used to calculate the median is 4501 given by the first popped operand. 4502 4503 The distance of the nearest non-blank neighbors is irrelevant in 4504 this interpolation. The neighbors of each blank pixel will be 4505 parsed in expanding circular rings (for 2D images) or spherical 4506 surfaces (for 3D cube) and each non-blank element over them is 4507 stored in memory. When the requested number of non-blank neighbors 4508 have been found, their median is used to replace that blank 4509 element. For example the line below replaces each blank element 4510 with the the median of the nearest 5 pixels. 4511 4512 $ astarithmetic image.fits 5 interpolate-medianngb 4513 4514 When you want to interpolate blank regions and you want each blank 4515 region to have a fixed value (for example the centers of saturated 4516 stars) this operator is not good. Because the pixels used to 4517 interpolate various parts of the region differ. For such 4518 scenarios, you may use ‘interpolate-maxofregion’ or 4519 ‘interpolate-inofregion’ (described below). 4520 4521‘interpolate-minngb’ 4522 Similar to ‘interpolate-medianngb’, but will fill the blank values 4523 of the dataset with the minimum value of the nearest neighbors. 4524 4525‘interpolate-maxngb’ 4526 Similar to ‘interpolate-medianngb’, but will fill the blank values 4527 of the dataset with the maximum value of the nearest neighbors. 4528 One useful implementation of this operator is to fill the saturated 4529 pixels of stars in images. 4530 4531‘interpolate-minofregion’ 4532 Interpolate all blank regions (consisting of many blank pixels that 4533 are touching) in the second popped operand with the minimum value 4534 of the pixels that are immediately bordering that region (a single 4535 value). The first popped operand is the connectivity (see 4536 description in ‘connected-components’). 4537 4538 For example with the command below all the connected blank regions 4539 of ‘image.fits’ will be filled. Its an image (2D dataset), so a 2 4540 connectivity means that the independent blank regions are defined 4541 by 8-connected neighbors. If connectivity was 1, the regions would 4542 be defined by 4-connectivity: blank regions that may only be 4543 touching on the corner of one pixel would be identified as separate 4544 regions. 4545 4546 $ astarithmetic image.fits 2 interpolate-minofregion 4547 4548‘interpolate-maxofregion’ 4549 Similar to ‘interpolate-minofregion’, but the maximum is used to 4550 fill the blank regions. 4551 4552 This operator can be useful in filling saturated pixels in stars 4553 for example. Recall that the ‘interpolate-maxngb’ operator looks 4554 for the maximum value with a given number of neighboring pixels and 4555 is more useful in small noisy regions. Therefore as the blank 4556 regions become larger, ‘interpolate-maxngb’ can cause a 4557 fragmentation in the connected blank region because the nearest 4558 neighbor to one part of the blank region, may not fall within the 4559 pixels searched for the other regions. With this option, the size 4560 of the blank region is irrelevant: all the pixels bordering the 4561 blank region are parsed and their maximum value is used for the 4562 whole region. 4563 4564 4565File: gnuastro.info, Node: Dimensionality changing operators, Next: Conditional operators, Prev: Interpolation operators, Up: Arithmetic operators 4566 45676.2.2.8 Dimensionality changing operators 4568......................................... 4569 4570Through these operators you can change the dimensions of the output 4571through certain statistics on the dimensions that should be removed. 4572For example, let’s assume you have a 3D data cube that has 300 by 300 4573pixels in the RA and Dec dimensions (first two dimensions), and 3600 4574slices along the wavelength (third dimension), so the whole cube is 4575$300\times300\times3600$ voxels (volume elements). To create a 4576narrow-band image that only contains 100 slices around a certain 4577wavelength, you can crop that section (using *note Crop::), giving you a 4578$300\times300\times100$ cube. You can now use the ‘collapse-sum’ 4579operator below to “collapse” all the 100 slices into one 2D image that 4580has $300\times300$ pixels. Every pixel in this 2D image will have the 4581flux of the sum of the 100 slices. 4582 4583‘collapse-sum’ 4584 Collapse the given dataset (second popped operand), by summing all 4585 elements along the first popped operand (a dimension in FITS 4586 standard: counting from one, from fastest dimension). The returned 4587 dataset has one dimension less compared to the input. 4588 4589 The output will have a double-precision floating point type 4590 irrespective of the input dataset’s type. Doing the operation in 4591 double-precision (64-bit) floating point will help the collapse 4592 (summation) be affected less by floating point errors. But 4593 afterwards, single-precision floating points are usually enough in 4594 real (noisy) datasets. So depending on the type of the input and 4595 its nature, it is recommended to use one of the type conversion 4596 operators on the returned dataset. 4597 4598 If any WCS is present, the returned dataset will also lack the 4599 respective dimension in its WCS matrix. Therefore, when the WCS is 4600 important for later processing, be sure that the input is aligned 4601 with the respective axes: all non-diagonal elements in the WCS 4602 matrix are zero. 4603 4604 One common application of this operator is the creation of pseudo 4605 broad-band or narrow-band 2D images from 3D data cubes. For 4606 example integral field unit (IFU) data products that have two 4607 spatial dimensions (first two FITS dimensions) and one spectral 4608 dimension (third FITS dimension). The command below will collapse 4609 the whole third dimension into a 2D array the size of the first two 4610 dimensions, and then convert the output to single-precision 4611 floating point (as discussed above). 4612 4613 $ astarithmetic cube.fits 3 collapse-sum float32 4614 4615‘collapse-mean’ 4616 Similar to ‘collapse-sum’, but the returned dataset will be the 4617 mean value along the collapsed dimension, not the sum. 4618 4619‘collapse-number’ 4620 Similar to ‘collapse-sum’, but the returned dataset will be the 4621 number of non-blank values along the collapsed dimension. The 4622 output will have a 32-bit signed integer type. If the input 4623 dataset doesn’t have blank values, all the elements in the returned 4624 dataset will have a single value (the length of the collapsed 4625 dimension). Therefore this is mostly relevant when there are blank 4626 values in the dataset. 4627 4628‘collapse-min’ 4629 Similar to ‘collapse-sum’, but the returned dataset will have the 4630 same numeric type as the input and will contain the minimum value 4631 for each pixel along the collapsed dimension. 4632 4633‘collapse-max’ 4634 Similar to ‘collapse-sum’, but the returned dataset will have the 4635 same numeric type as the input and will contain the maximum value 4636 for each pixel along the collapsed dimension. 4637 4638‘add-dimension’ 4639 Build a higher-dimensional dataset from all the input datasets 4640 stacked after one another (along the slowest dimension). The first 4641 popped operand has to be a single number. It is used by the 4642 operator to know how many operands it should pop from the stack 4643 (and the size of the output in the new dimension). The rest of the 4644 operands must have the same size and numerical data type. This 4645 operator currently only works for 2D input operands, please contact 4646 us if you want inputs to have different dimensions. 4647 4648 The output’s WCS (which should have a different dimensionality 4649 compared to the inputs) can be read from another file with the 4650 ‘--wcsfile’ option. If no file is specified for the WCS, the first 4651 dataset’s WCS will be used, you can later add/change the necessary 4652 WCS keywords with the FITS keyword modification features of the 4653 Fits program (see *note Fits::). 4654 4655 If your datasets don’t have the same type, you can use the type 4656 transformation operators of Arithmetic that are discussed below. 4657 Just beware of overflow if you are transforming to a smaller type, 4658 see *note Numeric data types::. 4659 4660 For example if you want to put the three ‘img1.fits’, ‘img2.fits’ 4661 and ‘img3.fits’ images (each a 2D dataset) into one 3D datacube, 4662 you can use this command: 4663 4664 $ astarithmetic img1.fits img2.fits img3.fits 3 add-dimension 4665 4666 4667File: gnuastro.info, Node: Conditional operators, Next: Mathematical morphology operators, Prev: Dimensionality changing operators, Up: Arithmetic operators 4668 46696.2.2.9 Conditional operators 4670............................. 4671 4672Conditional operators take two inputs and return a binary output that 4673can only have two values 0 (for pixels where the condition was false) or 46741 (for the pixels where the condition was true). Because of the binary 4675(2-valued) nature of their outputs, the output is therefore stored in an 4676‘unsigned char’ data type (see *note Numeric data types::) to speed up 4677process and take less space in your storage. There are two exceptions 4678to the general features above: ‘isblank’ only takes one input, and 4679‘where’ takes three, while not returning a binary output, see their 4680description for more. 4681 4682‘lt’ 4683 Less than: creates a binary output (values either 0 or 1) where 4684 each pixel will be 1 if the second popped operand is smaller than 4685 the first popped operand and 0 otherwise. If both operands are 4686 images, then all the pixels will be compared with their 4687 counterparts in the other image. 4688 4689 For example, the pixels in the output of the command below will 4690 have a value of 1 (true) if their value in ‘image1.fits’ is less 4691 than their value in ‘image2.fits’. Otherwise, their value will be 4692 0 (false). 4693 $ astarithmetic image1.fits image2.fits lt 4694 If only one operand is an image, then all the pixels will be 4695 compared with the single value (number) of the other operand. For 4696 example: 4697 $ astaithmetic image1.fits 1000 lt 4698 Finally if both are numbers, then the output is also just one 4699 number (0 or 1). 4700 $ astarithmetic 4 5 lt 4701 4702‘le’ 4703 Less or equal: similar to ‘lt’ (‘less than’ operator), but 4704 returning 1 when the second popped operand is smaller or equal to 4705 the first. For example 4706 $ astaithmetic image1.fits 1000 le 4707 4708‘gt’ 4709 Greater than: similar to ‘lt’ (‘less than’ operator), but returning 4710 1 when the second popped operand is greater than the first. For 4711 example 4712 $ astaithmetic image1.fits 1000 gt 4713 4714‘ge’ 4715 Greater or equal: similar to ‘lt’ (‘less than’ operator), but 4716 returning 1 when the second popped operand is larger or equal to 4717 the first. For example 4718 $ astaithmetic image1.fits 1000 ge 4719 4720‘eq’ 4721 Equality: similar to ‘lt’ (‘less than’ operator), but returning 1 4722 when the two popped operands are equal (to double precision 4723 floating point accuracy). 4724 $ astaithmetic image1.fits 1000 eq 4725 4726‘ne’ 4727 Non-Equality: similar to ‘lt’ (‘less than’ operator), but returning 4728 1 when the two popped operands are _not_ equal (to double precision 4729 floating point accuracy). 4730 $ astaithmetic image1.fits 1000 ne 4731 4732‘and’ 4733 Logical AND: returns 1 if both operands have a non-zero value and 0 4734 if both are zero. Both operands have to be the same kind: either 4735 both images or both numbers and it mostly makes meaningful values 4736 when the inputs are binary (with pixel values of 0 or 1). 4737 $ astarithmetic image1.fits image2.fits -g1 and 4738 4739 For example if you only want to see which pixels in an image have a 4740 value _between_ 50 (greater equal, or inclusive) and 200 (less 4741 than, or exclusive), you can use this command: 4742 $ astarithmetic image.fits set-i i 50 ge i 200 lt and 4743 4744‘or’ 4745 Logical OR: returns 1 if either one of the operands is non-zero and 4746 0 only when both operators are zero. Both operands have to be the 4747 same kind: either both images or both numbers. The usage is 4748 similar to ‘and’. 4749 4750 For example if you only want to see which pixels in an image have a 4751 value _outside of_ -100 (greater equal, or inclusive) and 200 (less 4752 than, or exclusive), you can use this command: 4753 $ astarithmetic image.fits set-i i -100 lt i 200 ge or 4754 4755‘not’ 4756 Logical NOT: returns 1 when the operand is 0 and 0 when the operand 4757 is non-zero. The operand can be an image or number, for an image, 4758 it is applied to each pixel separately. For example if you want to 4759 know which pixels are not blank, you can use not on the output of 4760 the ‘isblank’ operator described below: 4761 $ astarithmetic image.fits isblank not 4762 4763‘isblank’ 4764 Test for a blank value (see *note Blank pixels::). In essence, 4765 this is very similar to the conditional operators: the output is 4766 either 1 or 0 (see the ‘less than’ operator above). The difference 4767 is that it only needs one operand. For example: 4768 $ astarithmetic image.fits isblank 4769 Because of the definition of a blank pixel, a blank value is not 4770 even equal to itself, so you cannot use the equal operator above to 4771 select blank pixels. See the “Blank pixels” box below for more on 4772 Blank pixels in Arithmetic. 4773 4774‘where’ 4775 Change the input (pixel) value _where_/if a certain condition 4776 holds. The conditional operators above can be used to define the 4777 condition. Three operands are required for ‘where’. The input 4778 format is demonstrated in this simplified example: 4779 4780 $ astarithmetic modify.fits binary.fits if-true.fits where 4781 4782 The value of any pixel in ‘modify.fits’ that corresponds to a 4783 non-zero _and_ non-blank pixel of ‘binary.fits’ will be changed to 4784 the value of the same pixel in ‘if-true.fits’ (this may also be a 4785 number). The 3rd and 2nd popped operands (‘modify.fits’ and 4786 ‘binary.fits’ respectively, see *note Reverse polish notation::) 4787 have to have the same dimensions/size. ‘if-true.fits’ can be 4788 either a number, or have the same dimension/size as the other two. 4789 4790 The 2nd popped operand (‘binary.fits’) has to have ‘uint8’ (or 4791 ‘unsigned char’ in standard C) type (see *note Numeric data 4792 types::). It is treated as a binary dataset (with only two values: 4793 zero and non-zero, hence the name ‘binary.fits’ in this example). 4794 However, commonly you won’t be dealing with an actual FITS file of 4795 a condition/binary image. You will probably define the condition 4796 in the same run based on some other reference image and use the 4797 conditional and logical operators above to make a true/false (or 4798 one/zero) image for you internally. For example the case below: 4799 4800 $ astarithmetic in.fits reference.fits 100 gt new.fits where 4801 4802 In the example above, any of the ‘in.fits’ pixels that has a value 4803 in ‘reference.fits’ greater than ‘100’, will be replaced with the 4804 corresponding pixel in ‘new.fits’. Effectively the ‘reference.fits 4805 100 gt’ part created the condition/binary image which was added to 4806 the stack (in memory) and later used by ‘where’. The command above 4807 is thus equivalent to these two commands: 4808 4809 $ astarithmetic reference.fits 100 gt --output=binary.fits 4810 $ astarithmetic in.fits binary.fits new.fits where 4811 4812 Finally, the input operands are read and used independently, so you 4813 can use the same file more than once as any of the operands. 4814 4815 When the 1st popped operand to ‘where’ (‘if-true.fits’) is a single 4816 number, it may be a NaN value (or any blank value, depending on its 4817 type) like the example below (see *note Blank pixels::). When the 4818 number is blank, it will be converted to the blank value of the 4819 type of the 3rd popped operand (‘in.fits’). Hence, in the example 4820 below, all the pixels in ‘reference.fits’ that have a value greater 4821 than 100, will become blank in the natural data type of ‘in.fits’ 4822 (even though NaN values are only defined for floating point types). 4823 4824 $ astarithmetic in.fits reference.fits 100 gt nan where 4825 4826 4827File: gnuastro.info, Node: Mathematical morphology operators, Next: Bitwise operators, Prev: Conditional operators, Up: Arithmetic operators 4828 48296.2.2.10 Mathematical morphology operators 4830.......................................... 4831 4832From Wikipedia: “Mathematical morphology (MM) is a theory and technique 4833for the analysis and processing of geometrical structures, based on set 4834theory, lattice theory, topology, and random functions. MM is most 4835commonly applied to digital images”. In theory it extends a very large 4836body of research and methods in image processing, but currently in 4837Gnuastro it mainly applies to images that are binary (only have a value 4838of 0 or 1). For example you have applied the greater-than operator 4839(‘gt’, see *note Conditional operators::) to select all pixels in your 4840image that are larger than a value of 100. But they will all have a 4841value of 1, and you want to separate the various groups of pixels that 4842are connected (for example peaks of stars in your image). With the 4843‘connected-components’ operator, you can give each connected region of 4844the output of ‘gt’ a separate integer label. 4845 4846‘erode’ 4847 Erode the foreground pixels (with value ‘1’) of the input dataset 4848 (second popped operand). The first popped operand is the 4849 connectivity (see description in ‘connected-components’). Erosion 4850 is simply a flipping of all foreground pixels (with value ‘1’) to 4851 background (with value ‘0’) that are “touching” background pixels. 4852 “Touching” is defined by the connectivity. 4853 4854 In effect, this operator “carves off” the outer borders of the 4855 foreground, making them thinner. This operator assumes a binary 4856 dataset (all pixels are ‘0’ or ‘1’). For example, imagine that you 4857 have an astronomical image with a mean/sky value of 0 units and a 4858 standard deviation ($\sigma$) of 100 units and many galaxies in it. 4859 With the first command below, you can apply a threshold of 4860 $2\sigma$ on the image (by only keeping pixels that are greater 4861 than 200 using the ‘gt’ operator). The output of thresholding the 4862 image is a binary image (each pixel is either smaller or equal to 4863 the threshold or larger than it). You can then erode the binary 4864 image with the second command below to remove very small false 4865 positives (one or two pixel peaks). 4866 $ astarithmetic image.fits 100 gt -obinary.fits 4867 $ astarithmetic binary.fits 2 erode -oout.fits 4868 4869 In fact, you can merge these operations into one command thanks to 4870 the reverse polish notation (see *note Reverse polish notation::): 4871 $ astarithmetic image.fits 100 gt 2 erode -oout.fits 4872 4873 To see the effect of connectivity, try this: 4874 $ astarithmetic image.fits 100 gt 1 erode -oout-con-1.fits 4875 4876‘dilate’ 4877 Dilate the foreground pixels (with value ‘1’) of the binary input 4878 dataset (second popped operand). The first popped operand is the 4879 connectivity (see description in ‘connected-components’). Dilation 4880 is simply a flipping of all background pixels (with value ‘0’) to 4881 foreground (with value ‘1’) that are “touching” foreground pixels. 4882 “Touching” is defined by the connectivity. In effect, this expands 4883 the outer borders of the foreground. This operator assumes a 4884 binary dataset (all pixels are ‘0’ and ‘1’). The usage is similar 4885 to ‘erode’, for example: 4886 $ astarithmetic binary.fits 2 dilate -oout.fits 4887 4888‘connected-components’ 4889 Find the connected components in the input dataset (second popped 4890 operand). The first popped is the connectivity used in the 4891 connected components algorithm. The second popped operand is the 4892 dataset where connected components are to be found. It is assumed 4893 to be a binary image (with values of 0 or 1). It must have an 4894 8-bit unsigned integer type which is the format produced by 4895 conditional operators. This operator will return a labeled dataset 4896 where the non-zero pixels in the input will be labeled with a 4897 counter (starting from 1). 4898 4899 The connectivity is a number between 1 and the number of dimensions 4900 in the dataset (inclusive). 1 corresponds to the weakest 4901 (symmetric) connectivity between elements and the number of 4902 dimensions the strongest. For example on a 2D image, a 4903 connectivity of 1 corresponds to 4-connected neighbors and 2 4904 corresponds to 8-connected neighbors. 4905 4906 One example usage of this operator can be the identification of 4907 regions above a certain threshold, as in the command below. With 4908 this command, Arithmetic will first separate all pixels greater 4909 than 100 into a binary image (where pixels with a value of 1 are 4910 above that value). Afterwards, it will label all those that are 4911 connected. 4912 4913 $ astarithmetic in.fits 100 gt 2 connected-components 4914 4915 If your input dataset doesn’t have a binary type, but you know all 4916 its values are 0 or 1, you can use the ‘uint8’ operator (below) to 4917 convert it to binary. 4918 4919‘fill-holes’ 4920 Flip background (0) pixels surrounded by foreground (1) in a binary 4921 dataset. This operator takes two operands (similar to 4922 ‘connected-components’): the second is the binary (0 or 1 valued) 4923 dataset to fill holes in and the first popped operand is the 4924 connectivity (to define a hole). Imagine that in your dataset 4925 there are some holes with zero value inside the objects with one 4926 value (for example the output of the thresholding example of 4927 ‘erode’) and you want to fill the holes: 4928 $ astarithmetic binary.fits 2 fill-holes 4929 4930‘invert’ 4931 Invert an unsigned integer dataset (won’t work on other data types, 4932 see *note Numeric data types::). This is the only operator that 4933 ignores blank values (which are set to be the maximum values in the 4934 unsigned integer types). 4935 4936 This is useful in cases where the target(s) has(have) been imaged 4937 in absorption as raw formats (which are unsigned integer types). 4938 With this option, the maximum value for the given type will be 4939 subtracted from each pixel value, thus “inverting” the image, so 4940 the target(s) can be treated as emission. This can be useful when 4941 the higher-level analysis methods/tools only work on emission 4942 (positive skew in the noise, not negative). 4943 $ astarithmetic image.fits invert 4944 4945 4946File: gnuastro.info, Node: Bitwise operators, Next: Numerical type conversion operators, Prev: Mathematical morphology operators, Up: Arithmetic operators 4947 49486.2.2.11 Bitwise operators 4949.......................... 4950 4951Astronomical images are usually stored as an array multi-byte pixels 4952with different sizes for different precision levels (see *note Numeric 4953data types::). For example images from CCDs are usually in the unsigned 495416-bit integer type (each pixel takes 16 bits, or 2 bytes, of memory) 4955and fully reduced deep images have a 32-bit floating point type (each 4956pixel takes 32 bits or 4 bytes). 4957 4958 On the other hand, during the data reduction, we need to preserve a 4959lot of meta-data about some pixels. For example, if a cosmic ray had 4960hit the pixel during the exposure, or if the pixel was saturated, or is 4961known to have a problem, or if the optical vignetting is too strong on 4962it, and etc. A crude solution is to make a new image when checking for 4963each one of these things and make a binary image where we flag (set to 49641) pixels that satisfy any of these conditions above, and set the rest 4965to zero. However, processing pipelines sometimes need more than 20 4966flags to store important per-pixel meta-data, and recall that the 4967smallest numeric data type is one byte (or 8 bits, that can store up to 4968256 different values), while we only need two values for each flag! 4969This is a major waste of storage space! 4970 4971 A much more optimal solution is to use the bits within each pixel to 4972store different flags! In other words, if you have an 8-bit pixel, use 4973each bit as as a flag to mark if a certain condition has happened on a 4974certain pixel or not. For example, let’s set the following standard 4975based on the four cases mentioned above: the first bit will show that a 4976cosmic ray has hit that pixel. So if a pixel is only affected by cosmic 4977rays, it will have this sequence of bits (note that the bit-counting 4978starts from the right): ‘00000001’. The second bit shows that the pixel 4979was saturated (‘00000010’), the third bit shows that it has known 4980problems (‘00000100’) and the fourth bit shows that it was affected by 4981vignetting (‘00001000’). 4982 4983 Since each bit is independent, we can thus mark multiple metadata 4984about that pixel in the actual image, within a single “flag” or “mask” 4985pixel of a flag or mask image that has the same number of pixels. For 4986example a flag-pixel with the following bits ‘00001001’ shows that it 4987has been affected by cosmic rays _and_ it has been affected by 4988vignetting at the same time. The common data type to store these 4989flagging pixels are unsigned integer types (see *note Numeric data 4990types::). Therefore when you open an unsigned 8-bit flag image in a 4991viewer like DS9, you will see a single integer in each pixel that 4992actually has 8 layers of metadata in it! For example the integer you 4993will see for the bit sequences given above will respectively be: $2^0=1$ 4994(for a pixel that only has cosmic ray), $2^1=2$ (for a pixel that was 4995only saturated), $2^2=4$ (for a pixel that only has known problems), 4996$2^3=8$ (for a pixel that is only affected by vignetting) and $2^0 + 2^3 4997= 9$ (for a pixel that has a cosmic ray _and_ was affected by 4998vignetting). 4999 5000 You can later use this bit information to mark objects in your final 5001analysis or to mask certain pixels. For example you may want to set all 5002pixels affected by vignetting to NaN, but can interpolate over cosmic 5003rays. You therefore need ways to separate the pixels with a desired 5004flag(s) from the rest. It is possible to treat a flag pixel as a single 5005integer (and try to define certain ranges in value to select certain 5006flags). But a much more easier and robust way is to actually look at 5007each pixel as a sequence of bits (not as a single integer!) and use the 5008bitwise operators below for this job. For more on the theory behind 5009bitwise operators, see Wikipedia 5010(https://en.wikipedia.org/wiki/Bitwise_operation). 5011 5012‘bitand’ 5013 Bitwise AND operator: only bits with values of 1 in both popped 5014 operands will get the value of 1, the rest will be set to 0. For 5015 example (assuming numbers can be written as bit strings on the 5016 command-line): ‘00101000 00100010 bitand’ will give ‘00100000’. 5017 Note that the bitwise operators only work on integer type datasets. 5018 5019‘bitor’ 5020 Bitwise inclusive OR operator: The bits where at least one of the 5021 two popped operands has a 1 value get a value of 1, the others 0. 5022 For example (assuming numbers can be written as bit strings on the 5023 command-line): ‘00101000 00100010 bitand’ will give ‘00101010’. 5024 Note that the bitwise operators only work on integer type datasets. 5025 5026‘bitxor’ 5027 Bitwise exclusive OR operator: A bit will be 1 if it differs 5028 between the two popped operands. For example (assuming numbers can 5029 be written as bit strings on the command-line): ‘00101000 00100010 5030 bitand’ will give ‘00001010’. Note that the bitwise operators only 5031 work on integer type datasets. 5032 5033‘lshift’ 5034 Bitwise left shift operator: shift all the bits of the first 5035 operand to the left by a number of times given by the second 5036 operand. For example (assuming numbers can be written as bit 5037 strings on the command-line): ‘00101000 2 lshift’ will give 5038 ‘10100000’. This is equivalent to multiplication by 4. Note that 5039 the bitwise operators only work on integer type datasets. 5040 5041‘rshift’ 5042 Bitwise right shift operator: shift all the bits of the first 5043 operand to the right by a number of times given by the second 5044 operand. For example (assuming numbers can be written as bit 5045 strings on the command-line): ‘00101000 2 rshift’ will give 5046 ‘00001010’. Note that the bitwise operators only work on integer 5047 type datasets. 5048 5049‘bitnot’ 5050 Bitwise not (more formally known as one’s complement) operator: 5051 flip all the bits of the popped operand (note that this is the only 5052 unary, or single operand, bitwise operator). In other words, any 5053 bit with a value of ‘0’ is changed to ‘1’ and vice-versa. For 5054 example (assuming numbers can be written as bit strings on the 5055 command-line): ‘00101000 bitnot’ will give ‘11010111’. Note that 5056 the bitwise operators only work on integer type datasets/numbers. 5057 5058 5059File: gnuastro.info, Node: Numerical type conversion operators, Next: Adding noise operators, Prev: Bitwise operators, Up: Arithmetic operators 5060 50616.2.2.12 Numerical type conversion operators 5062............................................ 5063 5064With the operators below you can convert the numerical data type of your 5065input, see *note Numeric data types::. For example, let’s assume that 5066your colleague gives you thousands of single exposure images for 5067archival, but they have a double-precision floating point type! You 5068know that the statistical error a single-exposure image can never exceed 50696 or 7 significant digits, so you would prefer to archive them as a 5070single-precision floating point and save space on your computer (a 5071double-precision floating point is also double the file size!). You can 5072do this with the ‘float32’ operator described below. 5073 5074‘uint8’ 5075 Convert the type of the popped operand to 8-bit unsigned integer 5076 type (see *note Numeric data types::). The internal conversion of 5077 C will be used. 5078 5079‘int8’ 5080 Convert the type of the popped operand to 8-bit signed integer type 5081 (see *note Numeric data types::). The internal conversion of C 5082 will be used. 5083 5084‘uint16’ 5085 Convert the type of the popped operand to 16-bit unsigned integer 5086 type (see *note Numeric data types::). The internal conversion of 5087 C will be used. 5088 5089‘int16’ 5090 Convert the type of the popped operand to 16-bit signed integer 5091 (see *note Numeric data types::). The internal conversion of C 5092 will be used. 5093 5094‘uint32’ 5095 Convert the type of the popped operand to 32-bit unsigned integer 5096 type (see *note Numeric data types::). The internal conversion of 5097 C will be used. 5098 5099‘int32’ 5100 Convert the type of the popped operand to 32-bit signed integer 5101 type (see *note Numeric data types::). The internal conversion of 5102 C will be used. 5103 5104‘uint64’ 5105 Convert the type of the popped operand to 64-bit unsigned integer 5106 (see *note Numeric data types::). The internal conversion of C 5107 will be used. 5108 5109‘float32’ 5110 Convert the type of the popped operand to 32-bit (single precision) 5111 floating point (see *note Numeric data types::). The internal 5112 conversion of C will be used. 5113 5114‘float64’ 5115 Convert the type of the popped operand to 64-bit (double precision) 5116 floating point (see *note Numeric data types::). The internal 5117 conversion of C will be used. 5118 5119 5120File: gnuastro.info, Node: Adding noise operators, Next: Elliptical shape operators, Prev: Numerical type conversion operators, Up: Arithmetic operators 5121 51226.2.2.13 Adding noise operators 5123............................... 5124 5125When you simulate data (for example see *note Sufi simulates a 5126detection::), everything is ideal and there is no noise! The final step 5127of the process is to add simulated noise to the data. The operators in 5128this section are designed for that purpose. 5129 5130‘mknoise-sigma’ 5131 Add a fixed noise (Gaussian standard deviation) to each element of 5132 the input dataset. This operator takes two arguments: the 5133 top/first popped operand is the noise standard deviation, the next 5134 popped operand is the dataset that the noise should be added to. 5135 5136 When ‘--quiet’ isn’t given, a statement will be printed on each 5137 invocation of this operator (if there are multiple calls to the 5138 ‘mknoise-*’, the statement will be printed multiple times). It 5139 will show the random number generator function and seed that was 5140 used in that invocation, see *note Generating random numbers::. 5141 Reproducibility of the outputs can be ensured with the ‘--envseed’ 5142 option, see below for more. 5143 5144 For example with the first command below, ‘image.fits’ will be 5145 degraded by a noise of standard deviation 3 units. 5146 $ astarithmetic image.fits 3 mknoise-sigma 5147 5148 Alternatively, you can use this operator within column arithmetic 5149 in the Table program, to generate a random number like below 5150 (centered on 0, with $\sigma=3$) like the first command below. 5151 With the second command, you can put it into a shell variable for 5152 later usage. 5153 5154 $ echo 0 | asttable -c'arith $1 3 mknoise-sigma' 5155 $ value=$(echo 0 | asttable -c'arith $1 3 mknoise-sigma') 5156 $ echo $value 5157 5158 You can also use this operator in combination with AWK to easily 5159 generate an arbitrarily large table with random columns. In the 5160 example below, we’ll create a two column table with 20 rows. The 5161 first column will be centered on 5 and $\sigma_1=2$, the second 5162 will be centered on 10 and $\sigma_2=3$: 5163 5164 $ echo 5 10 \ 5165 | awk '{for(i=0;i<20;++i) print $1, $2}' \ 5166 | asttable -c'arith $1 2 mknoise-sigma' \ 5167 -c'arith $2 3 mknoise-sigma' 5168 5169 By adding an extra ‘--output=random.fits’, the table will be saved 5170 into a file called ‘random.fits’, and you can change the ‘i<20’ to 5171 ‘i<5000’ to have 5000 rows instead. Of course, if your input table 5172 has different values in the desired column the noisy distribution 5173 will be centered on each input element, but all will have the same 5174 scatter/sigma. 5175 5176 You can use the ‘--envseed’ option to fix the random number 5177 generator seed (and thus get a reproducible result). For more on 5178 ‘--envseed’, see *note Generating random numbers::. When using 5179 column arithmetic in Table, it may happen that multiple columns 5180 need random numbers (with any of the ‘mknoise-*’ operators) in one 5181 call of ‘asttable’. In such cases, the value given to 5182 ‘GSL_RNG_SEED’ is incremented by one on every call to the 5183 ‘mknoise-*’ operators. Without this increment, when the column 5184 values are the same (happens a lot, for no-noised datasets), the 5185 returned values for all columns will be identical. But this 5186 feature has a side-effect: that if the order of calling the 5187 ‘mknoise-*’ operators changes, the seeds used for each operator 5188 will change(1). 5189 5190‘mknoise-poisson’ 5191 Add Poisson noise to each element of the input dataset (see *note 5192 Photon counting noise::). This operator takes two arguments: the 5193 top/first popped operand is the background value (in units of 5194 electron counts), the next popped operand is the dataset that the 5195 noise should be added to. 5196 5197 Except for the noise-model, this operator is very similar to 5198 ‘mknoise-sigma’ and the examples there apply here too. The main 5199 difference with ‘mknoise-sigma’ is that in a Poisson distribution 5200 the scatter/sigma will depend on each element’s value. 5201 5202 For example, let’s assume you have made a mock image called 5203 ‘mock.fits’ with *note MakeProfiles:: and its assumed zeropoint is 5204 22.5 (for more on the zero point, see *note Brightness flux 5205 magnitude::). Let’s assume the background level for the Poisson 5206 noise has a value of 19 magnitudes. You can first use the 5207 ‘mag-to-counts’ operator to convert this background magnitude into 5208 counts, then feed the background value in counts to 5209 ‘mknoise-poisson’ operator: 5210 5211 $ astarithmetic mock.fits 19 22.5 mag-to-counts \ 5212 mknoise-poisson 5213 5214 Try changing the background value from 19 to 10 to see the effect! 5215 Recall that the tutorial *note Sufi simulates a detection:: shows 5216 how you can use MakeProfiles to build mock images. 5217 5218‘mknoise-uniform’ 5219 Add uniform noise to each element of the input dataset. This 5220 operator takes two arguments: the top/first popped operand is the 5221 width of the interval, the second popped operand is the dataset 5222 that the noise should be added to (each element will be the center 5223 of the interval). The returned random values may happen to be the 5224 minimum interval value, but will never be the maximum. Except for 5225 the noise-model, this operator behaves very similar to 5226 ‘mknoise-sigma’, see the explanation there for more. 5227 5228 For example with the command below, a random value will be selected 5229 between 10 to 14 (centered on 12, which is the only input data 5230 element, with a total width of 4). 5231 5232 echo 12 | asttable -c'arith $1 4 mknoise-uniform' 5233 5234 Similar to the example in ‘mknoise-sigma’, you can pipe the output 5235 of ‘echo’ to ‘awk’ before passing it to ‘asttable’ to generate a 5236 full column of uniformly selected values within the same interval. 5237 5238 ---------- Footnotes ---------- 5239 5240 (1) We have defined Task 15971 (https://savannah.gnu.org/task/?15971) 5241in Gnuastro’s project management system to address this. If you need 5242this feature please send us an email at ‘bug-gnuastro@gnu.org’ (to 5243motivate us in its implementation). 5244 5245 5246File: gnuastro.info, Node: Elliptical shape operators, Next: Building new dataset, Prev: Adding noise operators, Up: Arithmetic operators 5247 52486.2.2.14 Elliptical shape operators 5249................................... 5250 5251The operators here describe certain functions that will be necessary 5252when dealing with objects that have a certain elliptical shape. 5253 5254‘box-around-ellipse’ 5255 Return the width (along horizontal) and height (along vertical) of 5256 a box that encompasses an ellipse with the same center point. The 5257 top-popped operand is assumed to be the position angle (angle from 5258 the horizontal axis) in _degrees_. The second and third popped 5259 operands are the minor and major axis lengths respectively. This 5260 operator outputs two operands on the general stack. The first one 5261 is the width and the second (which will be the top one when this 5262 operator finishes) is the height. 5263 5264 If the value to the second popped operand (minor axis) is larger 5265 than the third (major axis), a NaN value will be written for both 5266 the width and height of that element and a warning will be printed 5267 (the warning can be disabled with the ‘--quiet’ option). 5268 5269 As an example, if your ellipse has a major axis length of 10 units, 5270 a minor axis length of 4 units and a position angle of 20 degrees, 5271 you can estimate the bounding box with this command: 5272 5273 $ echo "10 4 20" \ 5274 | asttable -c'arith $1 $2 $3 box-around-ellipse' 5275 5276 Alternatively if your three values are in separate FITS 5277 arrays/images, you can use the command below to have the width and 5278 height in similarly sized fits arrays. In this example ‘a.fits’ 5279 and ‘b.fits’ are respectively the major and minor axis lengths and 5280 ‘pa.fits’ is the position angle (in degrees). Also, in all three, 5281 we assume the first extension is used. After its done, the height 5282 of the box will be put in ‘h.fits’ and the width will be in 5283 ‘w.fits’. Just note that because this operator has two output 5284 datasets, you need to first write the height (top output operand) 5285 into a file and free it with the ‘tofilefree-’ operator, then write 5286 the width in the file given to ‘--output’. 5287 5288 $ astarithmetic a.fits b.fits pa.fits box-around-ellipse \ 5289 tofilefree-h.fits -ow.fits -g1 5290 5291 Finally, if you need to treat the width and height separately for 5292 further processing, you can call the ‘set-’ operator two times 5293 afterwards like below. Recall that the ‘set-’ operator will pop 5294 the top operand, and put it in memory with a certain name, bringing 5295 the next operand to the top of the stack. 5296 5297 For example let’s assume ‘catalog.fits’ has at least three columns 5298 ‘MAJOR’, ‘MINOR’ and ‘PA’ which specify the major axis, minor axis 5299 and position angle respectively. But you want the final width and 5300 height in 32-bit floating point numbers (not the default 64-bit, 5301 which may be too much precision in many scenarios). You can do 5302 this with the command below (note you can also break lines with 5303 <\>, within the single-quote environment) 5304 5305 $ asttable catalog.fits \ 5306 -c'arith MAJOR MINOR PA box-around-ellipse \ 5307 set-height set-width \ 5308 width float32 height float32' 5309 5310 5311File: gnuastro.info, Node: Building new dataset, Next: Operand storage in memory or a file, Prev: Elliptical shape operators, Up: Arithmetic operators 5312 53136.2.2.15 Building new dataset 5314............................. 5315 5316With the operator here, you can create a new dataset from scratch to 5317start certain operations without any input data. 5318 5319‘makenew’ 5320 Create a new dataset that only has zero values. The number of 5321 dimensions is read as the first popped operand and the number of 5322 elements along each dimension are the next popped operand (in 5323 reverse of the popping order). The type of the new dataset is an 5324 unsigned 8-bit integer and all pixel values have a value of zero. 5325 For example, if you want to create a new 100 by 200 pixel image, 5326 you can run this command: 5327 5328 $ astarithmetic 100 200 2 makenew 5329 5330 To further extend the example, you can use any of the noise-making 5331 operators to add noise to this new dataset (see *note Adding noise 5332 operators::), like the command below: 5333 5334 $ astarithmetic 100 200 2 makenew 5 mknoise-sigma 5335 5336 5337File: gnuastro.info, Node: Operand storage in memory or a file, Prev: Building new dataset, Up: Arithmetic operators 5338 53396.2.2.16 Operand storage in memory or a file 5340............................................ 5341 5342In your early days of using Gnuastro, to do multiple operations, it is 5343likely that you will simply call Arithmetic (or Table, with column 5344arithmetic) multiple times: feed the output file of the first call to 5345the second call. But as you get more proficient in the reverse polish 5346notation, you will find yourself combining many operations into one 5347call. This greatly speeds up your operation, because instead of writing 5348the dataset to a file in one command, and reading it in the next 5349command, it will just keep the intermediate dataset in memory! 5350 5351 But adding more complexity to your operations, can make them much 5352harder to debug, or extend even further. Therefore in this section we 5353have some special operators that behave differently from the rest: they 5354don’t touch the contents of the data, only where/how they are stored. 5355They are designed to do complex operations, without necessarily having a 5356complex command. 5357 5358‘set-AAA’ 5359 Set the characters after the dash (‘AAA’ in the case shown here) as 5360 a name for the first popped operand on the stack. The named 5361 dataset will be freed from memory as soon as it is no longer 5362 needed, or if the name is reset to refer to another dataset later 5363 in the command. This operator thus enables re-usability of a 5364 dataset without having to re-read it from a file every time it is 5365 necessary during a process. When a dataset is necessary more than 5366 once, this operator can thus help simplify reading/writing on the 5367 command-line (thus avoiding potential bugs), while also speeding up 5368 the processing. 5369 5370 Like all operators, this operator pops the top operand off of the 5371 main processing stack, but unlike other operands, it won’t add 5372 anything back to the stack immediately. It will keep the popped 5373 dataset in memory through a separate list of named datasets (not on 5374 the main stack). That list will be used to add/copy any requested 5375 dataset to the main processing stack when the name is called. 5376 5377 The name to give the popped dataset is part of the operator’s name. 5378 For example the ‘set-a’ operator of the command below, gives the 5379 name “‘a’” to the contents of ‘image.fits’. This name is then used 5380 instead of the actual filename to multiply the dataset by two. 5381 5382 $ astarithmetic image.fits set-a a 2 x 5383 5384 The name can be any string, but avoid strings ending with standard 5385 filename suffixes (for example ‘.fits’)(1). 5386 5387 One example of the usefulness of this operator is in the ‘where’ 5388 operator. For example, let’s assume you want to mask all pixels 5389 larger than ‘5’ in ‘image.fits’ (extension number 1) with a NaN 5390 value. Without setting a name for the dataset, you have to read 5391 the file two times from memory in a command like this: 5392 5393 $ astarithmetic image.fits image.fits 5 gt nan where -g1 5394 5395 But with this operator you can simply give ‘image.fits’ the name 5396 ‘i’ and simplify the command above to the more readable one below 5397 (which greatly helps when the filename is long): 5398 5399 $ astarithmetic image.fits set-i i i 5 gt nan where 5400 5401‘tofile-AAA’ 5402 Write the top operand on the operands stack into a file called 5403 ‘AAA’ (can be any FITS file name) without changing the operands 5404 stack. If you don’t need the dataset any more and would like to 5405 free it, see the ‘tofilefree’ operator below. 5406 5407 By default, any file that is given to this operator is deleted 5408 before Arithmetic actually starts working on the input datasets. 5409 The deletion can be deactivated with the ‘--dontdelete’ option (as 5410 in all Gnuastro programs, see *note Input output options::). If 5411 the same FITS file is given to this operator multiple times, it 5412 will contain multiple extensions (in the same order that it was 5413 called. 5414 5415 For example the operator ‘tofile-check.fits’ will write the top 5416 operand to ‘check.fits’. Since it doesn’t modify the operands 5417 stack, this operator is very convenient when you want to debug, or 5418 understanding, a string of operators and operands given to 5419 Arithmetic: simply put ‘tofile-AAA’ anywhere in the process to see 5420 what is happening behind the scenes without modifying the overall 5421 process. 5422 5423‘tofilefree-AAA’ 5424 Similar to the ‘tofile’ operator, with the only difference that the 5425 dataset that is written to a file is popped from the operand stack 5426 and freed from memory (cannot be used any more). 5427 5428 ---------- Footnotes ---------- 5429 5430 (1) A dataset name like ‘a.fits’ (which can be set with ‘set-a.fits’) 5431will cause confusion in the initial parser of Arithmetic. It will 5432assume this name is a FITS file, and if it is used multiple times, 5433Arithmetic will abort, complaining that you haven’t provided enough 5434HDUs. 5435 5436 5437File: gnuastro.info, Node: Invoking astarithmetic, Prev: Arithmetic operators, Up: Arithmetic 5438 54396.2.3 Invoking Arithmetic 5440------------------------- 5441 5442Arithmetic will do pixel to pixel arithmetic operations on the 5443individual pixels of input data and/or numbers. For the full list of 5444operators with explanations, please see *note Arithmetic operators::. 5445Any operand that only has a single element (number, or single pixel FITS 5446image) will be read as a number, the rest of the inputs must have the 5447same dimensions. The general template is: 5448 5449 $ astarithmetic [OPTION...] ASTRdata1 [ASTRdata2] OPERATOR ... 5450 5451One line examples: 5452 5453 ## Calculate (10.32-3.84)^2.7 quietly (will just print 155.329): 5454 $ astarithmetic -q 10.32 3.84 - 2.7 pow 5455 5456 ## Inverse the input image (1/pixel): 5457 $ astarithmetic 1 image.fits / --out=inverse.fits 5458 5459 ## Multiply each pixel in image by -1: 5460 $ astarithmetic image.fits -1 x --out=negative.fits 5461 5462 ## Subtract extension 4 from extension 1 (counting from zero): 5463 $ astarithmetic image.fits image.fits - --out=skysub.fits \ 5464 --hdu=1 --hdu=4 5465 5466 ## Add two images, then divide them by 2 (2 is read as floating point): 5467 ## Note that without the '.0', the '2' will be read/used as an integer. 5468 $ astarithmetic image1.fits image2.fits + 2.0 / --out=average.fits 5469 5470 ## Use Arithmetic's average operator: 5471 $ astarithmetic image1.fits image2.fits average --out=average.fits 5472 5473 ## Calculate the median of three images in three separate extensions: 5474 $ astarithmetic img1.fits img2.fits img3.fits median \ 5475 -h0 -h1 -h2 --out=median.fits 5476 5477 Arithmetic’s notation for giving operands to operators is fully 5478described in *note Reverse polish notation::. The output dataset is 5479last remaining operand on the stack. When the output dataset a single 5480number, it will be printed on the command-line. When the output is an 5481array, it will be stored as a file. 5482 5483 The name of the final file can be specified with the ‘--output’ 5484option, but if its not given, Arithmetic will use “automatic output” on 5485the name of the first FITS image encountered to generate an output file 5486name, see *note Automatic output::. By default, if the output file 5487already exists, it will be deleted before Arithmetic starts operation. 5488However, this can be disabled with the ‘--dontdelete’ option (see 5489below). At any point during Arithmetic’s operation, you can also write 5490the top operand on the stack to a file, using the ‘tofile’ or 5491‘tofilefree’ operators, see *note Arithmetic operators::. 5492 5493 By default, the world coordinate system (WCS) information of the 5494output dataset will be taken from the first input image (that contains a 5495WCS) on the command-line. This can be modified with the ‘--wcsfile’ and 5496‘--wcshdu’ options described below. When the ‘--quiet’ option isn’t 5497given, the name and extension of the dataset used for the output’s WCS 5498is printed on the command-line. 5499 5500 Through operators like those starting with ‘collapse-’, the 5501dimensionality of the inputs may not be the same as the outputs. By 5502default, when the output is 1D, Arithmetic will write it as a table, not 5503an image/array. The format of the output table (plain text or FITS 5504ASCII or binary) can be set with the ‘--tableformat’ option, see *note 5505Input output options::). You can disable this feature (write 1D arrays 5506as FITS images/arrays, or to the standard output) with the 5507‘--onedasimage’ or ‘--onedonstdout’ options. 5508 5509 See *note Common options:: for a review of the options in all 5510Gnuastro programs. Arithmetic just redefines the ‘--hdu’ and 5511‘--dontdelete’ options as explained below. 5512 5513‘-h INT/STR’ 5514‘--hdu INT/STR’ 5515 The header data unit of the input FITS images, see *note Input 5516 output options::. Unlike most options in Gnuastro (which will 5517 ultimately only have one value for this option), Arithmetic allows 5518 ‘--hdu’ to be called multiple times and the value of each 5519 invocation will be stored separately (for the unlimited number of 5520 input images you would like to use). Recall that for other 5521 programs this (common) option only takes a single value. So in 5522 other programs, if you specify it multiple times on the 5523 command-line, only the last value will be used and in the 5524 configuration files, it will be ignored if it already has a value. 5525 5526 The order of the values to ‘--hdu’ has to be in the same order as 5527 input FITS images. Options are first read from the command-line 5528 (from left to right), then top-down in each configuration file, see 5529 *note Configuration file precedence::. 5530 5531 If the number of HDUs is less than the number of input images, 5532 Arithmetic will abort and notify you. However, if there are more 5533 HDUs than FITS images, there is no problem: they will be used in 5534 the given order (every time a FITS image comes up on the stack) and 5535 the extra HDUs will be ignored in the end. So there is no problem 5536 with having extra HDUs in the configuration files and by default 5537 several HDUs with a value of ‘0’ are kept in the system-wide 5538 configuration file when you install Gnuastro. 5539 5540‘-g INT/STR’ 5541‘--globalhdu INT/STR’ 5542 Use the value to this option as the HDU of all input FITS files. 5543 This option is very convenient when you have many input files and 5544 the dataset of interest is in the same HDU of all the files. When 5545 this option is called, any values given to the ‘--hdu’ option 5546 (explained above) are ignored and will not be used. 5547 5548‘-w FITS’ 5549‘--wcsfile FITS’ 5550 FITS Filename containing the WCS structure that must be written to 5551 the output. The HDU/extension should be specified with ‘--wcshdu’. 5552 5553 When this option is used, the respective WCS will be read before 5554 any processing is done on the command-line and directly used in the 5555 final output. If the given file doesn’t have any WCS, then the 5556 default WCS (first file on the command-line with WCS) will be used 5557 in the output. 5558 5559 This option will mostly be used when the default file (first of the 5560 set of inputs) is not the one containing your desired WCS. But with 5561 this option, you can also use Arithmetic to rewrite/change the WCS 5562 of an existing FITS dataset from another file: 5563 5564 $ astarithmetic data.fits --wcsfile=other.fits -ofinal.fits 5565 5566‘-W STR’ 5567‘--wcshdu STR’ 5568 HDU/extension to read the WCS within the file given to ‘--wcsfile’. 5569 For more, see the description of ‘--wcsfile’. 5570 5571‘--envseed’ 5572 Use the environment for the random number generator settings in 5573 operators that need them (for example ‘mknoise-sigma’). This is 5574 very important for obtaining reproducible results, for more see 5575 *note Generating random numbers::. 5576 5577‘-O’ 5578‘--onedasimage’ 5579 When final dataset to write as output only has one dimension, write 5580 it as a FITS image/array. By default, if the output is 1D, it will 5581 be written as a table, see above. 5582 5583‘-s’ 5584‘--onedonstdout’ 5585 When final dataset to write as output only has one dimension, print 5586 it on the standard output, not in a file. By default, if the 5587 output is 1D, it will be written as a table, see above. 5588 5589‘-D’ 5590‘--dontdelete’ 5591 Don’t delete the output file, or files given to the ‘tofile’ or 5592 ‘tofilefree’ operators, if they already exist. Instead append the 5593 desired datasets to the extensions that already exist in the 5594 respective file. Note it doesn’t matter if the final output file 5595 name is given with the ‘--output’ option, or determined 5596 automatically. 5597 5598 Arithmetic treats this option differently from its default 5599 operation in other Gnuastro programs (see *note Input output 5600 options::). If the output file exists, when other Gnuastro 5601 programs are called with ‘--dontdelete’, they simply complain and 5602 abort. But when Arithmetic is called with ‘--dontdelete’, it will 5603 appended the dataset(s) to the existing extension(s) in the file. 5604 5605 Arithmetic accepts two kinds of input: images and numbers. Images 5606are considered to be any of the inputs that is a file name of a 5607recognized type (see *note Arguments::) and has more than one 5608element/pixel. Numbers on the command-line will be read into the 5609smallest type (see *note Numeric data types::) that can store them, so 5610‘-2’ will be read as a ‘char’ type (which is signed on most systems and 5611can thus keep negative values), ‘2500’ will be read as an ‘unsigned 5612short’ (all positive numbers will be read as unsigned), while 5613‘3.1415926535897’ will be read as a ‘double’ and ‘3.14’ will be read as 5614a ‘float’. To force a number to be read as float, put a ‘.’ after it 5615(possibly followed by a zero for easier readability), or add an ‘f’ 5616after it. Hence while ‘5’ will be read as an integer, ‘5.’, ‘5.0’ or 5617‘5f’ will be added to the stack as ‘float’ (see *note Reverse polish 5618notation::). 5619 5620 Unless otherwise stated (in *note Arithmetic operators::), the 5621operators can deal with numeric multiple data types (see *note Numeric 5622data types::). For example in “‘a.fits b.fits +’”, the image types can 5623be ‘long’ and ‘float’. In such cases, C’s internal type conversion will 5624be used. The output type will be set to the higher-ranking type of the 5625two inputs. Unsigned integer types have smaller ranking than their 5626signed counterparts and floating point types have higher ranking than 5627the integer types. So the internal C type conversions done in the 5628example above are equivalent to this piece of C: 5629 5630 size_t i; 5631 long a[100]; 5632 float b[100], out[100]; 5633 for(i=0;i<100;++i) out[i]=a[i]+b[i]; 5634 5635Relying on the default C type conversion significantly speeds up the 5636processing and also requires less RAM (when using very large images). 5637 5638 Some operators can only work on integer types (of any length, for 5639example bitwise operators) while others only work on floating point 5640types, (currently only the ‘pow’ operator). In such cases, if the 5641operand type(s) are different, an error will be printed. Arithmetic 5642also comes with internal type conversion operators which you can use to 5643convert the data into the appropriate type, see *note Arithmetic 5644operators::. 5645 5646 The hyphen (‘-’) can be used both to specify options (see *note 5647Options::) and also to specify a negative number which might be 5648necessary in your arithmetic. In order to enable you to do this, 5649Arithmetic will first parse all the input strings and if the first 5650character after a hyphen is a digit, then that hyphen is temporarily 5651replaced by the vertical tab character which is not commonly used. The 5652arguments are then parsed and these strings will not be specified as an 5653option. Then the given arguments are parsed and any vertical tabs are 5654replaced back with a hyphen so they can be read as negative numbers. 5655Therefore, as long as the names of the files you want to work on, don’t 5656start with a vertical tab followed by a digit, there is no problem. An 5657important consequence of this implementation is that you should not 5658write negative fractions like this: ‘-.3’, instead write them as ‘-0.3’. 5659 5660 Without any images, Arithmetic will act like a simple calculator and 5661print the resulting output number on the standard output like the first 5662example above. If you really want such calculator operations on the 5663command-line, AWK (GNU AWK is the most common implementation) is much 5664faster, easier and much more powerful. For example, the numerical 5665one-line example above can be done with the following command. In 5666general AWK is a fantastic tool and GNU AWK has a wonderful manual 5667(<https://www.gnu.org/software/gawk/manual/>). So if you often confront 5668situations like this, or have to work with large text tables/catalogs, 5669be sure to checkout AWK and simplify your life. 5670 5671 $ echo "" | awk '{print (10.32-3.84)^2.7}' 5672 155.329 5673 5674
