1""" 2Operations on regular files, special files, directories, and symlinks 3===================================================================== 4 5Salt States can aggressively manipulate files on a system. There are a number 6of ways in which files can be managed. 7 8Regular files can be enforced with the :mod:`file.managed 9<salt.states.file.managed>` state. This state downloads files from the salt 10master and places them on the target system. Managed files can be rendered as a 11jinja, mako, or wempy template, adding a dynamic component to file management. 12An example of :mod:`file.managed <salt.states.file.managed>` which makes use of 13the jinja templating system would look like this: 14 15.. code-block:: jinja 16 17 /etc/http/conf/http.conf: 18 file.managed: 19 - source: salt://apache/http.conf 20 - user: root 21 - group: root 22 - mode: 644 23 - attrs: ai 24 - template: jinja 25 - defaults: 26 custom_var: "default value" 27 other_var: 123 28 {% if grains['os'] == 'Ubuntu' %} 29 - context: 30 custom_var: "override" 31 {% endif %} 32 33It is also possible to use the :mod:`py renderer <salt.renderers.py>` as a 34templating option. The template would be a Python script which would need to 35contain a function called ``run()``, which returns a string. All arguments 36to the state will be made available to the Python script as globals. The 37returned string will be the contents of the managed file. For example: 38 39.. code-block:: python 40 41 def run(): 42 lines = ['foo', 'bar', 'baz'] 43 lines.extend([source, name, user, context]) # Arguments as globals 44 return '\\n\\n'.join(lines) 45 46.. note:: 47 48 The ``defaults`` and ``context`` arguments require extra indentation (four 49 spaces instead of the normal two) in order to create a nested dictionary. 50 :ref:`More information <nested-dict-indentation>`. 51 52If using a template, any user-defined template variables in the file defined in 53``source`` must be passed in using the ``defaults`` and/or ``context`` 54arguments. The general best practice is to place default values in 55``defaults``, with conditional overrides going into ``context``, as seen above. 56 57The template will receive a variable ``custom_var``, which would be accessed in 58the template using ``{{ custom_var }}``. If the operating system is Ubuntu, the 59value of the variable ``custom_var`` would be *override*, otherwise it is the 60default *default value* 61 62The ``source`` parameter can be specified as a list. If this is done, then the 63first file to be matched will be the one that is used. This allows you to have 64a default file on which to fall back if the desired file does not exist on the 65salt fileserver. Here's an example: 66 67.. code-block:: jinja 68 69 /etc/foo.conf: 70 file.managed: 71 - source: 72 - salt://foo.conf.{{ grains['fqdn'] }} 73 - salt://foo.conf.fallback 74 - user: foo 75 - group: users 76 - mode: 644 77 - attrs: i 78 - backup: minion 79 80.. note:: 81 82 Salt supports backing up managed files via the backup option. For more 83 details on this functionality please review the 84 :ref:`backup_mode documentation <file-state-backups>`. 85 86The ``source`` parameter can also specify a file in another Salt environment. 87In this example ``foo.conf`` in the ``dev`` environment will be used instead. 88 89.. code-block:: yaml 90 91 /etc/foo.conf: 92 file.managed: 93 - source: 94 - 'salt://foo.conf?saltenv=dev' 95 - user: foo 96 - group: users 97 - mode: '0644' 98 - attrs: i 99 100.. warning:: 101 102 When using a mode that includes a leading zero you must wrap the 103 value in single quotes. If the value is not wrapped in quotes it 104 will be read by YAML as an integer and evaluated as an octal. 105 106The ``names`` parameter, which is part of the state compiler, can be used to 107expand the contents of a single state declaration into multiple, single state 108declarations. Each item in the ``names`` list receives its own individual state 109``name`` and is converted into its own low-data structure. This is a convenient 110way to manage several files with similar attributes. 111 112.. code-block:: yaml 113 114 salt_master_conf: 115 file.managed: 116 - user: root 117 - group: root 118 - mode: '0644' 119 - names: 120 - /etc/salt/master.d/master.conf: 121 - source: salt://saltmaster/master.conf 122 - /etc/salt/minion.d/minion-99.conf: 123 - source: salt://saltmaster/minion.conf 124 125.. note:: 126 127 There is more documentation about this feature in the :ref:`Names declaration 128 <names-declaration>` section of the :ref:`Highstate docs <states-highstate>`. 129 130Special files can be managed via the ``mknod`` function. This function will 131create and enforce the permissions on a special file. The function supports the 132creation of character devices, block devices, and FIFO pipes. The function will 133create the directory structure up to the special file if it is needed on the 134minion. The function will not overwrite or operate on (change major/minor 135numbers) existing special files with the exception of user, group, and 136permissions. In most cases the creation of some special files require root 137permissions on the minion. This would require that the minion to be run as the 138root user. Here is an example of a character device: 139 140.. code-block:: yaml 141 142 /var/named/chroot/dev/random: 143 file.mknod: 144 - ntype: c 145 - major: 1 146 - minor: 8 147 - user: named 148 - group: named 149 - mode: 660 150 151Here is an example of a block device: 152 153.. code-block:: yaml 154 155 /var/named/chroot/dev/loop0: 156 file.mknod: 157 - ntype: b 158 - major: 7 159 - minor: 0 160 - user: named 161 - group: named 162 - mode: 660 163 164Here is an example of a fifo pipe: 165 166.. code-block:: yaml 167 168 /var/named/chroot/var/log/logfifo: 169 file.mknod: 170 - ntype: p 171 - user: named 172 - group: named 173 - mode: 660 174 175Directories can be managed via the ``directory`` function. This function can 176create and enforce the permissions on a directory. A directory statement will 177look like this: 178 179.. code-block:: yaml 180 181 /srv/stuff/substuf: 182 file.directory: 183 - user: fred 184 - group: users 185 - mode: 755 186 - makedirs: True 187 188If you need to enforce user and/or group ownership or permissions recursively 189on the directory's contents, you can do so by adding a ``recurse`` directive: 190 191.. code-block:: yaml 192 193 /srv/stuff/substuf: 194 file.directory: 195 - user: fred 196 - group: users 197 - mode: 755 198 - makedirs: True 199 - recurse: 200 - user 201 - group 202 - mode 203 204As a default, ``mode`` will resolve to ``dir_mode`` and ``file_mode``, to 205specify both directory and file permissions, use this form: 206 207.. code-block:: yaml 208 209 /srv/stuff/substuf: 210 file.directory: 211 - user: fred 212 - group: users 213 - file_mode: 744 214 - dir_mode: 755 215 - makedirs: True 216 - recurse: 217 - user 218 - group 219 - mode 220 221Symlinks can be easily created; the symlink function is very simple and only 222takes a few arguments: 223 224.. code-block:: yaml 225 226 /etc/grub.conf: 227 file.symlink: 228 - target: /boot/grub/grub.conf 229 230Recursive directory management can also be set via the ``recurse`` 231function. Recursive directory management allows for a directory on the salt 232master to be recursively copied down to the minion. This is a great tool for 233deploying large code and configuration systems. A state using ``recurse`` 234would look something like this: 235 236.. code-block:: yaml 237 238 /opt/code/flask: 239 file.recurse: 240 - source: salt://code/flask 241 - include_empty: True 242 243A more complex ``recurse`` example: 244 245.. code-block:: jinja 246 247 {% set site_user = 'testuser' %} 248 {% set site_name = 'test_site' %} 249 {% set project_name = 'test_proj' %} 250 {% set sites_dir = 'test_dir' %} 251 252 django-project: 253 file.recurse: 254 - name: {{ sites_dir }}/{{ site_name }}/{{ project_name }} 255 - user: {{ site_user }} 256 - dir_mode: 2775 257 - file_mode: '0644' 258 - template: jinja 259 - source: salt://project/templates_dir 260 - include_empty: True 261 262Retention scheduling can be applied to manage contents of backup directories. 263For example: 264 265.. code-block:: yaml 266 267 /var/backups/example_directory: 268 file.retention_schedule: 269 - strptime_format: example_name_%Y%m%dT%H%M%S.tar.bz2 270 - retain: 271 most_recent: 5 272 first_of_hour: 4 273 first_of_day: 14 274 first_of_week: 6 275 first_of_month: 6 276 first_of_year: all 277 278""" 279 280 281import copy 282import difflib 283import itertools 284import logging 285import os 286import posixpath 287import re 288import shutil 289import sys 290import time 291import traceback 292import urllib.parse 293from collections import defaultdict 294from collections.abc import Iterable, Mapping 295from datetime import date, datetime # python3 problem in the making? 296from itertools import zip_longest 297 298import salt.loader 299import salt.payload 300import salt.utils.data 301import salt.utils.dateutils 302import salt.utils.dictupdate 303import salt.utils.files 304import salt.utils.hashutils 305import salt.utils.path 306import salt.utils.platform 307import salt.utils.stringutils 308import salt.utils.templates 309import salt.utils.url 310import salt.utils.versions 311from salt.exceptions import CommandExecutionError 312from salt.serializers import DeserializationError 313from salt.state import get_accumulator_dir as _get_accumulator_dir 314from salt.utils.odict import OrderedDict 315 316if salt.utils.platform.is_windows(): 317 import salt.utils.win_dacl 318 import salt.utils.win_functions 319 import salt.utils.winapi 320 321if salt.utils.platform.is_windows(): 322 import pywintypes 323 import win32com.client 324 325log = logging.getLogger(__name__) 326 327COMMENT_REGEX = r"^([[:space:]]*){0}[[:space:]]?" 328__NOT_FOUND = object() 329 330__func_alias__ = { 331 "copy_": "copy", 332} 333 334 335def _get_accumulator_filepath(): 336 """ 337 Return accumulator data path. 338 """ 339 return os.path.join(_get_accumulator_dir(__opts__["cachedir"]), __instance_id__) 340 341 342def _load_accumulators(): 343 def _deserialize(path): 344 ret = {"accumulators": {}, "accumulators_deps": {}} 345 try: 346 with salt.utils.files.fopen(path, "rb") as f: 347 loaded = salt.payload.load(f) 348 return loaded if loaded else ret 349 except (OSError, NameError): 350 # NameError is a msgpack error from salt-ssh 351 return ret 352 353 loaded = _deserialize(_get_accumulator_filepath()) 354 355 return loaded["accumulators"], loaded["accumulators_deps"] 356 357 358def _persist_accummulators(accumulators, accumulators_deps): 359 accumm_data = {"accumulators": accumulators, "accumulators_deps": accumulators_deps} 360 361 try: 362 with salt.utils.files.fopen(_get_accumulator_filepath(), "w+b") as f: 363 salt.payload.dump(accumm_data, f) 364 except NameError: 365 # msgpack error from salt-ssh 366 pass 367 368 369def _check_user(user, group): 370 """ 371 Checks if the named user and group are present on the minion 372 """ 373 err = "" 374 if user: 375 uid = __salt__["file.user_to_uid"](user) 376 if uid == "": 377 err += "User {} is not available ".format(user) 378 if group: 379 gid = __salt__["file.group_to_gid"](group) 380 if gid == "": 381 err += "Group {} is not available".format(group) 382 return err 383 384 385def _is_valid_relpath(relpath, maxdepth=None): 386 """ 387 Performs basic sanity checks on a relative path. 388 389 Requires POSIX-compatible paths (i.e. the kind obtained through 390 cp.list_master or other such calls). 391 392 Ensures that the path does not contain directory transversal, and 393 that it does not exceed a stated maximum depth (if specified). 394 """ 395 # Check relpath surrounded by slashes, so that `..` can be caught as 396 # a path component at the start, end, and in the middle of the path. 397 sep, pardir = posixpath.sep, posixpath.pardir 398 if sep + pardir + sep in sep + relpath + sep: 399 return False 400 401 # Check that the relative path's depth does not exceed maxdepth 402 if maxdepth is not None: 403 path_depth = relpath.strip(sep).count(sep) 404 if path_depth > maxdepth: 405 return False 406 407 return True 408 409 410def _salt_to_os_path(path): 411 """ 412 Converts a path from the form received via salt master to the OS's native 413 path format. 414 """ 415 return os.path.normpath(path.replace(posixpath.sep, os.path.sep)) 416 417 418def _gen_recurse_managed_files( 419 name, 420 source, 421 keep_symlinks=False, 422 include_pat=None, 423 exclude_pat=None, 424 maxdepth=None, 425 include_empty=False, 426 **kwargs 427): 428 """ 429 Generate the list of files managed by a recurse state 430 """ 431 432 # Convert a relative path generated from salt master paths to an OS path 433 # using "name" as the base directory 434 def full_path(master_relpath): 435 return os.path.join(name, _salt_to_os_path(master_relpath)) 436 437 # Process symlinks and return the updated filenames list 438 def process_symlinks(filenames, symlinks): 439 for lname, ltarget in symlinks.items(): 440 srelpath = posixpath.relpath(lname, srcpath) 441 if not _is_valid_relpath(srelpath, maxdepth=maxdepth): 442 continue 443 if not salt.utils.stringutils.check_include_exclude( 444 srelpath, include_pat, exclude_pat 445 ): 446 continue 447 # Check for all paths that begin with the symlink 448 # and axe it leaving only the dirs/files below it. 449 # This needs to use list() otherwise they reference 450 # the same list. 451 _filenames = list(filenames) 452 for filename in _filenames: 453 if filename.startswith(lname): 454 log.debug( 455 "** skipping file ** %s, it intersects a symlink", filename 456 ) 457 filenames.remove(filename) 458 # Create the symlink along with the necessary dirs. 459 # The dir perms/ownership will be adjusted later 460 # if needed 461 managed_symlinks.add((srelpath, ltarget)) 462 463 # Add the path to the keep set in case clean is set to True 464 keep.add(full_path(srelpath)) 465 vdir.update(keep) 466 return filenames 467 468 managed_files = set() 469 managed_directories = set() 470 managed_symlinks = set() 471 keep = set() 472 vdir = set() 473 474 srcpath, senv = salt.utils.url.parse(source) 475 if senv is None: 476 senv = __env__ 477 if not srcpath.endswith(posixpath.sep): 478 # we're searching for things that start with this *directory*. 479 srcpath = srcpath + posixpath.sep 480 fns_ = __salt__["cp.list_master"](senv, srcpath) 481 482 # If we are instructed to keep symlinks, then process them. 483 if keep_symlinks: 484 # Make this global so that emptydirs can use it if needed. 485 symlinks = __salt__["cp.list_master_symlinks"](senv, srcpath) 486 fns_ = process_symlinks(fns_, symlinks) 487 488 for fn_ in fns_: 489 if not fn_.strip(): 490 continue 491 492 # fn_ here is the absolute (from file_roots) source path of 493 # the file to copy from; it is either a normal file or an 494 # empty dir(if include_empty==true). 495 496 relname = salt.utils.data.decode(posixpath.relpath(fn_, srcpath)) 497 if not _is_valid_relpath(relname, maxdepth=maxdepth): 498 continue 499 500 # Check if it is to be excluded. Match only part of the path 501 # relative to the target directory 502 if not salt.utils.stringutils.check_include_exclude( 503 relname, include_pat, exclude_pat 504 ): 505 continue 506 dest = full_path(relname) 507 dirname = os.path.dirname(dest) 508 keep.add(dest) 509 510 if dirname not in vdir: 511 # verify the directory perms if they are set 512 managed_directories.add(dirname) 513 vdir.add(dirname) 514 515 src = salt.utils.url.create(fn_, saltenv=senv) 516 managed_files.add((dest, src)) 517 518 if include_empty: 519 mdirs = __salt__["cp.list_master_dirs"](senv, srcpath) 520 for mdir in mdirs: 521 relname = posixpath.relpath(mdir, srcpath) 522 if not _is_valid_relpath(relname, maxdepth=maxdepth): 523 continue 524 if not salt.utils.stringutils.check_include_exclude( 525 relname, include_pat, exclude_pat 526 ): 527 continue 528 mdest = full_path(relname) 529 # Check for symlinks that happen to point to an empty dir. 530 if keep_symlinks: 531 islink = False 532 for link in symlinks: 533 if mdir.startswith(link, 0): 534 log.debug( 535 "** skipping empty dir ** %s, it intersectsa symlink", mdir 536 ) 537 islink = True 538 break 539 if islink: 540 continue 541 542 managed_directories.add(mdest) 543 keep.add(mdest) 544 545 return managed_files, managed_directories, managed_symlinks, keep 546 547 548def _gen_keep_files(name, require, walk_d=None): 549 """ 550 Generate the list of files that need to be kept when a dir based function 551 like directory or recurse has a clean. 552 """ 553 554 def _is_child(path, directory): 555 """ 556 Check whether ``path`` is child of ``directory`` 557 """ 558 path = os.path.abspath(path) 559 directory = os.path.abspath(directory) 560 561 relative = os.path.relpath(path, directory) 562 563 return not relative.startswith(os.pardir) 564 565 def _add_current_path(path): 566 _ret = set() 567 if os.path.isdir(path): 568 dirs, files = walk_d.get(path, ((), ())) 569 _ret.add(path) 570 for _name in files: 571 _ret.add(os.path.join(path, _name)) 572 for _name in dirs: 573 _ret.add(os.path.join(path, _name)) 574 return _ret 575 576 def _process_by_walk_d(name, ret): 577 if os.path.isdir(name): 578 walk_ret.update(_add_current_path(name)) 579 dirs, _ = walk_d.get(name, ((), ())) 580 for _d in dirs: 581 p = os.path.join(name, _d) 582 walk_ret.update(_add_current_path(p)) 583 _process_by_walk_d(p, ret) 584 585 def _process(name): 586 ret = set() 587 if os.path.isdir(name): 588 for root, dirs, files in salt.utils.path.os_walk(name): 589 ret.add(name) 590 for name in files: 591 ret.add(os.path.join(root, name)) 592 for name in dirs: 593 ret.add(os.path.join(root, name)) 594 return ret 595 596 keep = set() 597 if isinstance(require, list): 598 required_files = [comp for comp in require if "file" in comp] 599 for comp in required_files: 600 for low in __lowstate__: 601 # A requirement should match either the ID and the name of 602 # another state. 603 if low["name"] == comp["file"] or low["__id__"] == comp["file"]: 604 fn = low["name"] 605 fun = low["fun"] 606 if os.path.isdir(fn): 607 if _is_child(fn, name): 608 if fun == "recurse": 609 fkeep = _gen_recurse_managed_files(**low)[3] 610 log.debug("Keep from %s: %s", fn, fkeep) 611 keep.update(fkeep) 612 elif walk_d: 613 walk_ret = set() 614 _process_by_walk_d(fn, walk_ret) 615 keep.update(walk_ret) 616 else: 617 keep.update(_process(fn)) 618 else: 619 keep.add(fn) 620 log.debug("Files to keep from required states: %s", list(keep)) 621 return list(keep) 622 623 624def _check_file(name): 625 ret = True 626 msg = "" 627 628 if not os.path.isabs(name): 629 ret = False 630 msg = "Specified file {} is not an absolute path".format(name) 631 elif not os.path.exists(name): 632 ret = False 633 msg = "{}: file not found".format(name) 634 635 return ret, msg 636 637 638def _find_keep_files(root, keep): 639 """ 640 Compile a list of valid keep files (and directories). 641 Used by _clean_dir() 642 """ 643 real_keep = set() 644 real_keep.add(root) 645 if isinstance(keep, list): 646 for fn_ in keep: 647 if not os.path.isabs(fn_): 648 continue 649 fn_ = os.path.normcase(os.path.abspath(fn_)) 650 real_keep.add(fn_) 651 while True: 652 fn_ = os.path.abspath(os.path.dirname(fn_)) 653 real_keep.add(fn_) 654 drive, path = os.path.splitdrive(fn_) 655 if not path.lstrip(os.sep): 656 break 657 return real_keep 658 659 660def _clean_dir(root, keep, exclude_pat): 661 """ 662 Clean out all of the files and directories in a directory (root) while 663 preserving the files in a list (keep) and part of exclude_pat 664 """ 665 case_keep = None 666 if salt.utils.files.case_insensitive_filesystem(): 667 # Create a case-sensitive dict before doing comparisons 668 # if file system is case sensitive 669 case_keep = keep 670 671 root = os.path.normcase(root) 672 real_keep = _find_keep_files(root, keep) 673 removed = set() 674 675 def _delete_not_kept(nfn): 676 if nfn not in real_keep: 677 # -- check if this is a part of exclude_pat(only). No need to 678 # check include_pat 679 if not salt.utils.stringutils.check_include_exclude( 680 os.path.relpath(nfn, root), None, exclude_pat 681 ): 682 return 683 # Before we can accurately assess the removal of a file, we must 684 # check for systems with case sensitive files. If we originally 685 # meant to keep a file, but due to case sensitivity python would 686 # otherwise remove the file, check against the original list. 687 if case_keep: 688 for item in case_keep: 689 if item.casefold() == nfn.casefold(): 690 return 691 removed.add(nfn) 692 if not __opts__["test"]: 693 try: 694 os.remove(nfn) 695 except OSError: 696 __salt__["file.remove"](nfn) 697 698 for roots, dirs, files in salt.utils.path.os_walk(root): 699 for name in itertools.chain(dirs, files): 700 _delete_not_kept(os.path.join(roots, name)) 701 return list(removed) 702 703 704def _error(ret, err_msg): 705 ret["result"] = False 706 ret["comment"] = err_msg 707 return ret 708 709 710def _check_directory( 711 name, 712 user=None, 713 group=None, 714 recurse=False, 715 dir_mode=None, 716 file_mode=None, 717 clean=False, 718 require=False, 719 exclude_pat=None, 720 max_depth=None, 721 follow_symlinks=False, 722): 723 """ 724 Check what changes need to be made on a directory 725 """ 726 changes = {} 727 if recurse or clean: 728 assert max_depth is None or not clean 729 # walk path only once and store the result 730 walk_l = list(_depth_limited_walk(name, max_depth)) 731 # root: (dirs, files) structure, compatible for python2.6 732 walk_d = {} 733 for i in walk_l: 734 walk_d[i[0]] = (i[1], i[2]) 735 736 if recurse: 737 try: 738 recurse_set = _get_recurse_set(recurse) 739 except (TypeError, ValueError) as exc: 740 return False, "{}".format(exc), changes 741 if "user" not in recurse_set: 742 user = None 743 if "group" not in recurse_set: 744 group = None 745 if "mode" not in recurse_set: 746 dir_mode = None 747 file_mode = None 748 749 check_files = "ignore_files" not in recurse_set 750 check_dirs = "ignore_dirs" not in recurse_set 751 for root, dirs, files in walk_l: 752 if check_files: 753 for fname in files: 754 fchange = {} 755 path = os.path.join(root, fname) 756 stats = __salt__["file.stats"](path, None, follow_symlinks) 757 if user is not None and user != stats.get("user"): 758 fchange["user"] = user 759 if group is not None and group != stats.get("group"): 760 fchange["group"] = group 761 smode = salt.utils.files.normalize_mode(stats.get("mode")) 762 file_mode = salt.utils.files.normalize_mode(file_mode) 763 if ( 764 file_mode is not None 765 and file_mode != smode 766 and ( 767 # Ignore mode for symlinks on linux based systems where we can not 768 # change symlink file permissions 769 follow_symlinks 770 or stats.get("type") != "link" 771 or not salt.utils.platform.is_linux() 772 ) 773 ): 774 fchange["mode"] = file_mode 775 if fchange: 776 changes[path] = fchange 777 if check_dirs: 778 for name_ in dirs: 779 path = os.path.join(root, name_) 780 fchange = _check_dir_meta( 781 path, user, group, dir_mode, follow_symlinks 782 ) 783 if fchange: 784 changes[path] = fchange 785 # Recurse skips root (we always do dirs, not root), so always check root: 786 fchange = _check_dir_meta(name, user, group, dir_mode, follow_symlinks) 787 if fchange: 788 changes[name] = fchange 789 if clean: 790 keep = _gen_keep_files(name, require, walk_d) 791 792 def _check_changes(fname): 793 path = os.path.join(root, fname) 794 if path in keep: 795 return {} 796 else: 797 if not salt.utils.stringutils.check_include_exclude( 798 os.path.relpath(path, name), None, exclude_pat 799 ): 800 return {} 801 else: 802 return {path: {"removed": "Removed due to clean"}} 803 804 for root, dirs, files in walk_l: 805 for fname in files: 806 changes.update(_check_changes(fname)) 807 for name_ in dirs: 808 changes.update(_check_changes(name_)) 809 810 if not os.path.isdir(name): 811 changes[name] = {"directory": "new"} 812 if changes: 813 comments = ["The following files will be changed:\n"] 814 for fn_ in changes: 815 for key, val in changes[fn_].items(): 816 comments.append("{}: {} - {}\n".format(fn_, key, val)) 817 return None, "".join(comments), changes 818 return True, "The directory {} is in the correct state".format(name), changes 819 820 821def _check_directory_win( 822 name, 823 win_owner=None, 824 win_perms=None, 825 win_deny_perms=None, 826 win_inheritance=None, 827 win_perms_reset=None, 828): 829 """ 830 Check what changes need to be made on a directory 831 """ 832 if not os.path.isdir(name): 833 changes = {name: {"directory": "new"}} 834 else: 835 changes = salt.utils.win_dacl.check_perms( 836 obj_name=name, 837 obj_type="file", 838 ret={}, 839 owner=win_owner, 840 grant_perms=win_perms, 841 deny_perms=win_deny_perms, 842 inheritance=win_inheritance, 843 reset=win_perms_reset, 844 test_mode=True, 845 )["changes"] 846 847 if changes: 848 return None, 'The directory "{}" will be changed'.format(name), changes 849 850 return True, "The directory {} is in the correct state".format(name), changes 851 852 853def _check_dir_meta(name, user, group, mode, follow_symlinks=False): 854 """ 855 Check the changes in directory metadata 856 """ 857 try: 858 stats = __salt__["file.stats"](name, None, follow_symlinks) 859 except CommandExecutionError: 860 stats = {} 861 862 changes = {} 863 if not stats: 864 changes["directory"] = "new" 865 return changes 866 if user is not None and user != stats["user"] and user != stats.get("uid"): 867 changes["user"] = user 868 if group is not None and group != stats["group"] and group != stats.get("gid"): 869 changes["group"] = group 870 # Normalize the dir mode 871 smode = salt.utils.files.normalize_mode(stats["mode"]) 872 mode = salt.utils.files.normalize_mode(mode) 873 if ( 874 mode is not None 875 and mode != smode 876 and ( 877 # Ignore mode for symlinks on linux based systems where we can not 878 # change symlink file permissions 879 follow_symlinks 880 or stats.get("type") != "link" 881 or not salt.utils.platform.is_linux() 882 ) 883 ): 884 changes["mode"] = mode 885 return changes 886 887 888def _check_touch(name, atime, mtime): 889 """ 890 Check to see if a file needs to be updated or created 891 """ 892 ret = { 893 "result": None, 894 "comment": "", 895 "changes": {"new": name}, 896 } 897 if not os.path.exists(name): 898 ret["comment"] = "File {} is set to be created".format(name) 899 else: 900 stats = __salt__["file.stats"](name, follow_symlinks=False) 901 if (atime is not None and str(atime) != str(stats["atime"])) or ( 902 mtime is not None and str(mtime) != str(stats["mtime"]) 903 ): 904 ret["comment"] = "Times set to be updated on file {}".format(name) 905 ret["changes"] = {"touched": name} 906 else: 907 ret["result"] = True 908 ret["comment"] = "File {} exists and has the correct times".format(name) 909 return ret 910 911 912def _get_symlink_ownership(path): 913 if salt.utils.platform.is_windows(): 914 owner = salt.utils.win_dacl.get_owner(path) 915 return owner, owner 916 else: 917 return ( 918 __salt__["file.get_user"](path, follow_symlinks=False), 919 __salt__["file.get_group"](path, follow_symlinks=False), 920 ) 921 922 923def _check_symlink_ownership(path, user, group, win_owner): 924 """ 925 Check if the symlink ownership matches the specified user and group 926 """ 927 cur_user, cur_group = _get_symlink_ownership(path) 928 if salt.utils.platform.is_windows(): 929 return win_owner == cur_user 930 else: 931 return (cur_user == user) and (cur_group == group) 932 933 934def _set_symlink_ownership(path, user, group, win_owner): 935 """ 936 Set the ownership of a symlink and return a boolean indicating 937 success/failure 938 """ 939 if salt.utils.platform.is_windows(): 940 try: 941 salt.utils.win_dacl.set_owner(path, win_owner) 942 except CommandExecutionError: 943 pass 944 else: 945 try: 946 __salt__["file.lchown"](path, user, group) 947 except OSError: 948 pass 949 return _check_symlink_ownership(path, user, group, win_owner) 950 951 952def _symlink_check(name, target, force, user, group, win_owner): 953 """ 954 Check the symlink function 955 """ 956 changes = {} 957 if not os.path.exists(name) and not __salt__["file.is_link"](name): 958 changes["new"] = name 959 return ( 960 None, 961 "Symlink {} to {} is set for creation".format(name, target), 962 changes, 963 ) 964 if __salt__["file.is_link"](name): 965 if __salt__["file.readlink"](name) != target: 966 changes["change"] = name 967 return ( 968 None, 969 "Link {} target is set to be changed to {}".format(name, target), 970 changes, 971 ) 972 else: 973 result = True 974 msg = "The symlink {} is present".format(name) 975 if not _check_symlink_ownership(name, user, group, win_owner): 976 result = None 977 changes["ownership"] = "{}:{}".format(*_get_symlink_ownership(name)) 978 msg += ( 979 ", but the ownership of the symlink would be changed " 980 "from {2}:{3} to {0}:{1}".format( 981 user, group, *_get_symlink_ownership(name) 982 ) 983 ) 984 return result, msg, changes 985 else: 986 if force: 987 return ( 988 None, 989 "The file or directory {} is set for removal to " 990 "make way for a new symlink targeting {}".format(name, target), 991 changes, 992 ) 993 return ( 994 False, 995 "File or directory exists where the symlink {} " 996 "should be. Did you mean to use force?".format(name), 997 changes, 998 ) 999 1000 1001def _hardlink_same(name, target): 1002 """ 1003 Check to see if the inodes match for the name and the target 1004 """ 1005 res = __salt__["file.stats"](name, None, follow_symlinks=False) 1006 if "inode" not in res: 1007 return False 1008 name_i = res["inode"] 1009 1010 res = __salt__["file.stats"](target, None, follow_symlinks=False) 1011 if "inode" not in res: 1012 return False 1013 target_i = res["inode"] 1014 1015 return name_i == target_i 1016 1017 1018def _hardlink_check(name, target, force): 1019 """ 1020 Check the hardlink function 1021 """ 1022 changes = {} 1023 if not os.path.exists(target): 1024 msg = "Target {} for hard link does not exist".format(target) 1025 return False, msg, changes 1026 1027 elif os.path.isdir(target): 1028 msg = "Unable to hard link from directory {}".format(target) 1029 return False, msg, changes 1030 1031 if os.path.isdir(name): 1032 msg = "Unable to hard link to directory {}".format(name) 1033 return False, msg, changes 1034 1035 elif not os.path.exists(name): 1036 msg = "Hard link {} to {} is set for creation".format(name, target) 1037 changes["new"] = name 1038 return None, msg, changes 1039 1040 elif __salt__["file.is_hardlink"](name): 1041 if _hardlink_same(name, target): 1042 msg = "The hard link {} is presently targetting {}".format(name, target) 1043 return True, msg, changes 1044 1045 msg = "Link {} target is set to be changed to {}".format(name, target) 1046 changes["change"] = name 1047 return None, msg, changes 1048 1049 if force: 1050 msg = ( 1051 "The file or directory {} is set for removal to " 1052 "make way for a new hard link targeting {}".format(name, target) 1053 ) 1054 return None, msg, changes 1055 1056 msg = ( 1057 "File or directory exists where the hard link {} " 1058 "should be. Did you mean to use force?".format(name) 1059 ) 1060 return False, msg, changes 1061 1062 1063def _test_owner(kwargs, user=None): 1064 """ 1065 Convert owner to user, since other config management tools use owner, 1066 no need to punish people coming from other systems. 1067 PLEASE DO NOT DOCUMENT THIS! WE USE USER, NOT OWNER!!!! 1068 """ 1069 if user: 1070 return user 1071 if "owner" in kwargs: 1072 log.warning( 1073 'Use of argument owner found, "owner" is invalid, please use "user"' 1074 ) 1075 return kwargs["owner"] 1076 1077 return user 1078 1079 1080def _unify_sources_and_hashes( 1081 source=None, source_hash=None, sources=None, source_hashes=None 1082): 1083 """ 1084 Silly little function to give us a standard tuple list for sources and 1085 source_hashes 1086 """ 1087 if sources is None: 1088 sources = [] 1089 1090 if source_hashes is None: 1091 source_hashes = [] 1092 1093 if source and sources: 1094 return (False, "source and sources are mutually exclusive", []) 1095 1096 if source_hash and source_hashes: 1097 return (False, "source_hash and source_hashes are mutually exclusive", []) 1098 1099 if source: 1100 return (True, "", [(source, source_hash)]) 1101 1102 # Make a nice neat list of tuples exactly len(sources) long.. 1103 return True, "", list(zip_longest(sources, source_hashes[: len(sources)])) 1104 1105 1106def _get_template_texts( 1107 source_list=None, template="jinja", defaults=None, context=None, **kwargs 1108): 1109 """ 1110 Iterate a list of sources and process them as templates. 1111 Returns a list of 'chunks' containing the rendered templates. 1112 """ 1113 1114 ret = { 1115 "name": "_get_template_texts", 1116 "changes": {}, 1117 "result": True, 1118 "comment": "", 1119 "data": [], 1120 } 1121 1122 if source_list is None: 1123 return _error(ret, "_get_template_texts called with empty source_list") 1124 1125 txtl = [] 1126 1127 for (source, source_hash) in source_list: 1128 1129 tmpctx = defaults if defaults else {} 1130 if context: 1131 tmpctx.update(context) 1132 rndrd_templ_fn = __salt__["cp.get_template"]( 1133 source, "", template=template, saltenv=__env__, context=tmpctx, **kwargs 1134 ) 1135 log.debug( 1136 "cp.get_template returned %s (Called with: %s)", rndrd_templ_fn, source 1137 ) 1138 if rndrd_templ_fn: 1139 tmplines = None 1140 with salt.utils.files.fopen(rndrd_templ_fn, "rb") as fp_: 1141 tmplines = fp_.read() 1142 tmplines = salt.utils.stringutils.to_unicode(tmplines) 1143 tmplines = tmplines.splitlines(True) 1144 if not tmplines: 1145 msg = "Failed to read rendered template file {} ({})".format( 1146 rndrd_templ_fn, source 1147 ) 1148 log.debug(msg) 1149 ret["name"] = source 1150 return _error(ret, msg) 1151 txtl.append("".join(tmplines)) 1152 else: 1153 msg = "Failed to load template file {}".format(source) 1154 log.debug(msg) 1155 ret["name"] = source 1156 return _error(ret, msg) 1157 1158 ret["data"] = txtl 1159 return ret 1160 1161 1162def _validate_str_list(arg, encoding=None): 1163 """ 1164 ensure ``arg`` is a list of strings 1165 """ 1166 if isinstance(arg, bytes): 1167 ret = [salt.utils.stringutils.to_unicode(arg, encoding=encoding)] 1168 elif isinstance(arg, str): 1169 ret = [arg] 1170 elif isinstance(arg, Iterable) and not isinstance(arg, Mapping): 1171 ret = [] 1172 for item in arg: 1173 if isinstance(item, str): 1174 ret.append(item) 1175 else: 1176 ret.append(str(item)) 1177 else: 1178 ret = [str(arg)] 1179 return ret 1180 1181 1182def _get_shortcut_ownership(path): 1183 return __salt__["file.get_user"](path, follow_symlinks=False) 1184 1185 1186def _check_shortcut_ownership(path, user): 1187 """ 1188 Check if the shortcut ownership matches the specified user 1189 """ 1190 cur_user = _get_shortcut_ownership(path) 1191 return cur_user == user 1192 1193 1194def _set_shortcut_ownership(path, user): 1195 """ 1196 Set the ownership of a shortcut and return a boolean indicating 1197 success/failure 1198 """ 1199 try: 1200 __salt__["file.lchown"](path, user) 1201 except OSError: 1202 pass 1203 return _check_shortcut_ownership(path, user) 1204 1205 1206def _shortcut_check( 1207 name, target, arguments, working_dir, description, icon_location, force, user 1208): 1209 """ 1210 Check the shortcut function 1211 """ 1212 changes = {} 1213 if not os.path.exists(name): 1214 changes["new"] = name 1215 return ( 1216 None, 1217 'Shortcut "{}" to "{}" is set for creation'.format(name, target), 1218 changes, 1219 ) 1220 1221 if os.path.isfile(name): 1222 with salt.utils.winapi.Com(): 1223 shell = win32com.client.Dispatch("WScript.Shell") 1224 scut = shell.CreateShortcut(name) 1225 state_checks = [scut.TargetPath.lower() == target.lower()] 1226 if arguments is not None: 1227 state_checks.append(scut.Arguments == arguments) 1228 if working_dir is not None: 1229 state_checks.append( 1230 scut.WorkingDirectory.lower() == working_dir.lower() 1231 ) 1232 if description is not None: 1233 state_checks.append(scut.Description == description) 1234 if icon_location is not None: 1235 state_checks.append(scut.IconLocation.lower() == icon_location.lower()) 1236 1237 if not all(state_checks): 1238 changes["change"] = name 1239 return ( 1240 None, 1241 'Shortcut "{}" target is set to be changed to "{}"'.format( 1242 name, target 1243 ), 1244 changes, 1245 ) 1246 else: 1247 result = True 1248 msg = 'The shortcut "{}" is present'.format(name) 1249 if not _check_shortcut_ownership(name, user): 1250 result = None 1251 changes["ownership"] = "{}".format(_get_shortcut_ownership(name)) 1252 msg += ( 1253 ", but the ownership of the shortcut would be changed " 1254 "from {1} to {0}".format(user, _get_shortcut_ownership(name)) 1255 ) 1256 return result, msg, changes 1257 else: 1258 if force: 1259 return ( 1260 None, 1261 'The link or directory "{}" is set for removal to ' 1262 'make way for a new shortcut targeting "{}"'.format(name, target), 1263 changes, 1264 ) 1265 return ( 1266 False, 1267 'Link or directory exists where the shortcut "{}" ' 1268 "should be. Did you mean to use force?".format(name), 1269 changes, 1270 ) 1271 1272 1273def _makedirs( 1274 name, 1275 user=None, 1276 group=None, 1277 dir_mode=None, 1278 win_owner=None, 1279 win_perms=None, 1280 win_deny_perms=None, 1281 win_inheritance=None, 1282): 1283 """ 1284 Helper function for creating directories when the ``makedirs`` option is set 1285 to ``True``. Handles Unix and Windows based systems 1286 1287 .. versionadded:: 2017.7.8 1288 1289 Args: 1290 name (str): The directory path to create 1291 user (str): The linux user to own the directory 1292 group (str): The linux group to own the directory 1293 dir_mode (str): The linux mode to apply to the directory 1294 win_owner (str): The Windows user to own the directory 1295 win_perms (dict): A dictionary of grant permissions for Windows 1296 win_deny_perms (dict): A dictionary of deny permissions for Windows 1297 win_inheritance (bool): True to inherit permissions on Windows 1298 1299 Returns: 1300 bool: True if successful, otherwise False on Windows 1301 str: Error messages on failure on Linux 1302 None: On successful creation on Linux 1303 1304 Raises: 1305 CommandExecutionError: If the drive is not mounted on Windows 1306 """ 1307 if salt.utils.platform.is_windows(): 1308 # Make sure the drive is mapped before trying to create the 1309 # path in windows 1310 drive, path = os.path.splitdrive(name) 1311 if not os.path.isdir(drive): 1312 raise CommandExecutionError(drive) 1313 win_owner = win_owner if win_owner else user 1314 return __salt__["file.makedirs"]( 1315 path=name, 1316 owner=win_owner, 1317 grant_perms=win_perms, 1318 deny_perms=win_deny_perms, 1319 inheritance=win_inheritance, 1320 ) 1321 else: 1322 return __salt__["file.makedirs"]( 1323 path=name, user=user, group=group, mode=dir_mode 1324 ) 1325 1326 1327def hardlink( 1328 name, 1329 target, 1330 force=False, 1331 makedirs=False, 1332 user=None, 1333 group=None, 1334 dir_mode=None, 1335 **kwargs 1336): 1337 """ 1338 Create a hard link 1339 If the file already exists and is a hard link pointing to any location other 1340 than the specified target, the hard link will be replaced. If the hard link 1341 is a regular file or directory then the state will return False. If the 1342 regular file is desired to be replaced with a hard link pass force: True 1343 1344 name 1345 The location of the hard link to create 1346 target 1347 The location that the hard link points to 1348 force 1349 If the name of the hard link exists and force is set to False, the 1350 state will fail. If force is set to True, the file or directory in the 1351 way of the hard link file will be deleted to make room for the hard 1352 link, unless backupname is set, when it will be renamed 1353 makedirs 1354 If the location of the hard link does not already have a parent directory 1355 then the state will fail, setting makedirs to True will allow Salt to 1356 create the parent directory 1357 user 1358 The user to own any directories made if makedirs is set to true. This 1359 defaults to the user salt is running as on the minion 1360 group 1361 The group ownership set on any directories made if makedirs is set to 1362 true. This defaults to the group salt is running as on the minion. On 1363 Windows, this is ignored 1364 dir_mode 1365 If directories are to be created, passing this option specifies the 1366 permissions for those directories. 1367 """ 1368 name = os.path.expanduser(name) 1369 1370 # Make sure that leading zeros stripped by YAML loader are added back 1371 dir_mode = salt.utils.files.normalize_mode(dir_mode) 1372 1373 user = _test_owner(kwargs, user=user) 1374 ret = {"name": name, "changes": {}, "result": True, "comment": ""} 1375 if not name: 1376 return _error(ret, "Must provide name to file.hardlink") 1377 1378 if user is None: 1379 user = __opts__["user"] 1380 1381 if salt.utils.platform.is_windows(): 1382 if group is not None: 1383 log.warning( 1384 "The group argument for %s has been ignored as this " 1385 "is a Windows system.", 1386 name, 1387 ) 1388 group = user 1389 1390 if group is None: 1391 if "user.info" in __salt__: 1392 group = __salt__["file.gid_to_group"]( 1393 __salt__["user.info"](user).get("gid", 0) 1394 ) 1395 else: 1396 group = user 1397 1398 preflight_errors = [] 1399 uid = __salt__["file.user_to_uid"](user) 1400 gid = __salt__["file.group_to_gid"](group) 1401 1402 if uid == "": 1403 preflight_errors.append("User {} does not exist".format(user)) 1404 1405 if gid == "": 1406 preflight_errors.append("Group {} does not exist".format(group)) 1407 1408 if not os.path.isabs(name): 1409 preflight_errors.append( 1410 "Specified file {} is not an absolute path".format(name) 1411 ) 1412 1413 if not os.path.isabs(target): 1414 preflight_errors.append( 1415 "Specified target {} is not an absolute path".format(target) 1416 ) 1417 1418 if preflight_errors: 1419 msg = ". ".join(preflight_errors) 1420 if len(preflight_errors) > 1: 1421 msg += "." 1422 return _error(ret, msg) 1423 1424 if __opts__["test"]: 1425 tresult, tcomment, tchanges = _hardlink_check(name, target, force) 1426 ret["result"] = tresult 1427 ret["comment"] = tcomment 1428 ret["changes"] = tchanges 1429 return ret 1430 1431 # We use zip_longest here because there's a number of issues in pylint's 1432 # tracker that complains about not linking the zip builtin. 1433 for direction, item in zip_longest(["to", "from"], [name, target]): 1434 if os.path.isdir(item): 1435 msg = "Unable to hard link {} directory {}".format(direction, item) 1436 return _error(ret, msg) 1437 1438 if not os.path.exists(target): 1439 msg = "Target {} for hard link does not exist".format(target) 1440 return _error(ret, msg) 1441 1442 # Check that the directory to write the hard link to exists 1443 if not os.path.isdir(os.path.dirname(name)): 1444 if makedirs: 1445 __salt__["file.makedirs"](name, user=user, group=group, mode=dir_mode) 1446 1447 else: 1448 return _error( 1449 ret, 1450 "Directory {} for hard link is not present".format( 1451 os.path.dirname(name) 1452 ), 1453 ) 1454 1455 # If file is not a hard link and we're actually overwriting it, then verify 1456 # that this was forced. 1457 if os.path.isfile(name) and not __salt__["file.is_hardlink"](name): 1458 1459 # Remove whatever is in the way. This should then hit the else case 1460 # of the file.is_hardlink check below 1461 if force: 1462 os.remove(name) 1463 ret["changes"]["forced"] = "File for hard link was forcibly replaced" 1464 1465 # Otherwise throw an error 1466 else: 1467 return _error( 1468 ret, "File exists where the hard link {} should be".format(name) 1469 ) 1470 1471 # If the file is a hard link, then we can simply rewrite its target since 1472 # nothing is really being lost here. 1473 if __salt__["file.is_hardlink"](name): 1474 1475 # If the inodes point to the same thing, then there's nothing to do 1476 # except for let the user know that this has already happened. 1477 if _hardlink_same(name, target): 1478 ret["result"] = True 1479 ret["comment"] = "Target of hard link {} is already pointing to {}".format( 1480 name, target 1481 ) 1482 return ret 1483 1484 # First remove the old hard link since a reference to it already exists 1485 os.remove(name) 1486 1487 # Now we can remake it 1488 try: 1489 __salt__["file.link"](target, name) 1490 1491 # Or not... 1492 except CommandExecutionError as E: 1493 ret["result"] = False 1494 ret["comment"] = "Unable to set target of hard link {} -> {}: {}".format( 1495 name, target, E 1496 ) 1497 return ret 1498 1499 # Good to go 1500 ret["result"] = True 1501 ret["comment"] = "Set target of hard link {} -> {}".format(name, target) 1502 ret["changes"]["new"] = name 1503 1504 # The link is not present, so simply make it 1505 elif not os.path.exists(name): 1506 try: 1507 __salt__["file.link"](target, name) 1508 1509 # Or not... 1510 except CommandExecutionError as E: 1511 ret["result"] = False 1512 ret["comment"] = "Unable to create new hard link {} -> {}: {}".format( 1513 name, target, E 1514 ) 1515 return ret 1516 1517 # Made a new hard link, things are ok 1518 ret["result"] = True 1519 ret["comment"] = "Created new hard link {} -> {}".format(name, target) 1520 ret["changes"]["new"] = name 1521 1522 return ret 1523 1524 1525def symlink( 1526 name, 1527 target, 1528 force=False, 1529 backupname=None, 1530 makedirs=False, 1531 user=None, 1532 group=None, 1533 mode=None, 1534 win_owner=None, 1535 win_perms=None, 1536 win_deny_perms=None, 1537 win_inheritance=None, 1538 **kwargs 1539): 1540 """ 1541 Create a symbolic link (symlink, soft link) 1542 1543 If the file already exists and is a symlink pointing to any location other 1544 than the specified target, the symlink will be replaced. If an entry with 1545 the same name exists then the state will return False. If the existing 1546 entry is desired to be replaced with a symlink pass force: True, if it is 1547 to be renamed, pass a backupname. 1548 1549 name 1550 The location of the symlink to create 1551 1552 target 1553 The location that the symlink points to 1554 1555 force 1556 If the name of the symlink exists and is not a symlink and 1557 force is set to False, the state will fail. If force is set to 1558 True, the existing entry in the way of the symlink file 1559 will be deleted to make room for the symlink, unless 1560 backupname is set, when it will be renamed 1561 1562 .. versionchanged:: 3000 1563 Force will now remove all types of existing file system entries, 1564 not just files, directories and symlinks. 1565 1566 backupname 1567 If the name of the symlink exists and is not a symlink, it will be 1568 renamed to the backupname. If the backupname already 1569 exists and force is False, the state will fail. Otherwise, the 1570 backupname will be removed first. 1571 An absolute path OR a basename file/directory name must be provided. 1572 The latter will be placed relative to the symlink destination's parent 1573 directory. 1574 1575 makedirs 1576 If the location of the symlink does not already have a parent directory 1577 then the state will fail, setting makedirs to True will allow Salt to 1578 create the parent directory 1579 1580 user 1581 The user to own the file, this defaults to the user salt is running as 1582 on the minion 1583 1584 group 1585 The group ownership set for the file, this defaults to the group salt 1586 is running as on the minion. On Windows, this is ignored 1587 1588 mode 1589 The permissions to set on this file, aka 644, 0775, 4664. Not supported 1590 on Windows. 1591 1592 The default mode for new files and directories corresponds umask of salt 1593 process. For existing files and directories it's not enforced. 1594 1595 win_owner 1596 The owner of the symlink and directories if ``makedirs`` is True. If 1597 this is not passed, ``user`` will be used. If ``user`` is not passed, 1598 the account under which Salt is running will be used. 1599 1600 .. versionadded:: 2017.7.7 1601 1602 win_perms 1603 A dictionary containing permissions to grant 1604 1605 .. versionadded:: 2017.7.7 1606 1607 win_deny_perms 1608 A dictionary containing permissions to deny 1609 1610 .. versionadded:: 2017.7.7 1611 1612 win_inheritance 1613 True to inherit permissions from parent, otherwise False 1614 1615 .. versionadded:: 2017.7.7 1616 """ 1617 name = os.path.expanduser(name) 1618 1619 # Make sure that leading zeros stripped by YAML loader are added back 1620 mode = salt.utils.files.normalize_mode(mode) 1621 1622 user = _test_owner(kwargs, user=user) 1623 ret = {"name": name, "changes": {}, "result": True, "comment": ""} 1624 if not name: 1625 return _error(ret, "Must provide name to file.symlink") 1626 1627 if user is None: 1628 user = __opts__["user"] 1629 1630 if salt.utils.platform.is_windows(): 1631 1632 # Make sure the user exists in Windows 1633 # Salt default is 'root' 1634 if not __salt__["user.info"](user): 1635 # User not found, use the account salt is running under 1636 # If username not found, use System 1637 user = __salt__["user.current"]() 1638 if not user: 1639 user = "SYSTEM" 1640 1641 # If win_owner is not passed, use user 1642 if win_owner is None: 1643 win_owner = user if user else None 1644 1645 # Group isn't relevant to Windows, use win_perms/win_deny_perms 1646 if group is not None: 1647 log.warning( 1648 "The group argument for %s has been ignored as this " 1649 "is a Windows system. Please use the `win_*` parameters to set " 1650 "permissions in Windows.", 1651 name, 1652 ) 1653 group = user 1654 1655 if group is None: 1656 if "user.info" in __salt__: 1657 group = __salt__["file.gid_to_group"]( 1658 __salt__["user.info"](user).get("gid", 0) 1659 ) 1660 else: 1661 group = user 1662 1663 preflight_errors = [] 1664 if salt.utils.platform.is_windows(): 1665 # Make sure the passed owner exists 1666 try: 1667 salt.utils.win_functions.get_sid_from_name(win_owner) 1668 except CommandExecutionError as exc: 1669 preflight_errors.append("User {} does not exist".format(win_owner)) 1670 1671 # Make sure users passed in win_perms exist 1672 if win_perms: 1673 for name_check in win_perms: 1674 try: 1675 salt.utils.win_functions.get_sid_from_name(name_check) 1676 except CommandExecutionError as exc: 1677 preflight_errors.append("User {} does not exist".format(name_check)) 1678 1679 # Make sure users passed in win_deny_perms exist 1680 if win_deny_perms: 1681 for name_check in win_deny_perms: 1682 try: 1683 salt.utils.win_functions.get_sid_from_name(name_check) 1684 except CommandExecutionError as exc: 1685 preflight_errors.append("User {} does not exist".format(name_check)) 1686 else: 1687 uid = __salt__["file.user_to_uid"](user) 1688 gid = __salt__["file.group_to_gid"](group) 1689 1690 if uid == "": 1691 preflight_errors.append("User {} does not exist".format(user)) 1692 1693 if gid == "": 1694 preflight_errors.append("Group {} does not exist".format(group)) 1695 1696 if not os.path.isabs(name): 1697 preflight_errors.append( 1698 "Specified file {} is not an absolute path".format(name) 1699 ) 1700 1701 if preflight_errors: 1702 msg = ". ".join(preflight_errors) 1703 if len(preflight_errors) > 1: 1704 msg += "." 1705 return _error(ret, msg) 1706 1707 tresult, tcomment, tchanges = _symlink_check( 1708 name, target, force, user, group, win_owner 1709 ) 1710 1711 if not os.path.isdir(os.path.dirname(name)): 1712 if makedirs: 1713 if __opts__["test"]: 1714 tcomment += "\n{} will be created".format(os.path.dirname(name)) 1715 else: 1716 try: 1717 _makedirs( 1718 name=name, 1719 user=user, 1720 group=group, 1721 dir_mode=mode, 1722 win_owner=win_owner, 1723 win_perms=win_perms, 1724 win_deny_perms=win_deny_perms, 1725 win_inheritance=win_inheritance, 1726 ) 1727 except CommandExecutionError as exc: 1728 return _error(ret, "Drive {} is not mapped".format(exc.message)) 1729 else: 1730 if __opts__["test"]: 1731 tcomment += "\nDirectory {} for symlink is not present".format( 1732 os.path.dirname(name) 1733 ) 1734 else: 1735 return _error( 1736 ret, 1737 "Directory {} for symlink is not present".format( 1738 os.path.dirname(name) 1739 ), 1740 ) 1741 1742 if __opts__["test"]: 1743 ret["result"] = tresult 1744 ret["comment"] = tcomment 1745 ret["changes"] = tchanges 1746 return ret 1747 1748 if __salt__["file.is_link"](name): 1749 # The link exists, verify that it matches the target 1750 if os.path.normpath(__salt__["file.readlink"](name)) != os.path.normpath( 1751 target 1752 ): 1753 # The target is wrong, delete the link 1754 os.remove(name) 1755 else: 1756 if _check_symlink_ownership(name, user, group, win_owner): 1757 # The link looks good! 1758 if salt.utils.platform.is_windows(): 1759 ret["comment"] = "Symlink {} is present and owned by {}".format( 1760 name, win_owner 1761 ) 1762 else: 1763 ret["comment"] = "Symlink {} is present and owned by {}:{}".format( 1764 name, user, group 1765 ) 1766 else: 1767 if _set_symlink_ownership(name, user, group, win_owner): 1768 if salt.utils.platform.is_windows(): 1769 ret["comment"] = "Set ownership of symlink {} to {}".format( 1770 name, win_owner 1771 ) 1772 ret["changes"]["ownership"] = win_owner 1773 else: 1774 ret["comment"] = "Set ownership of symlink {} to {}:{}".format( 1775 name, user, group 1776 ) 1777 ret["changes"]["ownership"] = "{}:{}".format(user, group) 1778 else: 1779 ret["result"] = False 1780 if salt.utils.platform.is_windows(): 1781 ret[ 1782 "comment" 1783 ] += "Failed to set ownership of symlink {} to {}".format( 1784 name, win_owner 1785 ) 1786 else: 1787 ret[ 1788 "comment" 1789 ] += "Failed to set ownership of symlink {} to {}:{}".format( 1790 name, user, group 1791 ) 1792 return ret 1793 1794 elif os.path.exists(name): 1795 # It is not a link, but a file, dir, socket, FIFO etc. 1796 if backupname is not None: 1797 if not os.path.isabs(backupname): 1798 if backupname == os.path.basename(backupname): 1799 backupname = os.path.join( 1800 os.path.dirname(os.path.normpath(name)), backupname 1801 ) 1802 else: 1803 return _error( 1804 ret, 1805 "Backupname must be an absolute path or a file name: {}".format( 1806 backupname 1807 ), 1808 ) 1809 # Make a backup first 1810 if os.path.lexists(backupname): 1811 if not force: 1812 return _error( 1813 ret, 1814 "Symlink & backup dest exists and Force not set. {} -> {} -" 1815 " backup: {}".format(name, target, backupname), 1816 ) 1817 else: 1818 __salt__["file.remove"](backupname) 1819 try: 1820 __salt__["file.move"](name, backupname) 1821 except Exception as exc: # pylint: disable=broad-except 1822 ret["changes"] = {} 1823 log.debug( 1824 "Encountered error renaming %s to %s", 1825 name, 1826 backupname, 1827 exc_info=True, 1828 ) 1829 return _error( 1830 ret, 1831 "Unable to rename {} to backup {} -> : {}".format( 1832 name, backupname, exc 1833 ), 1834 ) 1835 elif force: 1836 # Remove whatever is in the way 1837 if __salt__["file.is_link"](name): 1838 __salt__["file.remove"](name) 1839 ret["changes"]["forced"] = "Symlink was forcibly replaced" 1840 else: 1841 __salt__["file.remove"](name) 1842 else: 1843 # Otherwise throw an error 1844 fs_entry_type = ( 1845 "File" 1846 if os.path.isfile(name) 1847 else "Directory" 1848 if os.path.isdir(name) 1849 else "File system entry" 1850 ) 1851 return _error( 1852 ret, 1853 "{} exists where the symlink {} should be".format(fs_entry_type, name), 1854 ) 1855 1856 if not os.path.exists(name): 1857 # The link is not present, make it 1858 try: 1859 __salt__["file.symlink"](target, name) 1860 except OSError as exc: 1861 ret["result"] = False 1862 ret["comment"] = "Unable to create new symlink {} -> {}: {}".format( 1863 name, target, exc 1864 ) 1865 return ret 1866 else: 1867 ret["comment"] = "Created new symlink {} -> {}".format(name, target) 1868 ret["changes"]["new"] = name 1869 1870 if not _check_symlink_ownership(name, user, group, win_owner): 1871 if not _set_symlink_ownership(name, user, group, win_owner): 1872 ret["result"] = False 1873 ret["comment"] += ", but was unable to set ownership to {}:{}".format( 1874 user, group 1875 ) 1876 return ret 1877 1878 1879def absent(name, **kwargs): 1880 """ 1881 Make sure that the named file or directory is absent. If it exists, it will 1882 be deleted. This will work to reverse any of the functions in the file 1883 state module. If a directory is supplied, it will be recursively deleted. 1884 1885 name 1886 The path which should be deleted 1887 """ 1888 name = os.path.expanduser(name) 1889 1890 ret = {"name": name, "changes": {}, "result": True, "comment": ""} 1891 if not name: 1892 return _error(ret, "Must provide name to file.absent") 1893 if not os.path.isabs(name): 1894 return _error(ret, "Specified file {} is not an absolute path".format(name)) 1895 if name == "/": 1896 return _error(ret, 'Refusing to make "/" absent') 1897 if os.path.isfile(name) or os.path.islink(name): 1898 if __opts__["test"]: 1899 ret["result"] = None 1900 ret["changes"]["removed"] = name 1901 ret["comment"] = "File {} is set for removal".format(name) 1902 return ret 1903 try: 1904 if salt.utils.platform.is_windows(): 1905 __salt__["file.remove"](name, force=True) 1906 else: 1907 __salt__["file.remove"](name) 1908 ret["comment"] = "Removed file {}".format(name) 1909 ret["changes"]["removed"] = name 1910 return ret 1911 except CommandExecutionError as exc: 1912 return _error(ret, "{}".format(exc)) 1913 1914 elif os.path.isdir(name): 1915 if __opts__["test"]: 1916 ret["result"] = None 1917 ret["changes"]["removed"] = name 1918 ret["comment"] = "Directory {} is set for removal".format(name) 1919 return ret 1920 try: 1921 if salt.utils.platform.is_windows(): 1922 __salt__["file.remove"](name, force=True) 1923 else: 1924 __salt__["file.remove"](name) 1925 ret["comment"] = "Removed directory {}".format(name) 1926 ret["changes"]["removed"] = name 1927 return ret 1928 except OSError: 1929 return _error(ret, "Failed to remove directory {}".format(name)) 1930 1931 ret["comment"] = "File {} is not present".format(name) 1932 return ret 1933 1934 1935def tidied(name, age=0, matches=None, rmdirs=False, size=0, **kwargs): 1936 """ 1937 Remove unwanted files based on specific criteria. Multiple criteria 1938 are OR’d together, so a file that is too large but is not old enough 1939 will still get tidied. 1940 1941 If neither age nor size is given all files which match a pattern in 1942 matches will be removed. 1943 1944 name 1945 The directory tree that should be tidied 1946 1947 age 1948 Maximum age in days after which files are considered for removal 1949 1950 matches 1951 List of regular expressions to restrict what gets removed. Default: ['.*'] 1952 1953 rmdirs 1954 Whether or not it's allowed to remove directories 1955 1956 size 1957 Maximum allowed file size. Files greater or equal to this size are 1958 removed. Doesn't apply to directories or symbolic links 1959 1960 .. code-block:: yaml 1961 1962 cleanup: 1963 file.tidied: 1964 - name: /tmp/salt_test 1965 - rmdirs: True 1966 - matches: 1967 - foo 1968 - b.*r 1969 """ 1970 name = os.path.expanduser(name) 1971 1972 ret = {"name": name, "changes": {}, "result": True, "comment": ""} 1973 1974 # Check preconditions 1975 if not os.path.isabs(name): 1976 return _error(ret, "Specified file {} is not an absolute path".format(name)) 1977 if not os.path.isdir(name): 1978 return _error(ret, "{} does not exist or is not a directory.".format(name)) 1979 1980 # Define some variables 1981 todelete = [] 1982 today = date.today() 1983 1984 # Compile regular expressions 1985 if matches is None: 1986 matches = [".*"] 1987 progs = [] 1988 for regex in matches: 1989 progs.append(re.compile(regex)) 1990 1991 # Helper to match a given name against one or more pre-compiled regular 1992 # expressions 1993 def _matches(name): 1994 for prog in progs: 1995 if prog.match(name): 1996 return True 1997 return False 1998 1999 # Iterate over given directory tree, depth-first 2000 for root, dirs, files in os.walk(top=name, topdown=False): 2001 # Check criteria for the found files and directories 2002 for elem in files + dirs: 2003 myage = 0 2004 mysize = 0 2005 deleteme = True 2006 path = os.path.join(root, elem) 2007 if os.path.islink(path): 2008 # Get age of symlink (not symlinked file) 2009 myage = abs(today - date.fromtimestamp(os.lstat(path).st_atime)) 2010 elif elem in dirs: 2011 # Get age of directory, check if directories should be deleted at all 2012 myage = abs(today - date.fromtimestamp(os.path.getatime(path))) 2013 deleteme = rmdirs 2014 else: 2015 # Get age and size of regular file 2016 myage = abs(today - date.fromtimestamp(os.path.getatime(path))) 2017 mysize = os.path.getsize(path) 2018 # Verify against given criteria, collect all elements that should be removed 2019 if ( 2020 (mysize >= size or myage.days >= age) 2021 and _matches(name=elem) 2022 and deleteme 2023 ): 2024 todelete.append(path) 2025 2026 # Now delete the stuff 2027 if todelete: 2028 if __opts__["test"]: 2029 ret["result"] = None 2030 ret["comment"] = "{} is set for tidy".format(name) 2031 ret["changes"] = {"removed": todelete} 2032 return ret 2033 ret["changes"]["removed"] = [] 2034 # Iterate over collected items 2035 try: 2036 for path in todelete: 2037 if salt.utils.platform.is_windows(): 2038 __salt__["file.remove"](path, force=True) 2039 else: 2040 __salt__["file.remove"](path) 2041 # Remember what we've removed, will appear in the summary 2042 ret["changes"]["removed"].append(path) 2043 except CommandExecutionError as exc: 2044 return _error(ret, "{}".format(exc)) 2045 # Set comment for the summary 2046 ret["comment"] = "Removed {} files or directories from directory {}".format( 2047 len(todelete), name 2048 ) 2049 else: 2050 # Set comment in case there was nothing to remove 2051 ret["comment"] = "Nothing to remove from directory {}".format(name) 2052 return ret 2053 2054 2055def exists(name, **kwargs): 2056 """ 2057 Verify that the named file or directory is present or exists. 2058 Ensures pre-requisites outside of Salt's purview 2059 (e.g., keytabs, private keys, etc.) have been previously satisfied before 2060 deployment. 2061 2062 This function does not create the file if it doesn't exist, it will return 2063 an error. 2064 2065 name 2066 Absolute path which must exist 2067 """ 2068 name = os.path.expanduser(name) 2069 2070 ret = {"name": name, "changes": {}, "result": True, "comment": ""} 2071 if not name: 2072 return _error(ret, "Must provide name to file.exists") 2073 if not os.path.exists(name): 2074 return _error(ret, "Specified path {} does not exist".format(name)) 2075 2076 ret["comment"] = "Path {} exists".format(name) 2077 return ret 2078 2079 2080def missing(name, **kwargs): 2081 """ 2082 Verify that the named file or directory is missing, this returns True only 2083 if the named file is missing but does not remove the file if it is present. 2084 2085 name 2086 Absolute path which must NOT exist 2087 """ 2088 name = os.path.expanduser(name) 2089 2090 ret = {"name": name, "changes": {}, "result": True, "comment": ""} 2091 if not name: 2092 return _error(ret, "Must provide name to file.missing") 2093 if os.path.exists(name): 2094 return _error(ret, "Specified path {} exists".format(name)) 2095 2096 ret["comment"] = "Path {} is missing".format(name) 2097 return ret 2098 2099 2100def managed( 2101 name, 2102 source=None, 2103 source_hash="", 2104 source_hash_name=None, 2105 keep_source=True, 2106 user=None, 2107 group=None, 2108 mode=None, 2109 attrs=None, 2110 template=None, 2111 makedirs=False, 2112 dir_mode=None, 2113 context=None, 2114 replace=True, 2115 defaults=None, 2116 backup="", 2117 show_changes=True, 2118 create=True, 2119 contents=None, 2120 tmp_dir="", 2121 tmp_ext="", 2122 contents_pillar=None, 2123 contents_grains=None, 2124 contents_newline=True, 2125 contents_delimiter=":", 2126 encoding=None, 2127 encoding_errors="strict", 2128 allow_empty=True, 2129 follow_symlinks=True, 2130 check_cmd=None, 2131 skip_verify=False, 2132 selinux=None, 2133 win_owner=None, 2134 win_perms=None, 2135 win_deny_perms=None, 2136 win_inheritance=True, 2137 win_perms_reset=False, 2138 verify_ssl=True, 2139 **kwargs 2140): 2141 r""" 2142 Manage a given file, this function allows for a file to be downloaded from 2143 the salt master and potentially run through a templating system. 2144 2145 name 2146 The location of the file to manage, as an absolute path. 2147 2148 source 2149 The source file to download to the minion, this source file can be 2150 hosted on either the salt master server (``salt://``), the salt minion 2151 local file system (``/``), or on an HTTP or FTP server (``http(s)://``, 2152 ``ftp://``). 2153 2154 Both HTTPS and HTTP are supported as well as downloading directly 2155 from Amazon S3 compatible URLs with both pre-configured and automatic 2156 IAM credentials. (see s3.get state documentation) 2157 File retrieval from Openstack Swift object storage is supported via 2158 swift://container/object_path URLs, see swift.get documentation. 2159 For files hosted on the salt file server, if the file is located on 2160 the master in the directory named spam, and is called eggs, the source 2161 string is salt://spam/eggs. If source is left blank or None 2162 (use ~ in YAML), the file will be created as an empty file and 2163 the content will not be managed. This is also the case when a file 2164 already exists and the source is undefined; the contents of the file 2165 will not be changed or managed. If source is left blank or None, please 2166 also set replaced to False to make your intention explicit. 2167 2168 2169 If the file is hosted on a HTTP or FTP server then the source_hash 2170 argument is also required. 2171 2172 A list of sources can also be passed in to provide a default source and 2173 a set of fallbacks. The first source in the list that is found to exist 2174 will be used and subsequent entries in the list will be ignored. Source 2175 list functionality only supports local files and remote files hosted on 2176 the salt master server or retrievable via HTTP, HTTPS, or FTP. 2177 2178 .. code-block:: yaml 2179 2180 file_override_example: 2181 file.managed: 2182 - source: 2183 - salt://file_that_does_not_exist 2184 - salt://file_that_exists 2185 2186 source_hash 2187 This can be one of the following: 2188 1. a source hash string 2189 2. the URI of a file that contains source hash strings 2190 2191 The function accepts the first encountered long unbroken alphanumeric 2192 string of correct length as a valid hash, in order from most secure to 2193 least secure: 2194 2195 .. code-block:: text 2196 2197 Type Length 2198 ====== ====== 2199 sha512 128 2200 sha384 96 2201 sha256 64 2202 sha224 56 2203 sha1 40 2204 md5 32 2205 2206 **Using a Source Hash File** 2207 The file can contain several checksums for several files. Each line 2208 must contain both the file name and the hash. If no file name is 2209 matched, the first hash encountered will be used, otherwise the most 2210 secure hash with the correct source file name will be used. 2211 2212 When using a source hash file the source_hash argument needs to be a 2213 url, the standard download urls are supported, ftp, http, salt etc: 2214 2215 Example: 2216 2217 .. code-block:: yaml 2218 2219 tomdroid-src-0.7.3.tar.gz: 2220 file.managed: 2221 - name: /tmp/tomdroid-src-0.7.3.tar.gz 2222 - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz 2223 - source_hash: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.hash 2224 2225 The following lines are all supported formats: 2226 2227 .. code-block:: text 2228 2229 /etc/rc.conf ef6e82e4006dee563d98ada2a2a80a27 2230 sha254c8525aee419eb649f0233be91c151178b30f0dff8ebbdcc8de71b1d5c8bcc06a /etc/resolv.conf 2231 ead48423703509d37c4a90e6a0d53e143b6fc268 2232 2233 Debian file type ``*.dsc`` files are also supported. 2234 2235 **Inserting the Source Hash in the SLS Data** 2236 2237 The source_hash can be specified as a simple checksum, like so: 2238 2239 .. code-block:: yaml 2240 2241 tomdroid-src-0.7.3.tar.gz: 2242 file.managed: 2243 - name: /tmp/tomdroid-src-0.7.3.tar.gz 2244 - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz 2245 - source_hash: 79eef25f9b0b2c642c62b7f737d4f53f 2246 2247 .. note:: 2248 Releases prior to 2016.11.0 must also include the hash type, like 2249 in the below example: 2250 2251 .. code-block:: yaml 2252 2253 tomdroid-src-0.7.3.tar.gz: 2254 file.managed: 2255 - name: /tmp/tomdroid-src-0.7.3.tar.gz 2256 - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz 2257 - source_hash: md5=79eef25f9b0b2c642c62b7f737d4f53f 2258 2259 Known issues: 2260 If the remote server URL has the hash file as an apparent 2261 sub-directory of the source file, the module will discover that it 2262 has already cached a directory where a file should be cached. For 2263 example: 2264 2265 .. code-block:: yaml 2266 2267 tomdroid-src-0.7.3.tar.gz: 2268 file.managed: 2269 - name: /tmp/tomdroid-src-0.7.3.tar.gz 2270 - source: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz 2271 - source_hash: https://launchpad.net/tomdroid/beta/0.7.3/+download/tomdroid-src-0.7.3.tar.gz/+md5 2272 2273 source_hash_name 2274 When ``source_hash`` refers to a hash file, Salt will try to find the 2275 correct hash by matching the filename/URI associated with that hash. By 2276 default, Salt will look for the filename being managed. When managing a 2277 file at path ``/tmp/foo.txt``, then the following line in a hash file 2278 would match: 2279 2280 .. code-block:: text 2281 2282 acbd18db4cc2f85cedef654fccc4a4d8 foo.txt 2283 2284 However, sometimes a hash file will include multiple similar paths: 2285 2286 .. code-block:: text 2287 2288 37b51d194a7513e45b56f6524f2d51f2 ./dir1/foo.txt 2289 acbd18db4cc2f85cedef654fccc4a4d8 ./dir2/foo.txt 2290 73feffa4b7f6bb68e44cf984c85f6e88 ./dir3/foo.txt 2291 2292 In cases like this, Salt may match the incorrect hash. This argument 2293 can be used to tell Salt which filename to match, to ensure that the 2294 correct hash is identified. For example: 2295 2296 .. code-block:: yaml 2297 2298 /tmp/foo.txt: 2299 file.managed: 2300 - source: https://mydomain.tld/dir2/foo.txt 2301 - source_hash: https://mydomain.tld/hashes 2302 - source_hash_name: ./dir2/foo.txt 2303 2304 .. note:: 2305 This argument must contain the full filename entry from the 2306 checksum file, as this argument is meant to disambiguate matches 2307 for multiple files that have the same basename. So, in the 2308 example above, simply using ``foo.txt`` would not match. 2309 2310 .. versionadded:: 2016.3.5 2311 2312 keep_source 2313 Set to ``False`` to discard the cached copy of the source file once the 2314 state completes. This can be useful for larger files to keep them from 2315 taking up space in minion cache. However, keep in mind that discarding 2316 the source file will result in the state needing to re-download the 2317 source file if the state is run again. 2318 2319 .. versionadded:: 2017.7.3 2320 2321 user 2322 The user to own the file, this defaults to the user salt is running as 2323 on the minion 2324 2325 group 2326 The group ownership set for the file, this defaults to the group salt 2327 is running as on the minion. On Windows, this is ignored 2328 2329 mode 2330 The permissions to set on this file, e.g. ``644``, ``0775``, or 2331 ``4664``. 2332 2333 The default mode for new files and directories corresponds to the 2334 umask of the salt process. The mode of existing files and directories 2335 will only be changed if ``mode`` is specified. 2336 2337 .. note:: 2338 This option is **not** supported on Windows. 2339 2340 .. versionchanged:: 2016.11.0 2341 This option can be set to ``keep``, and Salt will keep the mode 2342 from the Salt fileserver. This is only supported when the 2343 ``source`` URL begins with ``salt://``, or for files local to the 2344 minion. Because the ``source`` option cannot be used with any of 2345 the ``contents`` options, setting the ``mode`` to ``keep`` is also 2346 incompatible with the ``contents`` options. 2347 2348 .. note:: keep does not work with salt-ssh. 2349 2350 As a consequence of how the files are transferred to the minion, and 2351 the inability to connect back to the master with salt-ssh, salt is 2352 unable to stat the file as it exists on the fileserver and thus 2353 cannot mirror the mode on the salt-ssh minion 2354 2355 attrs 2356 The attributes to have on this file, e.g. ``a``, ``i``. The attributes 2357 can be any or a combination of the following characters: 2358 ``aAcCdDeijPsStTu``. 2359 2360 .. note:: 2361 This option is **not** supported on Windows. 2362 2363 .. versionadded:: 2018.3.0 2364 2365 template 2366 If this setting is applied, the named templating engine will be used to 2367 render the downloaded file. The following templates are supported: 2368 2369 - :mod:`cheetah<salt.renderers.cheetah>` 2370 - :mod:`genshi<salt.renderers.genshi>` 2371 - :mod:`jinja<salt.renderers.jinja>` 2372 - :mod:`mako<salt.renderers.mako>` 2373 - :mod:`py<salt.renderers.py>` 2374 - :mod:`wempy<salt.renderers.wempy>` 2375 2376 makedirs 2377 If set to ``True``, then the parent directories will be created to 2378 facilitate the creation of the named file. If ``False``, and the parent 2379 directory of the destination file doesn't exist, the state will fail. 2380 2381 dir_mode 2382 If directories are to be created, passing this option specifies the 2383 permissions for those directories. If this is not set, directories 2384 will be assigned permissions by adding the execute bit to the mode of 2385 the files. 2386 2387 The default mode for new files and directories corresponds umask of salt 2388 process. For existing files and directories it's not enforced. 2389 2390 replace 2391 If set to ``False`` and the file already exists, the file will not be 2392 modified even if changes would otherwise be made. Permissions and 2393 ownership will still be enforced, however. 2394 2395 context 2396 Overrides default context variables passed to the template. 2397 2398 defaults 2399 Default context passed to the template. 2400 2401 backup 2402 Overrides the default backup mode for this specific file. See 2403 :ref:`backup_mode documentation <file-state-backups>` for more details. 2404 2405 show_changes 2406 Output a unified diff of the old file and the new file. If ``False`` 2407 return a boolean if any changes were made. 2408 2409 create 2410 If set to ``False``, then the file will only be managed if the file 2411 already exists on the system. 2412 2413 contents 2414 Specify the contents of the file. Cannot be used in combination with 2415 ``source``. Ignores hashes and does not use a templating engine. 2416 2417 This value can be either a single string, a multiline YAML string or a 2418 list of strings. If a list of strings, then the strings will be joined 2419 together with newlines in the resulting file. For example, the below 2420 two example states would result in identical file contents: 2421 2422 .. code-block:: yaml 2423 2424 /path/to/file1: 2425 file.managed: 2426 - contents: 2427 - This is line 1 2428 - This is line 2 2429 2430 /path/to/file2: 2431 file.managed: 2432 - contents: | 2433 This is line 1 2434 This is line 2 2435 2436 2437 contents_pillar 2438 .. versionadded:: 0.17.0 2439 .. versionchanged:: 2016.11.0 2440 contents_pillar can also be a list, and the pillars will be 2441 concatenated together to form one file. 2442 2443 2444 Operates like ``contents``, but draws from a value stored in pillar, 2445 using the pillar path syntax used in :mod:`pillar.get 2446 <salt.modules.pillar.get>`. This is useful when the pillar value 2447 contains newlines, as referencing a pillar variable using a jinja/mako 2448 template can result in YAML formatting issues due to the newlines 2449 causing indentation mismatches. 2450 2451 For example, the following could be used to deploy an SSH private key: 2452 2453 .. code-block:: yaml 2454 2455 /home/deployer/.ssh/id_rsa: 2456 file.managed: 2457 - user: deployer 2458 - group: deployer 2459 - mode: 600 2460 - attrs: a 2461 - contents_pillar: userdata:deployer:id_rsa 2462 2463 This would populate ``/home/deployer/.ssh/id_rsa`` with the contents of 2464 ``pillar['userdata']['deployer']['id_rsa']``. An example of this pillar 2465 setup would be like so: 2466 2467 .. code-block:: yaml 2468 2469 userdata: 2470 deployer: 2471 id_rsa: | 2472 -----BEGIN RSA PRIVATE KEY----- 2473 MIIEowIBAAKCAQEAoQiwO3JhBquPAalQF9qP1lLZNXVjYMIswrMe2HcWUVBgh+vY 2474 U7sCwx/dH6+VvNwmCoqmNnP+8gTPKGl1vgAObJAnMT623dMXjVKwnEagZPRJIxDy 2475 B/HaAre9euNiY3LvIzBTWRSeMfT+rWvIKVBpvwlgGrfgz70m0pqxu+UyFbAGLin+ 2476 GpxzZAMaFpZw4sSbIlRuissXZj/sHpQb8p9M5IeO4Z3rjkCP1cxI 2477 -----END RSA PRIVATE KEY----- 2478 2479 .. note:: 2480 The private key above is shortened to keep the example brief, but 2481 shows how to do multiline string in YAML. The key is followed by a 2482 pipe character, and the multiline string is indented two more 2483 spaces. 2484 2485 To avoid the hassle of creating an indented multiline YAML string, 2486 the :mod:`file_tree external pillar <salt.pillar.file_tree>` can 2487 be used instead. However, this will not work for binary files in 2488 Salt releases before 2015.8.4. 2489 2490 contents_grains 2491 .. versionadded:: 2014.7.0 2492 2493 Operates like ``contents``, but draws from a value stored in grains, 2494 using the grains path syntax used in :mod:`grains.get 2495 <salt.modules.grains.get>`. This functionality works similarly to 2496 ``contents_pillar``, but with grains. 2497 2498 For example, the following could be used to deploy a "message of the day" 2499 file: 2500 2501 .. code-block:: yaml 2502 2503 write_motd: 2504 file.managed: 2505 - name: /etc/motd 2506 - contents_grains: motd 2507 2508 This would populate ``/etc/motd`` file with the contents of the ``motd`` 2509 grain. The ``motd`` grain is not a default grain, and would need to be 2510 set prior to running the state: 2511 2512 .. code-block:: bash 2513 2514 salt '*' grains.set motd 'Welcome! This system is managed by Salt.' 2515 2516 contents_newline 2517 .. versionadded:: 2014.7.0 2518 .. versionchanged:: 2015.8.4 2519 This option is now ignored if the contents being deployed contain 2520 binary data. 2521 2522 If ``True``, files managed using ``contents``, ``contents_pillar``, or 2523 ``contents_grains`` will have a newline added to the end of the file if 2524 one is not present. Setting this option to ``False`` will ensure the 2525 final line, or entry, does not contain a new line. If the last line, or 2526 entry in the file does contain a new line already, this option will not 2527 remove it. 2528 2529 contents_delimiter 2530 .. versionadded:: 2015.8.4 2531 2532 Can be used to specify an alternate delimiter for ``contents_pillar`` 2533 or ``contents_grains``. This delimiter will be passed through to 2534 :py:func:`pillar.get <salt.modules.pillar.get>` or :py:func:`grains.get 2535 <salt.modules.grains.get>` when retrieving the contents. 2536 2537 encoding 2538 If specified, then the specified encoding will be used. Otherwise, the 2539 file will be encoded using the system locale (usually UTF-8). See 2540 https://docs.python.org/3/library/codecs.html#standard-encodings for 2541 the list of available encodings. 2542 2543 .. versionadded:: 2017.7.0 2544 2545 encoding_errors 2546 Error encoding scheme. Default is ```'strict'```. 2547 See https://docs.python.org/2/library/codecs.html#codec-base-classes 2548 for the list of available schemes. 2549 2550 .. versionadded:: 2017.7.0 2551 2552 allow_empty 2553 .. versionadded:: 2015.8.4 2554 2555 If set to ``False``, then the state will fail if the contents specified 2556 by ``contents_pillar`` or ``contents_grains`` are empty. 2557 2558 follow_symlinks 2559 .. versionadded:: 2014.7.0 2560 2561 If the desired path is a symlink follow it and make changes to the 2562 file to which the symlink points. 2563 2564 check_cmd 2565 .. versionadded:: 2014.7.0 2566 2567 The specified command will be run with an appended argument of a 2568 *temporary* file containing the new managed contents. If the command 2569 exits with a zero status the new managed contents will be written to 2570 the managed destination. If the command exits with a nonzero exit 2571 code, the state will fail and no changes will be made to the file. 2572 2573 For example, the following could be used to verify sudoers before making 2574 changes: 2575 2576 .. code-block:: yaml 2577 2578 /etc/sudoers: 2579 file.managed: 2580 - user: root 2581 - group: root 2582 - mode: 0440 2583 - attrs: i 2584 - source: salt://sudoers/files/sudoers.jinja 2585 - template: jinja 2586 - check_cmd: /usr/sbin/visudo -c -f 2587 2588 **NOTE**: This ``check_cmd`` functions differently than the requisite 2589 ``check_cmd``. 2590 2591 tmp_dir 2592 Directory for temp file created by ``check_cmd``. Useful for checkers 2593 dependent on config file location (e.g. daemons restricted to their 2594 own config directories by an apparmor profile). 2595 2596 .. code-block:: yaml 2597 2598 /etc/dhcp/dhcpd.conf: 2599 file.managed: 2600 - user: root 2601 - group: root 2602 - mode: 0755 2603 - tmp_dir: '/etc/dhcp' 2604 - contents: "# Managed by Salt" 2605 - check_cmd: dhcpd -t -cf 2606 2607 tmp_ext 2608 Suffix for temp file created by ``check_cmd``. Useful for checkers 2609 dependent on config file extension (e.g. the init-checkconf upstart 2610 config checker). 2611 2612 .. code-block:: yaml 2613 2614 /etc/init/test.conf: 2615 file.managed: 2616 - user: root 2617 - group: root 2618 - mode: 0440 2619 - tmp_ext: '.conf' 2620 - contents: 2621 - 'description "Salt Minion"' 2622 - 'start on started mountall' 2623 - 'stop on shutdown' 2624 - 'respawn' 2625 - 'exec salt-minion' 2626 - check_cmd: init-checkconf -f 2627 2628 skip_verify 2629 If ``True``, hash verification of remote file sources (``http://``, 2630 ``https://``, ``ftp://``) will be skipped, and the ``source_hash`` 2631 argument will be ignored. 2632 2633 .. versionadded:: 2016.3.0 2634 2635 selinux 2636 Allows setting the selinux user, role, type, and range of a managed file 2637 2638 .. code-block:: yaml 2639 2640 /tmp/selinux.test 2641 file.managed: 2642 - user: root 2643 - selinux: 2644 seuser: system_u 2645 serole: object_r 2646 setype: system_conf_t 2647 seranage: s0 2648 2649 .. versionadded:: 3000 2650 2651 win_owner 2652 The owner of the directory. If this is not passed, user will be used. If 2653 user is not passed, the account under which Salt is running will be 2654 used. 2655 2656 .. versionadded:: 2017.7.0 2657 2658 win_perms 2659 A dictionary containing permissions to grant and their propagation. For 2660 example: ``{'Administrators': {'perms': 'full_control'}}`` Can be a 2661 single basic perm or a list of advanced perms. ``perms`` must be 2662 specified. ``applies_to`` does not apply to file objects. 2663 2664 .. versionadded:: 2017.7.0 2665 2666 win_deny_perms 2667 A dictionary containing permissions to deny and their propagation. For 2668 example: ``{'Administrators': {'perms': 'full_control'}}`` Can be a 2669 single basic perm or a list of advanced perms. ``perms`` must be 2670 specified. ``applies_to`` does not apply to file objects. 2671 2672 .. versionadded:: 2017.7.0 2673 2674 win_inheritance 2675 True to inherit permissions from the parent directory, False not to 2676 inherit permission. 2677 2678 .. versionadded:: 2017.7.0 2679 2680 win_perms_reset 2681 If ``True`` the existing DACL will be cleared and replaced with the 2682 settings defined in this function. If ``False``, new entries will be 2683 appended to the existing DACL. Default is ``False``. 2684 2685 .. versionadded:: 2018.3.0 2686 2687 Here's an example using the above ``win_*`` parameters: 2688 2689 .. code-block:: yaml 2690 2691 create_config_file: 2692 file.managed: 2693 - name: C:\config\settings.cfg 2694 - source: salt://settings.cfg 2695 - win_owner: Administrators 2696 - win_perms: 2697 # Basic Permissions 2698 dev_ops: 2699 perms: full_control 2700 # List of advanced permissions 2701 appuser: 2702 perms: 2703 - read_attributes 2704 - read_ea 2705 - create_folders 2706 - read_permissions 2707 joe_snuffy: 2708 perms: read 2709 - win_deny_perms: 2710 fred_snuffy: 2711 perms: full_control 2712 - win_inheritance: False 2713 2714 verify_ssl 2715 If ``False``, remote https file sources (``https://``) and source_hash 2716 will not attempt to validate the servers certificate. Default is True. 2717 2718 .. versionadded:: 3002 2719 """ 2720 if "env" in kwargs: 2721 # "env" is not supported; Use "saltenv". 2722 kwargs.pop("env") 2723 2724 name = os.path.expanduser(name) 2725 2726 ret = {"changes": {}, "comment": "", "name": name, "result": True} 2727 2728 if not name: 2729 return _error(ret, "Destination file name is required") 2730 2731 if mode is not None and salt.utils.platform.is_windows(): 2732 return _error(ret, "The 'mode' option is not supported on Windows") 2733 2734 if attrs is not None and salt.utils.platform.is_windows(): 2735 return _error(ret, "The 'attrs' option is not supported on Windows") 2736 2737 if selinux is not None and not salt.utils.platform.is_linux(): 2738 return _error(ret, "The 'selinux' option is only supported on Linux") 2739 2740 if selinux: 2741 seuser = selinux.get("seuser", None) 2742 serole = selinux.get("serole", None) 2743 setype = selinux.get("setype", None) 2744 serange = selinux.get("serange", None) 2745 else: 2746 seuser = serole = setype = serange = None 2747 2748 try: 2749 keep_mode = mode.lower() == "keep" 2750 if keep_mode: 2751 # We're not hard-coding the mode, so set it to None 2752 mode = None 2753 except AttributeError: 2754 keep_mode = False 2755 2756 # Make sure that any leading zeros stripped by YAML loader are added back 2757 mode = salt.utils.files.normalize_mode(mode) 2758 2759 contents_count = len( 2760 [x for x in (contents, contents_pillar, contents_grains) if x is not None] 2761 ) 2762 2763 if source and contents_count > 0: 2764 return _error( 2765 ret, 2766 "'source' cannot be used in combination with 'contents', " 2767 "'contents_pillar', or 'contents_grains'", 2768 ) 2769 elif keep_mode and contents_count > 0: 2770 return _error( 2771 ret, 2772 "Mode preservation cannot be used in combination with 'contents', " 2773 "'contents_pillar', or 'contents_grains'", 2774 ) 2775 elif contents_count > 1: 2776 return _error( 2777 ret, 2778 "Only one of 'contents', 'contents_pillar', and " 2779 "'contents_grains' is permitted", 2780 ) 2781 2782 # If no source is specified, set replace to False, as there is nothing 2783 # with which to replace the file. 2784 if not source and contents_count == 0 and replace: 2785 replace = False 2786 log.warning( 2787 "State for file: %s - Neither 'source' nor 'contents' nor " 2788 "'contents_pillar' nor 'contents_grains' was defined, yet " 2789 "'replace' was set to 'True'. As there is no source to " 2790 "replace the file with, 'replace' has been set to 'False' to " 2791 "avoid reading the file unnecessarily.", 2792 name, 2793 ) 2794 2795 if "file_mode" in kwargs: 2796 ret.setdefault("warnings", []).append( 2797 "The 'file_mode' argument will be ignored. " 2798 "Please use 'mode' instead to set file permissions." 2799 ) 2800 2801 # Use this below to avoid multiple '\0' checks and save some CPU cycles 2802 if contents_pillar is not None: 2803 if isinstance(contents_pillar, list): 2804 list_contents = [] 2805 for nextp in contents_pillar: 2806 nextc = __salt__["pillar.get"]( 2807 nextp, __NOT_FOUND, delimiter=contents_delimiter 2808 ) 2809 if nextc is __NOT_FOUND: 2810 return _error(ret, "Pillar {} does not exist".format(nextp)) 2811 list_contents.append(nextc) 2812 use_contents = os.linesep.join(list_contents) 2813 else: 2814 use_contents = __salt__["pillar.get"]( 2815 contents_pillar, __NOT_FOUND, delimiter=contents_delimiter 2816 ) 2817 if use_contents is __NOT_FOUND: 2818 return _error(ret, "Pillar {} does not exist".format(contents_pillar)) 2819 2820 elif contents_grains is not None: 2821 if isinstance(contents_grains, list): 2822 list_contents = [] 2823 for nextg in contents_grains: 2824 nextc = __salt__["grains.get"]( 2825 nextg, __NOT_FOUND, delimiter=contents_delimiter 2826 ) 2827 if nextc is __NOT_FOUND: 2828 return _error(ret, "Grain {} does not exist".format(nextc)) 2829 list_contents.append(nextc) 2830 use_contents = os.linesep.join(list_contents) 2831 else: 2832 use_contents = __salt__["grains.get"]( 2833 contents_grains, __NOT_FOUND, delimiter=contents_delimiter 2834 ) 2835 if use_contents is __NOT_FOUND: 2836 return _error(ret, "Grain {} does not exist".format(contents_grains)) 2837 2838 elif contents is not None: 2839 use_contents = contents 2840 2841 else: 2842 use_contents = None 2843 2844 if use_contents is not None: 2845 if not allow_empty and not use_contents: 2846 if contents_pillar: 2847 contents_id = "contents_pillar {}".format(contents_pillar) 2848 elif contents_grains: 2849 contents_id = "contents_grains {}".format(contents_grains) 2850 else: 2851 contents_id = "'contents'" 2852 return _error( 2853 ret, 2854 "{} value would result in empty contents. Set allow_empty " 2855 "to True to allow the managed file to be empty.".format(contents_id), 2856 ) 2857 2858 try: 2859 validated_contents = _validate_str_list(use_contents, encoding=encoding) 2860 if not validated_contents: 2861 return _error( 2862 ret, 2863 "Contents specified by contents/contents_pillar/" 2864 "contents_grains is not a string or list of strings, and " 2865 "is not binary data. SLS is likely malformed.", 2866 ) 2867 contents = "" 2868 for part in validated_contents: 2869 for line in part.splitlines(): 2870 contents += line.rstrip("\n").rstrip("\r") + os.linesep 2871 if not contents_newline: 2872 # If contents newline is set to False, strip out the newline 2873 # character and carriage return character 2874 contents = contents.rstrip("\n").rstrip("\r") 2875 2876 except UnicodeDecodeError: 2877 # Either something terrible happened, or we have binary data. 2878 if template: 2879 return _error( 2880 ret, 2881 "Contents specified by contents/contents_pillar/" 2882 "contents_grains appears to be binary data, and" 2883 " as will not be able to be treated as a Jinja" 2884 " template.", 2885 ) 2886 contents = use_contents 2887 if template: 2888 contents = __salt__["file.apply_template_on_contents"]( 2889 contents, 2890 template=template, 2891 context=context, 2892 defaults=defaults, 2893 saltenv=__env__, 2894 ) 2895 if not isinstance(contents, str): 2896 if "result" in contents: 2897 ret["result"] = contents["result"] 2898 else: 2899 ret["result"] = False 2900 if "comment" in contents: 2901 ret["comment"] = contents["comment"] 2902 else: 2903 ret["comment"] = "Error while applying template on contents" 2904 return ret 2905 2906 user = _test_owner(kwargs, user=user) 2907 if salt.utils.platform.is_windows(): 2908 2909 # If win_owner not passed, use user 2910 if win_owner is None: 2911 win_owner = user if user else None 2912 2913 # Group isn't relevant to Windows, use win_perms/win_deny_perms 2914 if group is not None: 2915 log.warning( 2916 "The group argument for %s has been ignored as this is " 2917 "a Windows system. Please use the `win_*` parameters to set " 2918 "permissions in Windows.", 2919 name, 2920 ) 2921 group = user 2922 2923 if not create: 2924 if not os.path.isfile(name): 2925 # Don't create a file that is not already present 2926 ret[ 2927 "comment" 2928 ] = "File {} is not present and is not set for creation".format(name) 2929 return ret 2930 u_check = _check_user(user, group) 2931 if u_check: 2932 # The specified user or group do not exist 2933 return _error(ret, u_check) 2934 if not os.path.isabs(name): 2935 return _error(ret, "Specified file {} is not an absolute path".format(name)) 2936 2937 if os.path.isdir(name): 2938 ret["comment"] = "Specified target {} is a directory".format(name) 2939 ret["result"] = False 2940 return ret 2941 2942 if context is None: 2943 context = {} 2944 elif not isinstance(context, dict): 2945 return _error(ret, "Context must be formed as a dict") 2946 if defaults and not isinstance(defaults, dict): 2947 return _error(ret, "Defaults must be formed as a dict") 2948 2949 if not replace and os.path.exists(name): 2950 ret_perms = {} 2951 # Check and set the permissions if necessary 2952 if salt.utils.platform.is_windows(): 2953 ret = __salt__["file.check_perms"]( 2954 path=name, 2955 ret=ret, 2956 owner=win_owner, 2957 grant_perms=win_perms, 2958 deny_perms=win_deny_perms, 2959 inheritance=win_inheritance, 2960 reset=win_perms_reset, 2961 ) 2962 else: 2963 ret, ret_perms = __salt__["file.check_perms"]( 2964 name, 2965 ret, 2966 user, 2967 group, 2968 mode, 2969 attrs, 2970 follow_symlinks, 2971 seuser=seuser, 2972 serole=serole, 2973 setype=setype, 2974 serange=serange, 2975 ) 2976 if __opts__["test"]: 2977 if ( 2978 mode 2979 and isinstance(ret_perms, dict) 2980 and "lmode" in ret_perms 2981 and mode != ret_perms["lmode"] 2982 ): 2983 ret["comment"] = ( 2984 "File {} will be updated with permissions " 2985 "{} from its current " 2986 "state of {}".format(name, mode, ret_perms["lmode"]) 2987 ) 2988 else: 2989 ret["comment"] = "File {} not updated".format(name) 2990 elif not ret["changes"] and ret["result"]: 2991 ret[ 2992 "comment" 2993 ] = "File {} exists with proper permissions. No changes made.".format(name) 2994 return ret 2995 2996 accum_data, _ = _load_accumulators() 2997 if name in accum_data: 2998 if not context: 2999 context = {} 3000 context["accumulator"] = accum_data[name] 3001 3002 try: 3003 if __opts__["test"]: 3004 if "file.check_managed_changes" in __salt__: 3005 ret["changes"] = __salt__["file.check_managed_changes"]( 3006 name, 3007 source, 3008 source_hash, 3009 source_hash_name, 3010 user, 3011 group, 3012 mode, 3013 attrs, 3014 template, 3015 context, 3016 defaults, 3017 __env__, 3018 contents, 3019 skip_verify, 3020 keep_mode, 3021 seuser=seuser, 3022 serole=serole, 3023 setype=setype, 3024 serange=serange, 3025 verify_ssl=verify_ssl, 3026 **kwargs 3027 ) 3028 3029 if salt.utils.platform.is_windows(): 3030 try: 3031 ret = __salt__["file.check_perms"]( 3032 path=name, 3033 ret=ret, 3034 owner=win_owner, 3035 grant_perms=win_perms, 3036 deny_perms=win_deny_perms, 3037 inheritance=win_inheritance, 3038 reset=win_perms_reset, 3039 ) 3040 except CommandExecutionError as exc: 3041 if exc.strerror.startswith("Path not found"): 3042 ret["changes"]["newfile"] = name 3043 3044 if isinstance(ret["changes"], tuple): 3045 ret["result"], ret["comment"] = ret["changes"] 3046 elif ret["changes"]: 3047 ret["result"] = None 3048 ret["comment"] = "The file {} is set to be changed".format(name) 3049 ret["comment"] += ( 3050 "\nNote: No changes made, actual changes may\n" 3051 "be different due to other states." 3052 ) 3053 if "diff" in ret["changes"] and not show_changes: 3054 ret["changes"]["diff"] = "<show_changes=False>" 3055 else: 3056 ret["result"] = True 3057 ret["comment"] = "The file {} is in the correct state".format(name) 3058 3059 return ret 3060 3061 # If the source is a list then find which file exists 3062 source, source_hash = __salt__["file.source_list"](source, source_hash, __env__) 3063 except CommandExecutionError as exc: 3064 ret["result"] = False 3065 ret["comment"] = "Unable to manage file: {}".format(exc) 3066 return ret 3067 3068 # Gather the source file from the server 3069 try: 3070 sfn, source_sum, comment_ = __salt__["file.get_managed"]( 3071 name, 3072 template, 3073 source, 3074 source_hash, 3075 source_hash_name, 3076 user, 3077 group, 3078 mode, 3079 attrs, 3080 __env__, 3081 context, 3082 defaults, 3083 skip_verify, 3084 verify_ssl=verify_ssl, 3085 **kwargs 3086 ) 3087 except Exception as exc: # pylint: disable=broad-except 3088 ret["changes"] = {} 3089 log.debug(traceback.format_exc()) 3090 return _error(ret, "Unable to manage file: {}".format(exc)) 3091 3092 tmp_filename = None 3093 3094 if check_cmd: 3095 tmp_filename = salt.utils.files.mkstemp(suffix=tmp_ext, dir=tmp_dir) 3096 3097 # if exists copy existing file to tmp to compare 3098 if __salt__["file.file_exists"](name): 3099 try: 3100 __salt__["file.copy"](name, tmp_filename) 3101 except Exception as exc: # pylint: disable=broad-except 3102 return _error( 3103 ret, 3104 "Unable to copy file {} to {}: {}".format(name, tmp_filename, exc), 3105 ) 3106 3107 try: 3108 ret = __salt__["file.manage_file"]( 3109 tmp_filename, 3110 sfn, 3111 ret, 3112 source, 3113 source_sum, 3114 user, 3115 group, 3116 mode, 3117 attrs, 3118 __env__, 3119 backup, 3120 makedirs, 3121 template, 3122 show_changes, 3123 contents, 3124 dir_mode, 3125 follow_symlinks, 3126 skip_verify, 3127 keep_mode, 3128 win_owner=win_owner, 3129 win_perms=win_perms, 3130 win_deny_perms=win_deny_perms, 3131 win_inheritance=win_inheritance, 3132 win_perms_reset=win_perms_reset, 3133 encoding=encoding, 3134 encoding_errors=encoding_errors, 3135 seuser=seuser, 3136 serole=serole, 3137 setype=setype, 3138 serange=serange, 3139 **kwargs 3140 ) 3141 except Exception as exc: # pylint: disable=broad-except 3142 ret["changes"] = {} 3143 log.debug(traceback.format_exc()) 3144 salt.utils.files.remove(tmp_filename) 3145 if not keep_source: 3146 if ( 3147 not sfn 3148 and source 3149 and urllib.parse.urlparse(source).scheme == "salt" 3150 ): 3151 # The file would not have been cached until manage_file was 3152 # run, so check again here for a cached copy. 3153 sfn = __salt__["cp.is_cached"](source, __env__) 3154 if sfn: 3155 salt.utils.files.remove(sfn) 3156 return _error(ret, "Unable to check_cmd file: {}".format(exc)) 3157 3158 # file being updated to verify using check_cmd 3159 if ret["changes"]: 3160 # Reset ret 3161 ret = {"changes": {}, "comment": "", "name": name, "result": True} 3162 3163 check_cmd_opts = {} 3164 if "shell" in __grains__: 3165 check_cmd_opts["shell"] = __grains__["shell"] 3166 3167 cret = mod_run_check_cmd(check_cmd, tmp_filename, **check_cmd_opts) 3168 if isinstance(cret, dict): 3169 ret.update(cret) 3170 salt.utils.files.remove(tmp_filename) 3171 return ret 3172 3173 # Since we generated a new tempfile and we are not returning here 3174 # lets change the original sfn to the new tempfile or else we will 3175 # get file not found 3176 3177 sfn = tmp_filename 3178 3179 else: 3180 ret = {"changes": {}, "comment": "", "name": name, "result": True} 3181 3182 if comment_ and contents is None: 3183 return _error(ret, comment_) 3184 else: 3185 try: 3186 return __salt__["file.manage_file"]( 3187 name, 3188 sfn, 3189 ret, 3190 source, 3191 source_sum, 3192 user, 3193 group, 3194 mode, 3195 attrs, 3196 __env__, 3197 backup, 3198 makedirs, 3199 template, 3200 show_changes, 3201 contents, 3202 dir_mode, 3203 follow_symlinks, 3204 skip_verify, 3205 keep_mode, 3206 win_owner=win_owner, 3207 win_perms=win_perms, 3208 win_deny_perms=win_deny_perms, 3209 win_inheritance=win_inheritance, 3210 win_perms_reset=win_perms_reset, 3211 encoding=encoding, 3212 encoding_errors=encoding_errors, 3213 seuser=seuser, 3214 serole=serole, 3215 setype=setype, 3216 serange=serange, 3217 **kwargs 3218 ) 3219 except Exception as exc: # pylint: disable=broad-except 3220 ret["changes"] = {} 3221 log.debug(traceback.format_exc()) 3222 return _error(ret, "Unable to manage file: {}".format(exc)) 3223 finally: 3224 if tmp_filename: 3225 salt.utils.files.remove(tmp_filename) 3226 if not keep_source: 3227 if ( 3228 not sfn 3229 and source 3230 and urllib.parse.urlparse(source).scheme == "salt" 3231 ): 3232 # The file would not have been cached until manage_file was 3233 # run, so check again here for a cached copy. 3234 sfn = __salt__["cp.is_cached"](source, __env__) 3235 if sfn: 3236 salt.utils.files.remove(sfn) 3237 3238 3239_RECURSE_TYPES = ["user", "group", "mode", "ignore_files", "ignore_dirs", "silent"] 3240 3241 3242def _get_recurse_set(recurse): 3243 """ 3244 Converse *recurse* definition to a set of strings. 3245 3246 Raises TypeError or ValueError when *recurse* has wrong structure. 3247 """ 3248 if not recurse: 3249 return set() 3250 if not isinstance(recurse, list): 3251 raise TypeError('"recurse" must be formed as a list of strings') 3252 try: 3253 recurse_set = set(recurse) 3254 except TypeError: # non-hashable elements 3255 recurse_set = None 3256 if recurse_set is None or not set(_RECURSE_TYPES) >= recurse_set: 3257 raise ValueError( 3258 'Types for "recurse" limited to {}.'.format( 3259 ", ".join('"{}"'.format(rtype) for rtype in _RECURSE_TYPES) 3260 ) 3261 ) 3262 if "ignore_files" in recurse_set and "ignore_dirs" in recurse_set: 3263 raise ValueError( 3264 'Must not specify "recurse" options "ignore_files"' 3265 ' and "ignore_dirs" at the same time.' 3266 ) 3267 return recurse_set 3268 3269 3270def _depth_limited_walk(top, max_depth=None): 3271 """ 3272 Walk the directory tree under root up till reaching max_depth. 3273 With max_depth=None (default), do not limit depth. 3274 """ 3275 for root, dirs, files in salt.utils.path.os_walk(top): 3276 if max_depth is not None: 3277 rel_depth = root.count(os.path.sep) - top.count(os.path.sep) 3278 if rel_depth >= max_depth: 3279 del dirs[:] 3280 yield (str(root), list(dirs), list(files)) 3281 3282 3283def directory( 3284 name, 3285 user=None, 3286 group=None, 3287 recurse=None, 3288 max_depth=None, 3289 dir_mode=None, 3290 file_mode=None, 3291 makedirs=False, 3292 clean=False, 3293 require=None, 3294 exclude_pat=None, 3295 follow_symlinks=False, 3296 force=False, 3297 backupname=None, 3298 allow_symlink=True, 3299 children_only=False, 3300 win_owner=None, 3301 win_perms=None, 3302 win_deny_perms=None, 3303 win_inheritance=True, 3304 win_perms_reset=False, 3305 **kwargs 3306): 3307 r""" 3308 Ensure that a named directory is present and has the right perms 3309 3310 name 3311 The location to create or manage a directory, as an absolute path 3312 3313 user 3314 The user to own the directory; this defaults to the user salt is 3315 running as on the minion 3316 3317 group 3318 The group ownership set for the directory; this defaults to the group 3319 salt is running as on the minion. On Windows, this is ignored 3320 3321 recurse 3322 Enforce user/group ownership and mode of directory recursively. Accepts 3323 a list of strings representing what you would like to recurse. If 3324 ``mode`` is defined, will recurse on both ``file_mode`` and ``dir_mode`` if 3325 they are defined. If ``ignore_files`` or ``ignore_dirs`` is included, files or 3326 directories will be left unchanged respectively. 3327 directories will be left unchanged respectively. If ``silent`` is defined, 3328 individual file/directory change notifications will be suppressed. 3329 3330 Example: 3331 3332 .. code-block:: yaml 3333 3334 /var/log/httpd: 3335 file.directory: 3336 - user: root 3337 - group: root 3338 - dir_mode: 755 3339 - file_mode: 644 3340 - recurse: 3341 - user 3342 - group 3343 - mode 3344 3345 Leave files or directories unchanged: 3346 3347 .. code-block:: yaml 3348 3349 /var/log/httpd: 3350 file.directory: 3351 - user: root 3352 - group: root 3353 - dir_mode: 755 3354 - file_mode: 644 3355 - recurse: 3356 - user 3357 - group 3358 - mode 3359 - ignore_dirs 3360 3361 .. versionadded:: 2015.5.0 3362 3363 max_depth 3364 Limit the recursion depth. The default is no limit=None. 3365 'max_depth' and 'clean' are mutually exclusive. 3366 3367 .. versionadded:: 2016.11.0 3368 3369 dir_mode / mode 3370 The permissions mode to set any directories created. Not supported on 3371 Windows. 3372 3373 The default mode for new files and directories corresponds umask of salt 3374 process. For existing files and directories it's not enforced. 3375 3376 file_mode 3377 The permissions mode to set any files created if 'mode' is run in 3378 'recurse'. This defaults to dir_mode. Not supported on Windows. 3379 3380 The default mode for new files and directories corresponds umask of salt 3381 process. For existing files and directories it's not enforced. 3382 3383 makedirs 3384 If the directory is located in a path without a parent directory, then 3385 the state will fail. If makedirs is set to True, then the parent 3386 directories will be created to facilitate the creation of the named 3387 file. 3388 3389 clean 3390 Make sure that only files that are set up by salt and required by this 3391 function are kept. If this option is set then everything in this 3392 directory will be deleted unless it is required. 3393 'clean' and 'max_depth' are mutually exclusive. 3394 3395 require 3396 Require other resources such as packages or files 3397 3398 exclude_pat 3399 When 'clean' is set to True, exclude this pattern from removal list 3400 and preserve in the destination. 3401 3402 follow_symlinks 3403 If the desired path is a symlink (or ``recurse`` is defined and a 3404 symlink is encountered while recursing), follow it and check the 3405 permissions of the directory/file to which the symlink points. 3406 3407 .. versionadded:: 2014.1.4 3408 3409 .. versionchanged:: 3001.1 3410 If set to False symlinks permissions are ignored on Linux systems 3411 because it does not support permissions modification. Symlinks 3412 permissions are always 0o777 on Linux. 3413 3414 force 3415 If the name of the directory exists and is not a directory and 3416 force is set to False, the state will fail. If force is set to 3417 True, the file in the way of the directory will be deleted to 3418 make room for the directory, unless backupname is set, 3419 then it will be renamed. 3420 3421 .. versionadded:: 2014.7.0 3422 3423 backupname 3424 If the name of the directory exists and is not a directory, it will be 3425 renamed to the backupname. If the backupname already 3426 exists and force is False, the state will fail. Otherwise, the 3427 backupname will be removed first. 3428 3429 .. versionadded:: 2014.7.0 3430 3431 allow_symlink 3432 If allow_symlink is True and the specified path is a symlink, it will be 3433 allowed to remain if it points to a directory. If allow_symlink is False 3434 then the state will fail, unless force is also set to True, in which case 3435 it will be removed or renamed, depending on the value of the backupname 3436 argument. 3437 3438 .. versionadded:: 2014.7.0 3439 3440 children_only 3441 If children_only is True the base of a path is excluded when performing 3442 a recursive operation. In case of /path/to/base, base will be ignored 3443 while all of /path/to/base/* are still operated on. 3444 3445 win_owner 3446 The owner of the directory. If this is not passed, user will be used. If 3447 user is not passed, the account under which Salt is running will be 3448 used. 3449 3450 .. versionadded:: 2017.7.0 3451 3452 win_perms 3453 A dictionary containing permissions to grant and their propagation. For 3454 example: ``{'Administrators': {'perms': 'full_control', 'applies_to': 3455 'this_folder_only'}}`` Can be a single basic perm or a list of advanced 3456 perms. ``perms`` must be specified. ``applies_to`` is optional and 3457 defaults to ``this_folder_subfolder_files``. 3458 3459 .. versionadded:: 2017.7.0 3460 3461 win_deny_perms 3462 A dictionary containing permissions to deny and their propagation. For 3463 example: ``{'Administrators': {'perms': 'full_control', 'applies_to': 3464 'this_folder_only'}}`` Can be a single basic perm or a list of advanced 3465 perms. 3466 3467 .. versionadded:: 2017.7.0 3468 3469 win_inheritance 3470 True to inherit permissions from the parent directory, False not to 3471 inherit permission. 3472 3473 .. versionadded:: 2017.7.0 3474 3475 win_perms_reset 3476 If ``True`` the existing DACL will be cleared and replaced with the 3477 settings defined in this function. If ``False``, new entries will be 3478 appended to the existing DACL. Default is ``False``. 3479 3480 .. versionadded:: 2018.3.0 3481 3482 Here's an example using the above ``win_*`` parameters: 3483 3484 .. code-block:: yaml 3485 3486 create_config_dir: 3487 file.directory: 3488 - name: 'C:\config\' 3489 - win_owner: Administrators 3490 - win_perms: 3491 # Basic Permissions 3492 dev_ops: 3493 perms: full_control 3494 # List of advanced permissions 3495 appuser: 3496 perms: 3497 - read_attributes 3498 - read_ea 3499 - create_folders 3500 - read_permissions 3501 applies_to: this_folder_only 3502 joe_snuffy: 3503 perms: read 3504 applies_to: this_folder_files 3505 - win_deny_perms: 3506 fred_snuffy: 3507 perms: full_control 3508 - win_inheritance: False 3509 """ 3510 name = os.path.expanduser(name) 3511 ret = {"name": name, "changes": {}, "result": True, "comment": ""} 3512 if not name: 3513 return _error(ret, "Must provide name to file.directory") 3514 # Remove trailing slash, if present and we're not working on "/" itself 3515 if name[-1] == "/" and name != "/": 3516 name = name[:-1] 3517 3518 if max_depth is not None and clean: 3519 return _error(ret, "Cannot specify both max_depth and clean") 3520 3521 user = _test_owner(kwargs, user=user) 3522 if salt.utils.platform.is_windows(): 3523 3524 # If win_owner not passed, use user 3525 if win_owner is None: 3526 win_owner = user if user else salt.utils.win_functions.get_current_user() 3527 3528 # Group isn't relevant to Windows, use win_perms/win_deny_perms 3529 if group is not None: 3530 log.warning( 3531 "The group argument for %s has been ignored as this is " 3532 "a Windows system. Please use the `win_*` parameters to set " 3533 "permissions in Windows.", 3534 name, 3535 ) 3536 group = user 3537 3538 if "mode" in kwargs and not dir_mode: 3539 dir_mode = kwargs.get("mode", []) 3540 3541 if not file_mode: 3542 file_mode = dir_mode 3543 3544 # Make sure that leading zeros stripped by YAML loader are added back 3545 dir_mode = salt.utils.files.normalize_mode(dir_mode) 3546 file_mode = salt.utils.files.normalize_mode(file_mode) 3547 3548 if salt.utils.platform.is_windows(): 3549 # Verify win_owner is valid on the target system 3550 try: 3551 salt.utils.win_dacl.get_sid(win_owner) 3552 except CommandExecutionError as exc: 3553 return _error(ret, exc) 3554 else: 3555 # Verify user and group are valid 3556 u_check = _check_user(user, group) 3557 if u_check: 3558 # The specified user or group do not exist 3559 return _error(ret, u_check) 3560 3561 # Must be an absolute path 3562 if not os.path.isabs(name): 3563 return _error(ret, "Specified file {} is not an absolute path".format(name)) 3564 3565 # Check for existing file or symlink 3566 if ( 3567 os.path.isfile(name) 3568 or (not allow_symlink and os.path.islink(name)) 3569 or (force and os.path.islink(name)) 3570 ): 3571 # Was a backupname specified 3572 if backupname is not None: 3573 # Make a backup first 3574 if os.path.lexists(backupname): 3575 if not force: 3576 return _error( 3577 ret, 3578 "File exists where the backup target {} should go".format( 3579 backupname 3580 ), 3581 ) 3582 else: 3583 __salt__["file.remove"](backupname) 3584 os.rename(name, backupname) 3585 elif force: 3586 # Remove whatever is in the way 3587 if os.path.isfile(name): 3588 if __opts__["test"]: 3589 ret["changes"]["forced"] = "File would be forcibly replaced" 3590 else: 3591 os.remove(name) 3592 ret["changes"]["forced"] = "File was forcibly replaced" 3593 elif __salt__["file.is_link"](name): 3594 if __opts__["test"]: 3595 ret["changes"]["forced"] = "Symlink would be forcibly replaced" 3596 else: 3597 __salt__["file.remove"](name) 3598 ret["changes"]["forced"] = "Symlink was forcibly replaced" 3599 else: 3600 if __opts__["test"]: 3601 ret["changes"]["forced"] = "Directory would be forcibly replaced" 3602 else: 3603 __salt__["file.remove"](name) 3604 ret["changes"]["forced"] = "Directory was forcibly replaced" 3605 else: 3606 if os.path.isfile(name): 3607 return _error( 3608 ret, "Specified location {} exists and is a file".format(name) 3609 ) 3610 elif os.path.islink(name): 3611 return _error( 3612 ret, "Specified location {} exists and is a symlink".format(name) 3613 ) 3614 3615 # Check directory? 3616 if salt.utils.platform.is_windows(): 3617 tresult, tcomment, tchanges = _check_directory_win( 3618 name=name, 3619 win_owner=win_owner, 3620 win_perms=win_perms, 3621 win_deny_perms=win_deny_perms, 3622 win_inheritance=win_inheritance, 3623 win_perms_reset=win_perms_reset, 3624 ) 3625 else: 3626 tresult, tcomment, tchanges = _check_directory( 3627 name, 3628 user, 3629 group, 3630 recurse or [], 3631 dir_mode, 3632 file_mode, 3633 clean, 3634 require, 3635 exclude_pat, 3636 max_depth, 3637 follow_symlinks, 3638 ) 3639 3640 if tchanges: 3641 ret["changes"].update(tchanges) 3642 3643 # Don't run through the reset of the function if there are no changes to be 3644 # made 3645 if __opts__["test"] or not ret["changes"]: 3646 ret["result"] = tresult 3647 ret["comment"] = tcomment 3648 return ret 3649 3650 if not os.path.isdir(name): 3651 # The dir does not exist, make it 3652 if not os.path.isdir(os.path.dirname(name)): 3653 # The parent directory does not exist, create them 3654 if makedirs: 3655 # Everything's good, create the parent Dirs 3656 try: 3657 _makedirs( 3658 name=name, 3659 user=user, 3660 group=group, 3661 dir_mode=dir_mode, 3662 win_owner=win_owner, 3663 win_perms=win_perms, 3664 win_deny_perms=win_deny_perms, 3665 win_inheritance=win_inheritance, 3666 ) 3667 except CommandExecutionError as exc: 3668 return _error(ret, "Drive {} is not mapped".format(exc.message)) 3669 else: 3670 return _error(ret, "No directory to create {} in".format(name)) 3671 3672 if salt.utils.platform.is_windows(): 3673 __salt__["file.mkdir"]( 3674 path=name, 3675 owner=win_owner, 3676 grant_perms=win_perms, 3677 deny_perms=win_deny_perms, 3678 inheritance=win_inheritance, 3679 reset=win_perms_reset, 3680 ) 3681 else: 3682 __salt__["file.mkdir"](name, user=user, group=group, mode=dir_mode) 3683 3684 if not os.path.isdir(name): 3685 return _error(ret, "Failed to create directory {}".format(name)) 3686 3687 ret["changes"][name] = {"directory": "new"} 3688 return ret 3689 3690 # issue 32707: skip this __salt__['file.check_perms'] call if children_only == True 3691 # Check permissions 3692 if not children_only: 3693 if salt.utils.platform.is_windows(): 3694 ret = __salt__["file.check_perms"]( 3695 path=name, 3696 ret=ret, 3697 owner=win_owner, 3698 grant_perms=win_perms, 3699 deny_perms=win_deny_perms, 3700 inheritance=win_inheritance, 3701 reset=win_perms_reset, 3702 ) 3703 else: 3704 ret, perms = __salt__["file.check_perms"]( 3705 name, ret, user, group, dir_mode, None, follow_symlinks 3706 ) 3707 3708 errors = [] 3709 if recurse or clean: 3710 # walk path only once and store the result 3711 walk_l = list(_depth_limited_walk(name, max_depth)) 3712 # root: (dirs, files) structure, compatible for python2.6 3713 walk_d = {} 3714 for i in walk_l: 3715 walk_d[i[0]] = (i[1], i[2]) 3716 3717 recurse_set = None 3718 if recurse: 3719 try: 3720 recurse_set = _get_recurse_set(recurse) 3721 except (TypeError, ValueError) as exc: 3722 ret["result"] = False 3723 ret["comment"] = "{}".format(exc) 3724 # NOTE: Should this be enough to stop the whole check altogether? 3725 if recurse_set: 3726 if "user" in recurse_set: 3727 if user or isinstance(user, int): 3728 uid = __salt__["file.user_to_uid"](user) 3729 # file.user_to_uid returns '' if user does not exist. Above 3730 # check for user is not fatal, so we need to be sure user 3731 # exists. 3732 if isinstance(uid, str): 3733 ret["result"] = False 3734 ret["comment"] = ( 3735 "Failed to enforce ownership for " 3736 "user {} (user does not " 3737 "exist)".format(user) 3738 ) 3739 else: 3740 ret["result"] = False 3741 ret["comment"] = ( 3742 "user not specified, but configured as " 3743 "a target for recursive ownership " 3744 "management" 3745 ) 3746 else: 3747 user = None 3748 if "group" in recurse_set: 3749 if group or isinstance(group, int): 3750 gid = __salt__["file.group_to_gid"](group) 3751 # As above with user, we need to make sure group exists. 3752 if isinstance(gid, str): 3753 ret["result"] = False 3754 ret[ 3755 "comment" 3756 ] = "Failed to enforce group ownership for group {}".format(group) 3757 else: 3758 ret["result"] = False 3759 ret["comment"] = ( 3760 "group not specified, but configured " 3761 "as a target for recursive ownership " 3762 "management" 3763 ) 3764 else: 3765 group = None 3766 3767 if "mode" not in recurse_set: 3768 file_mode = None 3769 dir_mode = None 3770 3771 if "silent" in recurse_set: 3772 ret["changes"] = {"recursion": "Changes silenced"} 3773 3774 check_files = "ignore_files" not in recurse_set 3775 check_dirs = "ignore_dirs" not in recurse_set 3776 3777 for root, dirs, files in walk_l: 3778 if check_files: 3779 for fn_ in files: 3780 full = os.path.join(root, fn_) 3781 try: 3782 if salt.utils.platform.is_windows(): 3783 ret = __salt__["file.check_perms"]( 3784 path=full, 3785 ret=ret, 3786 owner=win_owner, 3787 grant_perms=win_perms, 3788 deny_perms=win_deny_perms, 3789 inheritance=win_inheritance, 3790 reset=win_perms_reset, 3791 ) 3792 else: 3793 ret, _ = __salt__["file.check_perms"]( 3794 full, ret, user, group, file_mode, None, follow_symlinks 3795 ) 3796 except CommandExecutionError as exc: 3797 if not exc.strerror.startswith("Path not found"): 3798 errors.append(exc.strerror) 3799 3800 if check_dirs: 3801 for dir_ in dirs: 3802 full = os.path.join(root, dir_) 3803 try: 3804 if salt.utils.platform.is_windows(): 3805 ret = __salt__["file.check_perms"]( 3806 path=full, 3807 ret=ret, 3808 owner=win_owner, 3809 grant_perms=win_perms, 3810 deny_perms=win_deny_perms, 3811 inheritance=win_inheritance, 3812 reset=win_perms_reset, 3813 ) 3814 else: 3815 ret, _ = __salt__["file.check_perms"]( 3816 full, ret, user, group, dir_mode, None, follow_symlinks 3817 ) 3818 except CommandExecutionError as exc: 3819 if not exc.strerror.startswith("Path not found"): 3820 errors.append(exc.strerror) 3821 3822 if clean: 3823 keep = _gen_keep_files(name, require, walk_d) 3824 log.debug("List of kept files when use file.directory with clean: %s", keep) 3825 removed = _clean_dir(name, list(keep), exclude_pat) 3826 if removed: 3827 ret["changes"]["removed"] = removed 3828 ret["comment"] = "Files cleaned from directory {}".format(name) 3829 3830 # issue 32707: reflect children_only selection in comments 3831 if not ret["comment"]: 3832 if children_only: 3833 ret["comment"] = "Directory {}/* updated".format(name) 3834 else: 3835 if ret["changes"]: 3836 ret["comment"] = "Directory {} updated".format(name) 3837 3838 if __opts__["test"]: 3839 ret["comment"] = "Directory {} not updated".format(name) 3840 elif not ret["changes"] and ret["result"]: 3841 orig_comment = None 3842 if ret["comment"]: 3843 orig_comment = ret["comment"] 3844 3845 ret["comment"] = "Directory {} is in the correct state".format(name) 3846 if orig_comment: 3847 ret["comment"] = "\n".join([ret["comment"], orig_comment]) 3848 3849 if errors: 3850 ret["result"] = False 3851 ret["comment"] += "\n\nThe following errors were encountered:\n" 3852 for error in errors: 3853 ret["comment"] += "\n- {}".format(error) 3854 3855 return ret 3856 3857 3858def recurse( 3859 name, 3860 source, 3861 keep_source=True, 3862 clean=False, 3863 require=None, 3864 user=None, 3865 group=None, 3866 dir_mode=None, 3867 file_mode=None, 3868 sym_mode=None, 3869 template=None, 3870 context=None, 3871 replace=True, 3872 defaults=None, 3873 include_empty=False, 3874 backup="", 3875 include_pat=None, 3876 exclude_pat=None, 3877 maxdepth=None, 3878 keep_symlinks=False, 3879 force_symlinks=False, 3880 win_owner=None, 3881 win_perms=None, 3882 win_deny_perms=None, 3883 win_inheritance=True, 3884 **kwargs 3885): 3886 """ 3887 Recurse through a subdirectory on the master and copy said subdirectory 3888 over to the specified path. 3889 3890 name 3891 The directory to set the recursion in 3892 3893 source 3894 The source directory, this directory is located on the salt master file 3895 server and is specified with the salt:// protocol. If the directory is 3896 located on the master in the directory named spam, and is called eggs, 3897 the source string is salt://spam/eggs 3898 3899 keep_source 3900 Set to ``False`` to discard the cached copy of the source file once the 3901 state completes. This can be useful for larger files to keep them from 3902 taking up space in minion cache. However, keep in mind that discarding 3903 the source file will result in the state needing to re-download the 3904 source file if the state is run again. 3905 3906 .. versionadded:: 2017.7.3 3907 3908 clean 3909 Make sure that only files that are set up by salt and required by this 3910 function are kept. If this option is set then everything in this 3911 directory will be deleted unless it is required. 3912 3913 require 3914 Require other resources such as packages or files 3915 3916 user 3917 The user to own the directory. This defaults to the user salt is 3918 running as on the minion 3919 3920 group 3921 The group ownership set for the directory. This defaults to the group 3922 salt is running as on the minion. On Windows, this is ignored 3923 3924 dir_mode 3925 The permissions mode to set on any directories created. 3926 3927 The default mode for new files and directories corresponds umask of salt 3928 process. For existing files and directories it's not enforced. 3929 3930 .. note:: 3931 This option is **not** supported on Windows. 3932 3933 file_mode 3934 The permissions mode to set on any files created. 3935 3936 The default mode for new files and directories corresponds umask of salt 3937 process. For existing files and directories it's not enforced. 3938 3939 .. note:: 3940 This option is **not** supported on Windows. 3941 3942 .. versionchanged:: 2016.11.0 3943 This option can be set to ``keep``, and Salt will keep the mode 3944 from the Salt fileserver. This is only supported when the 3945 ``source`` URL begins with ``salt://``, or for files local to the 3946 minion. Because the ``source`` option cannot be used with any of 3947 the ``contents`` options, setting the ``mode`` to ``keep`` is also 3948 incompatible with the ``contents`` options. 3949 3950 sym_mode 3951 The permissions mode to set on any symlink created. 3952 3953 The default mode for new files and directories corresponds umask of salt 3954 process. For existing files and directories it's not enforced. 3955 3956 .. note:: 3957 This option is **not** supported on Windows. 3958 3959 template 3960 If this setting is applied, the named templating engine will be used to 3961 render the downloaded file. The following templates are supported: 3962 3963 - :mod:`cheetah<salt.renderers.cheetah>` 3964 - :mod:`genshi<salt.renderers.genshi>` 3965 - :mod:`jinja<salt.renderers.jinja>` 3966 - :mod:`mako<salt.renderers.mako>` 3967 - :mod:`py<salt.renderers.py>` 3968 - :mod:`wempy<salt.renderers.wempy>` 3969 3970 .. note:: 3971 3972 The template option is required when recursively applying templates. 3973 3974 replace 3975 If set to ``False`` and the file already exists, the file will not be 3976 modified even if changes would otherwise be made. Permissions and 3977 ownership will still be enforced, however. 3978 3979 context 3980 Overrides default context variables passed to the template. 3981 3982 defaults 3983 Default context passed to the template. 3984 3985 include_empty 3986 Set this to True if empty directories should also be created 3987 (default is False) 3988 3989 backup 3990 Overrides the default backup mode for all replaced files. See 3991 :ref:`backup_mode documentation <file-state-backups>` for more details. 3992 3993 include_pat 3994 When copying, include only this pattern, or list of patterns, from the 3995 source. Default is glob match; if prefixed with 'E@', then regexp match. 3996 Example: 3997 3998 .. code-block:: text 3999 4000 - include_pat: hello* :: glob matches 'hello01', 'hello02' 4001 ... but not 'otherhello' 4002 - include_pat: E@hello :: regexp matches 'otherhello', 4003 'hello01' ... 4004 4005 .. versionchanged:: 3001 4006 4007 List patterns are now supported 4008 4009 .. code-block:: text 4010 4011 - include_pat: 4012 - hello01 4013 - hello02 4014 4015 exclude_pat 4016 Exclude this pattern, or list of patterns, from the source when copying. 4017 If both `include_pat` and `exclude_pat` are supplied, then it will apply 4018 conditions cumulatively. i.e. first select based on include_pat, and 4019 then within that result apply exclude_pat. 4020 4021 Also, when 'clean=True', exclude this pattern from the removal 4022 list and preserve in the destination. 4023 Example: 4024 4025 .. code-block:: text 4026 4027 - exclude_pat: APPDATA* :: glob matches APPDATA.01, 4028 APPDATA.02,.. for exclusion 4029 - exclude_pat: E@(APPDATA)|(TEMPDATA) :: regexp matches APPDATA 4030 or TEMPDATA for exclusion 4031 4032 .. versionchanged:: 3001 4033 4034 List patterns are now supported 4035 4036 .. code-block:: text 4037 4038 - exclude_pat: 4039 - APPDATA.01 4040 - APPDATA.02 4041 4042 maxdepth 4043 When copying, only copy paths which are of depth `maxdepth` from the 4044 source path. 4045 Example: 4046 4047 .. code-block:: text 4048 4049 - maxdepth: 0 :: Only include files located in the source 4050 directory 4051 - maxdepth: 1 :: Only include files located in the source 4052 or immediate subdirectories 4053 4054 keep_symlinks 4055 Keep symlinks when copying from the source. This option will cause 4056 the copy operation to terminate at the symlink. If desire behavior 4057 similar to rsync, then set this to True. 4058 4059 force_symlinks 4060 Force symlink creation. This option will force the symlink creation. 4061 If a file or directory is obstructing symlink creation it will be 4062 recursively removed so that symlink creation can proceed. This 4063 option is usually not needed except in special circumstances. 4064 4065 win_owner 4066 The owner of the symlink and directories if ``makedirs`` is True. If 4067 this is not passed, ``user`` will be used. If ``user`` is not passed, 4068 the account under which Salt is running will be used. 4069 4070 .. versionadded:: 2017.7.7 4071 4072 win_perms 4073 A dictionary containing permissions to grant 4074 4075 .. versionadded:: 2017.7.7 4076 4077 win_deny_perms 4078 A dictionary containing permissions to deny 4079 4080 .. versionadded:: 2017.7.7 4081 4082 win_inheritance 4083 True to inherit permissions from parent, otherwise False 4084 4085 .. versionadded:: 2017.7.7 4086 4087 """ 4088 if "env" in kwargs: 4089 # "env" is not supported; Use "saltenv". 4090 kwargs.pop("env") 4091 4092 name = os.path.expanduser(salt.utils.data.decode(name)) 4093 4094 user = _test_owner(kwargs, user=user) 4095 if salt.utils.platform.is_windows(): 4096 if group is not None: 4097 log.warning( 4098 "The group argument for %s has been ignored as this " 4099 "is a Windows system.", 4100 name, 4101 ) 4102 group = user 4103 ret = { 4104 "name": name, 4105 "changes": {}, 4106 "result": True, 4107 "comment": {}, # { path: [comment, ...] } 4108 } 4109 4110 if "mode" in kwargs: 4111 ret["result"] = False 4112 ret["comment"] = ( 4113 "'mode' is not allowed in 'file.recurse'. Please use " 4114 "'file_mode' and 'dir_mode'." 4115 ) 4116 return ret 4117 4118 if ( 4119 any([x is not None for x in (dir_mode, file_mode, sym_mode)]) 4120 and salt.utils.platform.is_windows() 4121 ): 4122 return _error(ret, "mode management is not supported on Windows") 4123 4124 # Make sure that leading zeros stripped by YAML loader are added back 4125 dir_mode = salt.utils.files.normalize_mode(dir_mode) 4126 4127 try: 4128 keep_mode = file_mode.lower() == "keep" 4129 if keep_mode: 4130 # We're not hard-coding the mode, so set it to None 4131 file_mode = None 4132 except AttributeError: 4133 keep_mode = False 4134 4135 file_mode = salt.utils.files.normalize_mode(file_mode) 4136 4137 u_check = _check_user(user, group) 4138 if u_check: 4139 # The specified user or group do not exist 4140 return _error(ret, u_check) 4141 if not os.path.isabs(name): 4142 return _error(ret, "Specified file {} is not an absolute path".format(name)) 4143 4144 # expand source into source_list 4145 source_list = _validate_str_list(source) 4146 4147 for idx, val in enumerate(source_list): 4148 source_list[idx] = val.rstrip("/") 4149 4150 for precheck in source_list: 4151 if not precheck.startswith("salt://"): 4152 return _error( 4153 ret, 4154 "Invalid source '{}' (must be a salt:// URI)".format(precheck), 4155 ) 4156 4157 # Select the first source in source_list that exists 4158 try: 4159 source, source_hash = __salt__["file.source_list"](source_list, "", __env__) 4160 except CommandExecutionError as exc: 4161 ret["result"] = False 4162 ret["comment"] = "Recurse failed: {}".format(exc) 4163 return ret 4164 4165 # Check source path relative to fileserver root, make sure it is a 4166 # directory 4167 srcpath, senv = salt.utils.url.parse(source) 4168 if senv is None: 4169 senv = __env__ 4170 master_dirs = __salt__["cp.list_master_dirs"](saltenv=senv) 4171 if srcpath not in master_dirs and not any( 4172 x for x in master_dirs if x.startswith(srcpath + "/") 4173 ): 4174 ret["result"] = False 4175 ret["comment"] = ( 4176 "The directory '{}' does not exist on the salt fileserver " 4177 "in saltenv '{}'".format(srcpath, senv) 4178 ) 4179 return ret 4180 4181 # Verify the target directory 4182 if not os.path.isdir(name): 4183 if os.path.exists(name): 4184 # it is not a dir, but it exists - fail out 4185 return _error(ret, "The path {} exists and is not a directory".format(name)) 4186 if not __opts__["test"]: 4187 if salt.utils.platform.is_windows(): 4188 win_owner = win_owner if win_owner else user 4189 __salt__["file.makedirs_perms"]( 4190 path=name, 4191 owner=win_owner, 4192 grant_perms=win_perms, 4193 deny_perms=win_deny_perms, 4194 inheritance=win_inheritance, 4195 ) 4196 else: 4197 __salt__["file.makedirs_perms"]( 4198 name=name, user=user, group=group, mode=dir_mode 4199 ) 4200 4201 def add_comment(path, comment): 4202 comments = ret["comment"].setdefault(path, []) 4203 if isinstance(comment, str): 4204 comments.append(comment) 4205 else: 4206 comments.extend(comment) 4207 4208 def merge_ret(path, _ret): 4209 # Use the most "negative" result code (out of True, None, False) 4210 if _ret["result"] is False or ret["result"] is True: 4211 ret["result"] = _ret["result"] 4212 4213 # Only include comments about files that changed 4214 if _ret["result"] is not True and _ret["comment"]: 4215 add_comment(path, _ret["comment"]) 4216 4217 if _ret["changes"]: 4218 ret["changes"][path] = _ret["changes"] 4219 4220 def manage_file(path, source, replace): 4221 if clean and os.path.exists(path) and os.path.isdir(path) and replace: 4222 _ret = {"name": name, "changes": {}, "result": True, "comment": ""} 4223 if __opts__["test"]: 4224 _ret["comment"] = "Replacing directory {} with a file".format(path) 4225 _ret["result"] = None 4226 merge_ret(path, _ret) 4227 return 4228 else: 4229 __salt__["file.remove"](path) 4230 _ret["changes"] = {"diff": "Replaced directory with a new file"} 4231 merge_ret(path, _ret) 4232 4233 # Conflicts can occur if some kwargs are passed in here 4234 pass_kwargs = {} 4235 faults = ["mode", "makedirs"] 4236 for key in kwargs: 4237 if key not in faults: 4238 pass_kwargs[key] = kwargs[key] 4239 4240 _ret = managed( 4241 path, 4242 source=source, 4243 keep_source=keep_source, 4244 user=user, 4245 group=group, 4246 mode="keep" if keep_mode else file_mode, 4247 attrs=None, 4248 template=template, 4249 makedirs=True, 4250 replace=replace, 4251 context=context, 4252 defaults=defaults, 4253 backup=backup, 4254 **pass_kwargs 4255 ) 4256 merge_ret(path, _ret) 4257 4258 def manage_directory(path): 4259 if os.path.basename(path) == "..": 4260 return 4261 if clean and os.path.exists(path) and not os.path.isdir(path): 4262 _ret = {"name": name, "changes": {}, "result": True, "comment": ""} 4263 if __opts__["test"]: 4264 _ret["comment"] = "Replacing {} with a directory".format(path) 4265 _ret["result"] = None 4266 merge_ret(path, _ret) 4267 return 4268 else: 4269 __salt__["file.remove"](path) 4270 _ret["changes"] = {"diff": "Replaced file with a directory"} 4271 merge_ret(path, _ret) 4272 4273 _ret = directory( 4274 path, 4275 user=user, 4276 group=group, 4277 recurse=[], 4278 dir_mode=dir_mode, 4279 file_mode=None, 4280 makedirs=True, 4281 clean=False, 4282 require=None, 4283 ) 4284 merge_ret(path, _ret) 4285 4286 mng_files, mng_dirs, mng_symlinks, keep = _gen_recurse_managed_files( 4287 name, source, keep_symlinks, include_pat, exclude_pat, maxdepth, include_empty 4288 ) 4289 4290 for srelpath, ltarget in mng_symlinks: 4291 _ret = symlink( 4292 os.path.join(name, srelpath), 4293 ltarget, 4294 makedirs=True, 4295 force=force_symlinks, 4296 user=user, 4297 group=group, 4298 mode=sym_mode, 4299 ) 4300 if not _ret: 4301 continue 4302 merge_ret(os.path.join(name, srelpath), _ret) 4303 for dirname in mng_dirs: 4304 manage_directory(dirname) 4305 for dest, src in mng_files: 4306 manage_file(dest, src, replace) 4307 4308 if clean: 4309 # TODO: Use directory(clean=True) instead 4310 keep.update(_gen_keep_files(name, require)) 4311 removed = _clean_dir(name, list(keep), exclude_pat) 4312 if removed: 4313 if __opts__["test"]: 4314 if ret["result"]: 4315 ret["result"] = None 4316 add_comment("removed", removed) 4317 else: 4318 ret["changes"]["removed"] = removed 4319 4320 # Flatten comments until salt command line client learns 4321 # to display structured comments in a readable fashion 4322 ret["comment"] = "\n".join( 4323 "\n#### {} ####\n{}".format(k, v if isinstance(v, str) else "\n".join(v)) 4324 for (k, v) in ret["comment"].items() 4325 ).strip() 4326 4327 if not ret["comment"]: 4328 ret["comment"] = "Recursively updated {}".format(name) 4329 4330 if not ret["changes"] and ret["result"]: 4331 ret["comment"] = "The directory {} is in the correct state".format(name) 4332 4333 return ret 4334 4335 4336def retention_schedule(name, retain, strptime_format=None, timezone=None): 4337 """ 4338 Apply retention scheduling to backup storage directory. 4339 4340 .. versionadded:: 2016.11.0 4341 4342 :param name: 4343 The filesystem path to the directory containing backups to be managed. 4344 4345 :param retain: 4346 Delete the backups, except for the ones we want to keep. 4347 The N below should be an integer but may also be the special value of ``all``, 4348 which keeps all files matching the criteria. 4349 All of the retain options default to None, 4350 which means to not keep files based on this criteria. 4351 4352 :most_recent N: 4353 Keep the most recent N files. 4354 4355 :first_of_hour N: 4356 For the last N hours from now, keep the first file after the hour. 4357 4358 :first_of_day N: 4359 For the last N days from now, keep the first file after midnight. 4360 See also ``timezone``. 4361 4362 :first_of_week N: 4363 For the last N weeks from now, keep the first file after Sunday midnight. 4364 4365 :first_of_month N: 4366 For the last N months from now, keep the first file after the start of the month. 4367 4368 :first_of_year N: 4369 For the last N years from now, keep the first file after the start of the year. 4370 4371 :param strptime_format: 4372 A python strptime format string used to first match the filenames of backups 4373 and then parse the filename to determine the datetime of the file. 4374 https://docs.python.org/2/library/datetime.html#datetime.datetime.strptime 4375 Defaults to None, which considers all files in the directory to be backups eligible for deletion 4376 and uses ``os.path.getmtime()`` to determine the datetime. 4377 4378 :param timezone: 4379 The timezone to use when determining midnight. 4380 This is only used when datetime is pulled from ``os.path.getmtime()``. 4381 Defaults to ``None`` which uses the timezone from the locale. 4382 4383 Usage example: 4384 4385 .. code-block:: yaml 4386 4387 /var/backups/example_directory: 4388 file.retention_schedule: 4389 - retain: 4390 most_recent: 5 4391 first_of_hour: 4 4392 first_of_day: 7 4393 first_of_week: 6 # NotImplemented yet. 4394 first_of_month: 6 4395 first_of_year: all 4396 - strptime_format: example_name_%Y%m%dT%H%M%S.tar.bz2 4397 - timezone: None 4398 4399 """ 4400 name = os.path.expanduser(name) 4401 ret = { 4402 "name": name, 4403 "changes": {"retained": [], "deleted": [], "ignored": []}, 4404 "result": True, 4405 "comment": "", 4406 } 4407 if not name: 4408 return _error(ret, "Must provide name to file.retention_schedule") 4409 if not os.path.isdir(name): 4410 return _error(ret, "Name provided to file.retention must be a directory") 4411 4412 # get list of files in directory 4413 all_files = __salt__["file.readdir"](name) 4414 4415 # if strptime_format is set, filter through the list to find names which parse and get their datetimes. 4416 beginning_of_unix_time = datetime(1970, 1, 1) 4417 4418 def get_file_time_from_strptime(f): 4419 try: 4420 ts = datetime.strptime(f, strptime_format) 4421 ts_epoch = salt.utils.dateutils.total_seconds(ts - beginning_of_unix_time) 4422 return (ts, ts_epoch) 4423 except ValueError: 4424 # Files which don't match the pattern are not relevant files. 4425 return (None, None) 4426 4427 def get_file_time_from_mtime(f): 4428 if f == "." or f == "..": 4429 return (None, None) 4430 lstat = __salt__["file.lstat"](os.path.join(name, f)) 4431 if lstat: 4432 mtime = lstat["st_mtime"] 4433 return (datetime.fromtimestamp(mtime, timezone), mtime) 4434 else: # maybe it was deleted since we did the readdir? 4435 return (None, None) 4436 4437 get_file_time = ( 4438 get_file_time_from_strptime if strptime_format else get_file_time_from_mtime 4439 ) 4440 4441 # data structures are nested dicts: 4442 # files_by_ymd = year.month.day.hour.unixtime: filename 4443 # files_by_y_week_dow = year.week_of_year.day_of_week.unixtime: filename 4444 # http://the.randomengineer.com/2015/04/28/python-recursive-defaultdict/ 4445 # TODO: move to an ordered dict model and reduce the number of sorts in the rest of the code? 4446 def dict_maker(): 4447 return defaultdict(dict_maker) 4448 4449 files_by_ymd = dict_maker() 4450 files_by_y_week_dow = dict_maker() 4451 relevant_files = set() 4452 ignored_files = set() 4453 for f in all_files: 4454 ts, ts_epoch = get_file_time(f) 4455 if ts: 4456 files_by_ymd[ts.year][ts.month][ts.day][ts.hour][ts_epoch] = f 4457 week_of_year = ts.isocalendar()[1] 4458 files_by_y_week_dow[ts.year][week_of_year][ts.weekday()][ts_epoch] = f 4459 relevant_files.add(f) 4460 else: 4461 ignored_files.add(f) 4462 4463 # This is tightly coupled with the file_with_times data-structure above. 4464 RETAIN_TO_DEPTH = { 4465 "first_of_year": 1, 4466 "first_of_month": 2, 4467 "first_of_day": 3, 4468 "first_of_hour": 4, 4469 "most_recent": 5, 4470 } 4471 4472 def get_first(fwt): 4473 if isinstance(fwt, dict): 4474 first_sub_key = sorted(fwt.keys())[0] 4475 return get_first(fwt[first_sub_key]) 4476 else: 4477 return {fwt} 4478 4479 def get_first_n_at_depth(fwt, depth, n): 4480 if depth <= 0: 4481 return get_first(fwt) 4482 else: 4483 result_set = set() 4484 for k in sorted(fwt.keys(), reverse=True): 4485 needed = n - len(result_set) 4486 if needed < 1: 4487 break 4488 result_set |= get_first_n_at_depth(fwt[k], depth - 1, needed) 4489 return result_set 4490 4491 # for each retain criteria, add filenames which match the criteria to the retain set. 4492 retained_files = set() 4493 for retention_rule, keep_count in retain.items(): 4494 # This is kind of a hack, since 'all' should really mean all, 4495 # but I think it's a large enough number that even modern filesystems would 4496 # choke if they had this many files in a single directory. 4497 keep_count = sys.maxsize if "all" == keep_count else int(keep_count) 4498 if "first_of_week" == retention_rule: 4499 first_of_week_depth = 2 # year + week_of_year = 2 4500 # I'm adding 1 to keep_count below because it fixed an off-by one 4501 # issue in the tests. I don't understand why, and that bothers me. 4502 retained_files |= get_first_n_at_depth( 4503 files_by_y_week_dow, first_of_week_depth, keep_count + 1 4504 ) 4505 else: 4506 retained_files |= get_first_n_at_depth( 4507 files_by_ymd, RETAIN_TO_DEPTH[retention_rule], keep_count 4508 ) 4509 4510 deletable_files = list(relevant_files - retained_files) 4511 deletable_files.sort(reverse=True) 4512 changes = { 4513 "retained": sorted(list(retained_files), reverse=True), 4514 "deleted": deletable_files, 4515 "ignored": sorted(list(ignored_files), reverse=True), 4516 } 4517 ret["changes"] = changes 4518 4519 # TODO: track and report how much space was / would be reclaimed 4520 if __opts__["test"]: 4521 ret["comment"] = "{} backups would have been removed from {}.\n".format( 4522 len(deletable_files), name 4523 ) 4524 if deletable_files: 4525 ret["result"] = None 4526 else: 4527 for f in deletable_files: 4528 __salt__["file.remove"](os.path.join(name, f)) 4529 ret["comment"] = "{} backups were removed from {}.\n".format( 4530 len(deletable_files), name 4531 ) 4532 ret["changes"] = changes 4533 4534 return ret 4535 4536 4537def line( 4538 name, 4539 content=None, 4540 match=None, 4541 mode=None, 4542 location=None, 4543 before=None, 4544 after=None, 4545 show_changes=True, 4546 backup=False, 4547 quiet=False, 4548 indent=True, 4549 create=False, 4550 user=None, 4551 group=None, 4552 file_mode=None, 4553): 4554 """ 4555 Line-focused editing of a file. 4556 4557 .. versionadded:: 2015.8.0 4558 4559 .. note:: 4560 4561 ``file.line`` exists for historic reasons, and is not 4562 generally recommended. It has a lot of quirks. You may find 4563 ``file.replace`` to be more suitable. 4564 4565 ``file.line`` is most useful if you have single lines in a file, 4566 potentially a config file, that you would like to manage. It can 4567 remove, add, and replace lines. 4568 4569 name 4570 Filesystem path to the file to be edited. 4571 4572 content 4573 Content of the line. Allowed to be empty if mode=delete. 4574 4575 match 4576 Match the target line for an action by 4577 a fragment of a string or regular expression. 4578 4579 If neither ``before`` nor ``after`` are provided, and ``match`` 4580 is also ``None``, match falls back to the ``content`` value. 4581 4582 mode 4583 Defines how to edit a line. One of the following options is 4584 required: 4585 4586 - ensure 4587 If line does not exist, it will be added. If ``before`` 4588 and ``after`` are specified either zero lines, or lines 4589 that contain the ``content`` line are allowed to be in between 4590 ``before`` and ``after``. If there are lines, and none of 4591 them match then it will produce an error. 4592 - replace 4593 If line already exists, it will be replaced. 4594 - delete 4595 Delete the line, if found. 4596 - insert 4597 Nearly identical to ``ensure``. If a line does not exist, 4598 it will be added. 4599 4600 The differences are that multiple (and non-matching) lines are 4601 alloweed between ``before`` and ``after``, if they are 4602 specified. The line will always be inserted right before 4603 ``before``. ``insert`` also allows the use of ``location`` to 4604 specify that the line should be added at the beginning or end of 4605 the file. 4606 4607 .. note:: 4608 4609 If ``mode=insert`` is used, at least one of the following 4610 options must also be defined: ``location``, ``before``, or 4611 ``after``. If ``location`` is used, it takes precedence 4612 over the other two options. 4613 4614 location 4615 In ``mode=insert`` only, whether to place the ``content`` at the 4616 beginning or end of a the file. If ``location`` is provided, 4617 ``before`` and ``after`` are ignored. Valid locations: 4618 4619 - start 4620 Place the content at the beginning of the file. 4621 - end 4622 Place the content at the end of the file. 4623 4624 before 4625 Regular expression or an exact case-sensitive fragment of the string. 4626 Will be tried as **both** a regex **and** a part of the line. Must 4627 match **exactly** one line in the file. This value is only used in 4628 ``ensure`` and ``insert`` modes. The ``content`` will be inserted just 4629 before this line, matching its ``indent`` unless ``indent=False``. 4630 4631 after 4632 Regular expression or an exact case-sensitive fragment of the string. 4633 Will be tried as **both** a regex **and** a part of the line. Must 4634 match **exactly** one line in the file. This value is only used in 4635 ``ensure`` and ``insert`` modes. The ``content`` will be inserted 4636 directly after this line, unless ``before`` is also provided. If 4637 ``before`` is not matched, indentation will match this line, unless 4638 ``indent=False``. 4639 4640 show_changes 4641 Output a unified diff of the old file and the new file. 4642 If ``False`` return a boolean if any changes were made. 4643 Default is ``True`` 4644 4645 .. note:: 4646 Using this option will store two copies of the file in-memory 4647 (the original version and the edited version) in order to generate the diff. 4648 4649 backup 4650 Create a backup of the original file with the extension: 4651 "Year-Month-Day-Hour-Minutes-Seconds". 4652 4653 quiet 4654 Do not raise any exceptions. E.g. ignore the fact that the file that is 4655 tried to be edited does not exist and nothing really happened. 4656 4657 indent 4658 Keep indentation with the previous line. This option is not considered when 4659 the ``delete`` mode is specified. Default is ``True``. 4660 4661 create 4662 Create an empty file if doesn't exist. 4663 4664 .. versionadded:: 2016.11.0 4665 4666 user 4667 The user to own the file, this defaults to the user salt is running as 4668 on the minion. 4669 4670 .. versionadded:: 2016.11.0 4671 4672 group 4673 The group ownership set for the file, this defaults to the group salt 4674 is running as on the minion On Windows, this is ignored. 4675 4676 .. versionadded:: 2016.11.0 4677 4678 file_mode 4679 The permissions to set on this file, aka 644, 0775, 4664. Not supported 4680 on Windows. 4681 4682 .. versionadded:: 2016.11.0 4683 4684 If an equal sign (``=``) appears in an argument to a Salt command, it is 4685 interpreted as a keyword argument in the format of ``key=val``. That 4686 processing can be bypassed in order to pass an equal sign through to the 4687 remote shell command by manually specifying the kwarg: 4688 4689 .. code-block:: yaml 4690 4691 update_config: 4692 file.line: 4693 - name: /etc/myconfig.conf 4694 - mode: ensure 4695 - content: my key = my value 4696 - before: somekey.*? 4697 4698 4699 **Examples:** 4700 4701 Here's a simple config file. 4702 4703 .. code-block:: ini 4704 4705 [some_config] 4706 # Some config file 4707 # this line will go away 4708 4709 here=False 4710 away=True 4711 goodybe=away 4712 4713 And an sls file: 4714 4715 .. code-block:: yaml 4716 4717 remove_lines: 4718 file.line: 4719 - name: /some/file.conf 4720 - mode: delete 4721 - match: away 4722 4723 This will produce: 4724 4725 .. code-block:: ini 4726 4727 [some_config] 4728 # Some config file 4729 4730 here=False 4731 away=True 4732 goodbye=away 4733 4734 If that state is executed 2 more times, this will be the result: 4735 4736 .. code-block:: ini 4737 4738 [some_config] 4739 # Some config file 4740 4741 here=False 4742 4743 Given that original file with this state: 4744 4745 .. code-block:: yaml 4746 4747 replace_things: 4748 file.line: 4749 - name: /some/file.conf 4750 - mode: replace 4751 - match: away 4752 - content: here 4753 4754 Three passes will this state will result in this file: 4755 4756 .. code-block:: ini 4757 4758 [some_config] 4759 # Some config file 4760 here 4761 4762 here=False 4763 here 4764 here 4765 4766 Each pass replacing the first line found. 4767 4768 Given this file: 4769 4770 .. code-block:: text 4771 4772 insert after me 4773 something 4774 insert before me 4775 4776 The following state: 4777 4778 .. code-block:: yaml 4779 4780 insert_a_line: 4781 file.line: 4782 - name: /some/file.txt 4783 - mode: insert 4784 - after: insert after me 4785 - before: insert before me 4786 - content: thrice 4787 4788 If this state is executed 3 times, the result will be: 4789 4790 .. code-block:: text 4791 4792 insert after me 4793 something 4794 thrice 4795 thrice 4796 thrice 4797 insert before me 4798 4799 If the mode is ensure instead, it will fail each time. To succeed, we need 4800 to remove the incorrect line between before and after: 4801 4802 .. code-block:: text 4803 4804 insert after me 4805 insert before me 4806 4807 With an ensure mode, this will insert ``thrice`` the first time and 4808 make no changes for subsequent calls. For something simple this is 4809 fine, but if you have instead blocks like this: 4810 4811 .. code-block:: text 4812 4813 Begin SomeBlock 4814 foo = bar 4815 End 4816 4817 Begin AnotherBlock 4818 another = value 4819 End 4820 4821 And given this state: 4822 4823 .. code-block:: yaml 4824 4825 ensure_someblock: 4826 file.line: 4827 - name: /some/file.conf 4828 - mode: ensure 4829 - after: Begin SomeBlock 4830 - content: this = should be my content 4831 - before: End 4832 4833 This will fail because there are multiple ``End`` lines. Without that 4834 problem, it still would fail because there is a non-matching line, 4835 ``foo = bar``. Ensure **only** allows either zero, or the matching 4836 line present to be present in between ``before`` and ``after``. 4837 """ 4838 name = os.path.expanduser(name) 4839 ret = {"name": name, "changes": {}, "result": True, "comment": ""} 4840 if not name: 4841 return _error(ret, "Must provide name to file.line") 4842 4843 managed(name, create=create, user=user, group=group, mode=file_mode, replace=False) 4844 4845 check_res, check_msg = _check_file(name) 4846 if not check_res: 4847 return _error(ret, check_msg) 4848 4849 # We've set the content to be empty in the function params but we want to make sure 4850 # it gets passed when needed. Feature #37092 4851 mode = mode and mode.lower() or mode 4852 if mode is None: 4853 return _error(ret, "Mode was not defined. How to process the file?") 4854 4855 modeswithemptycontent = ["delete"] 4856 if mode not in modeswithemptycontent and content is None: 4857 return _error( 4858 ret, 4859 "Content can only be empty if mode is {}".format(modeswithemptycontent), 4860 ) 4861 del modeswithemptycontent 4862 4863 changes = __salt__["file.line"]( 4864 name, 4865 content, 4866 match=match, 4867 mode=mode, 4868 location=location, 4869 before=before, 4870 after=after, 4871 show_changes=show_changes, 4872 backup=backup, 4873 quiet=quiet, 4874 indent=indent, 4875 ) 4876 if changes: 4877 ret["changes"]["diff"] = changes 4878 if __opts__["test"]: 4879 ret["result"] = None 4880 ret["comment"] = "Changes would be made" 4881 else: 4882 ret["result"] = True 4883 ret["comment"] = "Changes were made" 4884 else: 4885 ret["result"] = True 4886 ret["comment"] = "No changes needed to be made" 4887 4888 return ret 4889 4890 4891def replace( 4892 name, 4893 pattern, 4894 repl, 4895 count=0, 4896 flags=8, 4897 bufsize=1, 4898 append_if_not_found=False, 4899 prepend_if_not_found=False, 4900 not_found_content=None, 4901 backup=".bak", 4902 show_changes=True, 4903 ignore_if_missing=False, 4904 backslash_literal=False, 4905): 4906 r""" 4907 Maintain an edit in a file. 4908 4909 .. versionadded:: 0.17.0 4910 4911 name 4912 Filesystem path to the file to be edited. If a symlink is specified, it 4913 will be resolved to its target. 4914 4915 pattern 4916 A regular expression, to be matched using Python's 4917 :py:func:`re.search`. 4918 4919 .. note:: 4920 4921 If you need to match a literal string that contains regex special 4922 characters, you may want to use salt's custom Jinja filter, 4923 ``regex_escape``. 4924 4925 .. code-block:: jinja 4926 4927 {{ 'http://example.com?foo=bar%20baz' | regex_escape }} 4928 4929 repl 4930 The replacement text 4931 4932 count 4933 Maximum number of pattern occurrences to be replaced. Defaults to 0. 4934 If count is a positive integer n, no more than n occurrences will be 4935 replaced, otherwise all occurrences will be replaced. 4936 4937 flags 4938 A list of flags defined in the ``re`` module documentation from the 4939 Python standard library. Each list item should be a string that will 4940 correlate to the human-friendly flag name. E.g., ``['IGNORECASE', 4941 'MULTILINE']``. Optionally, ``flags`` may be an int, with a value 4942 corresponding to the XOR (``|``) of all the desired flags. Defaults to 4943 ``8`` (which equates to ``['MULTILINE']``). 4944 4945 .. note:: 4946 4947 ``file.replace`` reads the entire file as a string to support 4948 multiline regex patterns. Therefore, when using anchors such as 4949 ``^`` or ``$`` in the pattern, those anchors may be relative to 4950 the line OR relative to the file. The default for ``file.replace`` 4951 is to treat anchors as relative to the line, which is implemented 4952 by setting the default value of ``flags`` to ``['MULTILINE']``. 4953 When overriding the default value for ``flags``, if 4954 ``'MULTILINE'`` is not present then anchors will be relative to 4955 the file. If the desired behavior is for anchors to be relative to 4956 the line, then simply add ``'MULTILINE'`` to the list of flags. 4957 4958 bufsize 4959 How much of the file to buffer into memory at once. The default value 4960 ``1`` processes one line at a time. The special value ``file`` may be 4961 specified which will read the entire file into memory before 4962 processing. 4963 4964 append_if_not_found 4965 If set to ``True``, and pattern is not found, then the content will be 4966 appended to the file. 4967 4968 .. versionadded:: 2014.7.0 4969 4970 prepend_if_not_found 4971 If set to ``True`` and pattern is not found, then the content will be 4972 prepended to the file. 4973 4974 .. versionadded:: 2014.7.0 4975 4976 not_found_content 4977 Content to use for append/prepend if not found. If ``None`` (default), 4978 uses ``repl``. Useful when ``repl`` uses references to group in 4979 pattern. 4980 4981 .. versionadded:: 2014.7.0 4982 4983 backup 4984 The file extension to use for a backup of the file before editing. Set 4985 to ``False`` to skip making a backup. 4986 4987 show_changes 4988 Output a unified diff of the old file and the new file. If ``False`` 4989 return a boolean if any changes were made. Returns a boolean or a 4990 string. 4991 4992 .. note: 4993 Using this option will store two copies of the file in memory (the 4994 original version and the edited version) in order to generate the 4995 diff. This may not normally be a concern, but could impact 4996 performance if used with large files. 4997 4998 ignore_if_missing 4999 .. versionadded:: 2016.3.4 5000 5001 Controls what to do if the file is missing. If set to ``False``, the 5002 state will display an error raised by the execution module. If set to 5003 ``True``, the state will simply report no changes. 5004 5005 backslash_literal 5006 .. versionadded:: 2016.11.7 5007 5008 Interpret backslashes as literal backslashes for the repl and not 5009 escape characters. This will help when using append/prepend so that 5010 the backslashes are not interpreted for the repl on the second run of 5011 the state. 5012 5013 For complex regex patterns, it can be useful to avoid the need for complex 5014 quoting and escape sequences by making use of YAML's multiline string 5015 syntax. 5016 5017 .. code-block:: yaml 5018 5019 complex_search_and_replace: 5020 file.replace: 5021 # <...snip...> 5022 - pattern: | 5023 CentOS \(2.6.32[^\\n]+\\n\s+root[^\\n]+\\n\)+ 5024 5025 .. note:: 5026 5027 When using YAML multiline string syntax in ``pattern:``, make sure to 5028 also use that syntax in the ``repl:`` part, or you might loose line 5029 feeds. 5030 5031 When regex capture groups are used in ``pattern:``, their captured value is 5032 available for reuse in the ``repl:`` part as a backreference (ex. ``\1``). 5033 5034 .. code-block:: yaml 5035 5036 add_login_group_to_winbind_ssh_access_list: 5037 file.replace: 5038 - name: '/etc/security/pam_winbind.conf' 5039 - pattern: '^(require_membership_of = )(.*)$' 5040 - repl: '\1\2,append-new-group-to-line' 5041 5042 .. note:: 5043 5044 The ``file.replace`` state uses Python's ``re`` module. 5045 For more advanced options, see https://docs.python.org/2/library/re.html 5046 """ 5047 name = os.path.expanduser(name) 5048 5049 ret = {"name": name, "changes": {}, "result": True, "comment": ""} 5050 if not name: 5051 return _error(ret, "Must provide name to file.replace") 5052 5053 check_res, check_msg = _check_file(name) 5054 if not check_res: 5055 if ignore_if_missing and "file not found" in check_msg: 5056 ret["comment"] = "No changes needed to be made" 5057 return ret 5058 else: 5059 return _error(ret, check_msg) 5060 5061 changes = __salt__["file.replace"]( 5062 name, 5063 pattern, 5064 repl, 5065 count=count, 5066 flags=flags, 5067 bufsize=bufsize, 5068 append_if_not_found=append_if_not_found, 5069 prepend_if_not_found=prepend_if_not_found, 5070 not_found_content=not_found_content, 5071 backup=backup, 5072 dry_run=__opts__["test"], 5073 show_changes=show_changes, 5074 ignore_if_missing=ignore_if_missing, 5075 backslash_literal=backslash_literal, 5076 ) 5077 5078 if changes: 5079 ret["changes"]["diff"] = changes 5080 if __opts__["test"]: 5081 ret["result"] = None 5082 ret["comment"] = "Changes would have been made" 5083 else: 5084 ret["result"] = True 5085 ret["comment"] = "Changes were made" 5086 else: 5087 ret["result"] = True 5088 ret["comment"] = "No changes needed to be made" 5089 5090 return ret 5091 5092 5093def keyvalue( 5094 name, 5095 key=None, 5096 value=None, 5097 key_values=None, 5098 separator="=", 5099 append_if_not_found=False, 5100 prepend_if_not_found=False, 5101 search_only=False, 5102 show_changes=True, 5103 ignore_if_missing=False, 5104 count=1, 5105 uncomment=None, 5106 key_ignore_case=False, 5107 value_ignore_case=False, 5108): 5109 """ 5110 Key/Value based editing of a file. 5111 5112 .. versionadded:: 3001 5113 5114 This function differs from ``file.replace`` in that it is able to search for 5115 keys, followed by a customizable separator, and replace the value with the 5116 given value. Should the value be the same as the one already in the file, no 5117 changes will be made. 5118 5119 Either supply both ``key`` and ``value`` parameters, or supply a dictionary 5120 with key / value pairs. It is an error to supply both. 5121 5122 name 5123 Name of the file to search/replace in. 5124 5125 key 5126 Key to search for when ensuring a value. Use in combination with a 5127 ``value`` parameter. 5128 5129 value 5130 Value to set for a given key. Use in combination with a ``key`` 5131 parameter. 5132 5133 key_values 5134 Dictionary of key / value pairs to search for and ensure values for. 5135 Used to specify multiple key / values at once. 5136 5137 separator 5138 Separator which separates key from value. 5139 5140 append_if_not_found 5141 Append the key/value to the end of the file if not found. Note that this 5142 takes precedence over ``prepend_if_not_found``. 5143 5144 prepend_if_not_found 5145 Prepend the key/value to the beginning of the file if not found. Note 5146 that ``append_if_not_found`` takes precedence. 5147 5148 show_changes 5149 Show a diff of the resulting removals and inserts. 5150 5151 ignore_if_missing 5152 Return with success even if the file is not found (or not readable). 5153 5154 count 5155 Number of occurrences to allow (and correct), default is 1. Set to -1 to 5156 replace all, or set to 0 to remove all lines with this key regardsless 5157 of its value. 5158 5159 .. note:: 5160 Any additional occurrences after ``count`` are removed. 5161 A count of -1 will only replace all occurrences that are currently 5162 uncommented already. Lines commented out will be left alone. 5163 5164 uncomment 5165 Disregard and remove supplied leading characters when finding keys. When 5166 set to None, lines that are commented out are left for what they are. 5167 5168 .. note:: 5169 The argument to ``uncomment`` is not a prefix string. Rather; it is a 5170 set of characters, each of which are stripped. 5171 5172 key_ignore_case 5173 Keys are matched case insensitively. When a value is changed the matched 5174 key is kept as-is. 5175 5176 value_ignore_case 5177 Values are checked case insensitively, trying to set e.g. 'Yes' while 5178 the current value is 'yes', will not result in changes when 5179 ``value_ignore_case`` is set to True. 5180 5181 An example of using ``file.keyvalue`` to ensure sshd does not allow 5182 for root to login with a password and at the same time setting the 5183 login-gracetime to 1 minute and disabling all forwarding: 5184 5185 .. code-block:: yaml 5186 5187 sshd_config_harden: 5188 file.keyvalue: 5189 - name: /etc/ssh/sshd_config 5190 - key_values: 5191 permitrootlogin: 'without-password' 5192 LoginGraceTime: '1m' 5193 DisableForwarding: 'yes' 5194 - separator: ' ' 5195 - uncomment: '# ' 5196 - key_ignore_case: True 5197 - append_if_not_found: True 5198 5199 The same example, except for only ensuring PermitRootLogin is set correctly. 5200 Thus being able to use the shorthand ``key`` and ``value`` parameters 5201 instead of ``key_values``. 5202 5203 .. code-block:: yaml 5204 5205 sshd_config_harden: 5206 file.keyvalue: 5207 - name: /etc/ssh/sshd_config 5208 - key: PermitRootLogin 5209 - value: without-password 5210 - separator: ' ' 5211 - uncomment: '# ' 5212 - key_ignore_case: True 5213 - append_if_not_found: True 5214 5215 .. note:: 5216 Notice how the key is not matched case-sensitively, this way it will 5217 correctly identify both 'PermitRootLogin' as well as 'permitrootlogin'. 5218 5219 """ 5220 name = os.path.expanduser(name) 5221 5222 # default return values 5223 ret = { 5224 "name": name, 5225 "changes": {}, 5226 "result": None, 5227 "comment": "", 5228 } 5229 5230 if not name: 5231 return _error(ret, "Must provide name to file.keyvalue") 5232 if key is not None and value is not None: 5233 if type(key_values) is dict: 5234 return _error( 5235 ret, "file.keyvalue can not combine key_values with key and value" 5236 ) 5237 key_values = {str(key): value} 5238 5239 elif not isinstance(key_values, dict) or not key_values: 5240 msg = "is not a dictionary" 5241 if not key_values: 5242 msg = "is empty" 5243 return _error( 5244 ret, 5245 "file.keyvalue key and value not supplied and key_values " + msg, 5246 ) 5247 5248 # try to open the file and only return a comment if ignore_if_missing is 5249 # enabled, also mark as an error if not 5250 file_contents = [] 5251 try: 5252 with salt.utils.files.fopen(name, "r") as fd: 5253 file_contents = fd.readlines() 5254 except OSError: 5255 ret["comment"] = "unable to open {n}".format(n=name) 5256 ret["result"] = True if ignore_if_missing else False 5257 return ret 5258 5259 # used to store diff combinations and check if anything has changed 5260 diff = [] 5261 # store the final content of the file in case it needs to be rewritten 5262 content = [] 5263 # target format is templated like this 5264 tmpl = "{key}{sep}{value}" + os.linesep 5265 # number of lines changed 5266 changes = 0 5267 # keep track of number of times a key was updated 5268 diff_count = {k: count for k in key_values.keys()} 5269 5270 # read all the lines from the file 5271 for line in file_contents: 5272 test_line = line.lstrip(uncomment) 5273 did_uncomment = True if len(line) > len(test_line) else False 5274 5275 if key_ignore_case: 5276 test_line = test_line.lower() 5277 5278 for key, value in key_values.items(): 5279 test_key = key.lower() if key_ignore_case else key 5280 # if the line starts with the key 5281 if test_line.startswith(test_key): 5282 # if the testline got uncommented then the real line needs to 5283 # be uncommented too, otherwhise there might be separation on 5284 # a character which is part of the comment set 5285 working_line = line.lstrip(uncomment) if did_uncomment else line 5286 5287 # try to separate the line into its' components 5288 line_key, line_sep, line_value = working_line.partition(separator) 5289 5290 # if separation was unsuccessful then line_sep is empty so 5291 # no need to keep trying. continue instead 5292 if line_sep != separator: 5293 continue 5294 5295 # start on the premises the key does not match the actual line 5296 keys_match = False 5297 if key_ignore_case: 5298 if line_key.lower() == test_key: 5299 keys_match = True 5300 else: 5301 if line_key == test_key: 5302 keys_match = True 5303 5304 # if the key was found in the line and separation was successful 5305 if keys_match: 5306 # trial and error have shown it's safest to strip whitespace 5307 # from values for the sake of matching 5308 line_value = line_value.strip() 5309 # make sure the value is an actual string at this point 5310 test_value = str(value).strip() 5311 # convert test_value and line_value to lowercase if need be 5312 if value_ignore_case: 5313 line_value = line_value.lower() 5314 test_value = test_value.lower() 5315 5316 # values match if they are equal at this point 5317 values_match = True if line_value == test_value else False 5318 5319 # in case a line had its comment removed there are some edge 5320 # cases that need considderation where changes are needed 5321 # regardless of values already matching. 5322 needs_changing = False 5323 if did_uncomment: 5324 # irrespective of a value, if it was commented out and 5325 # changes are still to be made, then it needs to be 5326 # commented in 5327 if diff_count[key] > 0: 5328 needs_changing = True 5329 # but if values did not match but there are really no 5330 # changes expected anymore either then leave this line 5331 elif not values_match: 5332 values_match = True 5333 else: 5334 # a line needs to be removed if it has been seen enough 5335 # times and was not commented out, regardless of value 5336 if diff_count[key] == 0: 5337 needs_changing = True 5338 5339 # then start checking to see if the value needs replacing 5340 if not values_match or needs_changing: 5341 # the old line always needs to go, so that will be 5342 # reflected in the diff (this is the original line from 5343 # the file being read) 5344 diff.append("- {}".format(line)) 5345 line = line[:0] 5346 5347 # any non-zero value means something needs to go back in 5348 # its place. negative values are replacing all lines not 5349 # commented out, positive values are having their count 5350 # reduced by one every replacement 5351 if diff_count[key] != 0: 5352 # rebuild the line using the key and separator found 5353 # and insert the correct value. 5354 line = str( 5355 tmpl.format(key=line_key, sep=line_sep, value=value) 5356 ) 5357 5358 # display a comment in case a value got converted 5359 # into a string 5360 if not isinstance(value, str): 5361 diff.append( 5362 "+ {} (from {} type){}".format( 5363 line.rstrip(), type(value).__name__, os.linesep 5364 ) 5365 ) 5366 else: 5367 diff.append("+ {}".format(line)) 5368 changes += 1 5369 # subtract one from the count if it was larger than 0, so 5370 # next lines are removed. if it is less than 0 then count is 5371 # ignored and all lines will be updated. 5372 if diff_count[key] > 0: 5373 diff_count[key] -= 1 5374 # at this point a continue saves going through the rest of 5375 # the keys to see if they match since this line already 5376 # matched the current key 5377 continue 5378 # with the line having been checked for all keys (or matched before all 5379 # keys needed searching), the line can be added to the content to be 5380 # written once the last checks have been performed 5381 content.append(line) 5382 # finally, close the file 5383 fd.close() 5384 5385 # if append_if_not_found was requested, then append any key/value pairs 5386 # still having a count left on them 5387 if append_if_not_found: 5388 tmpdiff = [] 5389 for key, value in key_values.items(): 5390 if diff_count[key] > 0: 5391 line = tmpl.format(key=key, sep=separator, value=value) 5392 tmpdiff.append("+ {}".format(line)) 5393 content.append(line) 5394 changes += 1 5395 if tmpdiff: 5396 tmpdiff.insert(0, "- <EOF>" + os.linesep) 5397 tmpdiff.append("+ <EOF>" + os.linesep) 5398 diff.extend(tmpdiff) 5399 # only if append_if_not_found was not set should prepend_if_not_found be 5400 # considered, benefit of this is that the number of counts left does not 5401 # mean there might be both a prepend and append happening 5402 elif prepend_if_not_found: 5403 did_diff = False 5404 for key, value in key_values.items(): 5405 if diff_count[key] > 0: 5406 line = tmpl.format(key=key, sep=separator, value=value) 5407 if not did_diff: 5408 diff.insert(0, " <SOF>" + os.linesep) 5409 did_diff = True 5410 diff.insert(1, "+ {}".format(line)) 5411 content.insert(0, line) 5412 changes += 1 5413 5414 # if a diff was made 5415 if changes > 0: 5416 # return comment of changes if test 5417 if __opts__["test"]: 5418 ret["comment"] = "File {n} is set to be changed ({c} lines)".format( 5419 n=name, c=changes 5420 ) 5421 if show_changes: 5422 # For some reason, giving an actual diff even in test=True mode 5423 # will be seen as both a 'changed' and 'unchanged'. this seems to 5424 # match the other modules behaviour though 5425 ret["changes"]["diff"] = "".join(diff) 5426 5427 # add changes to comments for now as well because of how 5428 # stateoutputter seems to handle changes etc. 5429 # See: https://github.com/saltstack/salt/issues/40208 5430 ret["comment"] += "\nPredicted diff:\n\r\t\t" 5431 ret["comment"] += "\r\t\t".join(diff) 5432 ret["result"] = None 5433 5434 # otherwise return the actual diff lines 5435 else: 5436 ret["comment"] = "Changed {c} lines".format(c=changes) 5437 if show_changes: 5438 ret["changes"]["diff"] = "".join(diff) 5439 else: 5440 ret["result"] = True 5441 return ret 5442 5443 # if not test=true, try and write the file 5444 if not __opts__["test"]: 5445 try: 5446 with salt.utils.files.fopen(name, "w") as fd: 5447 # write all lines to the file which was just truncated 5448 fd.writelines(content) 5449 fd.close() 5450 except OSError: 5451 # return an error if the file was not writable 5452 ret["comment"] = "{n} not writable".format(n=name) 5453 ret["result"] = False 5454 return ret 5455 # if all went well, then set result to true 5456 ret["result"] = True 5457 5458 return ret 5459 5460 5461def blockreplace( 5462 name, 5463 marker_start="#-- start managed zone --", 5464 marker_end="#-- end managed zone --", 5465 source=None, 5466 source_hash=None, 5467 template="jinja", 5468 sources=None, 5469 source_hashes=None, 5470 defaults=None, 5471 context=None, 5472 content="", 5473 append_if_not_found=False, 5474 prepend_if_not_found=False, 5475 backup=".bak", 5476 show_changes=True, 5477 append_newline=None, 5478 insert_before_match=None, 5479 insert_after_match=None, 5480): 5481 """ 5482 Maintain an edit in a file in a zone delimited by two line markers 5483 5484 .. versionadded:: 2014.1.0 5485 .. versionchanged:: 2017.7.5,2018.3.1 5486 ``append_newline`` argument added. Additionally, to improve 5487 idempotence, if the string represented by ``marker_end`` is found in 5488 the middle of the line, the content preceding the marker will be 5489 removed when the block is replaced. This allows one to remove 5490 ``append_newline: False`` from the SLS and have the block properly 5491 replaced if the end of the content block is immediately followed by the 5492 ``marker_end`` (i.e. no newline before the marker). 5493 5494 A block of content delimited by comments can help you manage several lines 5495 entries without worrying about old entries removal. This can help you 5496 maintaining an un-managed file containing manual edits. 5497 5498 .. note:: 5499 This function will store two copies of the file in-memory (the original 5500 version and the edited version) in order to detect changes and only 5501 edit the targeted file if necessary. 5502 5503 Additionally, you can use :py:func:`file.accumulated 5504 <salt.states.file.accumulated>` and target this state. All accumulated 5505 data dictionaries' content will be added in the content block. 5506 5507 name 5508 Filesystem path to the file to be edited 5509 5510 marker_start 5511 The line content identifying a line as the start of the content block. 5512 Note that the whole line containing this marker will be considered, so 5513 whitespace or extra content before or after the marker is included in 5514 final output 5515 5516 marker_end 5517 The line content identifying the end of the content block. As of 5518 versions 2017.7.5 and 2018.3.1, everything up to the text matching the 5519 marker will be replaced, so it's important to ensure that your marker 5520 includes the beginning of the text you wish to replace. 5521 5522 content 5523 The content to be used between the two lines identified by 5524 ``marker_start`` and ``marker_end`` 5525 5526 source 5527 The source file to download to the minion, this source file can be 5528 hosted on either the salt master server, or on an HTTP or FTP server. 5529 Both HTTPS and HTTP are supported as well as downloading directly 5530 from Amazon S3 compatible URLs with both pre-configured and automatic 5531 IAM credentials. (see s3.get state documentation) 5532 File retrieval from Openstack Swift object storage is supported via 5533 swift://container/object_path URLs, see swift.get documentation. 5534 For files hosted on the salt file server, if the file is located on 5535 the master in the directory named spam, and is called eggs, the source 5536 string is salt://spam/eggs. If source is left blank or None 5537 (use ~ in YAML), the file will be created as an empty file and 5538 the content will not be managed. This is also the case when a file 5539 already exists and the source is undefined; the contents of the file 5540 will not be changed or managed. 5541 5542 If the file is hosted on a HTTP or FTP server then the source_hash 5543 argument is also required. 5544 5545 A list of sources can also be passed in to provide a default source and 5546 a set of fallbacks. The first source in the list that is found to exist 5547 will be used and subsequent entries in the list will be ignored. 5548 5549 .. code-block:: yaml 5550 5551 file_override_example: 5552 file.blockreplace: 5553 - name: /etc/example.conf 5554 - source: 5555 - salt://file_that_does_not_exist 5556 - salt://file_that_exists 5557 5558 source_hash 5559 This can be one of the following: 5560 1. a source hash string 5561 2. the URI of a file that contains source hash strings 5562 5563 The function accepts the first encountered long unbroken alphanumeric 5564 string of correct length as a valid hash, in order from most secure to 5565 least secure: 5566 5567 .. code-block:: text 5568 5569 Type Length 5570 ====== ====== 5571 sha512 128 5572 sha384 96 5573 sha256 64 5574 sha224 56 5575 sha1 40 5576 md5 32 5577 5578 See the ``source_hash`` parameter description for :mod:`file.managed 5579 <salt.states.file.managed>` function for more details and examples. 5580 5581 template 5582 Templating engine to be used to render the downloaded file. The 5583 following engines are supported: 5584 5585 - :mod:`cheetah <salt.renderers.cheetah>` 5586 - :mod:`genshi <salt.renderers.genshi>` 5587 - :mod:`jinja <salt.renderers.jinja>` 5588 - :mod:`mako <salt.renderers.mako>` 5589 - :mod:`py <salt.renderers.py>` 5590 - :mod:`wempy <salt.renderers.wempy>` 5591 5592 context 5593 Overrides default context variables passed to the template 5594 5595 defaults 5596 Default context passed to the template 5597 5598 append_if_not_found 5599 If markers are not found and this option is set to ``True``, the 5600 content block will be appended to the file. 5601 5602 prepend_if_not_found 5603 If markers are not found and this option is set to ``True``, the 5604 content block will be prepended to the file. 5605 5606 insert_before_match 5607 If markers are not found, this parameter can be set to a regex which will 5608 insert the block before the first found occurrence in the file. 5609 5610 .. versionadded:: 3001 5611 5612 insert_after_match 5613 If markers are not found, this parameter can be set to a regex which will 5614 insert the block after the first found occurrence in the file. 5615 5616 .. versionadded:: 3001 5617 5618 backup 5619 The file extension to use for a backup of the file if any edit is made. 5620 Set this to ``False`` to skip making a backup. 5621 5622 dry_run 5623 If ``True``, do not make any edits to the file and simply return the 5624 changes that *would* be made. 5625 5626 show_changes 5627 Controls how changes are presented. If ``True``, the ``Changes`` 5628 section of the state return will contain a unified diff of the changes 5629 made. If False, then it will contain a boolean (``True`` if any changes 5630 were made, otherwise ``False``). 5631 5632 append_newline 5633 Controls whether or not a newline is appended to the content block. If 5634 the value of this argument is ``True`` then a newline will be added to 5635 the content block. If it is ``False``, then a newline will *not* be 5636 added to the content block. If it is unspecified, then a newline will 5637 only be added to the content block if it does not already end in a 5638 newline. 5639 5640 .. versionadded:: 2017.7.5,2018.3.1 5641 5642 Example of usage with an accumulator and with a variable: 5643 5644 .. code-block:: jinja 5645 5646 {% set myvar = 42 %} 5647 hosts-config-block-{{ myvar }}: 5648 file.blockreplace: 5649 - name: /etc/hosts 5650 - marker_start: "# START managed zone {{ myvar }} -DO-NOT-EDIT-" 5651 - marker_end: "# END managed zone {{ myvar }} --" 5652 - content: 'First line of content' 5653 - append_if_not_found: True 5654 - backup: '.bak' 5655 - show_changes: True 5656 5657 hosts-config-block-{{ myvar }}-accumulated1: 5658 file.accumulated: 5659 - filename: /etc/hosts 5660 - name: my-accumulator-{{ myvar }} 5661 - text: "text 2" 5662 - require_in: 5663 - file: hosts-config-block-{{ myvar }} 5664 5665 hosts-config-block-{{ myvar }}-accumulated2: 5666 file.accumulated: 5667 - filename: /etc/hosts 5668 - name: my-accumulator-{{ myvar }} 5669 - text: | 5670 text 3 5671 text 4 5672 - require_in: 5673 - file: hosts-config-block-{{ myvar }} 5674 5675 will generate and maintain a block of content in ``/etc/hosts``: 5676 5677 .. code-block:: text 5678 5679 # START managed zone 42 -DO-NOT-EDIT- 5680 First line of content 5681 text 2 5682 text 3 5683 text 4 5684 # END managed zone 42 -- 5685 """ 5686 name = os.path.expanduser(name) 5687 5688 ret = {"name": name, "changes": {}, "result": False, "comment": ""} 5689 if not name: 5690 return _error(ret, "Must provide name to file.blockreplace") 5691 5692 if sources is None: 5693 sources = [] 5694 if source_hashes is None: 5695 source_hashes = [] 5696 5697 (ok_, err, sl_) = _unify_sources_and_hashes( 5698 source=source, 5699 source_hash=source_hash, 5700 sources=sources, 5701 source_hashes=source_hashes, 5702 ) 5703 if not ok_: 5704 return _error(ret, err) 5705 5706 check_res, check_msg = _check_file(name) 5707 if not check_res: 5708 return _error(ret, check_msg) 5709 5710 accum_data, accum_deps = _load_accumulators() 5711 if name in accum_data: 5712 accumulator = accum_data[name] 5713 # if we have multiple accumulators for a file, only apply the one 5714 # required at a time 5715 deps = accum_deps.get(name, []) 5716 filtered = [ 5717 a for a in deps if __low__["__id__"] in deps[a] and a in accumulator 5718 ] 5719 if not filtered: 5720 filtered = [a for a in accumulator] 5721 for acc in filtered: 5722 acc_content = accumulator[acc] 5723 for line in acc_content: 5724 if content == "": 5725 content = line 5726 else: 5727 content += "\n" + line 5728 5729 if sl_: 5730 tmpret = _get_template_texts( 5731 source_list=sl_, template=template, defaults=defaults, context=context 5732 ) 5733 if not tmpret["result"]: 5734 return tmpret 5735 text = tmpret["data"] 5736 5737 for index, item in enumerate(text): 5738 content += str(item) 5739 5740 try: 5741 changes = __salt__["file.blockreplace"]( 5742 name, 5743 marker_start, 5744 marker_end, 5745 content=content, 5746 append_if_not_found=append_if_not_found, 5747 prepend_if_not_found=prepend_if_not_found, 5748 insert_before_match=insert_before_match, 5749 insert_after_match=insert_after_match, 5750 backup=backup, 5751 dry_run=__opts__["test"], 5752 show_changes=show_changes, 5753 append_newline=append_newline, 5754 ) 5755 except Exception as exc: # pylint: disable=broad-except 5756 log.exception("Encountered error managing block") 5757 ret[ 5758 "comment" 5759 ] = "Encountered error managing block: {}. See the log for details.".format(exc) 5760 return ret 5761 5762 if changes: 5763 ret["changes"]["diff"] = changes 5764 if __opts__["test"]: 5765 ret["result"] = None 5766 ret["comment"] = "Changes would be made" 5767 else: 5768 ret["result"] = True 5769 ret["comment"] = "Changes were made" 5770 else: 5771 ret["result"] = True 5772 ret["comment"] = "No changes needed to be made" 5773 5774 return ret 5775 5776 5777def comment(name, regex, char="#", backup=".bak"): 5778 """ 5779 Comment out specified lines in a file. 5780 5781 name 5782 The full path to the file to be edited 5783 regex 5784 A regular expression used to find the lines that are to be commented; 5785 this pattern will be wrapped in parenthesis and will move any 5786 preceding/trailing ``^`` or ``$`` characters outside the parenthesis 5787 (e.g., the pattern ``^foo$`` will be rewritten as ``^(foo)$``) 5788 Note that you _need_ the leading ^, otherwise each time you run 5789 highstate, another comment char will be inserted. 5790 char 5791 The character to be inserted at the beginning of a line in order to 5792 comment it out 5793 backup 5794 The file will be backed up before edit with this file extension 5795 5796 .. warning:: 5797 5798 This backup will be overwritten each time ``sed`` / ``comment`` / 5799 ``uncomment`` is called. Meaning the backup will only be useful 5800 after the first invocation. 5801 5802 Set to False/None to not keep a backup. 5803 5804 Usage: 5805 5806 .. code-block:: yaml 5807 5808 /etc/fstab: 5809 file.comment: 5810 - regex: ^bind 127.0.0.1 5811 5812 .. versionadded:: 0.9.5 5813 """ 5814 name = os.path.expanduser(name) 5815 5816 ret = {"name": name, "changes": {}, "result": False, "comment": ""} 5817 if not name: 5818 return _error(ret, "Must provide name to file.comment") 5819 5820 check_res, check_msg = _check_file(name) 5821 if not check_res: 5822 return _error(ret, check_msg) 5823 5824 # remove (?i)-like flags, ^ and $ 5825 unanchor_regex = re.sub(r"^(\(\?[iLmsux]\))?\^?(.*?)\$?$", r"\2", regex) 5826 5827 comment_regex = char + unanchor_regex 5828 5829 # Make sure the pattern appears in the file before continuing 5830 if not __salt__["file.search"](name, regex, multiline=True): 5831 if __salt__["file.search"](name, comment_regex, multiline=True): 5832 ret["comment"] = "Pattern already commented" 5833 ret["result"] = True 5834 return ret 5835 else: 5836 return _error(ret, "{}: Pattern not found".format(unanchor_regex)) 5837 5838 if __opts__["test"]: 5839 ret["changes"][name] = "updated" 5840 ret["comment"] = "File {} is set to be updated".format(name) 5841 ret["result"] = None 5842 return ret 5843 with salt.utils.files.fopen(name, "rb") as fp_: 5844 slines = fp_.read() 5845 slines = slines.decode(__salt_system_encoding__) 5846 slines = slines.splitlines(True) 5847 5848 # Perform the edit 5849 __salt__["file.comment_line"](name, regex, char, True, backup) 5850 5851 with salt.utils.files.fopen(name, "rb") as fp_: 5852 nlines = fp_.read() 5853 nlines = nlines.decode(__salt_system_encoding__) 5854 nlines = nlines.splitlines(True) 5855 5856 # Check the result 5857 ret["result"] = __salt__["file.search"](name, unanchor_regex, multiline=True) 5858 5859 if slines != nlines: 5860 if not __utils__["files.is_text"](name): 5861 ret["changes"]["diff"] = "Replace binary file" 5862 else: 5863 # Changes happened, add them 5864 ret["changes"]["diff"] = "".join(difflib.unified_diff(slines, nlines)) 5865 5866 if ret["result"]: 5867 ret["comment"] = "Commented lines successfully" 5868 else: 5869 ret["comment"] = "Expected commented lines not found" 5870 5871 return ret 5872 5873 5874def uncomment(name, regex, char="#", backup=".bak"): 5875 """ 5876 Uncomment specified commented lines in a file 5877 5878 name 5879 The full path to the file to be edited 5880 regex 5881 A regular expression used to find the lines that are to be uncommented. 5882 This regex should not include the comment character. A leading ``^`` 5883 character will be stripped for convenience (for easily switching 5884 between comment() and uncomment()). The regex will be searched for 5885 from the beginning of the line, ignoring leading spaces (we prepend 5886 '^[ \\t]*') 5887 char 5888 The character to remove in order to uncomment a line 5889 backup 5890 The file will be backed up before edit with this file extension; 5891 5892 .. warning:: 5893 5894 This backup will be overwritten each time ``sed`` / ``comment`` / 5895 ``uncomment`` is called. Meaning the backup will only be useful 5896 after the first invocation. 5897 5898 Set to False/None to not keep a backup. 5899 5900 Usage: 5901 5902 .. code-block:: yaml 5903 5904 /etc/adduser.conf: 5905 file.uncomment: 5906 - regex: EXTRA_GROUPS 5907 5908 .. versionadded:: 0.9.5 5909 """ 5910 name = os.path.expanduser(name) 5911 5912 ret = {"name": name, "changes": {}, "result": False, "comment": ""} 5913 if not name: 5914 return _error(ret, "Must provide name to file.uncomment") 5915 5916 check_res, check_msg = _check_file(name) 5917 if not check_res: 5918 return _error(ret, check_msg) 5919 5920 # Make sure the pattern appears in the file 5921 if __salt__["file.search"]( 5922 name, "{}[ \t]*{}".format(char, regex.lstrip("^")), multiline=True 5923 ): 5924 # Line exists and is commented 5925 pass 5926 elif __salt__["file.search"]( 5927 name, "^[ \t]*{}".format(regex.lstrip("^")), multiline=True 5928 ): 5929 ret["comment"] = "Pattern already uncommented" 5930 ret["result"] = True 5931 return ret 5932 else: 5933 return _error(ret, "{}: Pattern not found".format(regex)) 5934 5935 if __opts__["test"]: 5936 ret["changes"][name] = "updated" 5937 ret["comment"] = "File {} is set to be updated".format(name) 5938 ret["result"] = None 5939 return ret 5940 5941 with salt.utils.files.fopen(name, "rb") as fp_: 5942 slines = salt.utils.data.decode(fp_.readlines()) 5943 5944 # Perform the edit 5945 __salt__["file.comment_line"](name, regex, char, False, backup) 5946 5947 with salt.utils.files.fopen(name, "rb") as fp_: 5948 nlines = salt.utils.data.decode(fp_.readlines()) 5949 5950 # Check the result 5951 ret["result"] = __salt__["file.search"]( 5952 name, "^[ \t]*{}".format(regex.lstrip("^")), multiline=True 5953 ) 5954 5955 if slines != nlines: 5956 if not __utils__["files.is_text"](name): 5957 ret["changes"]["diff"] = "Replace binary file" 5958 else: 5959 # Changes happened, add them 5960 ret["changes"]["diff"] = "".join(difflib.unified_diff(slines, nlines)) 5961 5962 if ret["result"]: 5963 ret["comment"] = "Uncommented lines successfully" 5964 else: 5965 ret["comment"] = "Expected uncommented lines not found" 5966 5967 return ret 5968 5969 5970def append( 5971 name, 5972 text=None, 5973 makedirs=False, 5974 source=None, 5975 source_hash=None, 5976 template="jinja", 5977 sources=None, 5978 source_hashes=None, 5979 defaults=None, 5980 context=None, 5981 ignore_whitespace=True, 5982): 5983 """ 5984 Ensure that some text appears at the end of a file. 5985 5986 The text will not be appended if it already exists in the file. 5987 A single string of text or a list of strings may be appended. 5988 5989 name 5990 The location of the file to append to. 5991 5992 text 5993 The text to be appended, which can be a single string or a list 5994 of strings. 5995 5996 makedirs 5997 If the file is located in a path without a parent directory, 5998 then the state will fail. If makedirs is set to True, then 5999 the parent directories will be created to facilitate the 6000 creation of the named file. Defaults to False. 6001 6002 source 6003 A single source file to append. This source file can be hosted on either 6004 the salt master server, or on an HTTP or FTP server. Both HTTPS and 6005 HTTP are supported as well as downloading directly from Amazon S3 6006 compatible URLs with both pre-configured and automatic IAM credentials 6007 (see s3.get state documentation). File retrieval from Openstack Swift 6008 object storage is supported via swift://container/object_path URLs 6009 (see swift.get documentation). 6010 6011 For files hosted on the salt file server, if the file is located on 6012 the master in the directory named spam, and is called eggs, the source 6013 string is salt://spam/eggs. 6014 6015 If the file is hosted on an HTTP or FTP server, the source_hash argument 6016 is also required. 6017 6018 source_hash 6019 This can be one of the following: 6020 1. a source hash string 6021 2. the URI of a file that contains source hash strings 6022 6023 The function accepts the first encountered long unbroken alphanumeric 6024 string of correct length as a valid hash, in order from most secure to 6025 least secure: 6026 6027 .. code-block:: text 6028 6029 Type Length 6030 ====== ====== 6031 sha512 128 6032 sha384 96 6033 sha256 64 6034 sha224 56 6035 sha1 40 6036 md5 32 6037 6038 See the ``source_hash`` parameter description for :mod:`file.managed 6039 <salt.states.file.managed>` function for more details and examples. 6040 6041 template 6042 The named templating engine will be used to render the appended-to file. 6043 Defaults to ``jinja``. The following templates are supported: 6044 6045 - :mod:`cheetah<salt.renderers.cheetah>` 6046 - :mod:`genshi<salt.renderers.genshi>` 6047 - :mod:`jinja<salt.renderers.jinja>` 6048 - :mod:`mako<salt.renderers.mako>` 6049 - :mod:`py<salt.renderers.py>` 6050 - :mod:`wempy<salt.renderers.wempy>` 6051 6052 sources 6053 A list of source files to append. If the files are hosted on an HTTP or 6054 FTP server, the source_hashes argument is also required. 6055 6056 source_hashes 6057 A list of source_hashes corresponding to the sources list specified in 6058 the sources argument. 6059 6060 defaults 6061 Default context passed to the template. 6062 6063 context 6064 Overrides default context variables passed to the template. 6065 6066 ignore_whitespace 6067 .. versionadded:: 2015.8.4 6068 6069 Spaces and Tabs in text are ignored by default, when searching for the 6070 appending content, one space or multiple tabs are the same for salt. 6071 Set this option to ``False`` if you want to change this behavior. 6072 6073 Multi-line example: 6074 6075 .. code-block:: yaml 6076 6077 /etc/motd: 6078 file.append: 6079 - text: | 6080 Thou hadst better eat salt with the Philosophers of Greece, 6081 than sugar with the Courtiers of Italy. 6082 - Benjamin Franklin 6083 6084 Multiple lines of text: 6085 6086 .. code-block:: yaml 6087 6088 /etc/motd: 6089 file.append: 6090 - text: 6091 - Trust no one unless you have eaten much salt with him. 6092 - "Salt is born of the purest of parents: the sun and the sea." 6093 6094 Gather text from multiple template files: 6095 6096 .. code-block:: yaml 6097 6098 /etc/motd: 6099 file: 6100 - append 6101 - template: jinja 6102 - sources: 6103 - salt://motd/devops-messages.tmpl 6104 - salt://motd/hr-messages.tmpl 6105 - salt://motd/general-messages.tmpl 6106 6107 .. versionadded:: 0.9.5 6108 """ 6109 ret = {"name": name, "changes": {}, "result": False, "comment": ""} 6110 6111 if not name: 6112 return _error(ret, "Must provide name to file.append") 6113 6114 name = os.path.expanduser(name) 6115 6116 if sources is None: 6117 sources = [] 6118 6119 if source_hashes is None: 6120 source_hashes = [] 6121 6122 # Add sources and source_hashes with template support 6123 # NOTE: FIX 'text' and any 'source' are mutually exclusive as 'text' 6124 # is re-assigned in the original code. 6125 (ok_, err, sl_) = _unify_sources_and_hashes( 6126 source=source, 6127 source_hash=source_hash, 6128 sources=sources, 6129 source_hashes=source_hashes, 6130 ) 6131 if not ok_: 6132 return _error(ret, err) 6133 6134 if makedirs is True: 6135 dirname = os.path.dirname(name) 6136 if __opts__["test"]: 6137 ret["comment"] = "Directory {} is set to be updated".format(dirname) 6138 ret["result"] = None 6139 else: 6140 if not __salt__["file.directory_exists"](dirname): 6141 try: 6142 _makedirs(name=name) 6143 except CommandExecutionError as exc: 6144 return _error(ret, "Drive {} is not mapped".format(exc.message)) 6145 6146 check_res, check_msg, check_changes = ( 6147 _check_directory_win(dirname) 6148 if salt.utils.platform.is_windows() 6149 else _check_directory(dirname) 6150 ) 6151 6152 if not check_res: 6153 ret["changes"] = check_changes 6154 return _error(ret, check_msg) 6155 6156 check_res, check_msg = _check_file(name) 6157 if not check_res: 6158 # Try to create the file 6159 touch_ret = touch(name, makedirs=makedirs) 6160 if __opts__["test"]: 6161 return touch_ret 6162 retry_res, retry_msg = _check_file(name) 6163 if not retry_res: 6164 return _error(ret, check_msg) 6165 6166 # Follow the original logic and re-assign 'text' if using source(s)... 6167 if sl_: 6168 tmpret = _get_template_texts( 6169 source_list=sl_, template=template, defaults=defaults, context=context 6170 ) 6171 if not tmpret["result"]: 6172 return tmpret 6173 text = tmpret["data"] 6174 6175 text = _validate_str_list(text) 6176 6177 with salt.utils.files.fopen(name, "rb") as fp_: 6178 slines = fp_.read() 6179 slines = slines.decode(__salt_system_encoding__) 6180 slines = slines.splitlines() 6181 6182 append_lines = [] 6183 try: 6184 for chunk in text: 6185 if ignore_whitespace: 6186 if __salt__["file.search"]( 6187 name, 6188 salt.utils.stringutils.build_whitespace_split_regex(chunk), 6189 multiline=True, 6190 ): 6191 continue 6192 elif __salt__["file.search"](name, chunk, multiline=True): 6193 continue 6194 6195 for line_item in chunk.splitlines(): 6196 append_lines.append("{}".format(line_item)) 6197 6198 except TypeError: 6199 return _error(ret, "No text found to append. Nothing appended") 6200 6201 if __opts__["test"]: 6202 ret["comment"] = "File {} is set to be updated".format(name) 6203 ret["result"] = None 6204 nlines = list(slines) 6205 nlines.extend(append_lines) 6206 if slines != nlines: 6207 if not __utils__["files.is_text"](name): 6208 ret["changes"]["diff"] = "Replace binary file" 6209 else: 6210 # Changes happened, add them 6211 ret["changes"]["diff"] = "\n".join(difflib.unified_diff(slines, nlines)) 6212 else: 6213 ret["comment"] = "File {} is in correct state".format(name) 6214 ret["result"] = True 6215 return ret 6216 6217 if append_lines: 6218 __salt__["file.append"](name, args=append_lines) 6219 ret["comment"] = "Appended {} lines".format(len(append_lines)) 6220 else: 6221 ret["comment"] = "File {} is in correct state".format(name) 6222 6223 with salt.utils.files.fopen(name, "rb") as fp_: 6224 nlines = fp_.read() 6225 nlines = nlines.decode(__salt_system_encoding__) 6226 nlines = nlines.splitlines() 6227 6228 if slines != nlines: 6229 if not __utils__["files.is_text"](name): 6230 ret["changes"]["diff"] = "Replace binary file" 6231 else: 6232 # Changes happened, add them 6233 ret["changes"]["diff"] = "\n".join(difflib.unified_diff(slines, nlines)) 6234 6235 ret["result"] = True 6236 6237 return ret 6238 6239 6240def prepend( 6241 name, 6242 text=None, 6243 makedirs=False, 6244 source=None, 6245 source_hash=None, 6246 template="jinja", 6247 sources=None, 6248 source_hashes=None, 6249 defaults=None, 6250 context=None, 6251 header=None, 6252): 6253 """ 6254 Ensure that some text appears at the beginning of a file 6255 6256 The text will not be prepended again if it already exists in the file. You 6257 may specify a single line of text or a list of lines to append. 6258 6259 name 6260 The location of the file to append to. 6261 6262 text 6263 The text to be appended, which can be a single string or a list 6264 of strings. 6265 6266 makedirs 6267 If the file is located in a path without a parent directory, 6268 then the state will fail. If makedirs is set to True, then 6269 the parent directories will be created to facilitate the 6270 creation of the named file. Defaults to False. 6271 6272 source 6273 A single source file to append. This source file can be hosted on either 6274 the salt master server, or on an HTTP or FTP server. Both HTTPS and 6275 HTTP are supported as well as downloading directly from Amazon S3 6276 compatible URLs with both pre-configured and automatic IAM credentials 6277 (see s3.get state documentation). File retrieval from Openstack Swift 6278 object storage is supported via swift://container/object_path URLs 6279 (see swift.get documentation). 6280 6281 For files hosted on the salt file server, if the file is located on 6282 the master in the directory named spam, and is called eggs, the source 6283 string is salt://spam/eggs. 6284 6285 If the file is hosted on an HTTP or FTP server, the source_hash argument 6286 is also required. 6287 6288 source_hash 6289 This can be one of the following: 6290 1. a source hash string 6291 2. the URI of a file that contains source hash strings 6292 6293 The function accepts the first encountered long unbroken alphanumeric 6294 string of correct length as a valid hash, in order from most secure to 6295 least secure: 6296 6297 .. code-block:: text 6298 6299 Type Length 6300 ====== ====== 6301 sha512 128 6302 sha384 96 6303 sha256 64 6304 sha224 56 6305 sha1 40 6306 md5 32 6307 6308 See the ``source_hash`` parameter description for :mod:`file.managed 6309 <salt.states.file.managed>` function for more details and examples. 6310 6311 template 6312 The named templating engine will be used to render the appended-to file. 6313 Defaults to ``jinja``. The following templates are supported: 6314 6315 - :mod:`cheetah<salt.renderers.cheetah>` 6316 - :mod:`genshi<salt.renderers.genshi>` 6317 - :mod:`jinja<salt.renderers.jinja>` 6318 - :mod:`mako<salt.renderers.mako>` 6319 - :mod:`py<salt.renderers.py>` 6320 - :mod:`wempy<salt.renderers.wempy>` 6321 6322 sources 6323 A list of source files to append. If the files are hosted on an HTTP or 6324 FTP server, the source_hashes argument is also required. 6325 6326 source_hashes 6327 A list of source_hashes corresponding to the sources list specified in 6328 the sources argument. 6329 6330 defaults 6331 Default context passed to the template. 6332 6333 context 6334 Overrides default context variables passed to the template. 6335 6336 ignore_whitespace 6337 .. versionadded:: 2015.8.4 6338 6339 Spaces and Tabs in text are ignored by default, when searching for the 6340 appending content, one space or multiple tabs are the same for salt. 6341 Set this option to ``False`` if you want to change this behavior. 6342 6343 Multi-line example: 6344 6345 .. code-block:: yaml 6346 6347 /etc/motd: 6348 file.prepend: 6349 - text: | 6350 Thou hadst better eat salt with the Philosophers of Greece, 6351 than sugar with the Courtiers of Italy. 6352 - Benjamin Franklin 6353 6354 Multiple lines of text: 6355 6356 .. code-block:: yaml 6357 6358 /etc/motd: 6359 file.prepend: 6360 - text: 6361 - Trust no one unless you have eaten much salt with him. 6362 - "Salt is born of the purest of parents: the sun and the sea." 6363 6364 Optionally, require the text to appear exactly as specified 6365 (order and position). Combine with multi-line or multiple lines of input. 6366 6367 .. code-block:: yaml 6368 6369 /etc/motd: 6370 file.prepend: 6371 - header: True 6372 - text: 6373 - This will be the very first line in the file. 6374 - The 2nd line, regardless of duplicates elsewhere in the file. 6375 - These will be written anew if they do not appear verbatim. 6376 6377 Gather text from multiple template files: 6378 6379 .. code-block:: yaml 6380 6381 /etc/motd: 6382 file: 6383 - prepend 6384 - template: jinja 6385 - sources: 6386 - salt://motd/devops-messages.tmpl 6387 - salt://motd/hr-messages.tmpl 6388 - salt://motd/general-messages.tmpl 6389 6390 .. versionadded:: 2014.7.0 6391 """ 6392 name = os.path.expanduser(name) 6393 6394 ret = {"name": name, "changes": {}, "result": False, "comment": ""} 6395 if not name: 6396 return _error(ret, "Must provide name to file.prepend") 6397 6398 if sources is None: 6399 sources = [] 6400 6401 if source_hashes is None: 6402 source_hashes = [] 6403 6404 # Add sources and source_hashes with template support 6405 # NOTE: FIX 'text' and any 'source' are mutually exclusive as 'text' 6406 # is re-assigned in the original code. 6407 (ok_, err, sl_) = _unify_sources_and_hashes( 6408 source=source, 6409 source_hash=source_hash, 6410 sources=sources, 6411 source_hashes=source_hashes, 6412 ) 6413 if not ok_: 6414 return _error(ret, err) 6415 6416 if makedirs is True: 6417 dirname = os.path.dirname(name) 6418 if __opts__["test"]: 6419 ret["comment"] = "Directory {} is set to be updated".format(dirname) 6420 ret["result"] = None 6421 else: 6422 if not __salt__["file.directory_exists"](dirname): 6423 try: 6424 _makedirs(name=name) 6425 except CommandExecutionError as exc: 6426 return _error(ret, "Drive {} is not mapped".format(exc.message)) 6427 6428 check_res, check_msg, check_changes = ( 6429 _check_directory_win(dirname) 6430 if salt.utils.platform.is_windows() 6431 else _check_directory(dirname) 6432 ) 6433 6434 if not check_res: 6435 ret["changes"] = check_changes 6436 return _error(ret, check_msg) 6437 6438 check_res, check_msg = _check_file(name) 6439 if not check_res: 6440 # Try to create the file 6441 touch_ret = touch(name, makedirs=makedirs) 6442 if __opts__["test"]: 6443 return touch_ret 6444 retry_res, retry_msg = _check_file(name) 6445 if not retry_res: 6446 return _error(ret, check_msg) 6447 6448 # Follow the original logic and re-assign 'text' if using source(s)... 6449 if sl_: 6450 tmpret = _get_template_texts( 6451 source_list=sl_, template=template, defaults=defaults, context=context 6452 ) 6453 if not tmpret["result"]: 6454 return tmpret 6455 text = tmpret["data"] 6456 6457 text = _validate_str_list(text) 6458 6459 with salt.utils.files.fopen(name, "rb") as fp_: 6460 slines = fp_.read() 6461 slines = slines.decode(__salt_system_encoding__) 6462 slines = slines.splitlines(True) 6463 6464 count = 0 6465 test_lines = [] 6466 6467 preface = [] 6468 for chunk in text: 6469 6470 # if header kwarg is unset of False, use regex search 6471 if not header: 6472 if __salt__["file.search"]( 6473 name, 6474 salt.utils.stringutils.build_whitespace_split_regex(chunk), 6475 multiline=True, 6476 ): 6477 continue 6478 6479 lines = chunk.splitlines() 6480 6481 for line in lines: 6482 if __opts__["test"]: 6483 ret["comment"] = "File {} is set to be updated".format(name) 6484 ret["result"] = None 6485 test_lines.append("{}\n".format(line)) 6486 else: 6487 preface.append(line) 6488 count += 1 6489 6490 if __opts__["test"]: 6491 nlines = test_lines + slines 6492 if slines != nlines: 6493 if not __utils__["files.is_text"](name): 6494 ret["changes"]["diff"] = "Replace binary file" 6495 else: 6496 # Changes happened, add them 6497 ret["changes"]["diff"] = "".join(difflib.unified_diff(slines, nlines)) 6498 ret["result"] = None 6499 else: 6500 ret["comment"] = "File {} is in correct state".format(name) 6501 ret["result"] = True 6502 return ret 6503 6504 # if header kwarg is True, use verbatim compare 6505 if header: 6506 with salt.utils.files.fopen(name, "rb") as fp_: 6507 # read as many lines of target file as length of user input 6508 contents = fp_.read() 6509 contents = contents.decode(__salt_system_encoding__) 6510 contents = contents.splitlines(True) 6511 target_head = contents[0 : len(preface)] 6512 target_lines = [] 6513 # strip newline chars from list entries 6514 for chunk in target_head: 6515 target_lines += chunk.splitlines() 6516 # compare current top lines in target file with user input 6517 # and write user input if they differ 6518 if target_lines != preface: 6519 __salt__["file.prepend"](name, *preface) 6520 else: 6521 # clear changed lines counter if target file not modified 6522 count = 0 6523 else: 6524 __salt__["file.prepend"](name, *preface) 6525 6526 with salt.utils.files.fopen(name, "rb") as fp_: 6527 nlines = fp_.read() 6528 nlines = nlines.decode(__salt_system_encoding__) 6529 nlines = nlines.splitlines(True) 6530 6531 if slines != nlines: 6532 if not __utils__["files.is_text"](name): 6533 ret["changes"]["diff"] = "Replace binary file" 6534 else: 6535 # Changes happened, add them 6536 ret["changes"]["diff"] = "".join(difflib.unified_diff(slines, nlines)) 6537 6538 if count: 6539 ret["comment"] = "Prepended {} lines".format(count) 6540 else: 6541 ret["comment"] = "File {} is in correct state".format(name) 6542 ret["result"] = True 6543 return ret 6544 6545 6546def patch( 6547 name, 6548 source=None, 6549 source_hash=None, 6550 source_hash_name=None, 6551 skip_verify=False, 6552 template=None, 6553 context=None, 6554 defaults=None, 6555 options="", 6556 reject_file=None, 6557 strip=None, 6558 saltenv=None, 6559 **kwargs 6560): 6561 """ 6562 Ensure that a patch has been applied to the specified file or directory 6563 6564 .. versionchanged:: 2019.2.0 6565 The ``hash`` and ``dry_run_first`` options are now ignored, as the 6566 logic which determines whether or not the patch has already been 6567 applied no longer requires them. Additionally, this state now supports 6568 patch files that modify more than one file. To use these sort of 6569 patches, specify a directory (and, if necessary, the ``strip`` option) 6570 instead of a file. 6571 6572 .. note:: 6573 A suitable ``patch`` executable must be available on the minion. Also, 6574 keep in mind that the pre-check this state does to determine whether or 6575 not changes need to be made will create a temp file and send all patch 6576 output to that file. This means that, in the event that the patch would 6577 not have applied cleanly, the comment included in the state results will 6578 reference a temp file that will no longer exist once the state finishes 6579 running. 6580 6581 name 6582 The file or directory to which the patch should be applied 6583 6584 source 6585 The patch file to apply 6586 6587 .. versionchanged:: 2019.2.0 6588 The source can now be from any file source supported by Salt 6589 (``salt://``, ``http://``, ``https://``, ``ftp://``, etc.). 6590 Templating is also now supported. 6591 6592 source_hash 6593 Works the same way as in :py:func:`file.managed 6594 <salt.states.file.managed>`. 6595 6596 .. versionadded:: 2019.2.0 6597 6598 source_hash_name 6599 Works the same way as in :py:func:`file.managed 6600 <salt.states.file.managed>` 6601 6602 .. versionadded:: 2019.2.0 6603 6604 skip_verify 6605 Works the same way as in :py:func:`file.managed 6606 <salt.states.file.managed>` 6607 6608 .. versionadded:: 2019.2.0 6609 6610 template 6611 Works the same way as in :py:func:`file.managed 6612 <salt.states.file.managed>` 6613 6614 .. versionadded:: 2019.2.0 6615 6616 context 6617 Works the same way as in :py:func:`file.managed 6618 <salt.states.file.managed>` 6619 6620 .. versionadded:: 2019.2.0 6621 6622 defaults 6623 Works the same way as in :py:func:`file.managed 6624 <salt.states.file.managed>` 6625 6626 .. versionadded:: 2019.2.0 6627 6628 options 6629 Extra options to pass to patch. This should not be necessary in most 6630 cases. 6631 6632 .. note:: 6633 For best results, short opts should be separate from one another. 6634 The ``-N`` and ``-r``, and ``-o`` options are used internally by 6635 this state and cannot be used here. Additionally, instead of using 6636 ``-pN`` or ``--strip=N``, use the ``strip`` option documented 6637 below. 6638 6639 reject_file 6640 If specified, any rejected hunks will be written to this file. If not 6641 specified, then they will be written to a temp file which will be 6642 deleted when the state finishes running. 6643 6644 .. important:: 6645 The parent directory must exist. Also, this will overwrite the file 6646 if it is already present. 6647 6648 .. versionadded:: 2019.2.0 6649 6650 strip 6651 Number of directories to strip from paths in the patch file. For 6652 example, using the below SLS would instruct Salt to use ``-p1`` when 6653 applying the patch: 6654 6655 .. code-block:: yaml 6656 6657 /etc/myfile.conf: 6658 file.patch: 6659 - source: salt://myfile.patch 6660 - strip: 1 6661 6662 .. versionadded:: 2019.2.0 6663 In previous versions, ``-p1`` would need to be passed as part of 6664 the ``options`` value. 6665 6666 saltenv 6667 Specify the environment from which to retrieve the patch file indicated 6668 by the ``source`` parameter. If not provided, this defaults to the 6669 environment from which the state is being executed. 6670 6671 .. note:: 6672 Ignored when the patch file is from a non-``salt://`` source. 6673 6674 **Usage:** 6675 6676 .. code-block:: yaml 6677 6678 # Equivalent to ``patch --forward /opt/myfile.txt myfile.patch`` 6679 /opt/myfile.txt: 6680 file.patch: 6681 - source: salt://myfile.patch 6682 """ 6683 ret = {"name": name, "changes": {}, "result": False, "comment": ""} 6684 6685 if not salt.utils.path.which("patch"): 6686 ret["comment"] = "patch executable not found on minion" 6687 return ret 6688 6689 # is_dir should be defined if we proceed past the if/else block below, but 6690 # just in case, avoid a NameError. 6691 is_dir = False 6692 6693 if not name: 6694 ret["comment"] = "A file/directory to be patched is required" 6695 return ret 6696 else: 6697 try: 6698 name = os.path.expanduser(name) 6699 except Exception: # pylint: disable=broad-except 6700 ret["comment"] = "Invalid path '{}'".format(name) 6701 return ret 6702 else: 6703 if not os.path.isabs(name): 6704 ret["comment"] = "{} is not an absolute path".format(name) 6705 return ret 6706 elif not os.path.exists(name): 6707 ret["comment"] = "{} does not exist".format(name) 6708 return ret 6709 else: 6710 is_dir = os.path.isdir(name) 6711 6712 for deprecated_arg in ("hash", "dry_run_first"): 6713 if deprecated_arg in kwargs: 6714 ret.setdefault("warnings", []).append( 6715 "The '{}' argument is no longer used and has been ignored.".format( 6716 deprecated_arg 6717 ) 6718 ) 6719 6720 if reject_file is not None: 6721 try: 6722 reject_file_parent = os.path.dirname(reject_file) 6723 except Exception: # pylint: disable=broad-except 6724 ret["comment"] = "Invalid path '{}' for reject_file".format(reject_file) 6725 return ret 6726 else: 6727 if not os.path.isabs(reject_file_parent): 6728 ret["comment"] = "'{}' is not an absolute path".format(reject_file) 6729 return ret 6730 elif not os.path.isdir(reject_file_parent): 6731 ret["comment"] = ( 6732 "Parent directory for reject_file '{}' either does " 6733 "not exist, or is not a directory".format(reject_file) 6734 ) 6735 return ret 6736 6737 sanitized_options = [] 6738 options = salt.utils.args.shlex_split(options) 6739 index = 0 6740 max_index = len(options) - 1 6741 # Not using enumerate here because we may need to consume more than one 6742 # option if --strip is used. 6743 blacklisted_options = [] 6744 while index <= max_index: 6745 option = options[index] 6746 if not isinstance(option, str): 6747 option = str(option) 6748 6749 for item in ("-N", "--forward", "-r", "--reject-file", "-o", "--output"): 6750 if option.startswith(item): 6751 blacklisted = option 6752 break 6753 else: 6754 blacklisted = None 6755 6756 if blacklisted is not None: 6757 blacklisted_options.append(blacklisted) 6758 6759 if option.startswith("-p"): 6760 try: 6761 strip = int(option[2:]) 6762 except Exception: # pylint: disable=broad-except 6763 ret["comment"] = ( 6764 "Invalid format for '-p' CLI option. Consider using " 6765 "the 'strip' option for this state." 6766 ) 6767 return ret 6768 elif option.startswith("--strip"): 6769 if "=" in option: 6770 # Assume --strip=N 6771 try: 6772 strip = int(option.rsplit("=", 1)[-1]) 6773 except Exception: # pylint: disable=broad-except 6774 ret["comment"] = ( 6775 "Invalid format for '-strip' CLI option. Consider " 6776 "using the 'strip' option for this state." 6777 ) 6778 return ret 6779 else: 6780 # Assume --strip N and grab the next option in the list 6781 try: 6782 strip = int(options[index + 1]) 6783 except Exception: # pylint: disable=broad-except 6784 ret["comment"] = ( 6785 "Invalid format for '-strip' CLI option. Consider " 6786 "using the 'strip' option for this state." 6787 ) 6788 return ret 6789 else: 6790 # We need to increment again because we grabbed the next 6791 # option in the list. 6792 index += 1 6793 else: 6794 sanitized_options.append(option) 6795 6796 # Increment the index 6797 index += 1 6798 6799 if blacklisted_options: 6800 ret["comment"] = "The following CLI options are not allowed: {}".format( 6801 ", ".join(blacklisted_options) 6802 ) 6803 return ret 6804 6805 options = sanitized_options 6806 6807 try: 6808 source_match = __salt__["file.source_list"](source, source_hash, __env__)[0] 6809 except CommandExecutionError as exc: 6810 ret["result"] = False 6811 ret["comment"] = exc.strerror 6812 return ret 6813 else: 6814 # Passing the saltenv to file.managed to pull down the patch file is 6815 # not supported, because the saltenv is already being passed via the 6816 # state compiler and this would result in two values for that argument 6817 # (and a traceback). Therefore, we will add the saltenv to the source 6818 # URL to ensure we pull the file from the correct environment. 6819 if saltenv is not None: 6820 source_match_url, source_match_saltenv = salt.utils.url.parse(source_match) 6821 if source_match_url.startswith("salt://"): 6822 if source_match_saltenv is not None and source_match_saltenv != saltenv: 6823 ret.setdefault("warnings", []).append( 6824 "Ignoring 'saltenv' option in favor of saltenv " 6825 "included in the source URL." 6826 ) 6827 else: 6828 source_match += "?saltenv={}".format(saltenv) 6829 6830 cleanup = [] 6831 6832 try: 6833 patch_file = salt.utils.files.mkstemp() 6834 cleanup.append(patch_file) 6835 6836 try: 6837 orig_test = __opts__["test"] 6838 __opts__["test"] = False 6839 sys.modules[__salt__["file.patch"].__module__].__opts__["test"] = False 6840 result = managed( 6841 patch_file, 6842 source=source_match, 6843 source_hash=source_hash, 6844 source_hash_name=source_hash_name, 6845 skip_verify=skip_verify, 6846 template=template, 6847 context=context, 6848 defaults=defaults, 6849 ) 6850 except Exception as exc: # pylint: disable=broad-except 6851 msg = "Failed to cache patch file {}: {}".format( 6852 salt.utils.url.redact_http_basic_auth(source_match), exc 6853 ) 6854 log.exception(msg) 6855 ret["comment"] = msg 6856 return ret 6857 else: 6858 log.debug("file.managed: %s", result) 6859 finally: 6860 __opts__["test"] = orig_test 6861 sys.modules[__salt__["file.patch"].__module__].__opts__["test"] = orig_test 6862 6863 # TODO adding the not orig_test is just a patch 6864 # The call above to managed ends up in win_dacl utility and overwrites 6865 # the ret dict, specifically "result". This surfaces back here and is 6866 # providing an incorrect representation of the actual value. 6867 # This fix requires re-working the dacl utility when test mode is passed 6868 # to it from another function, such as this one, and it overwrites ret. 6869 if not orig_test and not result["result"]: 6870 log.debug( 6871 "failed to download %s", 6872 salt.utils.url.redact_http_basic_auth(source_match), 6873 ) 6874 return result 6875 6876 def _patch(patch_file, options=None, dry_run=False): 6877 patch_opts = copy.copy(sanitized_options) 6878 if options is not None: 6879 patch_opts.extend(options) 6880 return __salt__["file.patch"]( 6881 name, patch_file, options=patch_opts, dry_run=dry_run 6882 ) 6883 6884 if reject_file is not None: 6885 patch_rejects = reject_file 6886 else: 6887 # No rejects file specified, create a temp file 6888 patch_rejects = salt.utils.files.mkstemp() 6889 cleanup.append(patch_rejects) 6890 6891 patch_output = salt.utils.files.mkstemp() 6892 cleanup.append(patch_output) 6893 6894 # Older patch releases can only write patch output to regular files, 6895 # meaning that /dev/null can't be relied on. Also, if we ever want this 6896 # to work on Windows with patch.exe, /dev/null is a non-starter. 6897 # Therefore, redirect all patch output to a temp file, which we will 6898 # then remove. 6899 patch_opts = ["-N", "-r", patch_rejects, "-o", patch_output] 6900 if is_dir and strip is not None: 6901 patch_opts.append("-p{}".format(strip)) 6902 6903 pre_check = _patch(patch_file, patch_opts) 6904 if pre_check["retcode"] != 0: 6905 # Try to reverse-apply hunks from rejects file using a dry-run. 6906 # If this returns a retcode of 0, we know that the patch was 6907 # already applied. Rejects are written from the base of the 6908 # directory, so the strip option doesn't apply here. 6909 reverse_pass = _patch(patch_rejects, ["-R", "-f"], dry_run=True) 6910 already_applied = reverse_pass["retcode"] == 0 6911 6912 # Check if the patch command threw an error upon execution 6913 # and return the error here. According to gnu on patch-messages 6914 # patch exits with a status of 0 if successful 6915 # patch exits with a status of 1 if some hunks cannot be applied 6916 # patch exits with a status of 2 if something else went wrong 6917 # www.gnu.org/software/diffutils/manual/html_node/patch-Messages.html 6918 if pre_check["retcode"] == 2 and pre_check["stderr"]: 6919 ret["comment"] = pre_check["stderr"] 6920 ret["result"] = False 6921 return ret 6922 if already_applied: 6923 ret["comment"] = "Patch was already applied" 6924 ret["result"] = True 6925 return ret 6926 else: 6927 ret["comment"] = ( 6928 "Patch would not apply cleanly, no changes made. Results " 6929 "of dry-run are below." 6930 ) 6931 if reject_file is None: 6932 ret["comment"] += ( 6933 " Run state again using the reject_file option to " 6934 "save rejects to a persistent file." 6935 ) 6936 opts = copy.copy(__opts__) 6937 opts["color"] = False 6938 ret["comment"] += "\n\n" + salt.output.out_format( 6939 pre_check, "nested", opts, nested_indent=14 6940 ) 6941 return ret 6942 6943 if __opts__["test"]: 6944 ret["result"] = None 6945 ret["comment"] = "The patch would be applied" 6946 ret["changes"] = pre_check 6947 return ret 6948 6949 # If we've made it here, the patch should apply cleanly 6950 patch_opts = [] 6951 if is_dir and strip is not None: 6952 patch_opts.append("-p{}".format(strip)) 6953 ret["changes"] = _patch(patch_file, patch_opts) 6954 6955 if ret["changes"]["retcode"] == 0: 6956 ret["comment"] = "Patch successfully applied" 6957 ret["result"] = True 6958 else: 6959 ret["comment"] = "Failed to apply patch" 6960 6961 return ret 6962 6963 finally: 6964 # Clean up any temp files 6965 for path in cleanup: 6966 try: 6967 os.remove(path) 6968 except OSError as exc: 6969 if exc.errno != os.errno.ENOENT: 6970 log.error( 6971 "file.patch: Failed to remove temp file %s: %s", path, exc 6972 ) 6973 6974 6975def touch(name, atime=None, mtime=None, makedirs=False): 6976 """ 6977 Replicate the 'nix "touch" command to create a new empty 6978 file or update the atime and mtime of an existing file. 6979 6980 Note that if you just want to create a file and don't care about atime or 6981 mtime, you should use ``file.managed`` instead, as it is more 6982 feature-complete. (Just leave out the ``source``/``template``/``contents`` 6983 arguments, and it will just create the file and/or check its permissions, 6984 without messing with contents) 6985 6986 name 6987 name of the file 6988 6989 atime 6990 atime of the file 6991 6992 mtime 6993 mtime of the file 6994 6995 makedirs 6996 whether we should create the parent directory/directories in order to 6997 touch the file 6998 6999 Usage: 7000 7001 .. code-block:: yaml 7002 7003 /var/log/httpd/logrotate.empty: 7004 file.touch 7005 7006 .. versionadded:: 0.9.5 7007 """ 7008 name = os.path.expanduser(name) 7009 7010 ret = { 7011 "name": name, 7012 "changes": {}, 7013 } 7014 if not name: 7015 return _error(ret, "Must provide name to file.touch") 7016 if not os.path.isabs(name): 7017 return _error(ret, "Specified file {} is not an absolute path".format(name)) 7018 7019 if __opts__["test"]: 7020 ret.update(_check_touch(name, atime, mtime)) 7021 return ret 7022 7023 if makedirs: 7024 try: 7025 _makedirs(name=name) 7026 except CommandExecutionError as exc: 7027 return _error(ret, "Drive {} is not mapped".format(exc.message)) 7028 if not os.path.isdir(os.path.dirname(name)): 7029 return _error(ret, "Directory not present to touch file {}".format(name)) 7030 7031 extant = os.path.exists(name) 7032 7033 ret["result"] = __salt__["file.touch"](name, atime, mtime) 7034 if not extant and ret["result"]: 7035 ret["comment"] = "Created empty file {}".format(name) 7036 ret["changes"]["new"] = name 7037 elif extant and ret["result"]: 7038 ret["comment"] = "Updated times on {} {}".format( 7039 "directory" if os.path.isdir(name) else "file", name 7040 ) 7041 ret["changes"]["touched"] = name 7042 7043 return ret 7044 7045 7046def copy_( 7047 name, 7048 source, 7049 force=False, 7050 makedirs=False, 7051 preserve=False, 7052 user=None, 7053 group=None, 7054 mode=None, 7055 subdir=False, 7056 **kwargs 7057): 7058 """ 7059 If the file defined by the ``source`` option exists on the minion, copy it 7060 to the named path. The file will not be overwritten if it already exists, 7061 unless the ``force`` option is set to ``True``. 7062 7063 .. note:: 7064 This state only copies files from one location on a minion to another 7065 location on the same minion. For copying files from the master, use a 7066 :py:func:`file.managed <salt.states.file.managed>` state. 7067 7068 name 7069 The location of the file to copy to 7070 7071 source 7072 The location of the file to copy to the location specified with name 7073 7074 force 7075 If the target location is present then the file will not be moved, 7076 specify "force: True" to overwrite the target file 7077 7078 makedirs 7079 If the target subdirectories don't exist create them 7080 7081 preserve 7082 .. versionadded:: 2015.5.0 7083 7084 Set ``preserve: True`` to preserve user/group ownership and mode 7085 after copying. Default is ``False``. If ``preserve`` is set to ``True``, 7086 then user/group/mode attributes will be ignored. 7087 7088 user 7089 .. versionadded:: 2015.5.0 7090 7091 The user to own the copied file, this defaults to the user salt is 7092 running as on the minion. If ``preserve`` is set to ``True``, then 7093 this will be ignored 7094 7095 group 7096 .. versionadded:: 2015.5.0 7097 7098 The group to own the copied file, this defaults to the group salt is 7099 running as on the minion. If ``preserve`` is set to ``True`` or on 7100 Windows this will be ignored 7101 7102 mode 7103 .. versionadded:: 2015.5.0 7104 7105 The permissions to set on the copied file, aka 644, '0775', '4664'. 7106 If ``preserve`` is set to ``True``, then this will be ignored. 7107 Not supported on Windows. 7108 7109 The default mode for new files and directories corresponds umask of salt 7110 process. For existing files and directories it's not enforced. 7111 7112 subdir 7113 .. versionadded:: 2015.5.0 7114 7115 If the name is a directory then place the file inside the named 7116 directory 7117 7118 .. note:: 7119 The copy function accepts paths that are local to the Salt minion. 7120 This function does not support salt://, http://, or the other 7121 additional file paths that are supported by :mod:`states.file.managed 7122 <salt.states.file.managed>` and :mod:`states.file.recurse 7123 <salt.states.file.recurse>`. 7124 7125 Usage: 7126 7127 .. code-block:: yaml 7128 7129 # Use 'copy', not 'copy_' 7130 /etc/example.conf: 7131 file.copy: 7132 - source: /tmp/example.conf 7133 """ 7134 name = os.path.expanduser(name) 7135 source = os.path.expanduser(source) 7136 7137 ret = { 7138 "name": name, 7139 "changes": {}, 7140 "comment": 'Copied "{}" to "{}"'.format(source, name), 7141 "result": True, 7142 } 7143 if not name: 7144 return _error(ret, "Must provide name to file.copy") 7145 7146 changed = True 7147 if not os.path.isabs(name): 7148 return _error(ret, "Specified file {} is not an absolute path".format(name)) 7149 7150 if not os.path.exists(source): 7151 return _error(ret, 'Source file "{}" is not present'.format(source)) 7152 7153 if preserve: 7154 user = __salt__["file.get_user"](source) 7155 group = __salt__["file.get_group"](source) 7156 mode = __salt__["file.get_mode"](source) 7157 else: 7158 user = _test_owner(kwargs, user=user) 7159 if user is None: 7160 user = __opts__["user"] 7161 7162 if salt.utils.platform.is_windows(): 7163 if group is not None: 7164 log.warning( 7165 "The group argument for %s has been ignored as this is " 7166 "a Windows system.", 7167 name, 7168 ) 7169 group = user 7170 7171 if group is None: 7172 if "user.info" in __salt__: 7173 group = __salt__["file.gid_to_group"]( 7174 __salt__["user.info"](user).get("gid", 0) 7175 ) 7176 else: 7177 group = user 7178 7179 u_check = _check_user(user, group) 7180 if u_check: 7181 # The specified user or group do not exist 7182 return _error(ret, u_check) 7183 7184 if mode is None: 7185 mode = __salt__["file.get_mode"](source) 7186 7187 if os.path.isdir(name) and subdir: 7188 # If the target is a dir, and overwrite_dir is False, copy into the dir 7189 name = os.path.join(name, os.path.basename(source)) 7190 7191 if os.path.lexists(source) and os.path.lexists(name): 7192 # if this is a file which did not change, do not update 7193 if force and os.path.isfile(name): 7194 hash1 = salt.utils.hashutils.get_hash(name) 7195 hash2 = salt.utils.hashutils.get_hash(source) 7196 if hash1 == hash2: 7197 changed = True 7198 ret["comment"] = " ".join( 7199 [ret["comment"], "- files are identical but force flag is set"] 7200 ) 7201 if not force: 7202 changed = False 7203 elif not __opts__["test"] and changed: 7204 # Remove the destination to prevent problems later 7205 try: 7206 __salt__["file.remove"](name) 7207 except OSError: 7208 return _error( 7209 ret, 7210 'Failed to delete "{}" in preparation for forced move'.format(name), 7211 ) 7212 7213 if __opts__["test"]: 7214 if changed: 7215 ret["comment"] = 'File "{}" is set to be copied to "{}"'.format( 7216 source, name 7217 ) 7218 ret["result"] = None 7219 else: 7220 ret[ 7221 "comment" 7222 ] = 'The target file "{}" exists and will not be overwritten'.format(name) 7223 ret["result"] = True 7224 return ret 7225 7226 if not changed: 7227 ret[ 7228 "comment" 7229 ] = 'The target file "{}" exists and will not be overwritten'.format(name) 7230 ret["result"] = True 7231 return ret 7232 7233 # Run makedirs 7234 dname = os.path.dirname(name) 7235 if not os.path.isdir(dname): 7236 if makedirs: 7237 try: 7238 _makedirs(name=name, user=user, group=group, dir_mode=mode) 7239 except CommandExecutionError as exc: 7240 return _error(ret, "Drive {} is not mapped".format(exc.message)) 7241 else: 7242 return _error(ret, "The target directory {} is not present".format(dname)) 7243 # All tests pass, move the file into place 7244 try: 7245 if os.path.isdir(source): 7246 shutil.copytree(source, name, symlinks=True) 7247 for root, dirs, files in salt.utils.path.os_walk(name): 7248 for dir_ in dirs: 7249 __salt__["file.lchown"](os.path.join(root, dir_), user, group) 7250 for file_ in files: 7251 __salt__["file.lchown"](os.path.join(root, file_), user, group) 7252 else: 7253 shutil.copy(source, name) 7254 ret["changes"] = {name: source} 7255 # Preserve really means just keep the behavior of the cp command. If 7256 # the filesystem we're copying to is squashed or doesn't support chown 7257 # then we shouldn't be checking anything. 7258 if not preserve: 7259 if salt.utils.platform.is_windows(): 7260 # TODO: Add the other win_* parameters to this function 7261 check_ret = __salt__["file.check_perms"](path=name, ret=ret, owner=user) 7262 else: 7263 check_ret, perms = __salt__["file.check_perms"]( 7264 name, ret, user, group, mode 7265 ) 7266 if not check_ret["result"]: 7267 ret["result"] = check_ret["result"] 7268 ret["comment"] = check_ret["comment"] 7269 except OSError: 7270 return _error(ret, 'Failed to copy "{}" to "{}"'.format(source, name)) 7271 return ret 7272 7273 7274def rename(name, source, force=False, makedirs=False, **kwargs): 7275 """ 7276 If the source file exists on the system, rename it to the named file. The 7277 named file will not be overwritten if it already exists unless the force 7278 option is set to True. 7279 7280 name 7281 The location of the file to rename to 7282 7283 source 7284 The location of the file to move to the location specified with name 7285 7286 force 7287 If the target location is present then the file will not be moved, 7288 specify "force: True" to overwrite the target file 7289 7290 makedirs 7291 If the target subdirectories don't exist create them 7292 7293 """ 7294 name = os.path.expanduser(name) 7295 source = os.path.expanduser(source) 7296 7297 ret = {"name": name, "changes": {}, "comment": "", "result": True} 7298 if not name: 7299 return _error(ret, "Must provide name to file.rename") 7300 7301 if not os.path.isabs(name): 7302 return _error(ret, "Specified file {} is not an absolute path".format(name)) 7303 7304 if not os.path.lexists(source): 7305 ret["comment"] = 'Source file "{}" has already been moved out of place'.format( 7306 source 7307 ) 7308 return ret 7309 7310 if os.path.lexists(source) and os.path.lexists(name): 7311 if not force: 7312 ret[ 7313 "comment" 7314 ] = 'The target file "{}" exists and will not be overwritten'.format(name) 7315 return ret 7316 elif not __opts__["test"]: 7317 # Remove the destination to prevent problems later 7318 try: 7319 __salt__["file.remove"](name) 7320 except OSError: 7321 return _error( 7322 ret, 7323 'Failed to delete "{}" in preparation for forced move'.format(name), 7324 ) 7325 7326 if __opts__["test"]: 7327 ret["comment"] = 'File "{}" is set to be moved to "{}"'.format(source, name) 7328 ret["result"] = None 7329 return ret 7330 7331 # Run makedirs 7332 dname = os.path.dirname(name) 7333 if not os.path.isdir(dname): 7334 if makedirs: 7335 try: 7336 _makedirs(name=name) 7337 except CommandExecutionError as exc: 7338 return _error(ret, "Drive {} is not mapped".format(exc.message)) 7339 else: 7340 return _error(ret, "The target directory {} is not present".format(dname)) 7341 # All tests pass, move the file into place 7342 try: 7343 if os.path.islink(source): 7344 linkto = os.readlink(source) 7345 os.symlink(linkto, name) 7346 os.unlink(source) 7347 else: 7348 shutil.move(source, name) 7349 except OSError: 7350 return _error(ret, 'Failed to move "{}" to "{}"'.format(source, name)) 7351 7352 ret["comment"] = 'Moved "{}" to "{}"'.format(source, name) 7353 ret["changes"] = {name: source} 7354 return ret 7355 7356 7357def accumulated(name, filename, text, **kwargs): 7358 """ 7359 Prepare accumulator which can be used in template in file.managed state. 7360 Accumulator dictionary becomes available in template. It can also be used 7361 in file.blockreplace. 7362 7363 name 7364 Accumulator name 7365 7366 filename 7367 Filename which would receive this accumulator (see file.managed state 7368 documentation about ``name``) 7369 7370 text 7371 String or list for adding in accumulator 7372 7373 require_in / watch_in 7374 One of them required for sure we fill up accumulator before we manage 7375 the file. Probably the same as filename 7376 7377 Example: 7378 7379 Given the following: 7380 7381 .. code-block:: yaml 7382 7383 animals_doing_things: 7384 file.accumulated: 7385 - filename: /tmp/animal_file.txt 7386 - text: ' jumps over the lazy dog.' 7387 - require_in: 7388 - file: animal_file 7389 7390 animal_file: 7391 file.managed: 7392 - name: /tmp/animal_file.txt 7393 - source: salt://animal_file.txt 7394 - template: jinja 7395 7396 One might write a template for ``animal_file.txt`` like the following: 7397 7398 .. code-block:: jinja 7399 7400 The quick brown fox{% for animal in accumulator['animals_doing_things'] %}{{ animal }}{% endfor %} 7401 7402 Collectively, the above states and template file will produce: 7403 7404 .. code-block:: text 7405 7406 The quick brown fox jumps over the lazy dog. 7407 7408 Multiple accumulators can be "chained" together. 7409 7410 .. note:: 7411 The 'accumulator' data structure is a Python dictionary. 7412 Do not expect any loop over the keys in a deterministic order! 7413 """ 7414 ret = {"name": name, "changes": {}, "result": True, "comment": ""} 7415 if not name: 7416 return _error(ret, "Must provide name to file.accumulated") 7417 if text is None: 7418 ret["result"] = False 7419 ret["comment"] = "No text supplied for accumulator" 7420 return ret 7421 require_in = __low__.get("require_in", []) 7422 watch_in = __low__.get("watch_in", []) 7423 deps = require_in + watch_in 7424 if not [x for x in deps if "file" in x]: 7425 ret["result"] = False 7426 ret["comment"] = "Orphaned accumulator {} in {}:{}".format( 7427 name, __low__["__sls__"], __low__["__id__"] 7428 ) 7429 return ret 7430 if isinstance(text, str): 7431 text = (text,) 7432 elif isinstance(text, dict): 7433 text = (text,) 7434 accum_data, accum_deps = _load_accumulators() 7435 if filename not in accum_data: 7436 accum_data[filename] = {} 7437 if filename not in accum_deps: 7438 accum_deps[filename] = {} 7439 if name not in accum_deps[filename]: 7440 accum_deps[filename][name] = [] 7441 for accumulator in deps: 7442 if isinstance(accumulator, (dict, OrderedDict)): 7443 accum_deps[filename][name].extend(accumulator.values()) 7444 else: 7445 accum_deps[filename][name].extend(accumulator) 7446 if name not in accum_data[filename]: 7447 accum_data[filename][name] = [] 7448 for chunk in text: 7449 if chunk not in accum_data[filename][name]: 7450 accum_data[filename][name].append(chunk) 7451 ret["comment"] = "Accumulator {} for file {} was charged by text".format( 7452 name, filename 7453 ) 7454 _persist_accummulators(accum_data, accum_deps) 7455 return ret 7456 7457 7458def serialize( 7459 name, 7460 dataset=None, 7461 dataset_pillar=None, 7462 user=None, 7463 group=None, 7464 mode=None, 7465 backup="", 7466 makedirs=False, 7467 show_changes=True, 7468 create=True, 7469 merge_if_exists=False, 7470 encoding=None, 7471 encoding_errors="strict", 7472 serializer=None, 7473 serializer_opts=None, 7474 deserializer_opts=None, 7475 **kwargs 7476): 7477 """ 7478 Serializes dataset and store it into managed file. Useful for sharing 7479 simple configuration files. 7480 7481 name 7482 The location of the file to create 7483 7484 dataset 7485 The dataset that will be serialized 7486 7487 dataset_pillar 7488 Operates like ``dataset``, but draws from a value stored in pillar, 7489 using the pillar path syntax used in :mod:`pillar.get 7490 <salt.modules.pillar.get>`. This is useful when the pillar value 7491 contains newlines, as referencing a pillar variable using a jinja/mako 7492 template can result in YAML formatting issues due to the newlines 7493 causing indentation mismatches. 7494 7495 .. versionadded:: 2015.8.0 7496 7497 serializer (or formatter) 7498 Write the data as this format. See the list of 7499 :ref:`all-salt.serializers` for supported output formats. 7500 7501 .. versionchanged:: 3002 7502 ``serializer`` argument added as an alternative to ``formatter``. 7503 Both are accepted, but using both will result in an error. 7504 7505 encoding 7506 If specified, then the specified encoding will be used. Otherwise, the 7507 file will be encoded using the system locale (usually UTF-8). See 7508 https://docs.python.org/3/library/codecs.html#standard-encodings for 7509 the list of available encodings. 7510 7511 .. versionadded:: 2017.7.0 7512 7513 encoding_errors 7514 Error encoding scheme. Default is ```'strict'```. 7515 See https://docs.python.org/2/library/codecs.html#codec-base-classes 7516 for the list of available schemes. 7517 7518 .. versionadded:: 2017.7.0 7519 7520 user 7521 The user to own the directory, this defaults to the user salt is 7522 running as on the minion 7523 7524 group 7525 The group ownership set for the directory, this defaults to the group 7526 salt is running as on the minion 7527 7528 mode 7529 The permissions to set on this file, e.g. ``644``, ``0775``, or 7530 ``4664``. 7531 7532 The default mode for new files and directories corresponds umask of salt 7533 process. For existing files and directories it's not enforced. 7534 7535 .. note:: 7536 This option is **not** supported on Windows. 7537 7538 backup 7539 Overrides the default backup mode for this specific file. 7540 7541 makedirs 7542 Create parent directories for destination file. 7543 7544 .. versionadded:: 2014.1.3 7545 7546 show_changes 7547 Output a unified diff of the old file and the new file. If ``False`` 7548 return a boolean if any changes were made. 7549 7550 create 7551 Default is True, if create is set to False then the file will only be 7552 managed if the file already exists on the system. 7553 7554 merge_if_exists 7555 Default is False, if merge_if_exists is True then the existing file will 7556 be parsed and the dataset passed in will be merged with the existing 7557 content 7558 7559 .. versionadded:: 2014.7.0 7560 7561 serializer_opts 7562 Pass through options to serializer. For example: 7563 7564 .. code-block:: yaml 7565 7566 /etc/dummy/package.yaml 7567 file.serialize: 7568 - serializer: yaml 7569 - serializer_opts: 7570 - explicit_start: True 7571 - default_flow_style: True 7572 - indent: 4 7573 7574 The valid opts are the additional opts (i.e. not the data being 7575 serialized) for the function used to serialize the data. Documentation 7576 for the these functions can be found in the list below: 7577 7578 - For **yaml**: `yaml.dump()`_ 7579 - For **json**: `json.dumps()`_ 7580 - For **python**: `pprint.pformat()`_ 7581 - For **msgpack**: Run ``python -c 'import msgpack; help(msgpack.Packer)'`` 7582 to see the available options (``encoding``, ``unicode_errors``, etc.) 7583 7584 .. _`yaml.dump()`: https://pyyaml.org/wiki/PyYAMLDocumentation 7585 .. _`json.dumps()`: https://docs.python.org/2/library/json.html#json.dumps 7586 .. _`pprint.pformat()`: https://docs.python.org/2/library/pprint.html#pprint.pformat 7587 7588 deserializer_opts 7589 Like ``serializer_opts`` above, but only used when merging with an 7590 existing file (i.e. when ``merge_if_exists`` is set to ``True``). 7591 7592 The options specified here will be passed to the deserializer to load 7593 the existing data, before merging with the specified data and 7594 re-serializing. 7595 7596 .. code-block:: yaml 7597 7598 /etc/dummy/package.yaml 7599 file.serialize: 7600 - serializer: yaml 7601 - serializer_opts: 7602 - explicit_start: True 7603 - default_flow_style: True 7604 - indent: 4 7605 - deserializer_opts: 7606 - encoding: latin-1 7607 - merge_if_exists: True 7608 7609 The valid opts are the additional opts (i.e. not the data being 7610 deserialized) for the function used to deserialize the data. 7611 Documentation for the these functions can be found in the list below: 7612 7613 - For **yaml**: `yaml.load()`_ 7614 - For **json**: `json.loads()`_ 7615 7616 .. _`yaml.load()`: https://pyyaml.org/wiki/PyYAMLDocumentation 7617 .. _`json.loads()`: https://docs.python.org/2/library/json.html#json.loads 7618 7619 However, note that not all arguments are supported. For example, when 7620 deserializing JSON, arguments like ``parse_float`` and ``parse_int`` 7621 which accept a callable object cannot be handled in an SLS file. 7622 7623 .. versionadded:: 2019.2.0 7624 7625 For example, this state: 7626 7627 .. code-block:: yaml 7628 7629 /etc/dummy/package.json: 7630 file.serialize: 7631 - dataset: 7632 name: naive 7633 description: A package using naive versioning 7634 author: A confused individual <iam@confused.com> 7635 dependencies: 7636 express: '>= 1.2.0' 7637 optimist: '>= 0.1.0' 7638 engine: node 0.4.1 7639 - serializer: json 7640 7641 will manage the file ``/etc/dummy/package.json``: 7642 7643 .. code-block:: json 7644 7645 { 7646 "author": "A confused individual <iam@confused.com>", 7647 "dependencies": { 7648 "express": ">= 1.2.0", 7649 "optimist": ">= 0.1.0" 7650 }, 7651 "description": "A package using naive versioning", 7652 "engine": "node 0.4.1", 7653 "name": "naive" 7654 } 7655 """ 7656 if "env" in kwargs: 7657 # "env" is not supported; Use "saltenv". 7658 kwargs.pop("env") 7659 7660 name = os.path.expanduser(name) 7661 7662 # Set some defaults 7663 serializer_options = { 7664 "yaml.serialize": {"default_flow_style": False}, 7665 "json.serialize": {"indent": 2, "separators": (",", ": "), "sort_keys": True}, 7666 } 7667 deserializer_options = { 7668 "yaml.deserialize": {}, 7669 "json.deserialize": {}, 7670 } 7671 if encoding: 7672 serializer_options["yaml.serialize"].update({"allow_unicode": True}) 7673 serializer_options["json.serialize"].update({"ensure_ascii": False}) 7674 7675 ret = {"changes": {}, "comment": "", "name": name, "result": True} 7676 if not name: 7677 return _error(ret, "Must provide name to file.serialize") 7678 7679 if not create: 7680 if not os.path.isfile(name): 7681 # Don't create a file that is not already present 7682 ret[ 7683 "comment" 7684 ] = "File {} is not present and is not set for creation".format(name) 7685 return ret 7686 7687 formatter = kwargs.pop("formatter", None) 7688 if serializer and formatter: 7689 return _error(ret, "Only one of serializer and formatter are allowed") 7690 serializer = str(serializer or formatter or "yaml").lower() 7691 7692 if len([x for x in (dataset, dataset_pillar) if x]) > 1: 7693 return _error(ret, "Only one of 'dataset' and 'dataset_pillar' is permitted") 7694 7695 if dataset_pillar: 7696 dataset = __salt__["pillar.get"](dataset_pillar) 7697 7698 if dataset is None: 7699 return _error(ret, "Neither 'dataset' nor 'dataset_pillar' was defined") 7700 7701 if salt.utils.platform.is_windows(): 7702 if group is not None: 7703 log.warning( 7704 "The group argument for %s has been ignored as this " 7705 "is a Windows system.", 7706 name, 7707 ) 7708 group = user 7709 7710 serializer_name = "{}.serialize".format(serializer) 7711 deserializer_name = "{}.deserialize".format(serializer) 7712 7713 if serializer_name not in __serializers__: 7714 return { 7715 "changes": {}, 7716 "comment": ( 7717 "The {} serializer could not be found. It either does " 7718 "not exist or its prerequisites are not installed.".format(serializer) 7719 ), 7720 "name": name, 7721 "result": False, 7722 } 7723 7724 if serializer_opts: 7725 serializer_options.setdefault(serializer_name, {}).update( 7726 salt.utils.data.repack_dictlist(serializer_opts) 7727 ) 7728 7729 if deserializer_opts: 7730 deserializer_options.setdefault(deserializer_name, {}).update( 7731 salt.utils.data.repack_dictlist(deserializer_opts) 7732 ) 7733 7734 if merge_if_exists: 7735 if os.path.isfile(name): 7736 if deserializer_name not in __serializers__: 7737 return { 7738 "changes": {}, 7739 "comment": ( 7740 "merge_if_exists is not supported for the {} serializer".format( 7741 serializer 7742 ) 7743 ), 7744 "name": name, 7745 "result": False, 7746 } 7747 open_args = "r" 7748 if serializer == "plist": 7749 open_args += "b" 7750 with salt.utils.files.fopen(name, open_args) as fhr: 7751 try: 7752 existing_data = __serializers__[deserializer_name]( 7753 fhr, **deserializer_options.get(deserializer_name, {}) 7754 ) 7755 except (TypeError, DeserializationError) as exc: 7756 ret["result"] = False 7757 ret["comment"] = "Failed to deserialize existing data: {}".format( 7758 exc 7759 ) 7760 return False 7761 7762 if existing_data is not None: 7763 merged_data = salt.utils.dictupdate.merge_recurse( 7764 existing_data, dataset 7765 ) 7766 if existing_data == merged_data: 7767 ret["result"] = True 7768 ret["comment"] = "The file {} is in the correct state".format(name) 7769 return ret 7770 dataset = merged_data 7771 else: 7772 if deserializer_opts: 7773 ret.setdefault("warnings", []).append( 7774 "The 'deserializer_opts' option is ignored unless " 7775 "merge_if_exists is set to True." 7776 ) 7777 7778 contents = __serializers__[serializer_name]( 7779 dataset, **serializer_options.get(serializer_name, {}) 7780 ) 7781 7782 # Insert a newline, but only if the serialized contents are not a 7783 # bytestring. If it's a bytestring, it's almost certainly serialized into a 7784 # binary format that does not take kindly to additional bytes being foisted 7785 # upon it. 7786 try: 7787 contents += "\n" 7788 except TypeError: 7789 pass 7790 7791 # Make sure that any leading zeros stripped by YAML loader are added back 7792 mode = salt.utils.files.normalize_mode(mode) 7793 7794 if __opts__["test"]: 7795 ret["changes"] = __salt__["file.check_managed_changes"]( 7796 name=name, 7797 source=None, 7798 source_hash={}, 7799 source_hash_name=None, 7800 user=user, 7801 group=group, 7802 mode=mode, 7803 attrs=None, 7804 template=None, 7805 context=None, 7806 defaults=None, 7807 saltenv=__env__, 7808 contents=contents, 7809 skip_verify=False, 7810 **kwargs 7811 ) 7812 7813 if ret["changes"]: 7814 ret["result"] = None 7815 ret["comment"] = "Dataset will be serialized and stored into {}".format( 7816 name 7817 ) 7818 7819 if not show_changes: 7820 ret["changes"]["diff"] = "<show_changes=False>" 7821 else: 7822 ret["result"] = True 7823 ret["comment"] = "The file {} is in the correct state".format(name) 7824 return ret 7825 7826 return __salt__["file.manage_file"]( 7827 name=name, 7828 sfn="", 7829 ret=ret, 7830 source=None, 7831 source_sum={}, 7832 user=user, 7833 group=group, 7834 mode=mode, 7835 attrs=None, 7836 saltenv=__env__, 7837 backup=backup, 7838 makedirs=makedirs, 7839 template=None, 7840 show_changes=show_changes, 7841 encoding=encoding, 7842 encoding_errors=encoding_errors, 7843 contents=contents, 7844 ) 7845 7846 7847def mknod(name, ntype, major=0, minor=0, user=None, group=None, mode="0600"): 7848 """ 7849 Create a special file similar to the 'nix mknod command. The supported 7850 device types are ``p`` (fifo pipe), ``c`` (character device), and ``b`` 7851 (block device). Provide the major and minor numbers when specifying a 7852 character device or block device. A fifo pipe does not require this 7853 information. The command will create the necessary dirs if needed. If a 7854 file of the same name not of the same type/major/minor exists, it will not 7855 be overwritten or unlinked (deleted). This is logically in place as a 7856 safety measure because you can really shoot yourself in the foot here and 7857 it is the behavior of 'nix ``mknod``. It is also important to note that not 7858 just anyone can create special devices. Usually this is only done as root. 7859 If the state is executed as none other than root on a minion, you may 7860 receive a permission error. 7861 7862 name 7863 name of the file 7864 7865 ntype 7866 node type 'p' (fifo pipe), 'c' (character device), or 'b' 7867 (block device) 7868 7869 major 7870 major number of the device 7871 does not apply to a fifo pipe 7872 7873 minor 7874 minor number of the device 7875 does not apply to a fifo pipe 7876 7877 user 7878 owning user of the device/pipe 7879 7880 group 7881 owning group of the device/pipe 7882 7883 mode 7884 permissions on the device/pipe 7885 7886 Usage: 7887 7888 .. code-block:: yaml 7889 7890 /dev/chr: 7891 file.mknod: 7892 - ntype: c 7893 - major: 180 7894 - minor: 31 7895 - user: root 7896 - group: root 7897 - mode: 660 7898 7899 /dev/blk: 7900 file.mknod: 7901 - ntype: b 7902 - major: 8 7903 - minor: 999 7904 - user: root 7905 - group: root 7906 - mode: 660 7907 7908 /dev/fifo: 7909 file.mknod: 7910 - ntype: p 7911 - user: root 7912 - group: root 7913 - mode: 660 7914 7915 .. versionadded:: 0.17.0 7916 """ 7917 name = os.path.expanduser(name) 7918 7919 ret = {"name": name, "changes": {}, "comment": "", "result": False} 7920 if not name: 7921 return _error(ret, "Must provide name to file.mknod") 7922 7923 if ntype == "c": 7924 # Check for file existence 7925 if __salt__["file.file_exists"](name): 7926 ret["comment"] = ( 7927 "File {} exists and is not a character device. Refusing " 7928 "to continue".format(name) 7929 ) 7930 7931 # Check if it is a character device 7932 elif not __salt__["file.is_chrdev"](name): 7933 if __opts__["test"]: 7934 ret["comment"] = "Character device {} is set to be created".format(name) 7935 ret["result"] = None 7936 else: 7937 ret = __salt__["file.mknod"]( 7938 name, ntype, major, minor, user, group, mode 7939 ) 7940 7941 # Check the major/minor 7942 else: 7943 devmaj, devmin = __salt__["file.get_devmm"](name) 7944 if (major, minor) != (devmaj, devmin): 7945 ret["comment"] = ( 7946 "Character device {} exists and has a different " 7947 "major/minor {}/{}. Refusing to continue".format( 7948 name, devmaj, devmin 7949 ) 7950 ) 7951 # Check the perms 7952 else: 7953 ret = __salt__["file.check_perms"](name, None, user, group, mode)[0] 7954 if not ret["changes"]: 7955 ret[ 7956 "comment" 7957 ] = "Character device {} is in the correct state".format(name) 7958 7959 elif ntype == "b": 7960 # Check for file existence 7961 if __salt__["file.file_exists"](name): 7962 ret[ 7963 "comment" 7964 ] = "File {} exists and is not a block device. Refusing to continue".format( 7965 name 7966 ) 7967 7968 # Check if it is a block device 7969 elif not __salt__["file.is_blkdev"](name): 7970 if __opts__["test"]: 7971 ret["comment"] = "Block device {} is set to be created".format(name) 7972 ret["result"] = None 7973 else: 7974 ret = __salt__["file.mknod"]( 7975 name, ntype, major, minor, user, group, mode 7976 ) 7977 7978 # Check the major/minor 7979 else: 7980 devmaj, devmin = __salt__["file.get_devmm"](name) 7981 if (major, minor) != (devmaj, devmin): 7982 ret["comment"] = ( 7983 "Block device {} exists and has a different major/minor " 7984 "{}/{}. Refusing to continue".format(name, devmaj, devmin) 7985 ) 7986 # Check the perms 7987 else: 7988 ret = __salt__["file.check_perms"](name, None, user, group, mode)[0] 7989 if not ret["changes"]: 7990 ret["comment"] = "Block device {} is in the correct state".format( 7991 name 7992 ) 7993 7994 elif ntype == "p": 7995 # Check for file existence 7996 if __salt__["file.file_exists"](name): 7997 ret[ 7998 "comment" 7999 ] = "File {} exists and is not a fifo pipe. Refusing to continue".format( 8000 name 8001 ) 8002 8003 # Check if it is a fifo 8004 elif not __salt__["file.is_fifo"](name): 8005 if __opts__["test"]: 8006 ret["comment"] = "Fifo pipe {} is set to be created".format(name) 8007 ret["result"] = None 8008 else: 8009 ret = __salt__["file.mknod"]( 8010 name, ntype, major, minor, user, group, mode 8011 ) 8012 8013 # Check the perms 8014 else: 8015 ret = __salt__["file.check_perms"](name, None, user, group, mode)[0] 8016 if not ret["changes"]: 8017 ret["comment"] = "Fifo pipe {} is in the correct state".format(name) 8018 8019 else: 8020 ret["comment"] = ( 8021 "Node type unavailable: '{}'. Available node types are " 8022 "character ('c'), block ('b'), and pipe ('p')".format(ntype) 8023 ) 8024 8025 return ret 8026 8027 8028def mod_run_check_cmd(cmd, filename, **check_cmd_opts): 8029 """ 8030 Execute the check_cmd logic. 8031 8032 Return a result dict if ``check_cmd`` succeeds (check_cmd == 0) 8033 otherwise return True 8034 """ 8035 8036 log.debug("running our check_cmd") 8037 _cmd = "{} {}".format(cmd, filename) 8038 cret = __salt__["cmd.run_all"](_cmd, **check_cmd_opts) 8039 if cret["retcode"] != 0: 8040 ret = { 8041 "comment": "check_cmd execution failed", 8042 "skip_watch": True, 8043 "result": False, 8044 } 8045 8046 if cret.get("stdout"): 8047 ret["comment"] += "\n" + cret["stdout"] 8048 if cret.get("stderr"): 8049 ret["comment"] += "\n" + cret["stderr"] 8050 8051 return ret 8052 8053 # No reason to stop, return True 8054 return True 8055 8056 8057def decode( 8058 name, 8059 encoded_data=None, 8060 contents_pillar=None, 8061 encoding_type="base64", 8062 checksum="md5", 8063): 8064 """ 8065 Decode an encoded file and write it to disk 8066 8067 .. versionadded:: 2016.3.0 8068 8069 name 8070 Path of the file to be written. 8071 encoded_data 8072 The encoded file. Either this option or ``contents_pillar`` must be 8073 specified. 8074 contents_pillar 8075 A Pillar path to the encoded file. Uses the same path syntax as 8076 :py:func:`pillar.get <salt.modules.pillar.get>`. The 8077 :py:func:`hashutil.base64_encodefile 8078 <salt.modules.hashutil.base64_encodefile>` function can load encoded 8079 content into Pillar. Either this option or ``encoded_data`` must be 8080 specified. 8081 encoding_type 8082 The type of encoding. 8083 checksum 8084 The hashing algorithm to use to generate checksums. Wraps the 8085 :py:func:`hashutil.digest <salt.modules.hashutil.digest>` execution 8086 function. 8087 8088 Usage: 8089 8090 .. code-block:: yaml 8091 8092 write_base64_encoded_string_to_a_file: 8093 file.decode: 8094 - name: /tmp/new_file 8095 - encoding_type: base64 8096 - contents_pillar: mypillar:thefile 8097 8098 # or 8099 8100 write_base64_encoded_string_to_a_file: 8101 file.decode: 8102 - name: /tmp/new_file 8103 - encoding_type: base64 8104 - encoded_data: | 8105 Z2V0IHNhbHRlZAo= 8106 8107 Be careful with multi-line strings that the YAML indentation is correct. 8108 E.g., 8109 8110 .. code-block:: jinja 8111 8112 write_base64_encoded_string_to_a_file: 8113 file.decode: 8114 - name: /tmp/new_file 8115 - encoding_type: base64 8116 - encoded_data: | 8117 {{ salt.pillar.get('path:to:data') | indent(8) }} 8118 """ 8119 ret = {"name": name, "changes": {}, "result": False, "comment": ""} 8120 8121 if not (encoded_data or contents_pillar): 8122 raise CommandExecutionError( 8123 "Specify either the 'encoded_data' or 'contents_pillar' argument." 8124 ) 8125 elif encoded_data and contents_pillar: 8126 raise CommandExecutionError( 8127 "Specify only one 'encoded_data' or 'contents_pillar' argument." 8128 ) 8129 elif encoded_data: 8130 content = encoded_data 8131 elif contents_pillar: 8132 content = __salt__["pillar.get"](contents_pillar, False) 8133 if content is False: 8134 raise CommandExecutionError("Pillar data not found.") 8135 else: 8136 raise CommandExecutionError("No contents given.") 8137 8138 dest_exists = __salt__["file.file_exists"](name) 8139 if dest_exists: 8140 instr = __salt__["hashutil.base64_decodestring"](content) 8141 insum = __salt__["hashutil.digest"](instr, checksum) 8142 del instr # no need to keep in-memory after we have the hash 8143 outsum = __salt__["hashutil.digest_file"](name, checksum) 8144 8145 if insum != outsum: 8146 ret["changes"] = { 8147 "old": outsum, 8148 "new": insum, 8149 } 8150 8151 if not ret["changes"]: 8152 ret["comment"] = "File is in the correct state." 8153 ret["result"] = True 8154 8155 return ret 8156 8157 if __opts__["test"] is True: 8158 ret["comment"] = "File is set to be updated." 8159 ret["result"] = None 8160 return ret 8161 8162 ret["result"] = __salt__["hashutil.base64_decodefile"](content, name) 8163 ret["comment"] = "File was updated." 8164 8165 if not ret["changes"]: 8166 ret["changes"] = { 8167 "old": None, 8168 "new": __salt__["hashutil.digest_file"](name, checksum), 8169 } 8170 8171 return ret 8172 8173 8174def shortcut( 8175 name, 8176 target, 8177 arguments=None, 8178 working_dir=None, 8179 description=None, 8180 icon_location=None, 8181 force=False, 8182 backupname=None, 8183 makedirs=False, 8184 user=None, 8185 **kwargs 8186): 8187 """ 8188 Create a Windows shortcut 8189 8190 If the file already exists and is a shortcut pointing to any location other 8191 than the specified target, the shortcut will be replaced. If it is 8192 a regular file or directory then the state will return False. If the 8193 regular file or directory is desired to be replaced with a shortcut pass 8194 force: True, if it is to be renamed, pass a backupname. 8195 8196 name 8197 The location of the shortcut to create. Must end with either 8198 ".lnk" or ".url" 8199 8200 target 8201 The location that the shortcut points to 8202 8203 arguments 8204 Any arguments to pass in the shortcut 8205 8206 working_dir 8207 Working directory in which to execute target 8208 8209 description 8210 Description to set on shortcut 8211 8212 icon_location 8213 Location of shortcut's icon 8214 8215 force 8216 If the name of the shortcut exists and is not a file and 8217 force is set to False, the state will fail. If force is set to 8218 True, the link or directory in the way of the shortcut file 8219 will be deleted to make room for the shortcut, unless 8220 backupname is set, when it will be renamed 8221 8222 backupname 8223 If the name of the shortcut exists and is not a file, it will be 8224 renamed to the backupname. If the backupname already 8225 exists and force is False, the state will fail. Otherwise, the 8226 backupname will be removed first. 8227 8228 makedirs 8229 If the location of the shortcut does not already have a parent 8230 directory then the state will fail, setting makedirs to True will 8231 allow Salt to create the parent directory. Setting this to True will 8232 also create the parent for backupname if necessary. 8233 8234 user 8235 The user to own the file, this defaults to the user salt is running as 8236 on the minion 8237 8238 The default mode for new files and directories corresponds umask of salt 8239 process. For existing files and directories it's not enforced. 8240 """ 8241 user = _test_owner(kwargs, user=user) 8242 ret = {"name": name, "changes": {}, "result": True, "comment": ""} 8243 if not salt.utils.platform.is_windows(): 8244 return _error(ret, "Shortcuts are only supported on Windows") 8245 if not name: 8246 return _error(ret, "Must provide name to file.shortcut") 8247 if not name.endswith(".lnk") and not name.endswith(".url"): 8248 return _error(ret, 'Name must end with either ".lnk" or ".url"') 8249 8250 # Normalize paths; do this after error checks to avoid invalid input 8251 # getting expanded, e.g. '' turning into '.' 8252 name = os.path.realpath(os.path.expanduser(name)) 8253 if name.endswith(".lnk"): 8254 target = os.path.realpath(os.path.expanduser(target)) 8255 if working_dir: 8256 working_dir = os.path.realpath(os.path.expanduser(working_dir)) 8257 if icon_location: 8258 icon_location = os.path.realpath(os.path.expanduser(icon_location)) 8259 8260 if user is None: 8261 user = __opts__["user"] 8262 8263 # Make sure the user exists in Windows 8264 # Salt default is 'root' 8265 if not __salt__["user.info"](user): 8266 # User not found, use the account salt is running under 8267 # If username not found, use System 8268 user = __salt__["user.current"]() 8269 if not user: 8270 user = "SYSTEM" 8271 8272 preflight_errors = [] 8273 uid = __salt__["file.user_to_uid"](user) 8274 8275 if uid == "": 8276 preflight_errors.append("User {} does not exist".format(user)) 8277 8278 if not os.path.isabs(name): 8279 preflight_errors.append( 8280 "Specified file {} is not an absolute path".format(name) 8281 ) 8282 8283 if preflight_errors: 8284 msg = ". ".join(preflight_errors) 8285 if len(preflight_errors) > 1: 8286 msg += "." 8287 return _error(ret, msg) 8288 8289 tresult, tcomment, tchanges = _shortcut_check( 8290 name, target, arguments, working_dir, description, icon_location, force, user 8291 ) 8292 if __opts__["test"]: 8293 ret["result"] = tresult 8294 ret["comment"] = tcomment 8295 ret["changes"] = tchanges 8296 return ret 8297 8298 if not os.path.isdir(os.path.dirname(name)): 8299 if makedirs: 8300 try: 8301 _makedirs(name=name, user=user) 8302 except CommandExecutionError as exc: 8303 return _error(ret, "Drive {} is not mapped".format(exc.message)) 8304 else: 8305 return _error( 8306 ret, 8307 'Directory "{}" for shortcut is not present'.format( 8308 os.path.dirname(name) 8309 ), 8310 ) 8311 8312 if os.path.isdir(name) or os.path.islink(name): 8313 # It is not a shortcut, but a dir or symlink 8314 if backupname is not None: 8315 # Make a backup first 8316 if os.path.lexists(backupname): 8317 if not force: 8318 return _error( 8319 ret, 8320 "File exists where the backup target {} should go".format( 8321 backupname 8322 ), 8323 ) 8324 else: 8325 __salt__["file.remove"](backupname) 8326 time.sleep(1) # wait for asynchronous deletion 8327 if not os.path.isdir(os.path.dirname(backupname)): 8328 if makedirs: 8329 try: 8330 _makedirs(name=backupname) 8331 except CommandExecutionError as exc: 8332 return _error(ret, "Drive {} is not mapped".format(exc.message)) 8333 else: 8334 return _error( 8335 ret, 8336 'Directory does not exist for backup at "{}"'.format( 8337 os.path.dirname(backupname) 8338 ), 8339 ) 8340 os.rename(name, backupname) 8341 time.sleep(1) # wait for asynchronous rename 8342 elif force: 8343 # Remove whatever is in the way 8344 __salt__["file.remove"](name) 8345 ret["changes"]["forced"] = "Shortcut was forcibly replaced" 8346 time.sleep(1) # wait for asynchronous deletion 8347 else: 8348 # Otherwise throw an error 8349 return _error( 8350 ret, 8351 'Directory or symlink exists where the shortcut "{}" should be'.format( 8352 name 8353 ), 8354 ) 8355 8356 # This will just load the shortcut if it already exists 8357 # It won't create the file until calling scut.Save() 8358 with salt.utils.winapi.Com(): 8359 shell = win32com.client.Dispatch("WScript.Shell") 8360 scut = shell.CreateShortcut(name) 8361 8362 # The shortcut target will automatically be created with its 8363 # canonical capitalization; no way to override it, so ignore case 8364 state_checks = [scut.TargetPath.lower() == target.lower()] 8365 if arguments is not None: 8366 state_checks.append(scut.Arguments == arguments) 8367 if working_dir is not None: 8368 state_checks.append(scut.WorkingDirectory.lower() == working_dir.lower()) 8369 if description is not None: 8370 state_checks.append(scut.Description == description) 8371 if icon_location is not None: 8372 state_checks.append(scut.IconLocation.lower() == icon_location.lower()) 8373 8374 if __salt__["file.file_exists"](name): 8375 # The shortcut exists, verify that it matches the desired state 8376 if not all(state_checks): 8377 # The target is wrong, delete it 8378 os.remove(name) 8379 else: 8380 if _check_shortcut_ownership(name, user): 8381 # The shortcut looks good! 8382 ret["comment"] = "Shortcut {} is present and owned by {}".format( 8383 name, user 8384 ) 8385 else: 8386 if _set_shortcut_ownership(name, user): 8387 ret["comment"] = "Set ownership of shortcut {} to {}".format( 8388 name, user 8389 ) 8390 ret["changes"]["ownership"] = "{}".format(user) 8391 else: 8392 ret["result"] = False 8393 ret[ 8394 "comment" 8395 ] += "Failed to set ownership of shortcut {} to {}".format( 8396 name, user 8397 ) 8398 return ret 8399 8400 if not os.path.exists(name): 8401 # The shortcut is not present, make it 8402 try: 8403 scut.TargetPath = target 8404 if arguments is not None: 8405 scut.Arguments = arguments 8406 if working_dir is not None: 8407 scut.WorkingDirectory = working_dir 8408 if description is not None: 8409 scut.Description = description 8410 if icon_location is not None: 8411 scut.IconLocation = icon_location 8412 scut.Save() 8413 except (AttributeError, pywintypes.com_error) as exc: 8414 ret["result"] = False 8415 ret["comment"] = "Unable to create new shortcut {} -> {}: {}".format( 8416 name, target, exc 8417 ) 8418 return ret 8419 else: 8420 ret["comment"] = "Created new shortcut {} -> {}".format(name, target) 8421 ret["changes"]["new"] = name 8422 8423 if not _check_shortcut_ownership(name, user): 8424 if not _set_shortcut_ownership(name, user): 8425 ret["result"] = False 8426 ret["comment"] += ", but was unable to set ownership to {}".format( 8427 user 8428 ) 8429 return ret 8430 8431 8432def cached( 8433 name, source_hash="", source_hash_name=None, skip_verify=False, saltenv="base" 8434): 8435 """ 8436 .. versionadded:: 2017.7.3 8437 8438 Ensures that a file is saved to the minion's cache. This state is primarily 8439 invoked by other states to ensure that we do not re-download a source file 8440 if we do not need to. 8441 8442 name 8443 The URL of the file to be cached. To cache a file from an environment 8444 other than ``base``, either use the ``saltenv`` argument or include the 8445 saltenv in the URL (e.g. ``salt://path/to/file.conf?saltenv=dev``). 8446 8447 .. note:: 8448 A list of URLs is not supported, this must be a single URL. If a 8449 local file is passed here, then the state will obviously not try to 8450 download anything, but it will compare a hash if one is specified. 8451 8452 source_hash 8453 See the documentation for this same argument in the 8454 :py:func:`file.managed <salt.states.file.managed>` state. 8455 8456 .. note:: 8457 For remote files not originating from the ``salt://`` fileserver, 8458 such as http(s) or ftp servers, this state will not re-download the 8459 file if the locally-cached copy matches this hash. This is done to 8460 prevent unnecessary downloading on repeated runs of this state. To 8461 update the cached copy of a file, it is necessary to update this 8462 hash. 8463 8464 source_hash_name 8465 See the documentation for this same argument in the 8466 :py:func:`file.managed <salt.states.file.managed>` state. 8467 8468 skip_verify 8469 See the documentation for this same argument in the 8470 :py:func:`file.managed <salt.states.file.managed>` state. 8471 8472 .. note:: 8473 Setting this to ``True`` will result in a copy of the file being 8474 downloaded from a remote (http(s), ftp, etc.) source each time the 8475 state is run. 8476 8477 saltenv 8478 Used to specify the environment from which to download a file from the 8479 Salt fileserver (i.e. those with ``salt://`` URL). 8480 8481 8482 This state will in most cases not be useful in SLS files, but it is useful 8483 when writing a state or remote-execution module that needs to make sure 8484 that a file at a given URL has been downloaded to the cachedir. One example 8485 of this is in the :py:func:`archive.extracted <salt.states.file.extracted>` 8486 state: 8487 8488 .. code-block:: python 8489 8490 result = __states__['file.cached'](source_match, 8491 source_hash=source_hash, 8492 source_hash_name=source_hash_name, 8493 skip_verify=skip_verify, 8494 saltenv=__env__) 8495 8496 This will return a dictionary containing the state's return data, including 8497 a ``result`` key which will state whether or not the state was successful. 8498 Note that this will not catch exceptions, so it is best used within a 8499 try/except. 8500 8501 Once this state has been run from within another state or remote-execution 8502 module, the actual location of the cached file can be obtained using 8503 :py:func:`cp.is_cached <salt.modules.cp.is_cached>`: 8504 8505 .. code-block:: python 8506 8507 cached = __salt__['cp.is_cached'](source_match, saltenv=__env__) 8508 8509 This function will return the cached path of the file, or an empty string 8510 if the file is not present in the minion cache. 8511 """ 8512 ret = {"changes": {}, "comment": "", "name": name, "result": False} 8513 8514 try: 8515 parsed = urllib.parse.urlparse(name) 8516 except Exception: # pylint: disable=broad-except 8517 ret["comment"] = "Only URLs or local file paths are valid input" 8518 return ret 8519 8520 # This if statement will keep the state from proceeding if a remote source 8521 # is specified and no source_hash is presented (unless we're skipping hash 8522 # verification). 8523 if ( 8524 not skip_verify 8525 and not source_hash 8526 and parsed.scheme in salt.utils.files.REMOTE_PROTOS 8527 ): 8528 ret["comment"] = ( 8529 "Unable to verify upstream hash of source file {}, please set " 8530 "source_hash or set skip_verify to True".format( 8531 salt.utils.url.redact_http_basic_auth(name) 8532 ) 8533 ) 8534 return ret 8535 8536 if source_hash: 8537 # Get the hash and hash type from the input. This takes care of parsing 8538 # the hash out of a file containing checksums, if that is how the 8539 # source_hash was specified. 8540 try: 8541 source_sum = __salt__["file.get_source_sum"]( 8542 source=name, 8543 source_hash=source_hash, 8544 source_hash_name=source_hash_name, 8545 saltenv=saltenv, 8546 ) 8547 except CommandExecutionError as exc: 8548 ret["comment"] = exc.strerror 8549 return ret 8550 else: 8551 if not source_sum: 8552 # We shouldn't get here, problems in retrieving the hash in 8553 # file.get_source_sum should result in a CommandExecutionError 8554 # being raised, which we catch above. Nevertheless, we should 8555 # provide useful information in the event that 8556 # file.get_source_sum regresses. 8557 ret["comment"] = ( 8558 "Failed to get source hash from {}. This may be a bug. " 8559 "If this error persists, please report it and set " 8560 "skip_verify to True to work around it.".format(source_hash) 8561 ) 8562 return ret 8563 else: 8564 source_sum = {} 8565 8566 if parsed.scheme in salt.utils.files.LOCAL_PROTOS: 8567 # Source is a local file path 8568 full_path = os.path.realpath(os.path.expanduser(parsed.path)) 8569 if os.path.exists(full_path): 8570 if not skip_verify and source_sum: 8571 # Enforce the hash 8572 local_hash = __salt__["file.get_hash"]( 8573 full_path, source_sum.get("hash_type", __opts__["hash_type"]) 8574 ) 8575 if local_hash == source_sum["hsum"]: 8576 ret["result"] = True 8577 ret[ 8578 "comment" 8579 ] = "File {} is present on the minion and has hash {}".format( 8580 full_path, local_hash 8581 ) 8582 else: 8583 ret["comment"] = ( 8584 "File {} is present on the minion, but the hash ({}) " 8585 "does not match the specified hash ({})".format( 8586 full_path, local_hash, source_sum["hsum"] 8587 ) 8588 ) 8589 return ret 8590 else: 8591 ret["result"] = True 8592 ret["comment"] = "File {} is present on the minion".format(full_path) 8593 return ret 8594 else: 8595 ret["comment"] = "File {} is not present on the minion".format(full_path) 8596 return ret 8597 8598 local_copy = __salt__["cp.is_cached"](name, saltenv=saltenv) 8599 8600 if local_copy: 8601 # File is already cached 8602 pre_hash = __salt__["file.get_hash"]( 8603 local_copy, source_sum.get("hash_type", __opts__["hash_type"]) 8604 ) 8605 8606 if not skip_verify and source_sum: 8607 # Get the local copy's hash to compare with the hash that was 8608 # specified via source_hash. If it matches, we can exit early from 8609 # the state without going any further, because the file is cached 8610 # with the correct hash. 8611 if pre_hash == source_sum["hsum"]: 8612 ret["result"] = True 8613 ret["comment"] = "File is already cached to {} with hash {}".format( 8614 local_copy, pre_hash 8615 ) 8616 else: 8617 pre_hash = None 8618 8619 def _try_cache(path, checksum): 8620 """ 8621 This helper is not needed anymore in develop as the fileclient in the 8622 develop branch now has means of skipping a download if the existing 8623 hash matches one passed to cp.cache_file. Remove this helper and the 8624 code that invokes it, once we have merged forward into develop. 8625 """ 8626 if not path or not checksum: 8627 return True 8628 form = salt.utils.files.HASHES_REVMAP.get(len(checksum)) 8629 if form is None: 8630 # Shouldn't happen, an invalid checksum length should be caught 8631 # before we get here. But in the event this gets through, don't let 8632 # it cause any trouble, and just return True. 8633 return True 8634 try: 8635 return salt.utils.hashutils.get_hash(path, form=form) != checksum 8636 except (OSError, ValueError): 8637 # Again, shouldn't happen, but don't let invalid input/permissions 8638 # in the call to get_hash blow this up. 8639 return True 8640 8641 # Cache the file. Note that this will not actually download the file if 8642 # either of the following is true: 8643 # 1. source is a salt:// URL and the fileserver determines that the hash 8644 # of the minion's copy matches that of the fileserver. 8645 # 2. File is remote (http(s), ftp, etc.) and the specified source_hash 8646 # matches the cached copy. 8647 # Remote, non salt:// sources _will_ download if a copy of the file was 8648 # not already present in the minion cache. 8649 if _try_cache(local_copy, source_sum.get("hsum")): 8650 # The _try_cache helper is obsolete in the develop branch. Once merged 8651 # forward, remove the helper as well as this if statement, and dedent 8652 # the below block. 8653 try: 8654 local_copy = __salt__["cp.cache_file"]( 8655 name, saltenv=saltenv, source_hash=source_sum.get("hsum") 8656 ) 8657 except Exception as exc: # pylint: disable=broad-except 8658 ret["comment"] = salt.utils.url.redact_http_basic_auth(exc.__str__()) 8659 return ret 8660 8661 if not local_copy: 8662 ret[ 8663 "comment" 8664 ] = "Failed to cache {}, check minion log for more information".format( 8665 salt.utils.url.redact_http_basic_auth(name) 8666 ) 8667 return ret 8668 8669 post_hash = __salt__["file.get_hash"]( 8670 local_copy, source_sum.get("hash_type", __opts__["hash_type"]) 8671 ) 8672 8673 if pre_hash != post_hash: 8674 ret["changes"]["hash"] = {"old": pre_hash, "new": post_hash} 8675 8676 # Check the hash, if we're enforcing one. Note that this will be the first 8677 # hash check if the file was not previously cached, and the 2nd hash check 8678 # if it was cached and the 8679 if not skip_verify and source_sum: 8680 if post_hash == source_sum["hsum"]: 8681 ret["result"] = True 8682 ret["comment"] = "File is already cached to {} with hash {}".format( 8683 local_copy, post_hash 8684 ) 8685 else: 8686 ret["comment"] = ( 8687 "File is cached to {}, but the hash ({}) does not match " 8688 "the specified hash ({})".format( 8689 local_copy, post_hash, source_sum["hsum"] 8690 ) 8691 ) 8692 return ret 8693 8694 # We're not enforcing a hash, and we already know that the file was 8695 # successfully cached, so we know the state was successful. 8696 ret["result"] = True 8697 ret["comment"] = "File is cached to {}".format(local_copy) 8698 return ret 8699 8700 8701def not_cached(name, saltenv="base"): 8702 """ 8703 .. versionadded:: 2017.7.3 8704 8705 Ensures that a file is not present in the minion's cache, deleting it 8706 if found. This state is primarily invoked by other states to ensure 8707 that a fresh copy is fetched. 8708 8709 name 8710 The URL of the file to be removed from cache. To remove a file from 8711 cache in an environment other than ``base``, either use the ``saltenv`` 8712 argument or include the saltenv in the URL (e.g. 8713 ``salt://path/to/file.conf?saltenv=dev``). 8714 8715 .. note:: 8716 A list of URLs is not supported, this must be a single URL. If a 8717 local file is passed here, the state will take no action. 8718 8719 saltenv 8720 Used to specify the environment from which to download a file from the 8721 Salt fileserver (i.e. those with ``salt://`` URL). 8722 """ 8723 ret = {"changes": {}, "comment": "", "name": name, "result": False} 8724 8725 try: 8726 parsed = urllib.parse.urlparse(name) 8727 except Exception: # pylint: disable=broad-except 8728 ret["comment"] = "Only URLs or local file paths are valid input" 8729 return ret 8730 else: 8731 if parsed.scheme in salt.utils.files.LOCAL_PROTOS: 8732 full_path = os.path.realpath(os.path.expanduser(parsed.path)) 8733 ret["result"] = True 8734 ret["comment"] = "File {} is a local path, no action taken".format( 8735 full_path 8736 ) 8737 return ret 8738 8739 local_copy = __salt__["cp.is_cached"](name, saltenv=saltenv) 8740 8741 if local_copy: 8742 try: 8743 os.remove(local_copy) 8744 except Exception as exc: # pylint: disable=broad-except 8745 ret["comment"] = "Failed to delete {}: {}".format(local_copy, exc.__str__()) 8746 else: 8747 ret["result"] = True 8748 ret["changes"]["deleted"] = True 8749 ret["comment"] = "{} was deleted".format(local_copy) 8750 else: 8751 ret["result"] = True 8752 ret["comment"] = "{} is not cached".format(name) 8753 return ret 8754 8755 8756def mod_beacon(name, **kwargs): 8757 """ 8758 Create a beacon to monitor a file based on a beacon state argument. 8759 8760 .. note:: 8761 This state exists to support special handling of the ``beacon`` 8762 state argument for supported state functions. It should not be called directly. 8763 8764 """ 8765 sfun = kwargs.pop("sfun", None) 8766 supported_funcs = ["managed", "directory"] 8767 8768 if sfun in supported_funcs: 8769 if kwargs.get("beacon"): 8770 beacon_module = "inotify" 8771 8772 data = {} 8773 _beacon_data = kwargs.get("beacon_data", {}) 8774 8775 default_mask = ["create", "delete", "modify"] 8776 data["mask"] = _beacon_data.get("mask", default_mask) 8777 8778 if sfun == "directory": 8779 data["auto_add"] = _beacon_data.get("auto_add", True) 8780 data["recurse"] = _beacon_data.get("recurse", True) 8781 data["exclude"] = _beacon_data.get("exclude", []) 8782 8783 beacon_name = "beacon_{}_{}".format(beacon_module, name) 8784 beacon_kwargs = { 8785 "name": beacon_name, 8786 "files": {name: data}, 8787 "interval": _beacon_data.get("interval", 60), 8788 "coalesce": _beacon_data.get("coalesce", False), 8789 "beacon_module": beacon_module, 8790 } 8791 8792 ret = __states__["beacon.present"](**beacon_kwargs) 8793 return ret 8794 else: 8795 return { 8796 "name": name, 8797 "changes": {}, 8798 "comment": "Not adding beacon.", 8799 "result": True, 8800 } 8801 else: 8802 return { 8803 "name": name, 8804 "changes": {}, 8805 "comment": "file.{} does not work with the beacon state function".format( 8806 sfun 8807 ), 8808 "result": False, 8809 } 8810