1.. _developing_inventory: 2 3**************************** 4Developing dynamic inventory 5**************************** 6 7Ansible can pull inventory information from dynamic sources, including cloud sources, by using the supplied :ref:`inventory plugins <inventory_plugins>`. For details about how to pull inventory information, see :ref:`dynamic_inventory`. If the source you want is not currently covered by existing plugins, you can create your own inventory plugin as with any other plugin type. 8 9In previous versions, you had to create a script or program that could output JSON in the correct format when invoked with the proper arguments. 10You can still use and write inventory scripts, as we ensured backwards compatibility via the :ref:`script inventory plugin <script_inventory>` 11and there is no restriction on the programming language used. 12If you choose to write a script, however, you will need to implement some features yourself such as caching, configuration management, dynamic variable and group composition, and so on. 13If you use :ref:`inventory plugins <inventory_plugins>` instead, you can leverage the Ansible codebase and add these common features automatically. 14 15.. contents:: Topics 16 :local: 17 18 19.. _inventory_sources: 20 21Inventory sources 22================= 23 24Inventory sources are the input strings that inventory plugins work with. 25An inventory source can be a path to a file or to a script, or it can be raw data that the plugin can interpret. 26 27The table below shows some examples of inventory plugins and the source types that you can pass to them with ``-i`` on the command line. 28 29+--------------------------------------------+-----------------------------------------+ 30| Plugin | Source | 31+--------------------------------------------+-----------------------------------------+ 32| :ref:`host list <host_list_inventory>` | A comma-separated list of hosts | 33+--------------------------------------------+-----------------------------------------+ 34| :ref:`yaml <yaml_inventory>` | Path to a YAML format data file | 35+--------------------------------------------+-----------------------------------------+ 36| :ref:`constructed <constructed_inventory>` | Path to a YAML configuration file | 37+--------------------------------------------+-----------------------------------------+ 38| :ref:`ini <ini_inventory>` | Path to an INI formatted data file | 39+--------------------------------------------+-----------------------------------------+ 40| :ref:`virtualbox <virtualbox_inventory>` | Path to a YAML configuration file | 41+--------------------------------------------+-----------------------------------------+ 42| :ref:`script plugin <script_inventory>` | Path to an executable that outputs JSON | 43+--------------------------------------------+-----------------------------------------+ 44 45 46.. _developing_inventory_inventory_plugins: 47 48Inventory plugins 49================= 50 51Like most plugin types (except modules), inventory plugins must be developed in Python. They execute on the controller and should therefore adhere to the :ref:`control_node_requirements`. 52 53Most of the documentation in :ref:`developing_plugins` also applies here. You should read that document first for a general understanding and then come back to this document for specifics on inventory plugins. 54 55Normally, inventory plugins are executed at the start of a run, and before the playbooks, plays, or roles are loaded. 56However, you can use the ``meta: refresh_inventory`` task to clear the current inventory and execute the inventory plugins again, and this task will generate a new inventory. 57 58If you use the persistent cache, inventory plugins can also use the configured cache plugin to store and retrieve data. Caching inventory avoids making repeated and costly external calls. 59 60.. _developing_an_inventory_plugin: 61 62Developing an inventory plugin 63------------------------------ 64 65The first thing you want to do is use the base class: 66 67.. code-block:: python 68 69 from ansible.plugins.inventory import BaseInventoryPlugin 70 71 class InventoryModule(BaseInventoryPlugin): 72 73 NAME = 'myplugin' # used internally by Ansible, it should match the file name but not required 74 75If the inventory plugin is in a collection, the NAME should be in the 'namespace.collection_name.myplugin' format. The base class has a couple of methods that each plugin should implement and a few helpers for parsing the inventory source and updating the inventory. 76 77After you have the basic plugin working, you can incorporate other features by adding more base classes: 78 79.. code-block:: python 80 81 from ansible.plugins.inventory import BaseInventoryPlugin, Constructable, Cacheable 82 83 class InventoryModule(BaseInventoryPlugin, Constructable, Cacheable): 84 85 NAME = 'myplugin' 86 87For the bulk of the work in a plugin, we mostly want to deal with 2 methods ``verify_file`` and ``parse``. 88 89.. _inventory_plugin_verify_file: 90 91verify_file method 92^^^^^^^^^^^^^^^^^^ 93 94Ansible uses this method to quickly determine if the inventory source is usable by the plugin. The determination does not need to be 100% accurate, as there might be an overlap in what plugins can handle and by default Ansible will try the enabled plugins as per their sequence. 95 96.. code-block:: python 97 98 def verify_file(self, path): 99 ''' return true/false if this is possibly a valid file for this plugin to consume ''' 100 valid = False 101 if super(InventoryModule, self).verify_file(path): 102 # base class verifies that file exists and is readable by current user 103 if path.endswith(('virtualbox.yaml', 'virtualbox.yml', 'vbox.yaml', 'vbox.yml')): 104 valid = True 105 return valid 106 107In the above example, from the :ref:`virtualbox inventory plugin <virtualbox_inventory>`, we screen for specific file name patterns to avoid attempting to consume any valid YAML file. You can add any type of condition here, but the most common one is 'extension matching'. If you implement extension matching for YAML configuration files, the path suffix <plugin_name>.<yml|yaml> should be accepted. All valid extensions should be documented in the plugin description. 108 109The following is another example that does not use a 'file' but the inventory source string itself, 110from the :ref:`host list <host_list_inventory>` plugin: 111 112.. code-block:: python 113 114 def verify_file(self, path): 115 ''' don't call base class as we don't expect a path, but a host list ''' 116 host_list = path 117 valid = False 118 b_path = to_bytes(host_list, errors='surrogate_or_strict') 119 if not os.path.exists(b_path) and ',' in host_list: 120 # the path does NOT exist and there is a comma to indicate this is a 'host list' 121 valid = True 122 return valid 123 124This method is just to expedite the inventory process and avoid unnecessary parsing of sources that are easy to filter out before causing a parse error. 125 126.. _inventory_plugin_parse: 127 128parse method 129^^^^^^^^^^^^ 130 131This method does the bulk of the work in the plugin. 132It takes the following parameters: 133 134 * inventory: inventory object with existing data and the methods to add hosts/groups/variables to inventory 135 * loader: Ansible's DataLoader. The DataLoader can read files, auto load JSON/YAML and decrypt vaulted data, and cache read files. 136 * path: string with inventory source (this is usually a path, but is not required) 137 * cache: indicates whether the plugin should use or avoid caches (cache plugin and/or loader) 138 139 140The base class does some minimal assignment for reuse in other methods. 141 142.. code-block:: python 143 144 def parse(self, inventory, loader, path, cache=True): 145 146 self.loader = loader 147 self.inventory = inventory 148 self.templar = Templar(loader=loader) 149 150It is up to the plugin now to parse the provided inventory source and translate it into Ansible inventory. 151To facilitate this, the example below uses a few helper functions: 152 153.. code-block:: python 154 155 NAME = 'myplugin' 156 157 def parse(self, inventory, loader, path, cache=True): 158 159 # call base method to ensure properties are available for use with other helper methods 160 super(InventoryModule, self).parse(inventory, loader, path, cache) 161 162 # this method will parse 'common format' inventory sources and 163 # update any options declared in DOCUMENTATION as needed 164 config = self._read_config_data(path) 165 166 # if NOT using _read_config_data you should call set_options directly, 167 # to process any defined configuration for this plugin, 168 # if you don't define any options you can skip 169 #self.set_options() 170 171 # example consuming options from inventory source 172 mysession = apilib.session(user=self.get_option('api_user'), 173 password=self.get_option('api_pass'), 174 server=self.get_option('api_server') 175 ) 176 177 178 # make requests to get data to feed into inventory 179 mydata = mysession.getitall() 180 181 #parse data and create inventory objects: 182 for colo in mydata: 183 for server in mydata[colo]['servers']: 184 self.inventory.add_host(server['name']) 185 self.inventory.set_variable(server['name'], 'ansible_host', server['external_ip']) 186 187The specifics will vary depending on API and structure returned. Remember that if you get an inventory source error or any other issue, you should ``raise AnsibleParserError`` to let Ansible know that the source was invalid or the process failed. 188 189For examples on how to implement an inventory plugin, see the source code here: 190`lib/ansible/plugins/inventory <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/inventory>`_. 191 192.. _inventory_plugin_caching: 193 194inventory cache 195^^^^^^^^^^^^^^^ 196 197To cache the inventory, extend the inventory plugin documentation with the inventory_cache documentation fragment and use the Cacheable base class. 198 199.. code-block:: yaml 200 201 extends_documentation_fragment: 202 - inventory_cache 203 204.. code-block:: python 205 206 class InventoryModule(BaseInventoryPlugin, Constructable, Cacheable): 207 208 NAME = 'myplugin' 209 210Next, load the cache plugin specified by the user to read from and update the cache. If your inventory plugin uses YAML-based configuration files and the ``_read_config_data`` method, the cache plugin is loaded within that method. If your inventory plugin does not use ``_read_config_data``, you must load the cache explicitly with ``load_cache_plugin``. 211 212.. code-block:: python 213 214 NAME = 'myplugin' 215 216 def parse(self, inventory, loader, path, cache=True): 217 super(InventoryModule, self).parse(inventory, loader, path) 218 219 self.load_cache_plugin() 220 221Before using the cache plugin, you must retrieve a unique cache key by using the ``get_cache_key`` method. This task needs to be done by all inventory modules using the cache, so that you don't use/overwrite other parts of the cache. 222 223.. code-block:: python 224 225 def parse(self, inventory, loader, path, cache=True): 226 super(InventoryModule, self).parse(inventory, loader, path) 227 228 self.load_cache_plugin() 229 cache_key = self.get_cache_key(path) 230 231Now that you've enabled caching, loaded the correct plugin, and retrieved a unique cache key, you can set up the flow of data between the cache and your inventory using the ``cache`` parameter of the ``parse`` method. This value comes from the inventory manager and indicates whether the inventory is being refreshed (such as via ``--flush-cache`` or the meta task ``refresh_inventory``). Although the cache shouldn't be used to populate the inventory when being refreshed, the cache should be updated with the new inventory if the user has enabled caching. You can use ``self._cache`` like a dictionary. The following pattern allows refreshing the inventory to work in conjunction with caching. 232 233.. code-block:: python 234 235 def parse(self, inventory, loader, path, cache=True): 236 super(InventoryModule, self).parse(inventory, loader, path) 237 238 self.load_cache_plugin() 239 cache_key = self.get_cache_key(path) 240 241 # cache may be True or False at this point to indicate if the inventory is being refreshed 242 # get the user's cache option too to see if we should save the cache if it is changing 243 user_cache_setting = self.get_option('cache') 244 245 # read if the user has caching enabled and the cache isn't being refreshed 246 attempt_to_read_cache = user_cache_setting and cache 247 # update if the user has caching enabled and the cache is being refreshed; update this value to True if the cache has expired below 248 cache_needs_update = user_cache_setting and not cache 249 250 # attempt to read the cache if inventory isn't being refreshed and the user has caching enabled 251 if attempt_to_read_cache: 252 try: 253 results = self._cache[cache_key] 254 except KeyError: 255 # This occurs if the cache_key is not in the cache or if the cache_key expired, so the cache needs to be updated 256 cache_needs_update = True 257 258 if cache_needs_update: 259 results = self.get_inventory() 260 261 # set the cache 262 self._cache[cache_key] = results 263 264 self.populate(results) 265 266After the ``parse`` method is complete, the contents of ``self._cache`` is used to set the cache plugin if the contents of the cache have changed. 267 268You have three other cache methods available: 269 - ``set_cache_plugin`` forces the cache plugin to be set with the contents of ``self._cache``, before the ``parse`` method completes 270 - ``update_cache_if_changed`` sets the cache plugin only if ``self._cache`` has been modified, before the ``parse`` method completes 271 - ``clear_cache`` flushes the cache, ultimately by calling the cache plugin's ``flush()`` method, whose implementation is dependent upon the particular cache plugin in use. Note that if the user is using the same cache backend for facts and inventory, both will get flushed. To avoid this, the user can specify a distinct cache backend in their inventory plugin configuration. 272 273.. _inventory_source_common_format: 274 275Common format for inventory sources 276----------------------------------- 277 278To simplify development, most plugins use a standard YAML-based configuration file as the inventory source. The file has only one required field ``plugin``, which should contain the name of the plugin that is expected to consume the file. 279Depending on other common features used, you might need other fields, and you can add custom options in each plugin as required. 280For example, if you use the integrated caching, ``cache_plugin``, ``cache_timeout`` and other cache-related fields could be present. 281 282.. _inventory_development_auto: 283 284The 'auto' plugin 285----------------- 286 287From Ansible 2.5 onwards, we include the :ref:`auto inventory plugin <auto_inventory>` and enable it by default. If the ``plugin`` field in your standard configuration file matches the name of your inventory plugin, the ``auto`` inventory plugin will load your plugin. The 'auto' plugin makes it easier to use your plugin without having to update configurations. 288 289 290.. _inventory_scripts: 291.. _developing_inventory_scripts: 292 293Inventory scripts 294================= 295 296Even though we now have inventory plugins, we still support inventory scripts, not only for backwards compatibility but also to allow users to leverage other programming languages. 297 298 299.. _inventory_script_conventions: 300 301Inventory script conventions 302---------------------------- 303 304Inventory scripts must accept the ``--list`` and ``--host <hostname>`` arguments. Although other arguments are allowed, Ansible will not use them. 305Such arguments might still be useful for executing the scripts directly. 306 307When the script is called with the single argument ``--list``, the script must output to stdout a JSON-encoded hash or 308dictionary that contains all the groups to be managed. Each group's value should be either a hash or dictionary containing a list of each host, any child groups, and potential group variables, or simply a list of hosts:: 309 310 311 { 312 "group001": { 313 "hosts": ["host001", "host002"], 314 "vars": { 315 "var1": true 316 }, 317 "children": ["group002"] 318 }, 319 "group002": { 320 "hosts": ["host003","host004"], 321 "vars": { 322 "var2": 500 323 }, 324 "children":[] 325 } 326 327 } 328 329If any of the elements of a group are empty, they may be omitted from the output. 330 331When called with the argument ``--host <hostname>`` (where <hostname> is a host from above), the script must print either an empty JSON hash/dictionary, or a hash/dictionary of variables to make them available to templates and playbooks. For example:: 332 333 334 { 335 "VAR001": "VALUE", 336 "VAR002": "VALUE", 337 } 338 339Printing variables is optional. If the script does not print variables, it should print an empty hash or dictionary. 340 341.. _inventory_script_tuning: 342 343Tuning the external inventory script 344------------------------------------ 345 346.. versionadded:: 1.3 347 348The stock inventory script system mentioned above works for all versions of Ansible, but calling ``--host`` for every host can be rather inefficient, especially if it involves API calls to a remote subsystem. 349 350To avoid this inefficiency, if the inventory script returns a top-level element called "_meta", it is possible to return all the host variables in a single script execution. When this meta element contains a value for "hostvars", the inventory script will not be invoked with ``--host`` for each host. This behavior results in a significant performance increase for large numbers of hosts. 351 352The data to be added to the top-level JSON dictionary looks like this:: 353 354 { 355 356 # results of inventory script as above go here 357 # ... 358 359 "_meta": { 360 "hostvars": { 361 "host001": { 362 "var001" : "value" 363 }, 364 "host002": { 365 "var002": "value" 366 } 367 } 368 } 369 } 370 371To satisfy the requirements of using ``_meta``, to prevent ansible from calling your inventory with ``--host`` you must at least populate ``_meta`` with an empty ``hostvars`` dictionary. 372For example:: 373 374 { 375 376 # results of inventory script as above go here 377 # ... 378 379 "_meta": { 380 "hostvars": {} 381 } 382 } 383 384 385.. _replacing_inventory_ini_with_dynamic_provider: 386 387If you intend to replace an existing static inventory file with an inventory script, it must return a JSON object which contains an 'all' group that includes every host in the inventory as a member and every group in the inventory as a child. It should also include an 'ungrouped' group which contains all hosts which are not members of any other group. 388A skeleton example of this JSON object is: 389 390.. code-block:: json 391 392 { 393 "_meta": { 394 "hostvars": {} 395 }, 396 "all": { 397 "children": [ 398 "ungrouped" 399 ] 400 }, 401 "ungrouped": { 402 "children": [ 403 ] 404 } 405 } 406 407An easy way to see how this should look is using :ref:`ansible-inventory`, which also supports ``--list`` and ``--host`` parameters like an inventory script would. 408 409.. seealso:: 410 411 :ref:`developing_api` 412 Python API to Playbooks and Ad Hoc Task Execution 413 :ref:`developing_modules_general` 414 Get started with developing a module 415 :ref:`developing_plugins` 416 How to develop plugins 417 `AWX <https://github.com/ansible/awx>`_ 418 REST API endpoint and GUI for Ansible, syncs with dynamic inventory 419 `Development Mailing List <https://groups.google.com/group/ansible-devel>`_ 420 Mailing list for development topics 421 `irc.libera.chat <https://libera.chat/>`_ 422 #ansible IRC chat channel 423