1.. SPDX-License-Identifier: GPL-2.0 2 3Media Controller devices 4------------------------ 5 6Media Controller 7~~~~~~~~~~~~~~~~ 8 9The media controller userspace API is documented in 10:ref:`the Media Controller uAPI book <media_controller>`. This document focus 11on the kernel-side implementation of the media framework. 12 13Abstract media device model 14^^^^^^^^^^^^^^^^^^^^^^^^^^^ 15 16Discovering a device internal topology, and configuring it at runtime, is one 17of the goals of the media framework. To achieve this, hardware devices are 18modelled as an oriented graph of building blocks called entities connected 19through pads. 20 21An entity is a basic media hardware building block. It can correspond to 22a large variety of logical blocks such as physical hardware devices 23(CMOS sensor for instance), logical hardware devices (a building block 24in a System-on-Chip image processing pipeline), DMA channels or physical 25connectors. 26 27A pad is a connection endpoint through which an entity can interact with 28other entities. Data (not restricted to video) produced by an entity 29flows from the entity's output to one or more entity inputs. Pads should 30not be confused with physical pins at chip boundaries. 31 32A link is a point-to-point oriented connection between two pads, either 33on the same entity or on different entities. Data flows from a source 34pad to a sink pad. 35 36Media device 37^^^^^^^^^^^^ 38 39A media device is represented by a struct media_device 40instance, defined in ``include/media/media-device.h``. 41Allocation of the structure is handled by the media device driver, usually by 42embedding the :c:type:`media_device` instance in a larger driver-specific 43structure. 44 45Drivers initialise media device instances by calling 46:c:func:`media_device_init()`. After initialising a media device instance, it is 47registered by calling :c:func:`__media_device_register()` via the macro 48``media_device_register()`` and unregistered by calling 49:c:func:`media_device_unregister()`. An initialised media device must be 50eventually cleaned up by calling :c:func:`media_device_cleanup()`. 51 52Note that it is not allowed to unregister a media device instance that was not 53previously registered, or clean up a media device instance that was not 54previously initialised. 55 56Entities 57^^^^^^^^ 58 59Entities are represented by a struct media_entity 60instance, defined in ``include/media/media-entity.h``. The structure is usually 61embedded into a higher-level structure, such as 62:c:type:`v4l2_subdev` or :c:type:`video_device` 63instances, although drivers can allocate entities directly. 64 65Drivers initialize entity pads by calling 66:c:func:`media_entity_pads_init()`. 67 68Drivers register entities with a media device by calling 69:c:func:`media_device_register_entity()` 70and unregistered by calling 71:c:func:`media_device_unregister_entity()`. 72 73Interfaces 74^^^^^^^^^^ 75 76Interfaces are represented by a 77struct media_interface instance, defined in 78``include/media/media-entity.h``. Currently, only one type of interface is 79defined: a device node. Such interfaces are represented by a 80struct media_intf_devnode. 81 82Drivers initialize and create device node interfaces by calling 83:c:func:`media_devnode_create()` 84and remove them by calling: 85:c:func:`media_devnode_remove()`. 86 87Pads 88^^^^ 89Pads are represented by a struct media_pad instance, 90defined in ``include/media/media-entity.h``. Each entity stores its pads in 91a pads array managed by the entity driver. Drivers usually embed the array in 92a driver-specific structure. 93 94Pads are identified by their entity and their 0-based index in the pads 95array. 96 97Both information are stored in the struct media_pad, 98making the struct media_pad pointer the canonical way 99to store and pass link references. 100 101Pads have flags that describe the pad capabilities and state. 102 103``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data. 104``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data. 105 106.. note:: 107 108 One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must 109 be set for each pad. 110 111Links 112^^^^^ 113 114Links are represented by a struct media_link instance, 115defined in ``include/media/media-entity.h``. There are two types of links: 116 117**1. pad to pad links**: 118 119Associate two entities via their PADs. Each entity has a list that points 120to all links originating at or targeting any of its pads. 121A given link is thus stored twice, once in the source entity and once in 122the target entity. 123 124Drivers create pad to pad links by calling: 125:c:func:`media_create_pad_link()` and remove with 126:c:func:`media_entity_remove_links()`. 127 128**2. interface to entity links**: 129 130Associate one interface to a Link. 131 132Drivers create interface to entity links by calling: 133:c:func:`media_create_intf_link()` and remove with 134:c:func:`media_remove_intf_links()`. 135 136.. note:: 137 138 Links can only be created after having both ends already created. 139 140Links have flags that describe the link capabilities and state. The 141valid values are described at :c:func:`media_create_pad_link()` and 142:c:func:`media_create_intf_link()`. 143 144Graph traversal 145^^^^^^^^^^^^^^^ 146 147The media framework provides APIs to iterate over entities in a graph. 148 149To iterate over all entities belonging to a media device, drivers can use 150the media_device_for_each_entity macro, defined in 151``include/media/media-device.h``. 152 153.. code-block:: c 154 155 struct media_entity *entity; 156 157 media_device_for_each_entity(entity, mdev) { 158 // entity will point to each entity in turn 159 ... 160 } 161 162Drivers might also need to iterate over all entities in a graph that can be 163reached only through enabled links starting at a given entity. The media 164framework provides a depth-first graph traversal API for that purpose. 165 166.. note:: 167 168 Graphs with cycles (whether directed or undirected) are **NOT** 169 supported by the graph traversal API. To prevent infinite loops, the graph 170 traversal code limits the maximum depth to ``MEDIA_ENTITY_ENUM_MAX_DEPTH``, 171 currently defined as 16. 172 173Drivers initiate a graph traversal by calling 174:c:func:`media_graph_walk_start()` 175 176The graph structure, provided by the caller, is initialized to start graph 177traversal at the given entity. 178 179Drivers can then retrieve the next entity by calling 180:c:func:`media_graph_walk_next()` 181 182When the graph traversal is complete the function will return ``NULL``. 183 184Graph traversal can be interrupted at any moment. No cleanup function call 185is required and the graph structure can be freed normally. 186 187Helper functions can be used to find a link between two given pads, or a pad 188connected to another pad through an enabled link 189:c:func:`media_entity_find_link()` and 190:c:func:`media_entity_remote_pad()`. 191 192Use count and power handling 193^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 194 195Due to the wide differences between drivers regarding power management 196needs, the media controller does not implement power management. However, 197the struct media_entity includes a ``use_count`` 198field that media drivers 199can use to track the number of users of every entity for power management 200needs. 201 202The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by 203media drivers and must not be 204touched by entity drivers. Access to the field must be protected by the 205:c:type:`media_device`.\ ``graph_mutex`` lock. 206 207Links setup 208^^^^^^^^^^^ 209 210Link properties can be modified at runtime by calling 211:c:func:`media_entity_setup_link()`. 212 213Pipelines and media streams 214^^^^^^^^^^^^^^^^^^^^^^^^^^^ 215 216When starting streaming, drivers must notify all entities in the pipeline to 217prevent link states from being modified during streaming by calling 218:c:func:`media_pipeline_start()`. 219 220The function will mark all entities connected to the given entity through 221enabled links, either directly or indirectly, as streaming. 222 223The struct media_pipeline instance pointed to by 224the pipe argument will be stored in every entity in the pipeline. 225Drivers should embed the struct media_pipeline 226in higher-level pipeline structures and can then access the 227pipeline through the struct media_entity 228pipe field. 229 230Calls to :c:func:`media_pipeline_start()` can be nested. 231The pipeline pointer must be identical for all nested calls to the function. 232 233:c:func:`media_pipeline_start()` may return an error. In that case, 234it will clean up any of the changes it did by itself. 235 236When stopping the stream, drivers must notify the entities with 237:c:func:`media_pipeline_stop()`. 238 239If multiple calls to :c:func:`media_pipeline_start()` have been 240made the same number of :c:func:`media_pipeline_stop()` calls 241are required to stop streaming. 242The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last 243nested stop call. 244 245Link configuration will fail with ``-EBUSY`` by default if either end of the 246link is a streaming entity. Links that can be modified while streaming must 247be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag. 248 249If other operations need to be disallowed on streaming entities (such as 250changing entities configuration parameters) drivers can explicitly check the 251media_entity stream_count field to find out if an entity is streaming. This 252operation must be done with the media_device graph_mutex held. 253 254Link validation 255^^^^^^^^^^^^^^^ 256 257Link validation is performed by :c:func:`media_pipeline_start()` 258for any entity which has sink pads in the pipeline. The 259:c:type:`media_entity`.\ ``link_validate()`` callback is used for that 260purpose. In ``link_validate()`` callback, entity driver should check 261that the properties of the source pad of the connected entity and its own 262sink pad match. It is up to the type of the entity (and in the end, the 263properties of the hardware) what matching actually means. 264 265Subsystems should facilitate link validation by providing subsystem specific 266helper functions to provide easy access for commonly needed information, and 267in the end provide a way to use driver-specific callbacks. 268 269Media Controller Device Allocator API 270^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 271 272When the media device belongs to more than one driver, the shared media 273device is allocated with the shared struct device as the key for look ups. 274 275The shared media device should stay in registered state until the last 276driver unregisters it. In addition, the media device should be released when 277all the references are released. Each driver gets a reference to the media 278device during probe, when it allocates the media device. If media device is 279already allocated, the allocate API bumps up the refcount and returns the 280existing media device. The driver puts the reference back in its disconnect 281routine when it calls :c:func:`media_device_delete()`. 282 283The media device is unregistered and cleaned up from the kref put handler to 284ensure that the media device stays in registered state until the last driver 285unregisters the media device. 286 287**Driver Usage** 288 289Drivers should use the appropriate media-core routines to manage the shared 290media device life-time handling the two states: 2911. allocate -> register -> delete 2922. get reference to already registered device -> delete 293 294call :c:func:`media_device_delete()` routine to make sure the shared media 295device delete is handled correctly. 296 297**driver probe:** 298Call :c:func:`media_device_usb_allocate()` to allocate or get a reference 299Call :c:func:`media_device_register()`, if media devnode isn't registered 300 301**driver disconnect:** 302Call :c:func:`media_device_delete()` to free the media_device. Freeing is 303handled by the kref put handler. 304 305API Definitions 306^^^^^^^^^^^^^^^ 307 308.. kernel-doc:: include/media/media-device.h 309 310.. kernel-doc:: include/media/media-devnode.h 311 312.. kernel-doc:: include/media/media-entity.h 313 314.. kernel-doc:: include/media/media-request.h 315 316.. kernel-doc:: include/media/media-dev-allocator.h 317