1OBS Studio Backend Design 2========================= 3The OBS Studio backend is powered by the library libobs. Libobs 4provides the main pipeline, the video/audio subsystems, and the general 5framework for all plugins. 6 7 8Libobs Plugin Objects 9--------------------- 10Libobs is designed to be modular, where adding modules will add custom 11functionality. There are four libobs objects that you can make plugins 12for: 13 14- :ref:`plugins_sources` -- Sources are used to render video and/or 15 audio on stream. Things such as capturing displays/games/audio, 16 playing a video, showing an image, or playing audio. Sources can also 17 be used to implement audio and video filters. 18 19- :ref:`plugins_outputs` -- Outputs allow the ability to output the 20 currently rendering audio/video. Streaming and recording are two 21 common examples of outputs, but not the only types of outputs. 22 Outputs can receive the raw data or receive encoded data. 23 24- :ref:`plugins_encoders` -- Encoders are OBS-specific implementations 25 of video/audio encoders, which are used with outputs that use 26 encoders. x264, NVENC, Quicksync are examples of encoder 27 implementations. 28 29- :ref:`plugins_services` -- Services are custom implementations of 30 streaming services, which are used with outputs that stream. For 31 example, you could have a custom implementation for streaming to 32 Twitch, and another for YouTube to allow the ability to log in and use 33 their APIs to do things such as get the RTMP servers or control the 34 channel. 35 36*(Author's note: the service API is incomplete as of this writing)* 37 38 39Libobs Threads 40-------------- 41There are three primary threads spawned by libobs on initialization: 42 43- The obs_graphics_thread_ function used exclusively for rendering in 44 `libobs/obs-video.c`_ 45 46- The video_thread_ function used exclusively for video encoding/output 47 in `libobs/media-io/video-io.c`_ 48 49- The audio_thread_ function used for all audio 50 processing/encoding/output in `libobs/media-io/audio-io.c`_ 51 52*(Author's note: obs_graphics_thread was originally named 53obs_video_thread; it was renamed as of this writing to prevent confusion 54with video_thread)* 55 56 57.. _output_channels: 58 59Output Channels 60--------------- 61Rendering video or audio starts from output channels. You assign a 62source to an output channel via the :c:func:`obs_set_output_source()` 63function. The *channel* parameter can be any number from 640..(MAX_CHANNELS_-1). You may initially think that this is how you 65display multiple sources at once; however, sources are hierarchical. 66Sources such as scenes or transitions can have multiple sub-sources, and 67those sub-sources in turn can have sub-sources and so on (see 68:ref:`displaying_sources` for more information). Typically, you would 69use scenes to draw multiple sources as a group with specific transforms 70for each source, as a scene is just another type of source. The 71"channel" design allows for highly complex video presentation setups. 72The OBS Studio front-end has yet to even fully utilize this back-end 73design for its rendering, and currently only uses one output channel to 74render one scene at a time. It does however utilize additional channels 75for things such as global audio sources which are set in audio settings. 76 77*(Author's note: "Output channels" are not to be confused with output 78objects or audio channels. Output channels are used to set the sources 79you want to output, and output objects are used for actually 80streaming/recording/etc.)* 81 82 83General Video Pipeline Overview 84------------------------------- 85The video graphics pipeline is run from two threads: a dedicated 86graphics thread that renders preview displays as well as the final mix 87(the obs_graphics_thread_ function in `libobs/obs-video.c`_), and a 88dedicated thread specific to video encoding/output (the video_thread_ 89function in `libobs/media-io/video-io.c`_). 90 91Sources assigned to output channels will be drawn from channels 920..(MAX_CHANNELS_-1). They are drawn on to the final texture which will 93be used for output `[1]`_. Once all sources are drawn, the final 94texture is converted to whatever format that libobs is set to (typically 95a YUV format). After being converted to the back-end video format, it's 96then sent along with its timestamp to the current video handler, 97`obs_core_video::video`_. 98 99It then puts that raw frame in a queue of MAX_CACHE_SIZE_ in the `video 100output handler`_. A semaphore is posted, then the video-io thread will 101process frames as it's able. If the video frame queue is full, it will 102duplicate the last frame in the queue in an attempt to reduce video 103encoding complexity (and thus CPU usage) `[2]`_. This is why you may 104see frame skipping when the encoder can't keep up. Frames are sent to 105any raw outputs or video encoders that are currently active `[3]`_. 106 107If it's sent to a video encoder object (`libobs/obs-encoder.c`_), it 108encodes the frame and sends the encoded packet off to the outputs that 109encoder is connected to (which can be multiple). If the output takes 110both encoded video/audio, it puts the packets in an interleave queue to 111ensure encoded packets are sent in monotonic timestamp order `[4]`_. 112 113The encoded packet or raw frame is then sent to the output. 114 115 116General Audio Pipeline Overview 117------------------------------- 118The audio pipeline is run from a dedicated audio thread in the audio 119handler (the `audio_thread`_ function in `libobs/media-io/audio-io.c`_); 120assuming that AUDIO_OUTPUT_FRAMES_ is set to 1024, the audio thread 121"ticks" (processes audio data) once every 1024 audio samples (around 122every 21 millisecond intervals at 48khz), and calls the audio_callback_ 123function in `libobs/obs-audio.c`_ where most of the audio processing is 124accomplished. 125 126A source with audio will output its audio via the 127obs_source_output_audio_ function, and that audio data will be appended 128or inserted in to the circular buffer `obs_source::audio_input_buf`_. 129If the sample rate or channel count does not match what the back-end is 130set to, the audio is automatically remixed/resampled via swresample 131`[5]`_. Before insertion, audio data is also run through any audio 132filters attached to the source `[6]`_. 133 134Each audio tick, the audio thread takes a reference snapshot of the 135audio source tree (stores references of all sources that output/process 136audio) `[7]`_. On each audio leaf (audio source), it takes the closest 137audio (relative to the current audio thread timestamp) stored in the 138circular buffer `obs_source::audio_input_buf`_, and puts it in 139`obs_source::audio_output_buf`_. 140 141Then, the audio samples stored in `obs_source::audio_output_buf`_ of the 142leaves get sent through their parents in the source tree snapshot for 143mixing or processing at each source node in the hierarchy `[8]`_. 144Sources with multiple children such as scenes or transitions will 145mix/process their children's audio themselves via the 146`obs_source_info::audio_render`_ callback. This allows, for example, 147transitions to fade in the audio of one source and fade in the audio of 148a new source when they're transitioning between two sources. The mix or 149processed audio data is then stored in `obs_source::audio_output_buf`_ 150of that node similarly, and the process is repeated until the audio 151reaches the root nodes of the tree. 152 153Finally, when the audio has reached the base of the snapshot tree, the 154audio of all the sources in each output channel are mixed together for a 155final mix `[9]`_. That final mix is then sent to any raw outputs or 156audio encoders that are currently active `[10]`_. 157 158If it's sent to an audio encoder object (`libobs/obs-encoder.c`_), it 159encodes the audio data and sends the encoded packet off to the outputs 160that encoder is connected to (which can be multiple). If the output 161takes both encoded video/audio, it puts the packets in an interleave 162queue to ensure encoded packets are sent in monotonic timestamp order 163`[4]`_. 164 165The encoded packet or raw audio data is then sent to the output. 166 167.. _obs_graphics_thread: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-video.c#L588-L651 168.. _libobs/obs-audio.c: https://github.com/jp9000/obs-studio/blob/master/libobs/obs-audio.c 169.. _libobs/obs-video.c: https://github.com/jp9000/obs-studio/blob/master/libobs/obs-video.c 170.. _video_thread: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L169-L195 171.. _libobs/media-io/video-io.c: https://github.com/jp9000/obs-studio/blob/master/libobs/media-io/video-io.c 172.. _video output handler: https://github.com/jp9000/obs-studio/blob/master/libobs/media-io/video-io.c 173.. _audio_thread: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.c#L241-L282 174.. _libobs/media-io/audio-io.c: https://github.com/jp9000/obs-studio/blob/master/libobs/media-io/audio-io.c 175.. _MAX_CHANNELS: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-defs.h#L20-L21 176.. _[1]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-video.c#L99-L129 177.. _obs_core_video::video: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-internal.h#L250 178.. _MAX_CACHE_SIZE: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L34 179.. _[2]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L431-L434 180.. _[3]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/video-io.c#L115-L167 181.. _libobs/obs-encoder.c: https://github.com/jp9000/obs-studio/blob/master/libobs/obs-encoder.c 182.. _[4]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-output.c#L1382-L1439 183.. _AUDIO_OUTPUT_FRAMES: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.h#L30 184.. _audio_callback: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L367-L485 185.. _obs_source_output_audio: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2578-L2608 186.. _obs_source::audio_input_buf: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L1280-L1283 187.. _[5]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2561-L2563 188.. _[6]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.c#L2591 189.. _[7]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L393-L415 190.. _obs_source::audio_output_buf: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-internal.h#L580 191.. _[8]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L417-L423 192.. _obs_source_info::audio_render: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-source.h#L410-L412 193.. _[9]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/obs-audio.c#L436-L453 194.. _[10]: https://github.com/jp9000/obs-studio/blob/2c58185af3c85f4e594a4c067c9dfe5fa4b5b0a9/libobs/media-io/audio-io.c#L144-L165 195