1This directory contains some RTP payloaders/depayloaders for different payload 2types. Use one payloader/depayloder pair per payload. If several payloads can be 3payloaded/depayloaded by the same element, make different copies of it, one for 4each payload. 5 6The application/x-rtp mime type 7------------------------------- 8 9For valid RTP packets encapsulated in GstBuffers, we use the caps with 10mime type application/x-rtp. 11 12The following fields can or must (*) be specified in the structure: 13 14 * media: (String) [ "audio", "video", "application", "data", "control" ] 15 Defined in RFC 2327 in the SDP media announcement field. 16 Converted to lower case. 17 18 * payload: (int) [0, 127] 19 For audio and video, these will normally be a media payload type as 20 defined in the RTP Audio/Video Profile. For dynamicaly allocated 21 payload types, this value will be >= 96 and the encoding-name must be 22 set. 23 24 * clock-rate: (int) [0 - MAXINT] 25 The RTP clock rate. 26 27 encoding-name: (String) ANY 28 typically second part of the mime type. ex. MP4V-ES. only required if 29 payload type >= 96. Converted to upper case. 30 31 encoding-params: (String) ANY 32 extra encoding parameters (as in the SDP a=rtpmap: field). only required 33 if different from the default of the encoding-name. 34 Converted to lower-case. 35 36 ssrc: (uint) [0 - MAXINT] 37 The ssrc value currently in use. (default = the SSRC of the first RTP 38 packet) 39 40 timestamp-offset: (uint) [0 - MAXINT] 41 The RTP time representing time npt-start. (default = rtptime of first RTP 42 packet). 43 44 seqnum-offset: (uint) [0 - MAXINT] 45 The RTP sequence number representing the first rtp packet. When this 46 parameter is given, all sequence numbers below this seqnum should be 47 ignored. (default = seqnum of first RTP packet). 48 49 npt-start: (uint64) [0 - MAXINT] 50 The Normal Play Time for clock-base. This is the position in the stream and 51 is between 0 and the duration of the stream. This value is expressed in 52 nanoseconds GstClockTime. (default = 0) 53 54 npt-stop: (uint64) [0 - MAXINT] 55 The last position in the stream. This value is expressed in nanoseconds 56 GstClockTime. (default = -1, stop unknown) 57 58 play-speed: (gdouble) [-MIN - MAX] 59 The intended playback speed of the stream. The client is delivered data at 60 the adjusted speed. The client should adjust its playback speed with this 61 value and thus corresponds to the GStreamer rate field in the NEWSEGMENT 62 event. (default = 1.0) 63 64 play-scale: (gdouble) [-MIN - MAX] 65 The rate already applied to the stream. The client is delivered a stream 66 that is scaled by this amount. This value is used to adjust position 67 reporting and corresponds to the GStream applied-rate field in the 68 NEWSEGMENT event. (default = 1.0) 69 70 maxptime: (uint) [0, MAX] 71 The maxptime as defined in RFC 4566, this defines the maximum size of a 72 packet. It overrides the max-ptime property of payloaders. 73 74 Optional parameters as key/value pairs, media type specific. The value type 75 should be of type G_TYPE_STRING. The key is converted to lower-case. The 76 value is left in its original case. 77 A parameter with no value is converted to <param>=1. 78 79 Example: 80 81 "application/x-rtp", 82 "media", G_TYPE_STRING, "audio", -. 83 "payload", G_TYPE_INT, 96, | - required 84 "clock-rate", G_TYPE_INT, 8000, -' 85 "encoding-name", G_TYPE_STRING, "AMR", -. - required since payload >= 96 86 "encoding-params", G_TYPE_STRING, "1", -' - optional param for AMR 87 "octet-align", G_TYPE_STRING, "1", -. 88 "crc", G_TYPE_STRING, "0", | 89 "robust-sorting", G_TYPE_STRING, "0", | AMR specific params. 90 "interleaving", G_TYPE_STRING, "0", -' 91 92 Mapping of caps to and from SDP fields: 93 94 m=<media> <udp port> RTP/AVP <payload> -] media and payload from caps 95 a=rtpmap:<payload> <encoding-name>/<clock-rate>[/<encoding-params>] 96 -> when <payload> >= 96 97 a=fmtp:<payload> <param>=<value>;... 98 99 For above caps: 100 101 m=audio <udp port> RTP/AVP 96 102 a=rtpmap:96 AMR/8000/1 103 a=fmtp:96 octet-align=1;crc=0;robust-sorting=0;interleaving=0 104 105 Attributes are converted as follows: 106 107 IANA registered attribute names are prepended with 'a-' before putting them in 108 the caps. Unregistered keys (starting with 'x-') are copied directly into the 109 caps. 110 111 in RTSP, the SSRC is also sent. 112 113 The optional parameters in the SDP fields are case insensitive. In the caps we 114 always use the lowercase names so that the SDP -> caps mapping remains 115 possible. 116 117 Mapping of caps to NEWSEGMENT: 118 119 rate: <play-speed> 120 applied-rate: <play-scale> 121 format: GST_FORMAT_TIME 122 start: <clock-base> * GST_SECOND / <clock-rate> 123 stop: if <ntp-stop> != -1 124 <npt-stop> - <npt-start> + start 125 else 126 -1 127 time: <npt-start> 128 129 130Timestamping 131------------ 132 133RTP in GStreamer uses a combination of the RTP timestamps and GStreamer buffer 134timestamps to ensure proper synchronisation at the sender and the receiver end. 135 136In RTP applications, the synchronisation is most complex at the receiver side. 137 138At the sender side, the RTP timestamps are generated in the payloaders based on 139GStreamer timestamps. At the receiver, GStreamer timestamps are reconstructed 140from the RTP timestamps and the GStreamer timestamps in the jitterbuffer. This 141process is explained in more detail below. 142 143= synchronisation at the sender 144 145Individual streams at the sender are synchronised using GStreamer timestamps. 146The payloader at the sender will convert the GStreamer timestamp into an RTP 147timestamp using the following formula: 148 149 RTP = ((RT - RT-base) * clock-rate / GST_SECOND) + RTP-offset 150 151 RTP: the RTP timestamp for the stream. This value is truncated to 152 32 bits. 153 RT: the GStreamer running time corresponding to the timestamp of the 154 packet to payload 155 RT-base: the GStreamer running time of the first packet encoded 156 clock-rate: the clock-rate of the stream 157 RTP-offset: a random RTP offset 158 159The RTP timestamp corresponding to RT-base is the clock-base (see caps above). 160 161In addition to setting an RTP timestamp in the RTP packet, the payloader is also 162responsible for putting the GStreamer timestamp on the resulting output buffer. 163This timestamp is used for further synchronisation at the sender pipeline, such 164as for sending out the packet on the network. 165 166Notice that the absolute timing information is lost; if the sender is sending 167multiple streams, the RTP timestamps in the packets do not contain enough 168information to synchronize them in the receiver. The receiver can however use 169the RTP timestamps to reconstruct the timing of the stream as it was created by 170the sender according to the sender's clock. 171 172Because the payloaded packet contains both an RTP timestamp and a GStreamer 173timestamp, it is possible for an RTP session manager to derive the relation 174between the RTP and GST timestamps. This information is used by a session 175manager to create SR reports. The NTP time in the report will contain the 176running time converted to NTP time and the corresponding RTP timestamp. 177 178Note that at the sender side, the RTP and GStreamer timestamp both increment at 179the same rate, the sender rate. This rate depends on the global pipeline clock 180of the sender. 181 182Some pipelines to illustrate the process: 183 184 gst-launch-1.0 v4l2src ! videoconvert ! avenc_h263p ! rtph263ppay ! udpsink 185 186 v4l2src puts a GStreamer timestamp on the video frames base on the current 187 running_time. The encoder encodes and passed the timestamp on. The payloader 188 generates an RTP timestamp using the above formula and puts it in the RTP 189 packet. It also copies the incoming GStreamer timestamp on the output RTP 190 packet. udpsink synchronizes on the gstreamer timestamp before pushing out the 191 packet. 192 193 194= synchronisation at the receiver 195 196The receiver is responsible for timestamping the received RTP packet with the 197running_time of the clock at the time the packet was received. This GStreamer 198timestamp reflects the receiver rate and depends on the global pipeline clock of 199the receiver. The gstreamer timestamp of the received RTP packet contains a 200certain amount of jitter introduced by the network. 201 202The most simple option for the receiver is to depayload the RTP packet and play 203it back as soon as possible, this is with the timestamp when it was received 204from the network. For the above sender pipeline this would be done with the 205following pipeline: 206 207 gst-launch-1.0 udpsrc caps="application/x-rtp, media=(string)video, 208 clock-rate=(int)90000, encoding-name=(string)H263-1998" ! rtph263pdepay ! 209 avdec_h263 ! autovideosink 210 211It is important that the depayloader copies the incoming GStreamer timestamp 212directly to the depayloaded output buffer. It should never attempt to perform 213any logic with the RTP timestamp, this task is for the jitterbuffer as we will 214see next. 215 216The above pipeline does not attempt to deal with reordered packets or network 217jitter, which could result in jerky playback in the case of high jitter or 218corrupted video in the case of packet loss or reordering. This functionality is 219performed by the gstrtpjitterbuffer in GStreamer. 220 221The task of the gstrtpjitterbuffer element is to: 222 223 - deal with reordered packets based on the seqnum 224 - calculate the drift between the sender and receiver clocks using the 225 GStreamer timestamps (receiver clock rate) and RTP timestamps (sender clock 226 rate). 227 228To deal with reordered packet, the jitterbuffer holds on to the received RTP 229packets in a queue for a configurable amount of time, called the latency. 230 231The jitterbuffer also eliminates network jitter and then tracks the drift 232between the local clock (as expressed in the GStreamer timestamps) and the 233remote clock (as expressed in the RTP timestamps). It will remove the jitter 234and will apply the drift correction to the GStreamer timestamp before pushing 235the buffer downstream. The result is that the depayloader receives a smoothed 236GStreamer timestamp on the RTP packet, which is copied to the depayloaded data. 237 238The following pipeline illustrates a receiver with a jitterbuffer. 239 240 gst-launch-1.0 udpsrc caps="application/x-rtp, media=(string)video, 241 clock-rate=(int)90000, encoding-name=(string)H263-1998" ! 242 rtpjitterbuffer latency=100 ! rtph263pdepay ! avdec_h263 ! autovideosink 243 244The latency property on the jitterbuffer controls the amount of delay (in 245milliseconds) to apply to the outgoing packets. A higher latency will produce 246smoother playback in networks with high jitter but cause a higher latency. 247Choosing a good value for the latency is a tradeoff between the quality and 248latency. The better the network, the lower the latency can be set. 249 250 251usage with UDP 252-------------- 253 254To correctly and completely use the RTP payloaders on the sender and the 255receiver you need to write an application. It is not possible to write a full 256blown RTP server with a single gst-launch-1.0 line. 257 258That said, it is possible to do something functional with a few gst-launch 259lines. The biggest problem when constructing a correct gst-launch-1.0 line lies on 260the receiver end. 261 262The receiver needs to know about the type of the RTP data along with a set of 263RTP configuration parameters. This information is usually transmitted to the 264client using some sort of session description language (SDP) over some reliable 265channel (HTTP/RTSP/...). 266 267All of the required parameters to connect and use the RTP session on the 268server can be found in the caps on the server end. The client receives this 269information in some way (caps are converted to and from SDP, as explained above, 270for example). 271 272Some gst-launch-1.0 lines: 273 274 gst-launch-1.0 -v videotestsrc ! videoconvert ! avenc_h263p ! rtph263ppay ! udpsink 275 276 Setting pipeline to PAUSED ... 277 /pipeline0/videotestsrc0.src: caps = video/x-raw, format=(string)I420, 278 width=(int)320, height=(int)240, framerate=(fraction)30/1 279 Pipeline is PREROLLING ... 280 .... 281 /pipeline0/udpsink0.sink: caps = application/x-rtp, media=(string)video, 282 payload=(int)96, clock-rate=(int)90000, encoding-name=(string)H263-1998, 283 ssrc=(guint)527842345, clock-base=(guint)1150776941, seqnum-base=(guint)30982 284 .... 285 Pipeline is PREROLLED ... 286 Setting pipeline to PLAYING ... 287 New clock: GstSystemClock 288 289 Write down the caps on the udpsink and set them as the caps of the UDP 290 receiver: 291 292 gst-launch-1.0 -v udpsrc caps="application/x-rtp, media=(string)video, 293 payload=(int)96, clock-rate=(int)90000, encoding-name=(string)H263-1998, 294 ssrc=(guint)527842345, clock-base=(guint)1150776941, seqnum-base=(guint)30982" 295 ! rtph263pdepay ! avdec_h263 ! autovideosink 296 297 The receiver now displays an h263 image. Since there is no jitterbuffer in the 298 pipeline, frames will be displayed at the time when they are received. This can 299 result in jerky playback in the case of high network jitter or currupted video 300 when packets are dropped or reordered. 301 302 Stream a quicktime file with mpeg4 video and AAC audio on port 5000 and port 303 5002. 304 305 gst-launch-1.0 -v filesrc location=~/data/sincity.mp4 ! qtdemux name=d ! queue ! rtpmp4vpay ! udpsink port=5000 306 d. ! queue ! rtpmp4gpay ! udpsink port=5002 307 .... 308 /pipeline0/udpsink0.sink: caps = application/x-rtp, media=(string)video, 309 payload=(int)96, clock-rate=(int)90000, encoding-name=(string)MP4V-ES, 310 ssrc=(guint)1162703703, clock-base=(guint)816135835, seqnum-base=(guint)9294, 311 profile-level-id=(string)3, config=(string)000001b003000001b50900000100000001200086c5d4c307d314043c1463000001b25876694430303334 312 /pipeline0/udpsink1.sink: caps = application/x-rtp, media=(string)audio, 313 payload=(int)96, clock-rate=(int)44100, encoding-name=(string)MPEG4-GENERIC, 314 ssrc=(guint)3246149898, clock-base=(guint)4134514058, seqnum-base=(guint)57633, 315 encoding-params=(string)2, streamtype=(string)5, profile-level-id=(string)1, 316 mode=(string)aac-hbr, config=(string)1210, sizelength=(string)13, 317 indexlength=(string)3, indexdeltalength=(string)3 318 .... 319 320 Again copy the caps on both sinks to the receiver launch line 321 322 gst-launch-1.0 323 udpsrc port=5000 caps="application/x-rtp, media=(string)video, payload=(int)96, 324 clock-rate=(int)90000, encoding-name=(string)MP4V-ES, ssrc=(guint)1162703703, 325 clock-base=(guint)816135835, seqnum-base=(guint)9294, profile-level-id=(string)3, 326 config=(string)000001b003000001b50900000100000001200086c5d4c307d314043c1463000001b25876694430303334" 327 ! rtpmp4vdepay ! ffdec_mpeg4 ! autovideosink sync=false 328 udpsrc port=5002 caps="application/x-rtp, media=(string)audio, payload=(int)96, 329 clock-rate=(int)44100, encoding-name=(string)MPEG4-GENERIC, ssrc=(guint)3246149898, 330 clock-base=(guint)4134514058, seqnum-base=(guint)57633, encoding-params=(string)2, 331 streamtype=(string)5, profile-level-id=(string)1, mode=(string)aac-hbr, 332 config=(string)1210, sizelength=(string)13, indexlength=(string)3, 333 indexdeltalength=(string)3" 334 ! rtpmp4gdepay ! faad ! alsasink sync=false 335 336 The caps on the udpsinks can be retrieved when the server pipeline prerolled to 337 PAUSED. 338 339 The above pipeline sets sync=false on the audio and video sink which means that 340 no synchronisation will be performed in the sinks, they play the data when it 341 arrives. If you want to enable synchronisation in the sinks it is highly 342 recommended to use a gstrtpjitterbuffer after the udpsrc elements. 343 344 Even when sync is enabled, the two different streams will not play synchronised 345 against eachother because the receiver does not have enough information to 346 perform this task. For this you need to add the rtpbin element in both the 347 sender and receiver pipeline and use additional sources and sinks to transmit 348 RTCP packets used for inter-stream synchronisation. 349 350 The caps on the receiver side can be set on the UDP source elements when the 351 pipeline went to PAUSED. In that state no data is received from the UDP sources 352 as they are live sources and only produce data in PLAYING. 353 354 355Relevant RFCs 356------------- 357 3583550 RTP: A Transport Protocol for Real-Time Applications. ( 1889 Obsolete ) 359 3602198 RTP Payload for Redundant Audio Data. 3613119 A More Loss-Tolerant RTP Payload Format for MP3 Audio. 362 3632793 RTP Payload for Text Conversation. 364 3652032 RTP Payload Format for H.261 Video Streams. 3662190 RTP Payload Format for H.263 Video Streams. 3672250 RTP Payload Format for MPEG1/MPEG2 Video. 3682343 RTP Payload Format for Bundled MPEG. 3692429 RTP Payload Format for the 1998 Version of ITU-T Rec. H.263 Video 3702431 RTP Payload Format for BT.656 Video Encoding. 3712435 RTP Payload Format for JPEG-compressed Video. 3723016 RTP Payload Format for MPEG-4 Audio/Visual Streams. 3733047 RTP Payload Format for ITU-T Recommendation G.722.1. 3743189 RTP Payload Format for DV (IEC 61834) Video. 3753190 RTP Payload Format for 12-bit DAT Audio and 20- and 24-bit Linear Sampled Audio. 3763389 Real-time Transport Protocol (RTP) Payload for Comfort Noise (CN) 3772733 An RTP Payload Format for Generic Forward Error Correction. 3782833 RTP Payload for DTMF Digits, Telephony Tones and Telephony 379 Signals. 3802862 RTP Payload Format for Real-Time Pointers. 3813351 RTP Profile for Audio and Video Conferences with Minimal Control. ( 1890 Obsolete ) 3823555 MIME Type Registration of RTP Payload Formats. 383 3842508 Compressing IP/UDP/RTP Headers for Low-Speed Serial Links. 3851305 Network Time Protocol (Version 3) Specification, Implementation and Analysis. 3863339 Date and Time on the Internet: Timestamps. 3872246 The TLS Protocol Version 1.0 3883546 Transport Layer Security (TLS) Extensions. ( Updates 2246 ) 389 390do we care? 391----------- 392 3932029 RTP Payload Format of Sun's CellB Video Encoding. 394 395useful 396------ 397 398http://www.iana.org/assignments/rtp-parameters 399