1tsdemux/tsparse TODO
2--------------------
3
4* Perfomance
5  * Bufferlist : Creating/Destroying very small buffers is too
6  costly. Switch to pre-/re-allocating outgoing buffers in which we
7  copy the data.
8  * Adapter : Use gst_adapter_peek()/_flush() instead of constantly
9  creating buffers.
10
11* Latency
12  * Calculate the actual latency instead of returning a fixed
13  value. The latency (for live streams) is the difference between the
14  currently inputted buffer timestamp (can be stored in the
15  packetizer) and the buffer we're pushing out.
16  This value should be reported/updated (leave a bit of extra margin
17  in addition to the calculated value).
18
19* mpegtsparser
20  * SERIOUS room for improvement performance-wise (see callgrind),
21  mostly related to performance issues mentionned above.
22
23* Random-access seeking
24  * Do minimal parsing of video headers to detect keyframes and use
25  that to compute the keyframe intervals. Use that interval to offset
26  the seek position in order to maximize the chance of pushing out the
27  requested frames.
28
29
30Synchronization, Scheduling and Timestamping
31--------------------------------------------
32
33  A mpeg-ts demuxer can be used in a variety of situations:
34  * lives streaming over DVB, UDP, RTP,..
35  * play-as-you-download like HTTP Live Streaming or UPNP/DLNA
36  * random-access local playback, file, Bluray, ...
37
38  Those use-cases can be categorized in 3 different categories:
39  * Push-based scheduling with live sources [0]
40  * Push-based scheduling with non-live sources
41  * Pull-based scheduling with fast random-access
42
43  Due to the nature of timing within the mpeg-ts format, we need to
44pay extra attention to the outgoing NEWSEGMENT event and buffer
45timestamps in order to guarantee proper playback and synchronization
46of the stream.
47
48  In the following, 'timestamps' correspond to GStreamer
49  buffer/segment values. The mpeg-ts PCR/DTS/PTS values are indicated
50  with their actual name.
51
52 1) Live push-based scheduling
53
54  The NEWSEGMENT event will be in time format and is forwarded as is,
55  and the values are cached locally.
56
57  Since the clock is running when the upstream buffers are captured,
58  the outgoing buffer timestamps need to correspond to the incoming
59  buffer timestamp values.
60
61    => mpegtspacketizer keeps track of PCR and input timestamp and
62       extrapolates a clock skew using the EPTLA algorithm.
63
64    => The outgoing buffers will be timestamped with their PTS values
65       (overflow corrected) corrected by that calculated clock skew.
66
67  A latency is introduced between the time the buffer containing the
68  first bit of a Access Unit is received in the demuxer and the moment
69  the demuxer pushed out the buffer corresponding to that Access Unit.
70
71    => That latency needs to be reported.
72
73  According to the ISO/IEC 13818-1:2007 specifications, D.0.1 Timing
74  mode, the "coded audio and video that represent sound and pictures
75  that are to be presented simultaneously may be separated in time
76  within the coded bit stream by ==>as much as one second<=="
77
78    => The algorithm to calculate the latency should take that into
79       account.
80
81
82 2) Non-live push-based scheduling
83
84  If the upstream NEWSEGMENT is in time format, the NEWSEGMENT event
85  is forwarded as is, and the values are cached locally.
86
87  If upstream does provide a NEWSEGMENT in another format, we need to
88  compute one by taking the default values:
89    start : 0
90    stop  : GST_CLOCK_TIME_NONE
91    time  : 0
92
93  Since no prerolling is happening downstream and the incoming buffers
94  do not have capture timestamps, we need to ensure the first buffer
95  we push out corresponds to the base segment start runing time.
96
97    => The packetizer keeps track of PCR locations and offsets in
98       addition to the clock skew (in the case of upstream buffers
99       being timestamped, which is the case for HLS).
100
101    => The demuxer indicates to the packetizer when he sees the
102       'beginning' of the program (i.e. the first valid PAT/PMT
103       combination). The packetizer will then use that location as
104       "timestamp 0", or "reference position/PCR".
105
106    => The lowest DTS is passed to the packetizer to be converted to
107       timestamp. That value is computed in the same way as live
108       streams if upstream buffers have timestamps, or will be
109       subtracted from the reference PCR.
110
111    => The outgoing buffers will be timestamped with their PTS values
112       (overflow corrected) adjusted by the packetizer.
113
114  Latency is reported just as with the live use-case.
115
116
117 3) Random access pull-mode
118
119  We do not get a NEWSEGMENT event from upstream, we therefore need to
120  compute the outgoing values.
121
122    => The outgoing values for the newsegment are calculated like for
123       the non-live push-based mode when upstream doesn't provide
124       timestamp'ed buffers.
125
126    => The outgoing buffer timestamps are timestamped with their PTS
127       values (overflow corrected) adjusted by the packetizer.
128
129
130
131[0] When talking about live sources, we mean this in the GStreamer
132definition of live sources, which is to say sources where if we miss
133the capture, we will miss the data to be captured. Sources which do
134internal buffering (like TCP connections or file descriptors) are
135*NOT* live sources.
136