1# Low Level Information for the SRT project
2
3## Introduction
4
5This document contains information on various topics related to
6the SRT source code, including descriptions of some cross-source analysis that would
7not be obvious for a source code reviewer. It's not a complete documentation of
8anything, rather a collection of various kind of information retrieved during
9development and even reverse engineering.
10
11## Mutex locking
12
13This analysis is a result of detected lots of cascade mutex locking in the
14SRT code. A more detailed analysis would be required as to which mutex is
15going to protect what kind of data later.
16
17Here is the info collected so far:
18
19### Data structures
20
21The overall structure of the object database, involving sockets and groups
22is as follows (pseudo-language):
23
24```
25CUDTUnited (singleton) {
26     CONTAINER<CUDTSocket> m_Sockets;
27     CONTAINER<CUDTSocket> m_ClosedSockets;
28     CONTAINER<CUDTGroup> m_Groups;
29     CONTAINER<CUDTGroup> m_ClosedGroups;
30}
31
32CUDTGroup {
33     type SocketData { CUDTSocket* ps; SRTSOCKET id; int state; ... }
34     CONTAINER<SocketData> m_Group;
35}
36```
37
38Dead sockets (either closed manually or broken after connection) are
39moved first from `m_Sockets` to `m_ClosedSockets`. The GC thread will take
40care to delete them physically after making sure all inside facilities
41do not contain any remaining data of interest.
42
43Groups may only be manually closed, however a closed group is moved
44to `m_ClosedGroups`. The GC thread will take care to delete them, as long
45as their usage counter is 0. Every call to an API function (as well as
46TSBPD thread) increases the usage counter in the beginning and decreases
47upon exit. A group may be closed in one thread and still being used in
48another. The group will persist up to the time when the current API function
49using it exits and decreases the usage counter back to 0.
50
51Containers and contents guarded by mutex:
52
53`CUDTUnited::m_GlobControlLock` - guards all containers in CUDTUnited.
54
55`CUDTSocket::m_ControlLock` - guards internal operation performed on particular
56socket, with its existence assumed (this is because a socket will always exist
57until it's deleted while being in `m_ClosedSockets`, and when the socket is in
58`m_ClosedSockets` it will not be deleted until it's free from any operation,
59while the socket is assumed nonexistent for any newly called API function even
60if it exists physically, but is moved to `m_ClosedSockets`).
61
62`CUDTGroup::m_GroupLock` - guards the `m_Group` container inside a group that
63collects member sockets.
64
65There are unfortunately many situations when multiple locks have to be applied
66at a time. This is then the hierarchy of the mutexes that must be preserved
67everywhere in the code.
68
69As mutexes cannot be really ordered unanimously, below are two trees, with also
70some possible branches inside. The mutex marked with (T) is terminal, that is,
71no other locks shall be allowed in the section where this mutex is locked.
72
73### Mutex ordering information
74
75Note that the list isn't exactly complete, but it should contain all
76mutexes for which the locking order must be preserved.
77
78```
79
80 - CUDTSocket::m_ControlLock
81
82 - CUDT::m_ConnectionLock
83
84        - CRendezvousQueue::m_RIDVectorLock
85
86 - CUDTUnited::m_GlobControlLock
87
88 - CUDTGroup::m_GroupLock
89
90    - CUDT::m_RecvAckLock  || CEPoll::m_EPollLock(T)
91
92----------------
93 - CUDTUnited::m_GlobControlLock
94
95        - CUDTGroup::m_GroupLock  || CSndUList::m_ListLock(T)
96
97     - CUDT::m_ConnectionLock
98
99              - CRendezvousQueue::m_RIDVectorLock
100
101 - CUDT::m_SendLock
102
103    - CUDT::m_RecvLock
104
105       - CUDT::m_RecvBufferLock
106
107 - CUDT::m_RecvAckLock || CUDT::m_SendBlockLock
108------------------
109
110ANALYSIS ON: m_ConnectionLock
111
112
113-- CUDT::startConnect flow
114
115CUDTUnited::connectIn -- > [LOCKED s->m_ControlLock]
116    CUDT::open -- > [MAYBE_LOCKED m_ConnectionLock, if bind() not called]
117        CUDT::clearData --> [LOCKED m_StatsLock]
118    CUDTUnited::updateMux  -- > [LOCKED m_GlobControlLock]
119    {
120       [SCOPE UNLOCK s->m_ControlLock, if blocking mode]
121       CUDT::startConnect  -- > [LOCKED m_ConnectionLock]
122           CRcvQueue::registerConnector
123                CRendezvousQueue::insert --> [LOCKED CRendezvousQueue::m_RIDVectorLock]
124    }
125END.
126
127-- CUDT::groupConnect flow
128
129CUDT::groupConnect (no locks)
130   CUDT::setOpt [LOCKS m_ConnectionLock, m_SendLock, m_RecvLock]
131   { [LOCKS m_GlobControlLock]
132        CUDTGroup::add [LOCKS m_GroupLock]
133   }
134CUDT::connectIn --> continue with startConnect flow
135
136-- CUDTUnited::listen (API function)
137
138CUDTUnited::listen
139    CUDTUnited::locateSocket [LOCKS m_GlobControlLock]
140    {
141        [SCOPE LOCK s->m_ControlLock]
142        CUDT::setListenState -- > [LOCKED m_ConnectionLock]
143             CRcvQueue::setListener -- > [LOCKED m_LSLock]
144    }
145
146-- CUDT::processAsyncConnectRequest
147
148CRcvQueue::worker ->
149...
150CRcvQueue::worker_TryAsyncRend_OrStore
151     CUDT::processAsyncConnectResponse -- > [LOCKED m_ConnectionLock]
152         CUDT::processConnectResponse
153             CUDT::postConnect
154                 CUDT::interpretSrtHandshake ->
155                 [IF group extension found]
156                 CUDT::interpretGroup
157                 {
158                     [SCOPE LOCK m_GlobControlLock]
159                     [IF Responder]
160                     {
161                         CUDT::makeMePeerOf
162                             [LOCKS m_GroupLock]
163                                 CUDTGroup::syncWithSocket
164                             CUDTGroup::find --> [LOCKED m_GroupLock]
165                     }
166                     debugGroup -- > [LOCKED m_GroupLock]
167                 }
168
169
170-- CUDT::acceptAndRespond
171
172CRcvQueue::worker_ProcessConnectionRequest
173{
174     [SCOPE LOCK m_LSLock]
175     CUDT::processConnectRequest
176         CUDTUnited::newConnection
177             locateSocket -- > [LOCKED m_GlobControlLock]
178             locatePeer -- > [LOCKED m_GlobControlLock]
179             [IF failure, LOCK m_AcceptLock]
180             generateSocketID --> [LOCKED m_IDLock]
181             CUDT::open  -- > [LOCKED m_ConnectionLock]
182             CUDT::updateListenerMux  -- > [LOCKED m_GlobControlLock]
183             CUDT::acceptAndRespond --> [LOCKED m_ConnectionLock]
184                 CUDT::interpretSrtHandshake ->
185                 [IF group extension found]
186                 CUDT::interpretGroup
187                 {
188                     [SCOPE LOCK m_GlobControlLock]
189                     [IF Responder]
190                     {
191                         CUDT::makeMePeerOf
192                             [LOCKS m_GroupLock]
193                                 CUDTGroup::syncWithSocket
194                             CUDTGroup::find --> [LOCKED m_GroupLock]
195                     }
196                     debugGroup -- > [LOCKED m_GroupLock]
197                 }
198                 {
199                     [SCOPE LOCK m_GlobControlLock]
200                     CUDT::synchronizeWithGroup -- > [LOCKED m_GroupLock]
201                 }
202                 CRcvQueue::setNewEntry -- > [LOCKED CRcvQueue::m_IDLock]
203                 {
204                     [SCOPE LOCK m_GlobControlLock]
205                     {
206                         [SCOPE LOCK m_GroupLock]
207                     }
208                 }
209             {
210                 [SCOPE LOCK m_AcceptLock]
211                     CEPoll::update_events
212             }
213             [IF Rollback]
214             CUDT::closeInternal
215                 [LOCKING m_EPollLock]
216                 {
217                     [SCOPE LOCK m_ConnectionLock]
218                         [SCOPE LOCK m_SendLock]
219                             [SCOPE LOCK m_RecvLock]
220                                 [LOCKING m_RcvBufferLock]
221                 }
222             {
223                 [SCOPE LOCK m_GlobControlLock]
224                 CUDT::removeFromGroup --> [LOCKED m_GroupLock]
225             }
226
227         CEPoll::update_events
228}
229
230-- CUDT::bstats: TRT-LOCKED m_ConnectionLock
231
232-- CUDT::packData
233
234CSndQueue::worker
235 CSndUList::pop -- > [LOCKED m_ListLock]
236     CUDT::packData
237
238```
239