1# Low Level Information for the SRT project 2 3## Introduction 4 5This document contains information on various topics related to 6the SRT source code, including descriptions of some cross-source analysis that would 7not be obvious for a source code reviewer. It's not a complete documentation of 8anything, rather a collection of various kind of information retrieved during 9development and even reverse engineering. 10 11## Mutex locking 12 13This analysis is a result of detected lots of cascade mutex locking in the 14SRT code. A more detailed analysis would be required as to which mutex is 15going to protect what kind of data later. 16 17Here is the info collected so far: 18 19### Data structures 20 21The overall structure of the object database, involving sockets and groups 22is as follows (pseudo-language): 23 24``` 25CUDTUnited (singleton) { 26 CONTAINER<CUDTSocket> m_Sockets; 27 CONTAINER<CUDTSocket> m_ClosedSockets; 28 CONTAINER<CUDTGroup> m_Groups; 29 CONTAINER<CUDTGroup> m_ClosedGroups; 30} 31 32CUDTGroup { 33 type SocketData { CUDTSocket* ps; SRTSOCKET id; int state; ... } 34 CONTAINER<SocketData> m_Group; 35} 36``` 37 38Dead sockets (either closed manually or broken after connection) are 39moved first from `m_Sockets` to `m_ClosedSockets`. The GC thread will take 40care to delete them physically after making sure all inside facilities 41do not contain any remaining data of interest. 42 43Groups may only be manually closed, however a closed group is moved 44to `m_ClosedGroups`. The GC thread will take care to delete them, as long 45as their usage counter is 0. Every call to an API function (as well as 46TSBPD thread) increases the usage counter in the beginning and decreases 47upon exit. A group may be closed in one thread and still being used in 48another. The group will persist up to the time when the current API function 49using it exits and decreases the usage counter back to 0. 50 51Containers and contents guarded by mutex: 52 53`CUDTUnited::m_GlobControlLock` - guards all containers in CUDTUnited. 54 55`CUDTSocket::m_ControlLock` - guards internal operation performed on particular 56socket, with its existence assumed (this is because a socket will always exist 57until it's deleted while being in `m_ClosedSockets`, and when the socket is in 58`m_ClosedSockets` it will not be deleted until it's free from any operation, 59while the socket is assumed nonexistent for any newly called API function even 60if it exists physically, but is moved to `m_ClosedSockets`). 61 62`CUDTGroup::m_GroupLock` - guards the `m_Group` container inside a group that 63collects member sockets. 64 65There are unfortunately many situations when multiple locks have to be applied 66at a time. This is then the hierarchy of the mutexes that must be preserved 67everywhere in the code. 68 69As mutexes cannot be really ordered unanimously, below are two trees, with also 70some possible branches inside. The mutex marked with (T) is terminal, that is, 71no other locks shall be allowed in the section where this mutex is locked. 72 73### Mutex ordering information 74 75Note that the list isn't exactly complete, but it should contain all 76mutexes for which the locking order must be preserved. 77 78``` 79 80 - CUDTSocket::m_ControlLock 81 82 - CUDT::m_ConnectionLock 83 84 - CRendezvousQueue::m_RIDVectorLock 85 86 - CUDTUnited::m_GlobControlLock 87 88 - CUDTGroup::m_GroupLock 89 90 - CUDT::m_RecvAckLock || CEPoll::m_EPollLock(T) 91 92---------------- 93 - CUDTUnited::m_GlobControlLock 94 95 - CUDTGroup::m_GroupLock || CSndUList::m_ListLock(T) 96 97 - CUDT::m_ConnectionLock 98 99 - CRendezvousQueue::m_RIDVectorLock 100 101 - CUDT::m_SendLock 102 103 - CUDT::m_RecvLock 104 105 - CUDT::m_RecvBufferLock 106 107 - CUDT::m_RecvAckLock || CUDT::m_SendBlockLock 108------------------ 109 110ANALYSIS ON: m_ConnectionLock 111 112 113-- CUDT::startConnect flow 114 115CUDTUnited::connectIn -- > [LOCKED s->m_ControlLock] 116 CUDT::open -- > [MAYBE_LOCKED m_ConnectionLock, if bind() not called] 117 CUDT::clearData --> [LOCKED m_StatsLock] 118 CUDTUnited::updateMux -- > [LOCKED m_GlobControlLock] 119 { 120 [SCOPE UNLOCK s->m_ControlLock, if blocking mode] 121 CUDT::startConnect -- > [LOCKED m_ConnectionLock] 122 CRcvQueue::registerConnector 123 CRendezvousQueue::insert --> [LOCKED CRendezvousQueue::m_RIDVectorLock] 124 } 125END. 126 127-- CUDT::groupConnect flow 128 129CUDT::groupConnect (no locks) 130 CUDT::setOpt [LOCKS m_ConnectionLock, m_SendLock, m_RecvLock] 131 { [LOCKS m_GlobControlLock] 132 CUDTGroup::add [LOCKS m_GroupLock] 133 } 134CUDT::connectIn --> continue with startConnect flow 135 136-- CUDTUnited::listen (API function) 137 138CUDTUnited::listen 139 CUDTUnited::locateSocket [LOCKS m_GlobControlLock] 140 { 141 [SCOPE LOCK s->m_ControlLock] 142 CUDT::setListenState -- > [LOCKED m_ConnectionLock] 143 CRcvQueue::setListener -- > [LOCKED m_LSLock] 144 } 145 146-- CUDT::processAsyncConnectRequest 147 148CRcvQueue::worker -> 149... 150CRcvQueue::worker_TryAsyncRend_OrStore 151 CUDT::processAsyncConnectResponse -- > [LOCKED m_ConnectionLock] 152 CUDT::processConnectResponse 153 CUDT::postConnect 154 CUDT::interpretSrtHandshake -> 155 [IF group extension found] 156 CUDT::interpretGroup 157 { 158 [SCOPE LOCK m_GlobControlLock] 159 [IF Responder] 160 { 161 CUDT::makeMePeerOf 162 [LOCKS m_GroupLock] 163 CUDTGroup::syncWithSocket 164 CUDTGroup::find --> [LOCKED m_GroupLock] 165 } 166 debugGroup -- > [LOCKED m_GroupLock] 167 } 168 169 170-- CUDT::acceptAndRespond 171 172CRcvQueue::worker_ProcessConnectionRequest 173{ 174 [SCOPE LOCK m_LSLock] 175 CUDT::processConnectRequest 176 CUDTUnited::newConnection 177 locateSocket -- > [LOCKED m_GlobControlLock] 178 locatePeer -- > [LOCKED m_GlobControlLock] 179 [IF failure, LOCK m_AcceptLock] 180 generateSocketID --> [LOCKED m_IDLock] 181 CUDT::open -- > [LOCKED m_ConnectionLock] 182 CUDT::updateListenerMux -- > [LOCKED m_GlobControlLock] 183 CUDT::acceptAndRespond --> [LOCKED m_ConnectionLock] 184 CUDT::interpretSrtHandshake -> 185 [IF group extension found] 186 CUDT::interpretGroup 187 { 188 [SCOPE LOCK m_GlobControlLock] 189 [IF Responder] 190 { 191 CUDT::makeMePeerOf 192 [LOCKS m_GroupLock] 193 CUDTGroup::syncWithSocket 194 CUDTGroup::find --> [LOCKED m_GroupLock] 195 } 196 debugGroup -- > [LOCKED m_GroupLock] 197 } 198 { 199 [SCOPE LOCK m_GlobControlLock] 200 CUDT::synchronizeWithGroup -- > [LOCKED m_GroupLock] 201 } 202 CRcvQueue::setNewEntry -- > [LOCKED CRcvQueue::m_IDLock] 203 { 204 [SCOPE LOCK m_GlobControlLock] 205 { 206 [SCOPE LOCK m_GroupLock] 207 } 208 } 209 { 210 [SCOPE LOCK m_AcceptLock] 211 CEPoll::update_events 212 } 213 [IF Rollback] 214 CUDT::closeInternal 215 [LOCKING m_EPollLock] 216 { 217 [SCOPE LOCK m_ConnectionLock] 218 [SCOPE LOCK m_SendLock] 219 [SCOPE LOCK m_RecvLock] 220 [LOCKING m_RcvBufferLock] 221 } 222 { 223 [SCOPE LOCK m_GlobControlLock] 224 CUDT::removeFromGroup --> [LOCKED m_GroupLock] 225 } 226 227 CEPoll::update_events 228} 229 230-- CUDT::bstats: TRT-LOCKED m_ConnectionLock 231 232-- CUDT::packData 233 234CSndQueue::worker 235 CSndUList::pop -- > [LOCKED m_ListLock] 236 CUDT::packData 237 238``` 239