1.. SPDX-License-Identifier: GPL-2.0
2
3======================
4RxRPC Network Protocol
5======================
6
7The RxRPC protocol driver provides a reliable two-phase transport on top of UDP
8that can be used to perform RxRPC remote operations.  This is done over sockets
9of AF_RXRPC family, using sendmsg() and recvmsg() with control data to send and
10receive data, aborts and errors.
11
12Contents of this document:
13
14 (#) Overview.
15
16 (#) RxRPC protocol summary.
17
18 (#) AF_RXRPC driver model.
19
20 (#) Control messages.
21
22 (#) Socket options.
23
24 (#) Security.
25
26 (#) Example client usage.
27
28 (#) Example server usage.
29
30 (#) AF_RXRPC kernel interface.
31
32 (#) Configurable parameters.
33
34
35Overview
36========
37
38RxRPC is a two-layer protocol.  There is a session layer which provides
39reliable virtual connections using UDP over IPv4 (or IPv6) as the transport
40layer, but implements a real network protocol; and there's the presentation
41layer which renders structured data to binary blobs and back again using XDR
42(as does SunRPC)::
43
44		+-------------+
45		| Application |
46		+-------------+
47		|     XDR     |		Presentation
48		+-------------+
49		|    RxRPC    |		Session
50		+-------------+
51		|     UDP     |		Transport
52		+-------------+
53
54
55AF_RXRPC provides:
56
57 (1) Part of an RxRPC facility for both kernel and userspace applications by
58     making the session part of it a Linux network protocol (AF_RXRPC).
59
60 (2) A two-phase protocol.  The client transmits a blob (the request) and then
61     receives a blob (the reply), and the server receives the request and then
62     transmits the reply.
63
64 (3) Retention of the reusable bits of the transport system set up for one call
65     to speed up subsequent calls.
66
67 (4) A secure protocol, using the Linux kernel's key retention facility to
68     manage security on the client end.  The server end must of necessity be
69     more active in security negotiations.
70
71AF_RXRPC does not provide XDR marshalling/presentation facilities.  That is
72left to the application.  AF_RXRPC only deals in blobs.  Even the operation ID
73is just the first four bytes of the request blob, and as such is beyond the
74kernel's interest.
75
76
77Sockets of AF_RXRPC family are:
78
79 (1) created as type SOCK_DGRAM;
80
81 (2) provided with a protocol of the type of underlying transport they're going
82     to use - currently only PF_INET is supported.
83
84
85The Andrew File System (AFS) is an example of an application that uses this and
86that has both kernel (filesystem) and userspace (utility) components.
87
88
89RxRPC Protocol Summary
90======================
91
92An overview of the RxRPC protocol:
93
94 (#) RxRPC sits on top of another networking protocol (UDP is the only option
95     currently), and uses this to provide network transport.  UDP ports, for
96     example, provide transport endpoints.
97
98 (#) RxRPC supports multiple virtual "connections" from any given transport
99     endpoint, thus allowing the endpoints to be shared, even to the same
100     remote endpoint.
101
102 (#) Each connection goes to a particular "service".  A connection may not go
103     to multiple services.  A service may be considered the RxRPC equivalent of
104     a port number.  AF_RXRPC permits multiple services to share an endpoint.
105
106 (#) Client-originating packets are marked, thus a transport endpoint can be
107     shared between client and server connections (connections have a
108     direction).
109
110 (#) Up to a billion connections may be supported concurrently between one
111     local transport endpoint and one service on one remote endpoint.  An RxRPC
112     connection is described by seven numbers::
113
114	Local address	}
115	Local port	} Transport (UDP) address
116	Remote address	}
117	Remote port	}
118	Direction
119	Connection ID
120	Service ID
121
122 (#) Each RxRPC operation is a "call".  A connection may make up to four
123     billion calls, but only up to four calls may be in progress on a
124     connection at any one time.
125
126 (#) Calls are two-phase and asymmetric: the client sends its request data,
127     which the service receives; then the service transmits the reply data
128     which the client receives.
129
130 (#) The data blobs are of indefinite size, the end of a phase is marked with a
131     flag in the packet.  The number of packets of data making up one blob may
132     not exceed 4 billion, however, as this would cause the sequence number to
133     wrap.
134
135 (#) The first four bytes of the request data are the service operation ID.
136
137 (#) Security is negotiated on a per-connection basis.  The connection is
138     initiated by the first data packet on it arriving.  If security is
139     requested, the server then issues a "challenge" and then the client
140     replies with a "response".  If the response is successful, the security is
141     set for the lifetime of that connection, and all subsequent calls made
142     upon it use that same security.  In the event that the server lets a
143     connection lapse before the client, the security will be renegotiated if
144     the client uses the connection again.
145
146 (#) Calls use ACK packets to handle reliability.  Data packets are also
147     explicitly sequenced per call.
148
149 (#) There are two types of positive acknowledgment: hard-ACKs and soft-ACKs.
150     A hard-ACK indicates to the far side that all the data received to a point
151     has been received and processed; a soft-ACK indicates that the data has
152     been received but may yet be discarded and re-requested.  The sender may
153     not discard any transmittable packets until they've been hard-ACK'd.
154
155 (#) Reception of a reply data packet implicitly hard-ACK's all the data
156     packets that make up the request.
157
158 (#) An call is complete when the request has been sent, the reply has been
159     received and the final hard-ACK on the last packet of the reply has
160     reached the server.
161
162 (#) An call may be aborted by either end at any time up to its completion.
163
164
165AF_RXRPC Driver Model
166=====================
167
168About the AF_RXRPC driver:
169
170 (#) The AF_RXRPC protocol transparently uses internal sockets of the transport
171     protocol to represent transport endpoints.
172
173 (#) AF_RXRPC sockets map onto RxRPC connection bundles.  Actual RxRPC
174     connections are handled transparently.  One client socket may be used to
175     make multiple simultaneous calls to the same service.  One server socket
176     may handle calls from many clients.
177
178 (#) Additional parallel client connections will be initiated to support extra
179     concurrent calls, up to a tunable limit.
180
181 (#) Each connection is retained for a certain amount of time [tunable] after
182     the last call currently using it has completed in case a new call is made
183     that could reuse it.
184
185 (#) Each internal UDP socket is retained [tunable] for a certain amount of
186     time [tunable] after the last connection using it discarded, in case a new
187     connection is made that could use it.
188
189 (#) A client-side connection is only shared between calls if they have
190     the same key struct describing their security (and assuming the calls
191     would otherwise share the connection).  Non-secured calls would also be
192     able to share connections with each other.
193
194 (#) A server-side connection is shared if the client says it is.
195
196 (#) ACK'ing is handled by the protocol driver automatically, including ping
197     replying.
198
199 (#) SO_KEEPALIVE automatically pings the other side to keep the connection
200     alive [TODO].
201
202 (#) If an ICMP error is received, all calls affected by that error will be
203     aborted with an appropriate network error passed through recvmsg().
204
205
206Interaction with the user of the RxRPC socket:
207
208 (#) A socket is made into a server socket by binding an address with a
209     non-zero service ID.
210
211 (#) In the client, sending a request is achieved with one or more sendmsgs,
212     followed by the reply being received with one or more recvmsgs.
213
214 (#) The first sendmsg for a request to be sent from a client contains a tag to
215     be used in all other sendmsgs or recvmsgs associated with that call.  The
216     tag is carried in the control data.
217
218 (#) connect() is used to supply a default destination address for a client
219     socket.  This may be overridden by supplying an alternate address to the
220     first sendmsg() of a call (struct msghdr::msg_name).
221
222 (#) If connect() is called on an unbound client, a random local port will
223     bound before the operation takes place.
224
225 (#) A server socket may also be used to make client calls.  To do this, the
226     first sendmsg() of the call must specify the target address.  The server's
227     transport endpoint is used to send the packets.
228
229 (#) Once the application has received the last message associated with a call,
230     the tag is guaranteed not to be seen again, and so it can be used to pin
231     client resources.  A new call can then be initiated with the same tag
232     without fear of interference.
233
234 (#) In the server, a request is received with one or more recvmsgs, then the
235     the reply is transmitted with one or more sendmsgs, and then the final ACK
236     is received with a last recvmsg.
237
238 (#) When sending data for a call, sendmsg is given MSG_MORE if there's more
239     data to come on that call.
240
241 (#) When receiving data for a call, recvmsg flags MSG_MORE if there's more
242     data to come for that call.
243
244 (#) When receiving data or messages for a call, MSG_EOR is flagged by recvmsg
245     to indicate the terminal message for that call.
246
247 (#) A call may be aborted by adding an abort control message to the control
248     data.  Issuing an abort terminates the kernel's use of that call's tag.
249     Any messages waiting in the receive queue for that call will be discarded.
250
251 (#) Aborts, busy notifications and challenge packets are delivered by recvmsg,
252     and control data messages will be set to indicate the context.  Receiving
253     an abort or a busy message terminates the kernel's use of that call's tag.
254
255 (#) The control data part of the msghdr struct is used for a number of things:
256
257     (#) The tag of the intended or affected call.
258
259     (#) Sending or receiving errors, aborts and busy notifications.
260
261     (#) Notifications of incoming calls.
262
263     (#) Sending debug requests and receiving debug replies [TODO].
264
265 (#) When the kernel has received and set up an incoming call, it sends a
266     message to server application to let it know there's a new call awaiting
267     its acceptance [recvmsg reports a special control message].  The server
268     application then uses sendmsg to assign a tag to the new call.  Once that
269     is done, the first part of the request data will be delivered by recvmsg.
270
271 (#) The server application has to provide the server socket with a keyring of
272     secret keys corresponding to the security types it permits.  When a secure
273     connection is being set up, the kernel looks up the appropriate secret key
274     in the keyring and then sends a challenge packet to the client and
275     receives a response packet.  The kernel then checks the authorisation of
276     the packet and either aborts the connection or sets up the security.
277
278 (#) The name of the key a client will use to secure its communications is
279     nominated by a socket option.
280
281
282Notes on sendmsg:
283
284 (#) MSG_WAITALL can be set to tell sendmsg to ignore signals if the peer is
285     making progress at accepting packets within a reasonable time such that we
286     manage to queue up all the data for transmission.  This requires the
287     client to accept at least one packet per 2*RTT time period.
288
289     If this isn't set, sendmsg() will return immediately, either returning
290     EINTR/ERESTARTSYS if nothing was consumed or returning the amount of data
291     consumed.
292
293
294Notes on recvmsg:
295
296 (#) If there's a sequence of data messages belonging to a particular call on
297     the receive queue, then recvmsg will keep working through them until:
298
299     (a) it meets the end of that call's received data,
300
301     (b) it meets a non-data message,
302
303     (c) it meets a message belonging to a different call, or
304
305     (d) it fills the user buffer.
306
307     If recvmsg is called in blocking mode, it will keep sleeping, awaiting the
308     reception of further data, until one of the above four conditions is met.
309
310 (2) MSG_PEEK operates similarly, but will return immediately if it has put any
311     data in the buffer rather than sleeping until it can fill the buffer.
312
313 (3) If a data message is only partially consumed in filling a user buffer,
314     then the remainder of that message will be left on the front of the queue
315     for the next taker.  MSG_TRUNC will never be flagged.
316
317 (4) If there is more data to be had on a call (it hasn't copied the last byte
318     of the last data message in that phase yet), then MSG_MORE will be
319     flagged.
320
321
322Control Messages
323================
324
325AF_RXRPC makes use of control messages in sendmsg() and recvmsg() to multiplex
326calls, to invoke certain actions and to report certain conditions.  These are:
327
328	=======================	=== ===========	===============================
329	MESSAGE ID		SRT DATA	MEANING
330	=======================	=== ===========	===============================
331	RXRPC_USER_CALL_ID	sr- User ID	App's call specifier
332	RXRPC_ABORT		srt Abort code	Abort code to issue/received
333	RXRPC_ACK		-rt n/a		Final ACK received
334	RXRPC_NET_ERROR		-rt error num	Network error on call
335	RXRPC_BUSY		-rt n/a		Call rejected (server busy)
336	RXRPC_LOCAL_ERROR	-rt error num	Local error encountered
337	RXRPC_NEW_CALL		-r- n/a		New call received
338	RXRPC_ACCEPT		s-- n/a		Accept new call
339	RXRPC_EXCLUSIVE_CALL	s-- n/a		Make an exclusive client call
340	RXRPC_UPGRADE_SERVICE	s-- n/a		Client call can be upgraded
341	RXRPC_TX_LENGTH		s-- data len	Total length of Tx data
342	=======================	=== ===========	===============================
343
344	(SRT = usable in Sendmsg / delivered by Recvmsg / Terminal message)
345
346 (#) RXRPC_USER_CALL_ID
347
348     This is used to indicate the application's call ID.  It's an unsigned long
349     that the app specifies in the client by attaching it to the first data
350     message or in the server by passing it in association with an RXRPC_ACCEPT
351     message.  recvmsg() passes it in conjunction with all messages except
352     those of the RXRPC_NEW_CALL message.
353
354 (#) RXRPC_ABORT
355
356     This is can be used by an application to abort a call by passing it to
357     sendmsg, or it can be delivered by recvmsg to indicate a remote abort was
358     received.  Either way, it must be associated with an RXRPC_USER_CALL_ID to
359     specify the call affected.  If an abort is being sent, then error EBADSLT
360     will be returned if there is no call with that user ID.
361
362 (#) RXRPC_ACK
363
364     This is delivered to a server application to indicate that the final ACK
365     of a call was received from the client.  It will be associated with an
366     RXRPC_USER_CALL_ID to indicate the call that's now complete.
367
368 (#) RXRPC_NET_ERROR
369
370     This is delivered to an application to indicate that an ICMP error message
371     was encountered in the process of trying to talk to the peer.  An
372     errno-class integer value will be included in the control message data
373     indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call
374     affected.
375
376 (#) RXRPC_BUSY
377
378     This is delivered to a client application to indicate that a call was
379     rejected by the server due to the server being busy.  It will be
380     associated with an RXRPC_USER_CALL_ID to indicate the rejected call.
381
382 (#) RXRPC_LOCAL_ERROR
383
384     This is delivered to an application to indicate that a local error was
385     encountered and that a call has been aborted because of it.  An
386     errno-class integer value will be included in the control message data
387     indicating the problem, and an RXRPC_USER_CALL_ID will indicate the call
388     affected.
389
390 (#) RXRPC_NEW_CALL
391
392     This is delivered to indicate to a server application that a new call has
393     arrived and is awaiting acceptance.  No user ID is associated with this,
394     as a user ID must subsequently be assigned by doing an RXRPC_ACCEPT.
395
396 (#) RXRPC_ACCEPT
397
398     This is used by a server application to attempt to accept a call and
399     assign it a user ID.  It should be associated with an RXRPC_USER_CALL_ID
400     to indicate the user ID to be assigned.  If there is no call to be
401     accepted (it may have timed out, been aborted, etc.), then sendmsg will
402     return error ENODATA.  If the user ID is already in use by another call,
403     then error EBADSLT will be returned.
404
405 (#) RXRPC_EXCLUSIVE_CALL
406
407     This is used to indicate that a client call should be made on a one-off
408     connection.  The connection is discarded once the call has terminated.
409
410 (#) RXRPC_UPGRADE_SERVICE
411
412     This is used to make a client call to probe if the specified service ID
413     may be upgraded by the server.  The caller must check msg_name returned to
414     recvmsg() for the service ID actually in use.  The operation probed must
415     be one that takes the same arguments in both services.
416
417     Once this has been used to establish the upgrade capability (or lack
418     thereof) of the server, the service ID returned should be used for all
419     future communication to that server and RXRPC_UPGRADE_SERVICE should no
420     longer be set.
421
422 (#) RXRPC_TX_LENGTH
423
424     This is used to inform the kernel of the total amount of data that is
425     going to be transmitted by a call (whether in a client request or a
426     service response).  If given, it allows the kernel to encrypt from the
427     userspace buffer directly to the packet buffers, rather than copying into
428     the buffer and then encrypting in place.  This may only be given with the
429     first sendmsg() providing data for a call.  EMSGSIZE will be generated if
430     the amount of data actually given is different.
431
432     This takes a parameter of __s64 type that indicates how much will be
433     transmitted.  This may not be less than zero.
434
435The symbol RXRPC__SUPPORTED is defined as one more than the highest control
436message type supported.  At run time this can be queried by means of the
437RXRPC_SUPPORTED_CMSG socket option (see below).
438
439
440==============
441SOCKET OPTIONS
442==============
443
444AF_RXRPC sockets support a few socket options at the SOL_RXRPC level:
445
446 (#) RXRPC_SECURITY_KEY
447
448     This is used to specify the description of the key to be used.  The key is
449     extracted from the calling process's keyrings with request_key() and
450     should be of "rxrpc" type.
451
452     The optval pointer points to the description string, and optlen indicates
453     how long the string is, without the NUL terminator.
454
455 (#) RXRPC_SECURITY_KEYRING
456
457     Similar to above but specifies a keyring of server secret keys to use (key
458     type "keyring").  See the "Security" section.
459
460 (#) RXRPC_EXCLUSIVE_CONNECTION
461
462     This is used to request that new connections should be used for each call
463     made subsequently on this socket.  optval should be NULL and optlen 0.
464
465 (#) RXRPC_MIN_SECURITY_LEVEL
466
467     This is used to specify the minimum security level required for calls on
468     this socket.  optval must point to an int containing one of the following
469     values:
470
471     (a) RXRPC_SECURITY_PLAIN
472
473	 Encrypted checksum only.
474
475     (b) RXRPC_SECURITY_AUTH
476
477	 Encrypted checksum plus packet padded and first eight bytes of packet
478	 encrypted - which includes the actual packet length.
479
480     (c) RXRPC_SECURITY_ENCRYPT
481
482	 Encrypted checksum plus entire packet padded and encrypted, including
483	 actual packet length.
484
485 (#) RXRPC_UPGRADEABLE_SERVICE
486
487     This is used to indicate that a service socket with two bindings may
488     upgrade one bound service to the other if requested by the client.  optval
489     must point to an array of two unsigned short ints.  The first is the
490     service ID to upgrade from and the second the service ID to upgrade to.
491
492 (#) RXRPC_SUPPORTED_CMSG
493
494     This is a read-only option that writes an int into the buffer indicating
495     the highest control message type supported.
496
497
498========
499SECURITY
500========
501
502Currently, only the kerberos 4 equivalent protocol has been implemented
503(security index 2 - rxkad).  This requires the rxkad module to be loaded and,
504on the client, tickets of the appropriate type to be obtained from the AFS
505kaserver or the kerberos server and installed as "rxrpc" type keys.  This is
506normally done using the klog program.  An example simple klog program can be
507found at:
508
509	http://people.redhat.com/~dhowells/rxrpc/klog.c
510
511The payload provided to add_key() on the client should be of the following
512form::
513
514	struct rxrpc_key_sec2_v1 {
515		uint16_t	security_index;	/* 2 */
516		uint16_t	ticket_length;	/* length of ticket[] */
517		uint32_t	expiry;		/* time at which expires */
518		uint8_t		kvno;		/* key version number */
519		uint8_t		__pad[3];
520		uint8_t		session_key[8];	/* DES session key */
521		uint8_t		ticket[0];	/* the encrypted ticket */
522	};
523
524Where the ticket blob is just appended to the above structure.
525
526
527For the server, keys of type "rxrpc_s" must be made available to the server.
528They have a description of "<serviceID>:<securityIndex>" (eg: "52:2" for an
529rxkad key for the AFS VL service).  When such a key is created, it should be
530given the server's secret key as the instantiation data (see the example
531below).
532
533	add_key("rxrpc_s", "52:2", secret_key, 8, keyring);
534
535A keyring is passed to the server socket by naming it in a sockopt.  The server
536socket then looks the server secret keys up in this keyring when secure
537incoming connections are made.  This can be seen in an example program that can
538be found at:
539
540	http://people.redhat.com/~dhowells/rxrpc/listen.c
541
542
543====================
544EXAMPLE CLIENT USAGE
545====================
546
547A client would issue an operation by:
548
549 (1) An RxRPC socket is set up by::
550
551	client = socket(AF_RXRPC, SOCK_DGRAM, PF_INET);
552
553     Where the third parameter indicates the protocol family of the transport
554     socket used - usually IPv4 but it can also be IPv6 [TODO].
555
556 (2) A local address can optionally be bound::
557
558	struct sockaddr_rxrpc srx = {
559		.srx_family	= AF_RXRPC,
560		.srx_service	= 0,  /* we're a client */
561		.transport_type	= SOCK_DGRAM,	/* type of transport socket */
562		.transport.sin_family	= AF_INET,
563		.transport.sin_port	= htons(7000), /* AFS callback */
564		.transport.sin_address	= 0,  /* all local interfaces */
565	};
566	bind(client, &srx, sizeof(srx));
567
568     This specifies the local UDP port to be used.  If not given, a random
569     non-privileged port will be used.  A UDP port may be shared between
570     several unrelated RxRPC sockets.  Security is handled on a basis of
571     per-RxRPC virtual connection.
572
573 (3) The security is set::
574
575	const char *key = "AFS:cambridge.redhat.com";
576	setsockopt(client, SOL_RXRPC, RXRPC_SECURITY_KEY, key, strlen(key));
577
578     This issues a request_key() to get the key representing the security
579     context.  The minimum security level can be set::
580
581	unsigned int sec = RXRPC_SECURITY_ENCRYPT;
582	setsockopt(client, SOL_RXRPC, RXRPC_MIN_SECURITY_LEVEL,
583		   &sec, sizeof(sec));
584
585 (4) The server to be contacted can then be specified (alternatively this can
586     be done through sendmsg)::
587
588	struct sockaddr_rxrpc srx = {
589		.srx_family	= AF_RXRPC,
590		.srx_service	= VL_SERVICE_ID,
591		.transport_type	= SOCK_DGRAM,	/* type of transport socket */
592		.transport.sin_family	= AF_INET,
593		.transport.sin_port	= htons(7005), /* AFS volume manager */
594		.transport.sin_address	= ...,
595	};
596	connect(client, &srx, sizeof(srx));
597
598 (5) The request data should then be posted to the server socket using a series
599     of sendmsg() calls, each with the following control message attached:
600
601	==================	===================================
602	RXRPC_USER_CALL_ID	specifies the user ID for this call
603	==================	===================================
604
605     MSG_MORE should be set in msghdr::msg_flags on all but the last part of
606     the request.  Multiple requests may be made simultaneously.
607
608     An RXRPC_TX_LENGTH control message can also be specified on the first
609     sendmsg() call.
610
611     If a call is intended to go to a destination other than the default
612     specified through connect(), then msghdr::msg_name should be set on the
613     first request message of that call.
614
615 (6) The reply data will then be posted to the server socket for recvmsg() to
616     pick up.  MSG_MORE will be flagged by recvmsg() if there's more reply data
617     for a particular call to be read.  MSG_EOR will be set on the terminal
618     read for a call.
619
620     All data will be delivered with the following control message attached:
621
622	RXRPC_USER_CALL_ID	- specifies the user ID for this call
623
624     If an abort or error occurred, this will be returned in the control data
625     buffer instead, and MSG_EOR will be flagged to indicate the end of that
626     call.
627
628A client may ask for a service ID it knows and ask that this be upgraded to a
629better service if one is available by supplying RXRPC_UPGRADE_SERVICE on the
630first sendmsg() of a call.  The client should then check srx_service in the
631msg_name filled in by recvmsg() when collecting the result.  srx_service will
632hold the same value as given to sendmsg() if the upgrade request was ignored by
633the service - otherwise it will be altered to indicate the service ID the
634server upgraded to.  Note that the upgraded service ID is chosen by the server.
635The caller has to wait until it sees the service ID in the reply before sending
636any more calls (further calls to the same destination will be blocked until the
637probe is concluded).
638
639
640Example Server Usage
641====================
642
643A server would be set up to accept operations in the following manner:
644
645 (1) An RxRPC socket is created by::
646
647	server = socket(AF_RXRPC, SOCK_DGRAM, PF_INET);
648
649     Where the third parameter indicates the address type of the transport
650     socket used - usually IPv4.
651
652 (2) Security is set up if desired by giving the socket a keyring with server
653     secret keys in it::
654
655	keyring = add_key("keyring", "AFSkeys", NULL, 0,
656			  KEY_SPEC_PROCESS_KEYRING);
657
658	const char secret_key[8] = {
659		0xa7, 0x83, 0x8a, 0xcb, 0xc7, 0x83, 0xec, 0x94 };
660	add_key("rxrpc_s", "52:2", secret_key, 8, keyring);
661
662	setsockopt(server, SOL_RXRPC, RXRPC_SECURITY_KEYRING, "AFSkeys", 7);
663
664     The keyring can be manipulated after it has been given to the socket. This
665     permits the server to add more keys, replace keys, etc. while it is live.
666
667 (3) A local address must then be bound::
668
669	struct sockaddr_rxrpc srx = {
670		.srx_family	= AF_RXRPC,
671		.srx_service	= VL_SERVICE_ID, /* RxRPC service ID */
672		.transport_type	= SOCK_DGRAM,	/* type of transport socket */
673		.transport.sin_family	= AF_INET,
674		.transport.sin_port	= htons(7000), /* AFS callback */
675		.transport.sin_address	= 0,  /* all local interfaces */
676	};
677	bind(server, &srx, sizeof(srx));
678
679     More than one service ID may be bound to a socket, provided the transport
680     parameters are the same.  The limit is currently two.  To do this, bind()
681     should be called twice.
682
683 (4) If service upgrading is required, first two service IDs must have been
684     bound and then the following option must be set::
685
686	unsigned short service_ids[2] = { from_ID, to_ID };
687	setsockopt(server, SOL_RXRPC, RXRPC_UPGRADEABLE_SERVICE,
688		   service_ids, sizeof(service_ids));
689
690     This will automatically upgrade connections on service from_ID to service
691     to_ID if they request it.  This will be reflected in msg_name obtained
692     through recvmsg() when the request data is delivered to userspace.
693
694 (5) The server is then set to listen out for incoming calls::
695
696	listen(server, 100);
697
698 (6) The kernel notifies the server of pending incoming connections by sending
699     it a message for each.  This is received with recvmsg() on the server
700     socket.  It has no data, and has a single dataless control message
701     attached::
702
703	RXRPC_NEW_CALL
704
705     The address that can be passed back by recvmsg() at this point should be
706     ignored since the call for which the message was posted may have gone by
707     the time it is accepted - in which case the first call still on the queue
708     will be accepted.
709
710 (7) The server then accepts the new call by issuing a sendmsg() with two
711     pieces of control data and no actual data:
712
713	==================	==============================
714	RXRPC_ACCEPT		indicate connection acceptance
715	RXRPC_USER_CALL_ID	specify user ID for this call
716	==================	==============================
717
718 (8) The first request data packet will then be posted to the server socket for
719     recvmsg() to pick up.  At that point, the RxRPC address for the call can
720     be read from the address fields in the msghdr struct.
721
722     Subsequent request data will be posted to the server socket for recvmsg()
723     to collect as it arrives.  All but the last piece of the request data will
724     be delivered with MSG_MORE flagged.
725
726     All data will be delivered with the following control message attached:
727
728
729	==================	===================================
730	RXRPC_USER_CALL_ID	specifies the user ID for this call
731	==================	===================================
732
733 (9) The reply data should then be posted to the server socket using a series
734     of sendmsg() calls, each with the following control messages attached:
735
736	==================	===================================
737	RXRPC_USER_CALL_ID	specifies the user ID for this call
738	==================	===================================
739
740     MSG_MORE should be set in msghdr::msg_flags on all but the last message
741     for a particular call.
742
743(10) The final ACK from the client will be posted for retrieval by recvmsg()
744     when it is received.  It will take the form of a dataless message with two
745     control messages attached:
746
747	==================	===================================
748	RXRPC_USER_CALL_ID	specifies the user ID for this call
749	RXRPC_ACK		indicates final ACK (no data)
750	==================	===================================
751
752     MSG_EOR will be flagged to indicate that this is the final message for
753     this call.
754
755(11) Up to the point the final packet of reply data is sent, the call can be
756     aborted by calling sendmsg() with a dataless message with the following
757     control messages attached:
758
759	==================	===================================
760	RXRPC_USER_CALL_ID	specifies the user ID for this call
761	RXRPC_ABORT		indicates abort code (4 byte data)
762	==================	===================================
763
764     Any packets waiting in the socket's receive queue will be discarded if
765     this is issued.
766
767Note that all the communications for a particular service take place through
768the one server socket, using control messages on sendmsg() and recvmsg() to
769determine the call affected.
770
771
772AF_RXRPC Kernel Interface
773=========================
774
775The AF_RXRPC module also provides an interface for use by in-kernel utilities
776such as the AFS filesystem.  This permits such a utility to:
777
778 (1) Use different keys directly on individual client calls on one socket
779     rather than having to open a whole slew of sockets, one for each key it
780     might want to use.
781
782 (2) Avoid having RxRPC call request_key() at the point of issue of a call or
783     opening of a socket.  Instead the utility is responsible for requesting a
784     key at the appropriate point.  AFS, for instance, would do this during VFS
785     operations such as open() or unlink().  The key is then handed through
786     when the call is initiated.
787
788 (3) Request the use of something other than GFP_KERNEL to allocate memory.
789
790 (4) Avoid the overhead of using the recvmsg() call.  RxRPC messages can be
791     intercepted before they get put into the socket Rx queue and the socket
792     buffers manipulated directly.
793
794To use the RxRPC facility, a kernel utility must still open an AF_RXRPC socket,
795bind an address as appropriate and listen if it's to be a server socket, but
796then it passes this to the kernel interface functions.
797
798The kernel interface functions are as follows:
799
800 (#) Begin a new client call::
801
802	struct rxrpc_call *
803	rxrpc_kernel_begin_call(struct socket *sock,
804				struct sockaddr_rxrpc *srx,
805				struct key *key,
806				unsigned long user_call_ID,
807				s64 tx_total_len,
808				gfp_t gfp,
809				rxrpc_notify_rx_t notify_rx,
810				bool upgrade,
811				bool intr,
812				unsigned int debug_id);
813
814     This allocates the infrastructure to make a new RxRPC call and assigns
815     call and connection numbers.  The call will be made on the UDP port that
816     the socket is bound to.  The call will go to the destination address of a
817     connected client socket unless an alternative is supplied (srx is
818     non-NULL).
819
820     If a key is supplied then this will be used to secure the call instead of
821     the key bound to the socket with the RXRPC_SECURITY_KEY sockopt.  Calls
822     secured in this way will still share connections if at all possible.
823
824     The user_call_ID is equivalent to that supplied to sendmsg() in the
825     control data buffer.  It is entirely feasible to use this to point to a
826     kernel data structure.
827
828     tx_total_len is the amount of data the caller is intending to transmit
829     with this call (or -1 if unknown at this point).  Setting the data size
830     allows the kernel to encrypt directly to the packet buffers, thereby
831     saving a copy.  The value may not be less than -1.
832
833     notify_rx is a pointer to a function to be called when events such as
834     incoming data packets or remote aborts happen.
835
836     upgrade should be set to true if a client operation should request that
837     the server upgrade the service to a better one.  The resultant service ID
838     is returned by rxrpc_kernel_recv_data().
839
840     intr should be set to true if the call should be interruptible.  If this
841     is not set, this function may not return until a channel has been
842     allocated; if it is set, the function may return -ERESTARTSYS.
843
844     debug_id is the call debugging ID to be used for tracing.  This can be
845     obtained by atomically incrementing rxrpc_debug_id.
846
847     If this function is successful, an opaque reference to the RxRPC call is
848     returned.  The caller now holds a reference on this and it must be
849     properly ended.
850
851 (#) End a client call::
852
853	void rxrpc_kernel_end_call(struct socket *sock,
854				   struct rxrpc_call *call);
855
856     This is used to end a previously begun call.  The user_call_ID is expunged
857     from AF_RXRPC's knowledge and will not be seen again in association with
858     the specified call.
859
860 (#) Send data through a call::
861
862	typedef void (*rxrpc_notify_end_tx_t)(struct sock *sk,
863					      unsigned long user_call_ID,
864					      struct sk_buff *skb);
865
866	int rxrpc_kernel_send_data(struct socket *sock,
867				   struct rxrpc_call *call,
868				   struct msghdr *msg,
869				   size_t len,
870				   rxrpc_notify_end_tx_t notify_end_rx);
871
872     This is used to supply either the request part of a client call or the
873     reply part of a server call.  msg.msg_iovlen and msg.msg_iov specify the
874     data buffers to be used.  msg_iov may not be NULL and must point
875     exclusively to in-kernel virtual addresses.  msg.msg_flags may be given
876     MSG_MORE if there will be subsequent data sends for this call.
877
878     The msg must not specify a destination address, control data or any flags
879     other than MSG_MORE.  len is the total amount of data to transmit.
880
881     notify_end_rx can be NULL or it can be used to specify a function to be
882     called when the call changes state to end the Tx phase.  This function is
883     called with the call-state spinlock held to prevent any reply or final ACK
884     from being delivered first.
885
886 (#) Receive data from a call::
887
888	int rxrpc_kernel_recv_data(struct socket *sock,
889				   struct rxrpc_call *call,
890				   void *buf,
891				   size_t size,
892				   size_t *_offset,
893				   bool want_more,
894				   u32 *_abort,
895				   u16 *_service)
896
897      This is used to receive data from either the reply part of a client call
898      or the request part of a service call.  buf and size specify how much
899      data is desired and where to store it.  *_offset is added on to buf and
900      subtracted from size internally; the amount copied into the buffer is
901      added to *_offset before returning.
902
903      want_more should be true if further data will be required after this is
904      satisfied and false if this is the last item of the receive phase.
905
906      There are three normal returns: 0 if the buffer was filled and want_more
907      was true; 1 if the buffer was filled, the last DATA packet has been
908      emptied and want_more was false; and -EAGAIN if the function needs to be
909      called again.
910
911      If the last DATA packet is processed but the buffer contains less than
912      the amount requested, EBADMSG is returned.  If want_more wasn't set, but
913      more data was available, EMSGSIZE is returned.
914
915      If a remote ABORT is detected, the abort code received will be stored in
916      ``*_abort`` and ECONNABORTED will be returned.
917
918      The service ID that the call ended up with is returned into *_service.
919      This can be used to see if a call got a service upgrade.
920
921 (#) Abort a call??
922
923     ::
924
925	void rxrpc_kernel_abort_call(struct socket *sock,
926				     struct rxrpc_call *call,
927				     u32 abort_code);
928
929     This is used to abort a call if it's still in an abortable state.  The
930     abort code specified will be placed in the ABORT message sent.
931
932 (#) Intercept received RxRPC messages::
933
934	typedef void (*rxrpc_interceptor_t)(struct sock *sk,
935					    unsigned long user_call_ID,
936					    struct sk_buff *skb);
937
938	void
939	rxrpc_kernel_intercept_rx_messages(struct socket *sock,
940					   rxrpc_interceptor_t interceptor);
941
942     This installs an interceptor function on the specified AF_RXRPC socket.
943     All messages that would otherwise wind up in the socket's Rx queue are
944     then diverted to this function.  Note that care must be taken to process
945     the messages in the right order to maintain DATA message sequentiality.
946
947     The interceptor function itself is provided with the address of the socket
948     and handling the incoming message, the ID assigned by the kernel utility
949     to the call and the socket buffer containing the message.
950
951     The skb->mark field indicates the type of message:
952
953	===============================	=======================================
954	Mark				Meaning
955	===============================	=======================================
956	RXRPC_SKB_MARK_DATA		Data message
957	RXRPC_SKB_MARK_FINAL_ACK	Final ACK received for an incoming call
958	RXRPC_SKB_MARK_BUSY		Client call rejected as server busy
959	RXRPC_SKB_MARK_REMOTE_ABORT	Call aborted by peer
960	RXRPC_SKB_MARK_NET_ERROR	Network error detected
961	RXRPC_SKB_MARK_LOCAL_ERROR	Local error encountered
962	RXRPC_SKB_MARK_NEW_CALL		New incoming call awaiting acceptance
963	===============================	=======================================
964
965     The remote abort message can be probed with rxrpc_kernel_get_abort_code().
966     The two error messages can be probed with rxrpc_kernel_get_error_number().
967     A new call can be accepted with rxrpc_kernel_accept_call().
968
969     Data messages can have their contents extracted with the usual bunch of
970     socket buffer manipulation functions.  A data message can be determined to
971     be the last one in a sequence with rxrpc_kernel_is_data_last().  When a
972     data message has been used up, rxrpc_kernel_data_consumed() should be
973     called on it.
974
975     Messages should be handled to rxrpc_kernel_free_skb() to dispose of.  It
976     is possible to get extra refs on all types of message for later freeing,
977     but this may pin the state of a call until the message is finally freed.
978
979 (#) Accept an incoming call::
980
981	struct rxrpc_call *
982	rxrpc_kernel_accept_call(struct socket *sock,
983				 unsigned long user_call_ID);
984
985     This is used to accept an incoming call and to assign it a call ID.  This
986     function is similar to rxrpc_kernel_begin_call() and calls accepted must
987     be ended in the same way.
988
989     If this function is successful, an opaque reference to the RxRPC call is
990     returned.  The caller now holds a reference on this and it must be
991     properly ended.
992
993 (#) Reject an incoming call::
994
995	int rxrpc_kernel_reject_call(struct socket *sock);
996
997     This is used to reject the first incoming call on the socket's queue with
998     a BUSY message.  -ENODATA is returned if there were no incoming calls.
999     Other errors may be returned if the call had been aborted (-ECONNABORTED)
1000     or had timed out (-ETIME).
1001
1002 (#) Allocate a null key for doing anonymous security::
1003
1004	struct key *rxrpc_get_null_key(const char *keyname);
1005
1006     This is used to allocate a null RxRPC key that can be used to indicate
1007     anonymous security for a particular domain.
1008
1009 (#) Get the peer address of a call::
1010
1011	void rxrpc_kernel_get_peer(struct socket *sock, struct rxrpc_call *call,
1012				   struct sockaddr_rxrpc *_srx);
1013
1014     This is used to find the remote peer address of a call.
1015
1016 (#) Set the total transmit data size on a call::
1017
1018	void rxrpc_kernel_set_tx_length(struct socket *sock,
1019					struct rxrpc_call *call,
1020					s64 tx_total_len);
1021
1022     This sets the amount of data that the caller is intending to transmit on a
1023     call.  It's intended to be used for setting the reply size as the request
1024     size should be set when the call is begun.  tx_total_len may not be less
1025     than zero.
1026
1027 (#) Get call RTT::
1028
1029	u64 rxrpc_kernel_get_rtt(struct socket *sock, struct rxrpc_call *call);
1030
1031     Get the RTT time to the peer in use by a call.  The value returned is in
1032     nanoseconds.
1033
1034 (#) Check call still alive::
1035
1036	bool rxrpc_kernel_check_life(struct socket *sock,
1037				     struct rxrpc_call *call,
1038				     u32 *_life);
1039	void rxrpc_kernel_probe_life(struct socket *sock,
1040				     struct rxrpc_call *call);
1041
1042     The first function passes back in ``*_life`` a number that is updated when
1043     ACKs are received from the peer (notably including PING RESPONSE ACKs
1044     which we can elicit by sending PING ACKs to see if the call still exists
1045     on the server).  The caller should compare the numbers of two calls to see
1046     if the call is still alive after waiting for a suitable interval.  It also
1047     returns true as long as the call hasn't yet reached the completed state.
1048
1049     This allows the caller to work out if the server is still contactable and
1050     if the call is still alive on the server while waiting for the server to
1051     process a client operation.
1052
1053     The second function causes a ping ACK to be transmitted to try to provoke
1054     the peer into responding, which would then cause the value returned by the
1055     first function to change.  Note that this must be called in TASK_RUNNING
1056     state.
1057
1058 (#) Get reply timestamp::
1059
1060	bool rxrpc_kernel_get_reply_time(struct socket *sock,
1061					 struct rxrpc_call *call,
1062					 ktime_t *_ts)
1063
1064     This allows the timestamp on the first DATA packet of the reply of a
1065     client call to be queried, provided that it is still in the Rx ring.  If
1066     successful, the timestamp will be stored into ``*_ts`` and true will be
1067     returned; false will be returned otherwise.
1068
1069 (#) Get remote client epoch::
1070
1071	u32 rxrpc_kernel_get_epoch(struct socket *sock,
1072				   struct rxrpc_call *call)
1073
1074     This allows the epoch that's contained in packets of an incoming client
1075     call to be queried.  This value is returned.  The function always
1076     successful if the call is still in progress.  It shouldn't be called once
1077     the call has expired.  Note that calling this on a local client call only
1078     returns the local epoch.
1079
1080     This value can be used to determine if the remote client has been
1081     restarted as it shouldn't change otherwise.
1082
1083 (#) Set the maxmimum lifespan on a call::
1084
1085	void rxrpc_kernel_set_max_life(struct socket *sock,
1086				       struct rxrpc_call *call,
1087				       unsigned long hard_timeout)
1088
1089     This sets the maximum lifespan on a call to hard_timeout (which is in
1090     jiffies).  In the event of the timeout occurring, the call will be
1091     aborted and -ETIME or -ETIMEDOUT will be returned.
1092
1093 (#) Apply the RXRPC_MIN_SECURITY_LEVEL sockopt to a socket from within in the
1094     kernel::
1095
1096       int rxrpc_sock_set_min_security_level(struct sock *sk,
1097					     unsigned int val);
1098
1099     This specifies the minimum security level required for calls on this
1100     socket.
1101
1102
1103Configurable Parameters
1104=======================
1105
1106The RxRPC protocol driver has a number of configurable parameters that can be
1107adjusted through sysctls in /proc/net/rxrpc/:
1108
1109 (#) req_ack_delay
1110
1111     The amount of time in milliseconds after receiving a packet with the
1112     request-ack flag set before we honour the flag and actually send the
1113     requested ack.
1114
1115     Usually the other side won't stop sending packets until the advertised
1116     reception window is full (to a maximum of 255 packets), so delaying the
1117     ACK permits several packets to be ACK'd in one go.
1118
1119 (#) soft_ack_delay
1120
1121     The amount of time in milliseconds after receiving a new packet before we
1122     generate a soft-ACK to tell the sender that it doesn't need to resend.
1123
1124 (#) idle_ack_delay
1125
1126     The amount of time in milliseconds after all the packets currently in the
1127     received queue have been consumed before we generate a hard-ACK to tell
1128     the sender it can free its buffers, assuming no other reason occurs that
1129     we would send an ACK.
1130
1131 (#) resend_timeout
1132
1133     The amount of time in milliseconds after transmitting a packet before we
1134     transmit it again, assuming no ACK is received from the receiver telling
1135     us they got it.
1136
1137 (#) max_call_lifetime
1138
1139     The maximum amount of time in seconds that a call may be in progress
1140     before we preemptively kill it.
1141
1142 (#) dead_call_expiry
1143
1144     The amount of time in seconds before we remove a dead call from the call
1145     list.  Dead calls are kept around for a little while for the purpose of
1146     repeating ACK and ABORT packets.
1147
1148 (#) connection_expiry
1149
1150     The amount of time in seconds after a connection was last used before we
1151     remove it from the connection list.  While a connection is in existence,
1152     it serves as a placeholder for negotiated security; when it is deleted,
1153     the security must be renegotiated.
1154
1155 (#) transport_expiry
1156
1157     The amount of time in seconds after a transport was last used before we
1158     remove it from the transport list.  While a transport is in existence, it
1159     serves to anchor the peer data and keeps the connection ID counter.
1160
1161 (#) rxrpc_rx_window_size
1162
1163     The size of the receive window in packets.  This is the maximum number of
1164     unconsumed received packets we're willing to hold in memory for any
1165     particular call.
1166
1167 (#) rxrpc_rx_mtu
1168
1169     The maximum packet MTU size that we're willing to receive in bytes.  This
1170     indicates to the peer whether we're willing to accept jumbo packets.
1171
1172 (#) rxrpc_rx_jumbo_max
1173
1174     The maximum number of packets that we're willing to accept in a jumbo
1175     packet.  Non-terminal packets in a jumbo packet must contain a four byte
1176     header plus exactly 1412 bytes of data.  The terminal packet must contain
1177     a four byte header plus any amount of data.  In any event, a jumbo packet
1178     may not exceed rxrpc_rx_mtu in size.
1179