1============
2Introduction
3============
4
5The RapidIO standard is a packet-based fabric interconnect standard designed for
6use in embedded systems. Development of the RapidIO standard is directed by the
7RapidIO Trade Association (RTA). The current version of the RapidIO specification
8is publicly available for download from the RTA web-site [1].
9
10This document describes the basics of the Linux RapidIO subsystem and provides
11information on its major components.
12
131 Overview
14==========
15
16Because the RapidIO subsystem follows the Linux device model it is integrated
17into the kernel similarly to other buses by defining RapidIO-specific device and
18bus types and registering them within the device model.
19
20The Linux RapidIO subsystem is architecture independent and therefore defines
21architecture-specific interfaces that provide support for common RapidIO
22subsystem operations.
23
242. Core Components
25==================
26
27A typical RapidIO network is a combination of endpoints and switches.
28Each of these components is represented in the subsystem by an associated data
29structure. The core logical components of the RapidIO subsystem are defined
30in include/linux/rio.h file.
31
322.1 Master Port
33---------------
34
35A master port (or mport) is a RapidIO interface controller that is local to the
36processor executing the Linux code. A master port generates and receives RapidIO
37packets (transactions). In the RapidIO subsystem each master port is represented
38by a rio_mport data structure. This structure contains master port specific
39resources such as mailboxes and doorbells. The rio_mport also includes a unique
40host device ID that is valid when a master port is configured as an enumerating
41host.
42
43RapidIO master ports are serviced by subsystem specific mport device drivers
44that provide functionality defined for this subsystem. To provide a hardware
45independent interface for RapidIO subsystem operations, rio_mport structure
46includes rio_ops data structure which contains pointers to hardware specific
47implementations of RapidIO functions.
48
492.2 Device
50----------
51
52A RapidIO device is any endpoint (other than mport) or switch in the network.
53All devices are presented in the RapidIO subsystem by corresponding rio_dev data
54structure. Devices form one global device list and per-network device lists
55(depending on number of available mports and networks).
56
572.3 Switch
58----------
59
60A RapidIO switch is a special class of device that routes packets between its
61ports towards their final destination. The packet destination port within a
62switch is defined by an internal routing table. A switch is presented in the
63RapidIO subsystem by rio_dev data structure expanded by additional rio_switch
64data structure, which contains switch specific information such as copy of the
65routing table and pointers to switch specific functions.
66
67The RapidIO subsystem defines the format and initialization method for subsystem
68specific switch drivers that are designed to provide hardware-specific
69implementation of common switch management routines.
70
712.4 Network
72-----------
73
74A RapidIO network is a combination of interconnected endpoint and switch devices.
75Each RapidIO network known to the system is represented by corresponding rio_net
76data structure. This structure includes lists of all devices and local master
77ports that form the same network. It also contains a pointer to the default
78master port that is used to communicate with devices within the network.
79
802.5 Device Drivers
81------------------
82
83RapidIO device-specific drivers follow Linux Kernel Driver Model and are
84intended to support specific RapidIO devices attached to the RapidIO network.
85
862.6 Subsystem Interfaces
87------------------------
88
89RapidIO interconnect specification defines features that may be used to provide
90one or more common service layers for all participating RapidIO devices. These
91common services may act separately from device-specific drivers or be used by
92device-specific drivers. Example of such service provider is the RIONET driver
93which implements Ethernet-over-RapidIO interface. Because only one driver can be
94registered for a device, all common RapidIO services have to be registered as
95subsystem interfaces. This allows to have multiple common services attached to
96the same device without blocking attachment of a device-specific driver.
97
983. Subsystem Initialization
99===========================
100
101In order to initialize the RapidIO subsystem, a platform must initialize and
102register at least one master port within the RapidIO network. To register mport
103within the subsystem controller driver's initialization code calls function
104rio_register_mport() for each available master port.
105
106After all active master ports are registered with a RapidIO subsystem,
107an enumeration and/or discovery routine may be called automatically or
108by user-space command.
109
110RapidIO subsystem can be configured to be built as a statically linked or
111modular component of the kernel (see details below).
112
1134. Enumeration and Discovery
114============================
115
1164.1 Overview
117------------
118
119RapidIO subsystem configuration options allow users to build enumeration and
120discovery methods as statically linked components or loadable modules.
121An enumeration/discovery method implementation and available input parameters
122define how any given method can be attached to available RapidIO mports:
123simply to all available mports OR individually to the specified mport device.
124
125Depending on selected enumeration/discovery build configuration, there are
126several methods to initiate an enumeration and/or discovery process:
127
128  (a) Statically linked enumeration and discovery process can be started
129  automatically during kernel initialization time using corresponding module
130  parameters. This was the original method used since introduction of RapidIO
131  subsystem. Now this method relies on enumerator module parameter which is
132  'rio-scan.scan' for existing basic enumeration/discovery method.
133  When automatic start of enumeration/discovery is used a user has to ensure
134  that all discovering endpoints are started before the enumerating endpoint
135  and are waiting for enumeration to be completed.
136  Configuration option CONFIG_RAPIDIO_DISC_TIMEOUT defines time that discovering
137  endpoint waits for enumeration to be completed. If the specified timeout
138  expires the discovery process is terminated without obtaining RapidIO network
139  information. NOTE: a timed out discovery process may be restarted later using
140  a user-space command as it is described below (if the given endpoint was
141  enumerated successfully).
142
143  (b) Statically linked enumeration and discovery process can be started by
144  a command from user space. This initiation method provides more flexibility
145  for a system startup compared to the option (a) above. After all participating
146  endpoints have been successfully booted, an enumeration process shall be
147  started first by issuing a user-space command, after an enumeration is
148  completed a discovery process can be started on all remaining endpoints.
149
150  (c) Modular enumeration and discovery process can be started by a command from
151  user space. After an enumeration/discovery module is loaded, a network scan
152  process can be started by issuing a user-space command.
153  Similar to the option (b) above, an enumerator has to be started first.
154
155  (d) Modular enumeration and discovery process can be started by a module
156  initialization routine. In this case an enumerating module shall be loaded
157  first.
158
159When a network scan process is started it calls an enumeration or discovery
160routine depending on the configured role of a master port: host or agent.
161
162Enumeration is performed by a master port if it is configured as a host port by
163assigning a host destination ID greater than or equal to zero. The host
164destination ID can be assigned to a master port using various methods depending
165on RapidIO subsystem build configuration:
166
167  (a) For a statically linked RapidIO subsystem core use command line parameter
168  "rapidio.hdid=" with a list of destination ID assignments in order of mport
169  device registration. For example, in a system with two RapidIO controllers
170  the command line parameter "rapidio.hdid=-1,7" will result in assignment of
171  the host destination ID=7 to the second RapidIO controller, while the first
172  one will be assigned destination ID=-1.
173
174  (b) If the RapidIO subsystem core is built as a loadable module, in addition
175  to the method shown above, the host destination ID(s) can be specified using
176  traditional methods of passing module parameter "hdid=" during its loading:
177
178  - from command line: "modprobe rapidio hdid=-1,7", or
179  - from modprobe configuration file using configuration command "options",
180    like in this example: "options rapidio hdid=-1,7". An example of modprobe
181    configuration file is provided in the section below.
182
183NOTES:
184  (i) if "hdid=" parameter is omitted all available mport will be assigned
185  destination ID = -1;
186
187  (ii) the "hdid=" parameter in systems with multiple mports can have
188  destination ID assignments omitted from the end of list (default = -1).
189
190If the host device ID for a specific master port is set to -1, the discovery
191process will be performed for it.
192
193The enumeration and discovery routines use RapidIO maintenance transactions
194to access the configuration space of devices.
195
196NOTE: If RapidIO switch-specific device drivers are built as loadable modules
197they must be loaded before enumeration/discovery process starts.
198This requirement is cased by the fact that enumeration/discovery methods invoke
199vendor-specific callbacks on early stages.
200
2014.2 Automatic Start of Enumeration and Discovery
202------------------------------------------------
203
204Automatic enumeration/discovery start method is applicable only to built-in
205enumeration/discovery RapidIO configuration selection. To enable automatic
206enumeration/discovery start by existing basic enumerator method set use boot
207command line parameter "rio-scan.scan=1".
208
209This configuration requires synchronized start of all RapidIO endpoints that
210form a network which will be enumerated/discovered. Discovering endpoints have
211to be started before an enumeration starts to ensure that all RapidIO
212controllers have been initialized and are ready to be discovered. Configuration
213parameter CONFIG_RAPIDIO_DISC_TIMEOUT defines time (in seconds) which
214a discovering endpoint will wait for enumeration to be completed.
215
216When automatic enumeration/discovery start is selected, basic method's
217initialization routine calls rio_init_mports() to perform enumeration or
218discovery for all known mport devices.
219
220Depending on RapidIO network size and configuration this automatic
221enumeration/discovery start method may be difficult to use due to the
222requirement for synchronized start of all endpoints.
223
2244.3 User-space Start of Enumeration and Discovery
225-------------------------------------------------
226
227User-space start of enumeration and discovery can be used with built-in and
228modular build configurations. For user-space controlled start RapidIO subsystem
229creates the sysfs write-only attribute file '/sys/bus/rapidio/scan'. To initiate
230an enumeration or discovery process on specific mport device, a user needs to
231write mport_ID (not RapidIO destination ID) into that file. The mport_ID is a
232sequential number (0 ... RIO_MAX_MPORTS) assigned during mport device
233registration. For example for machine with single RapidIO controller, mport_ID
234for that controller always will be 0.
235
236To initiate RapidIO enumeration/discovery on all available mports a user may
237write '-1' (or RIO_MPORT_ANY) into the scan attribute file.
238
2394.4 Basic Enumeration Method
240----------------------------
241
242This is an original enumeration/discovery method which is available since
243first release of RapidIO subsystem code. The enumeration process is
244implemented according to the enumeration algorithm outlined in the RapidIO
245Interconnect Specification: Annex I [1].
246
247This method can be configured as statically linked or loadable module.
248The method's single parameter "scan" allows to trigger the enumeration/discovery
249process from module initialization routine.
250
251This enumeration/discovery method can be started only once and does not support
252unloading if it is built as a module.
253
254The enumeration process traverses the network using a recursive depth-first
255algorithm. When a new device is found, the enumerator takes ownership of that
256device by writing into the Host Device ID Lock CSR. It does this to ensure that
257the enumerator has exclusive right to enumerate the device. If device ownership
258is successfully acquired, the enumerator allocates a new rio_dev structure and
259initializes it according to device capabilities.
260
261If the device is an endpoint, a unique device ID is assigned to it and its value
262is written into the device's Base Device ID CSR.
263
264If the device is a switch, the enumerator allocates an additional rio_switch
265structure to store switch specific information. Then the switch's vendor ID and
266device ID are queried against a table of known RapidIO switches. Each switch
267table entry contains a pointer to a switch-specific initialization routine that
268initializes pointers to the rest of switch specific operations, and performs
269hardware initialization if necessary. A RapidIO switch does not have a unique
270device ID; it relies on hopcount and routing for device ID of an attached
271endpoint if access to its configuration registers is required. If a switch (or
272chain of switches) does not have any endpoint (except enumerator) attached to
273it, a fake device ID will be assigned to configure a route to that switch.
274In the case of a chain of switches without endpoint, one fake device ID is used
275to configure a route through the entire chain and switches are differentiated by
276their hopcount value.
277
278For both endpoints and switches the enumerator writes a unique component tag
279into device's Component Tag CSR. That unique value is used by the error
280management notification mechanism to identify a device that is reporting an
281error management event.
282
283Enumeration beyond a switch is completed by iterating over each active egress
284port of that switch. For each active link, a route to a default device ID
285(0xFF for 8-bit systems and 0xFFFF for 16-bit systems) is temporarily written
286into the routing table. The algorithm recurs by calling itself with hopcount + 1
287and the default device ID in order to access the device on the active port.
288
289After the host has completed enumeration of the entire network it releases
290devices by clearing device ID locks (calls rio_clear_locks()). For each endpoint
291in the system, it sets the Discovered bit in the Port General Control CSR
292to indicate that enumeration is completed and agents are allowed to execute
293passive discovery of the network.
294
295The discovery process is performed by agents and is similar to the enumeration
296process that is described above. However, the discovery process is performed
297without changes to the existing routing because agents only gather information
298about RapidIO network structure and are building an internal map of discovered
299devices. This way each Linux-based component of the RapidIO subsystem has
300a complete view of the network. The discovery process can be performed
301simultaneously by several agents. After initializing its RapidIO master port
302each agent waits for enumeration completion by the host for the configured wait
303time period. If this wait time period expires before enumeration is completed,
304an agent skips RapidIO discovery and continues with remaining kernel
305initialization.
306
3074.5 Adding New Enumeration/Discovery Method
308-------------------------------------------
309
310RapidIO subsystem code organization allows addition of new enumeration/discovery
311methods as new configuration options without significant impact to the core
312RapidIO code.
313
314A new enumeration/discovery method has to be attached to one or more mport
315devices before an enumeration/discovery process can be started. Normally,
316method's module initialization routine calls rio_register_scan() to attach
317an enumerator to a specified mport device (or devices). The basic enumerator
318implementation demonstrates this process.
319
3204.6 Using Loadable RapidIO Switch Drivers
321-----------------------------------------
322
323In the case when RapidIO switch drivers are built as loadable modules a user
324must ensure that they are loaded before the enumeration/discovery starts.
325This process can be automated by specifying pre- or post- dependencies in the
326RapidIO-specific modprobe configuration file as shown in the example below.
327
328File /etc/modprobe.d/rapidio.conf::
329
330  # Configure RapidIO subsystem modules
331
332  # Set enumerator host destination ID (overrides kernel command line option)
333  options rapidio hdid=-1,2
334
335  # Load RapidIO switch drivers immediately after rapidio core module was loaded
336  softdep rapidio post: idt_gen2 idtcps tsi57x
337
338  # OR :
339
340  # Load RapidIO switch drivers just before rio-scan enumerator module is loaded
341  softdep rio-scan pre: idt_gen2 idtcps tsi57x
342
343  --------------------------
344
345NOTE:
346  In the example above, one of "softdep" commands must be removed or
347  commented out to keep required module loading sequence.
348
3495. References
350=============
351
352[1] RapidIO Trade Association. RapidIO Interconnect Specifications.
353    http://www.rapidio.org.
354
355[2] Rapidio TA. Technology Comparisons.
356    http://www.rapidio.org/education/technology_comparisons/
357
358[3] RapidIO support for Linux.
359    https://lwn.net/Articles/139118/
360
361[4] Matt Porter. RapidIO for Linux. Ottawa Linux Symposium, 2005
362    https://www.kernel.org/doc/ols/2005/ols2005v2-pages-43-56.pdf
363