1*97162a1eSMauro Carvalho Chehab==================== 2*97162a1eSMauro Carvalho ChehabUserspace MAD access 3*97162a1eSMauro Carvalho Chehab==================== 4*97162a1eSMauro Carvalho Chehab 5*97162a1eSMauro Carvalho ChehabDevice files 6*97162a1eSMauro Carvalho Chehab============ 7*97162a1eSMauro Carvalho Chehab 8*97162a1eSMauro Carvalho Chehab Each port of each InfiniBand device has a "umad" device and an 9*97162a1eSMauro Carvalho Chehab "issm" device attached. For example, a two-port HCA will have two 10*97162a1eSMauro Carvalho Chehab umad devices and two issm devices, while a switch will have one 11*97162a1eSMauro Carvalho Chehab device of each type (for switch port 0). 12*97162a1eSMauro Carvalho Chehab 13*97162a1eSMauro Carvalho ChehabCreating MAD agents 14*97162a1eSMauro Carvalho Chehab=================== 15*97162a1eSMauro Carvalho Chehab 16*97162a1eSMauro Carvalho Chehab A MAD agent can be created by filling in a struct ib_user_mad_reg_req 17*97162a1eSMauro Carvalho Chehab and then calling the IB_USER_MAD_REGISTER_AGENT ioctl on a file 18*97162a1eSMauro Carvalho Chehab descriptor for the appropriate device file. If the registration 19*97162a1eSMauro Carvalho Chehab request succeeds, a 32-bit id will be returned in the structure. 20*97162a1eSMauro Carvalho Chehab For example:: 21*97162a1eSMauro Carvalho Chehab 22*97162a1eSMauro Carvalho Chehab struct ib_user_mad_reg_req req = { /* ... */ }; 23*97162a1eSMauro Carvalho Chehab ret = ioctl(fd, IB_USER_MAD_REGISTER_AGENT, (char *) &req); 24*97162a1eSMauro Carvalho Chehab if (!ret) 25*97162a1eSMauro Carvalho Chehab my_agent = req.id; 26*97162a1eSMauro Carvalho Chehab else 27*97162a1eSMauro Carvalho Chehab perror("agent register"); 28*97162a1eSMauro Carvalho Chehab 29*97162a1eSMauro Carvalho Chehab Agents can be unregistered with the IB_USER_MAD_UNREGISTER_AGENT 30*97162a1eSMauro Carvalho Chehab ioctl. Also, all agents registered through a file descriptor will 31*97162a1eSMauro Carvalho Chehab be unregistered when the descriptor is closed. 32*97162a1eSMauro Carvalho Chehab 33*97162a1eSMauro Carvalho Chehab 2014 34*97162a1eSMauro Carvalho Chehab a new registration ioctl is now provided which allows additional 35*97162a1eSMauro Carvalho Chehab fields to be provided during registration. 36*97162a1eSMauro Carvalho Chehab Users of this registration call are implicitly setting the use of 37*97162a1eSMauro Carvalho Chehab pkey_index (see below). 38*97162a1eSMauro Carvalho Chehab 39*97162a1eSMauro Carvalho ChehabReceiving MADs 40*97162a1eSMauro Carvalho Chehab============== 41*97162a1eSMauro Carvalho Chehab 42*97162a1eSMauro Carvalho Chehab MADs are received using read(). The receive side now supports 43*97162a1eSMauro Carvalho Chehab RMPP. The buffer passed to read() must be at least one 44*97162a1eSMauro Carvalho Chehab struct ib_user_mad + 256 bytes. For example: 45*97162a1eSMauro Carvalho Chehab 46*97162a1eSMauro Carvalho Chehab If the buffer passed is not large enough to hold the received 47*97162a1eSMauro Carvalho Chehab MAD (RMPP), the errno is set to ENOSPC and the length of the 48*97162a1eSMauro Carvalho Chehab buffer needed is set in mad.length. 49*97162a1eSMauro Carvalho Chehab 50*97162a1eSMauro Carvalho Chehab Example for normal MAD (non RMPP) reads:: 51*97162a1eSMauro Carvalho Chehab 52*97162a1eSMauro Carvalho Chehab struct ib_user_mad *mad; 53*97162a1eSMauro Carvalho Chehab mad = malloc(sizeof *mad + 256); 54*97162a1eSMauro Carvalho Chehab ret = read(fd, mad, sizeof *mad + 256); 55*97162a1eSMauro Carvalho Chehab if (ret != sizeof mad + 256) { 56*97162a1eSMauro Carvalho Chehab perror("read"); 57*97162a1eSMauro Carvalho Chehab free(mad); 58*97162a1eSMauro Carvalho Chehab } 59*97162a1eSMauro Carvalho Chehab 60*97162a1eSMauro Carvalho Chehab Example for RMPP reads:: 61*97162a1eSMauro Carvalho Chehab 62*97162a1eSMauro Carvalho Chehab struct ib_user_mad *mad; 63*97162a1eSMauro Carvalho Chehab mad = malloc(sizeof *mad + 256); 64*97162a1eSMauro Carvalho Chehab ret = read(fd, mad, sizeof *mad + 256); 65*97162a1eSMauro Carvalho Chehab if (ret == -ENOSPC)) { 66*97162a1eSMauro Carvalho Chehab length = mad.length; 67*97162a1eSMauro Carvalho Chehab free(mad); 68*97162a1eSMauro Carvalho Chehab mad = malloc(sizeof *mad + length); 69*97162a1eSMauro Carvalho Chehab ret = read(fd, mad, sizeof *mad + length); 70*97162a1eSMauro Carvalho Chehab } 71*97162a1eSMauro Carvalho Chehab if (ret < 0) { 72*97162a1eSMauro Carvalho Chehab perror("read"); 73*97162a1eSMauro Carvalho Chehab free(mad); 74*97162a1eSMauro Carvalho Chehab } 75*97162a1eSMauro Carvalho Chehab 76*97162a1eSMauro Carvalho Chehab In addition to the actual MAD contents, the other struct ib_user_mad 77*97162a1eSMauro Carvalho Chehab fields will be filled in with information on the received MAD. For 78*97162a1eSMauro Carvalho Chehab example, the remote LID will be in mad.lid. 79*97162a1eSMauro Carvalho Chehab 80*97162a1eSMauro Carvalho Chehab If a send times out, a receive will be generated with mad.status set 81*97162a1eSMauro Carvalho Chehab to ETIMEDOUT. Otherwise when a MAD has been successfully received, 82*97162a1eSMauro Carvalho Chehab mad.status will be 0. 83*97162a1eSMauro Carvalho Chehab 84*97162a1eSMauro Carvalho Chehab poll()/select() may be used to wait until a MAD can be read. 85*97162a1eSMauro Carvalho Chehab 86*97162a1eSMauro Carvalho ChehabSending MADs 87*97162a1eSMauro Carvalho Chehab============ 88*97162a1eSMauro Carvalho Chehab 89*97162a1eSMauro Carvalho Chehab MADs are sent using write(). The agent ID for sending should be 90*97162a1eSMauro Carvalho Chehab filled into the id field of the MAD, the destination LID should be 91*97162a1eSMauro Carvalho Chehab filled into the lid field, and so on. The send side does support 92*97162a1eSMauro Carvalho Chehab RMPP so arbitrary length MAD can be sent. For example:: 93*97162a1eSMauro Carvalho Chehab 94*97162a1eSMauro Carvalho Chehab struct ib_user_mad *mad; 95*97162a1eSMauro Carvalho Chehab 96*97162a1eSMauro Carvalho Chehab mad = malloc(sizeof *mad + mad_length); 97*97162a1eSMauro Carvalho Chehab 98*97162a1eSMauro Carvalho Chehab /* fill in mad->data */ 99*97162a1eSMauro Carvalho Chehab 100*97162a1eSMauro Carvalho Chehab mad->hdr.id = my_agent; /* req.id from agent registration */ 101*97162a1eSMauro Carvalho Chehab mad->hdr.lid = my_dest; /* in network byte order... */ 102*97162a1eSMauro Carvalho Chehab /* etc. */ 103*97162a1eSMauro Carvalho Chehab 104*97162a1eSMauro Carvalho Chehab ret = write(fd, &mad, sizeof *mad + mad_length); 105*97162a1eSMauro Carvalho Chehab if (ret != sizeof *mad + mad_length) 106*97162a1eSMauro Carvalho Chehab perror("write"); 107*97162a1eSMauro Carvalho Chehab 108*97162a1eSMauro Carvalho ChehabTransaction IDs 109*97162a1eSMauro Carvalho Chehab=============== 110*97162a1eSMauro Carvalho Chehab 111*97162a1eSMauro Carvalho Chehab Users of the umad devices can use the lower 32 bits of the 112*97162a1eSMauro Carvalho Chehab transaction ID field (that is, the least significant half of the 113*97162a1eSMauro Carvalho Chehab field in network byte order) in MADs being sent to match 114*97162a1eSMauro Carvalho Chehab request/response pairs. The upper 32 bits are reserved for use by 115*97162a1eSMauro Carvalho Chehab the kernel and will be overwritten before a MAD is sent. 116*97162a1eSMauro Carvalho Chehab 117*97162a1eSMauro Carvalho ChehabP_Key Index Handling 118*97162a1eSMauro Carvalho Chehab==================== 119*97162a1eSMauro Carvalho Chehab 120*97162a1eSMauro Carvalho Chehab The old ib_umad interface did not allow setting the P_Key index for 121*97162a1eSMauro Carvalho Chehab MADs that are sent and did not provide a way for obtaining the P_Key 122*97162a1eSMauro Carvalho Chehab index of received MADs. A new layout for struct ib_user_mad_hdr 123*97162a1eSMauro Carvalho Chehab with a pkey_index member has been defined; however, to preserve binary 124*97162a1eSMauro Carvalho Chehab compatibility with older applications, this new layout will not be used 125*97162a1eSMauro Carvalho Chehab unless one of IB_USER_MAD_ENABLE_PKEY or IB_USER_MAD_REGISTER_AGENT2 ioctl's 126*97162a1eSMauro Carvalho Chehab are called before a file descriptor is used for anything else. 127*97162a1eSMauro Carvalho Chehab 128*97162a1eSMauro Carvalho Chehab In September 2008, the IB_USER_MAD_ABI_VERSION will be incremented 129*97162a1eSMauro Carvalho Chehab to 6, the new layout of struct ib_user_mad_hdr will be used by 130*97162a1eSMauro Carvalho Chehab default, and the IB_USER_MAD_ENABLE_PKEY ioctl will be removed. 131*97162a1eSMauro Carvalho Chehab 132*97162a1eSMauro Carvalho ChehabSetting IsSM Capability Bit 133*97162a1eSMauro Carvalho Chehab=========================== 134*97162a1eSMauro Carvalho Chehab 135*97162a1eSMauro Carvalho Chehab To set the IsSM capability bit for a port, simply open the 136*97162a1eSMauro Carvalho Chehab corresponding issm device file. If the IsSM bit is already set, 137*97162a1eSMauro Carvalho Chehab then the open call will block until the bit is cleared (or return 138*97162a1eSMauro Carvalho Chehab immediately with errno set to EAGAIN if the O_NONBLOCK flag is 139*97162a1eSMauro Carvalho Chehab passed to open()). The IsSM bit will be cleared when the issm file 140*97162a1eSMauro Carvalho Chehab is closed. No read, write or other operations can be performed on 141*97162a1eSMauro Carvalho Chehab the issm file. 142*97162a1eSMauro Carvalho Chehab 143*97162a1eSMauro Carvalho Chehab/dev files 144*97162a1eSMauro Carvalho Chehab========== 145*97162a1eSMauro Carvalho Chehab 146*97162a1eSMauro Carvalho Chehab To create the appropriate character device files automatically with 147*97162a1eSMauro Carvalho Chehab udev, a rule like:: 148*97162a1eSMauro Carvalho Chehab 149*97162a1eSMauro Carvalho Chehab KERNEL=="umad*", NAME="infiniband/%k" 150*97162a1eSMauro Carvalho Chehab KERNEL=="issm*", NAME="infiniband/%k" 151*97162a1eSMauro Carvalho Chehab 152*97162a1eSMauro Carvalho Chehab can be used. This will create device nodes named:: 153*97162a1eSMauro Carvalho Chehab 154*97162a1eSMauro Carvalho Chehab /dev/infiniband/umad0 155*97162a1eSMauro Carvalho Chehab /dev/infiniband/issm0 156*97162a1eSMauro Carvalho Chehab 157*97162a1eSMauro Carvalho Chehab for the first port, and so on. The InfiniBand device and port 158*97162a1eSMauro Carvalho Chehab associated with these devices can be determined from the files:: 159*97162a1eSMauro Carvalho Chehab 160*97162a1eSMauro Carvalho Chehab /sys/class/infiniband_mad/umad0/ibdev 161*97162a1eSMauro Carvalho Chehab /sys/class/infiniband_mad/umad0/port 162*97162a1eSMauro Carvalho Chehab 163*97162a1eSMauro Carvalho Chehab and:: 164*97162a1eSMauro Carvalho Chehab 165*97162a1eSMauro Carvalho Chehab /sys/class/infiniband_mad/issm0/ibdev 166*97162a1eSMauro Carvalho Chehab /sys/class/infiniband_mad/issm0/port 167