1--- 2layout: page 3title: fi_av(3) 4tagline: Libfabric Programmer's Manual 5--- 6{% include JB/setup %} 7 8# NAME 9 10fi_av \- Address vector operations 11 12fi_av_open / fi_close 13: Open or close an address vector 14 15fi_av_bind 16: Associate an address vector with an event queue. 17 18fi_av_insert / fi_av_insertsvc / fi_av_remove 19: Insert/remove an address into/from the address vector. 20 21fi_av_lookup 22: Retrieve an address stored in the address vector. 23 24fi_av_straddr 25: Convert an address into a printable string. 26 27# SYNOPSIS 28 29```c 30#include <rdma/fi_domain.h> 31 32int fi_av_open(struct fid_domain *domain, struct fi_av_attr *attr, 33 struct fid_av **av, void *context); 34 35int fi_close(struct fid *av); 36 37int fi_av_bind(struct fid_av *av, struct fid *eq, uint64_t flags); 38 39int fi_av_insert(struct fid_av *av, void *addr, size_t count, 40 fi_addr_t *fi_addr, uint64_t flags, void *context); 41 42int fi_av_insertsvc(struct fid_av *av, const char *node, 43 const char *service, fi_addr_t *fi_addr, uint64_t flags, 44 void *context); 45 46int fi_av_insertsym(struct fid_av *av, const char *node, 47 size_t nodecnt, const char *service, size_t svccnt, 48 fi_addr_t *fi_addr, uint64_t flags, void *context); 49 50int fi_av_remove(struct fid_av *av, fi_addr_t *fi_addr, size_t count, 51 uint64_t flags); 52 53int fi_av_lookup(struct fid_av *av, fi_addr_t fi_addr, 54 void *addr, size_t *addrlen); 55 56fi_addr_t fi_rx_addr(fi_addr_t fi_addr, int rx_index, 57 int rx_ctx_bits); 58 59const char * fi_av_straddr(struct fid_av *av, const void *addr, 60 char *buf, size_t *len); 61``` 62 63# ARGUMENTS 64 65*domain* 66: Resource domain 67 68*av* 69: Address vector 70 71*eq* 72: Event queue 73 74*attr* 75: Address vector attributes 76 77*context* 78: User specified context associated with the address vector or insert 79 operation. 80 81*addr* 82: Buffer containing one or more addresses to insert into address vector. 83 84*addrlen* 85: On input, specifies size of addr buffer. On output, stores number 86 of bytes written to addr buffer. 87 88*fi_addr* 89: For insert, a reference to an array where returned fabric addresses 90 will be written. For remove, one or more fabric addresses to remove. 91 92*count* 93: Number of addresses to insert/remove from an AV. 94 95*flags* 96: Additional flags to apply to the operation. 97 98# DESCRIPTION 99 100Address vectors are used to map higher level addresses, which may be 101more natural for an application to use, into fabric specific 102addresses. The mapping of addresses is fabric and provider specific, 103but may involve lengthy address resolution and fabric management 104protocols. AV operations are synchronous by default, but may be set 105to operate asynchronously by specifying the FI_EVENT flag to 106`fi_av_open`. When requesting asynchronous operation, the application 107must first bind an event queue to the AV before inserting addresses. 108 109## fi_av_open 110 111fi_av_open allocates or opens an address vector. The properties and 112behavior of the address vector are defined by `struct fi_av_attr`. 113 114```c 115struct fi_av_attr { 116 enum fi_av_type type; /* type of AV */ 117 int rx_ctx_bits; /* address bits to identify rx ctx */ 118 size_t count; /* # entries for AV */ 119 size_t ep_per_node; /* # endpoints per fabric address */ 120 const char *name; /* system name of AV */ 121 void *map_addr; /* base mmap address */ 122 uint64_t flags; /* operation flags */ 123}; 124``` 125 126*type* 127: An AV type corresponds to a conceptual implementation of an address 128 vector. The type specifies how an application views data stored in 129 the AV, including how it may be accessed. Valid values are: 130 131- *FI_AV_MAP* 132: Addresses which are inserted into an AV are mapped to a native 133 fabric address for use by the application. The use of FI_AV_MAP 134 requires that an application store the returned fi_addr_t value 135 that is associated with each inserted address. The advantage of 136 using FI_AV_MAP is that the returned fi_addr_t value may contain 137 encoded address data, which is immediately available when 138 processing data transfer requests. This can eliminate or reduce 139 the number of memory lookups needed when initiating a transfer. 140 The disadvantage of FI_AV_MAP is the increase in memory usage 141 needed to store the returned addresses. Addresses are stored in 142 the AV using a provider specific mechanism, including, but not 143 limited to a tree, hash table, or maintained on the heap. 144 145- *FI_AV_TABLE* 146: Addresses which are inserted into an AV of type FI_AV_TABLE are 147 accessible using a simple index. Conceptually, the AV may be 148 treated as an array of addresses, though the provider may implement 149 the AV using a variety of mechanisms. When FI_AV_TABLE is used, the 150 returned fi_addr_t is an index, with the index for an inserted 151 address the same as its insertion order into the table. The index 152 of the first address inserted into an FI_AV_TABLE will be 0, and 153 successive insertions will be given sequential indices. Sequential 154 indices will be assigned across insertion calls on the same AV. 155 156- *FI_AV_UNSPEC* 157: Provider will choose its preferred AV type. The AV type used will 158 be returned through the type field in fi_av_attr. 159 160*Receive Context Bits (rx_ctx_bits)* 161: The receive context bits field is only for use with scalable 162 endpoints. It indicates the number of bits reserved in a returned 163 fi_addr_t, which will be used to identify a specific target receive 164 context. See fi_rx_addr() and fi_endpoint(3) for additional details 165 on receive contexts. The requested number of bits should be 166 selected such that 2 ^ rx_ctx_bits >= rx_ctx_cnt for the endpoint. 167 168*count* 169: Indicates the expected number of addresses that will be inserted 170 into the AV. The provider uses this to optimize resource 171 allocations. 172 173*ep_per_node* 174: This field indicates the number of endpoints that will be associated 175 with a specific fabric, or network, address. If the number of 176 endpoints per node is unknown, this value should be set to 0. The 177 provider uses this value to optimize resource allocations. For 178 example, distributed, parallel applications may set this to the 179 number of processes allocated per node, times the number of 180 endpoints each process will open. 181 182*name* 183: An optional system name associated with the address vector to create 184 or open. Address vectors may be shared across multiple processes 185 which access the same named domain on the same node. The name field 186 allows the underlying provider to identify a shared AV. 187 188 If the name field is non-NULL and the AV is not opened for read-only 189 access, a named AV will be created, if it does not already exist. 190 191*map_addr* 192: The map_addr determines the base fi_addr_t address that a provider 193 should use when sharing an AV of type FI_AV_MAP between processes. 194 Processes that provide the same value for map_addr to a shared AV 195 may use the same fi_addr_t values returned from an fi_av_insert call. 196 197 The map_addr may be used by the provider to mmap memory allocated 198 for a shared AV between processes; however, the provider is not 199 required to use the map_addr in this fashion. The only requirement 200 is that an fi_addr_t returned as part of an fi_av_insert call on one 201 process is usable on another process which opens an AV of the same 202 name at the same map_addr value. The relationship between the 203 map_addr and any returned fi_addr_t is not defined. 204 205 If name is non-NULL and map_addr is 0, then the map_addr used by the 206 provider will be returned through the attribute structure. The 207 map_addr field is ignored if name is NULL. 208 209*flags* 210: The following flags may be used when opening an AV. 211 212- *FI_EVENT* 213: When the flag FI_EVENT is specified, all insert operations on this 214 AV will occur asynchronously. There will be one EQ error entry 215 generated for each failed address insertion, followed by one 216 non-error event indicating that the insertion operation has 217 completed. There will always be one non-error completion event for 218 each insert operation, even if all addresses fail. The context 219 field in all completions will be the context specified to the insert 220 call, and the data field in the final completion entry will report 221 the number of addresses successfully inserted. 222 If an error occurs during the asynchronous insertion, an error 223 completion entry is returned (see [`fi_eq`(3)](fi_eq.3.html) for a 224 discussion of the fi_eq_err_entry error completion struct). The 225 context field of the error completion will be the context that was 226 specified in the insert call; the data field will contain the index 227 of the failed address. There will be one error completion returned 228 for each address that fails to insert into the AV. 229 230 If an AV is opened with FI_EVENT, any insertions attempted before an 231 EQ is bound to the AV will fail with -FI_ENOEQ. 232 233 Error completions for failed insertions will contain the index of 234 the failed address in the index field of the error completion entry. 235 236 Note that the order of delivery of insert completions may not match 237 the order in which the calls to fi_av_insert were made. The only 238 guarantee is that all error completions for a given call to 239 fi_av_insert will precede the single associated non-error 240 completion. 241 242- *FI_READ* 243: Opens an AV for read-only access. An AV opened for read-only access 244 must be named (name attribute specified), and the AV must exist. 245 246- *FI_SYMMETRIC* 247: Indicates that each node will be associated with the same number of 248 endpoints, the same transport addresses will be allocated on each 249 node, and the transport addresses will be sequential. This feature 250 targets distributed applications on large fabrics and allows for 251 highly-optimized storage of remote endpoint addressing. 252 253## fi_close 254 255The fi_close call is used to release all resources associated with an 256address vector. Note that any events queued on an event queue referencing 257the AV are left untouched. It is recommended that callers retrieve all 258events associated with the AV before closing it. 259 260When closing the address vector, there must be no opened endpoints associated 261with the AV. If resources are still associated with the AV when attempting to 262close, the call will return -FI_EBUSY. 263 264## fi_av_bind 265 266Associates an event queue with the AV. If an AV has been opened with 267`FI_EVENT`, then an event queue must be bound to the AV before any 268insertion calls are attempted. Any calls to insert addresses before 269an event queue has been bound will fail with `-FI_ENOEQ`. Flags are 270reserved for future use and must be 0. 271 272## fi_av_insert 273 274The fi_av_insert call inserts zero or more addresses into an AV. The 275number of addresses is specified through the count parameter. The 276addr parameter references an array of addresses to insert into the AV. 277Addresses inserted into an address vector must be in the same format 278as specified in the addr_format field of the fi_info struct provided when 279opening the corresponding domain. When using the `FI_ADDR_STR` format, 280the `addr` parameter should reference an array of strings (char \*\*). 281 282For AV's of type FI_AV_MAP, once inserted addresses have been mapped, 283the mapped values are written into the buffer referenced by fi_addr. 284The fi_addr buffer must remain valid until the AV insertion has 285completed and an event has been generated to an associated event 286queue. The value of the returned fi_addr should be considered opaque 287by the application for AVs of type FI_AV_MAP. The returned value may 288point to an internal structure or a provider specific encoding of 289low-level addressing data, for example. In the latter case, use of 290FI_AV_MAP may be able to avoid memory references during data transfer 291operations. 292 293For AV's of type FI_AV_TABLE, addresses are placed into the table in 294order. An address is inserted at the lowest index that corresponds to 295an unused table location, with indices starting at 0. That is, the 296first address inserted may be referenced at index 0, the second at index 2971, and so forth. When addresses are inserted into an AV table, 298the assigned fi_addr values will be simple indices 299corresponding to the entry into the table where the address was 300inserted. Index values accumulate across successive insert calls 301in the order the calls are made, not necessarily in the order the 302insertions complete. 303 304Because insertions occur at a pre-determined index, the fi_addr 305parameter may be NULL. If fi_addr is non-NULL, it must reference 306an array of fi_addr_t, and the buffer must remain valid until the 307insertion operation completes. Note that if fi_addr is NULL and 308synchronous operation is requested without using FI_SYNC_ERR flag, individual 309insertion failures cannot be reported and the application must use 310other calls, such as `fi_av_lookup` to learn which specific addresses 311failed to insert. Since fi_av_remove is provider-specific, it is recommended 312that calls to fi_av_insert following a call to fi_av_remove always reference a 313valid buffer in the fi_addr parameter. Otherwise it may be difficult to 314determine what the next assigned index will be. 315 316*flags* 317: The following flag may be passed to AV insertion calls: fi_av_insert, 318 fi_av_insertsvc, or fi_av_insertsym. 319 320- *FI_MORE* 321: In order to allow optimized address insertion, the application may 322 specify the FI_MORE flag to the insert call to give a hint to the 323 provider that more insertion requests will follow, allowing the 324 provider to aggregate insertion requests if desired. An application 325 may make any number of insertion calls with FI_MORE set, provided 326 that they are followed by an insertion call without FI_MORE. This 327 signifies to the provider that the insertion list is complete. 328 Providers are free to ignore FI_MORE. 329 330- *FI_SYNC_ERR* 331: This flag applies to synchronous insertions only, and is used to 332 retrieve error details of failed insertions. If set, the context 333 parameter of insertion calls references an array of integers, with 334 context set to address of the first element of the array. 335 The resulting status of attempting to insert each address will be 336 written to the corresponding array location. Successful insertions 337 will be updated to 0. Failures will contain a fabric errno code. 338 339## fi_av_insertsvc 340 341The fi_av_insertsvc call behaves similar to fi_av_insert, but allows the 342application to specify the node and service names, similar to the 343fi_getinfo inputs, rather than an encoded address. The node and service 344parameters are defined the same as fi_getinfo(3). Node should be a string 345that corresponds to a hostname or network address. The service string 346corresponds to a textual representation of a transport address. 347Applications may also pass in an `FI_ADDR_STR` formatted address as the 348node parameter. In such cases, the service parameter must be NULL. See 349fi_getinfo.3 for details on using `FI_ADDR_STR`. Supported flags are the 350same as for fi_av_insert. 351 352## fi_av_insertsym 353 354fi_av_insertsym performs a symmetric insert that inserts a sequential 355range of nodes and/or service addresses into an AV. The svccnt 356parameter indicates the number of transport (endpoint) addresses to 357insert into the AV for each node address, with the service parameter 358specifying the starting transport address. Inserted transport 359addresses will be of the range {service, service + svccnt - 1}, 360inclusive. All service addresses for a node will be inserted before 361the next node is inserted. 362 363The nodecnt parameter indicates the number of node (network) addresses 364to insert into the AV, with the node parameter specifying the starting 365node address. Inserted node addresses will be of the range {node, 366node + nodecnt - 1}, inclusive. If node is a non-numeric string, such 367as a hostname, it must contain a numeric suffix if nodecnt > 1. 368 369As an example, if node = "10.1.1.1", nodecnt = 2, service = "5000", 370and svccnt = 2, the following addresses will be inserted into the AV 371in the order shown: 10.1.1.1:5000, 10.1.1.1:5001, 10.1.1.2:5000, 37210.1.1.2:5001. If node were replaced by the hostname "host10", the 373addresses would be: host10:5000, host10:5001, host11:5000, 374host11:5001. 375 376The total number of inserted addresses will be nodecnt x svccnt. 377 378Supported flags are the same as for fi_av_insert. 379 380## fi_av_remove 381 382fi_av_remove removes a set of addresses from an address vector. All 383resources associated with the indicated addresses are released. 384The removed address - either the mapped address (in the case of FI_AV_MAP) 385or index (FI_AV_TABLE) - is invalid until it is returned again by a 386new fi_av_insert. 387 388The behavior of operations in progress that reference the removed addresses 389is undefined. 390 391The use of fi_av_remove is an optimization that applications may use 392to free memory allocated with addresses that will no longer be 393accessed. Inserted addresses are not required to be removed. 394fi_av_close will automatically cleanup any resources associated with 395addresses remaining in the AV when it is invoked. 396 397Flags are reserved for future use and must be 0. 398 399## fi_av_lookup 400 401This call returns the address stored in the address vector that 402corresponds to the given fi_addr. The returned address is the same 403format as those stored by the AV. On input, the addrlen parameter 404should indicate the size of the addr buffer. If the actual address is 405larger than what can fit into the buffer, it will be truncated. On 406output, addrlen is set to the size of the buffer needed to store the 407address, which may be larger than the input value. 408 409## fi_rx_addr 410 411This function is used to convert an endpoint address, returned by 412fi_av_insert, into an address that specifies a target receive context. 413The specified fi_addr parameter must either be a value returned from 414fi_av_insert, in the case of FI_AV_MAP, or an index, in the case of 415FI_AV_TABLE. The value for rx_ctx_bits must match that specified in 416the AV attributes for the given address. 417 418Connected endpoints that support multiple receive contexts, but are 419not associated with address vectors should specify FI_ADDR_NOTAVAIL 420for the fi_addr parameter. 421 422## fi_av_straddr 423 424The fi_av_straddr function converts the provided address into a 425printable string. The specified address must be of the same format as 426those stored by the AV, though the address itself is not required to 427have been inserted. On input, the len parameter should specify the 428size of the buffer referenced by buf. On output, addrlen is set to the 429size of the buffer needed to store the address. This size may be 430larger than the input len. If the provided buffer is too small, the 431results will be truncated. fi_av_straddr returns a pointer to buf. 432 433# NOTES 434 435Providers may implement AV's using a variety of mechanisms. 436Specifically, a provider may begin resolving inserted addresses as 437soon as they have been added to an AV, even if asynchronous operation 438has been specified. Similarly, a provider may lazily release 439resources from removed entries. 440 441# RETURN VALUES 442 443Insertion calls for an AV opened for synchronous operation will return 444the number of addresses that were successfully inserted. In the case of 445failure, the return value will be less than the number of addresses that 446was specified. 447 448Insertion calls for an AV opened for asynchronous operation (with FI_EVENT 449flag specified) will return 0 if the operation was successfully initiated. 450In the case of failure, a negative fabric errno will be returned. Providers 451are allowed to abort insertion operations in the case of an error. Addresses 452that are not inserted because they were aborted will fail with an error code 453of FI_ECANCELED. 454 455In both the synchronous and asynchronous modes of operation, the fi_addr 456buffer associated with a failed or aborted insertion will be set to 457FI_ADDR_NOTAVAIL. 458 459All other calls return 0 on success, or a negative value corresponding to 460fabric errno on error. 461Fabric errno values are defined in 462`rdma/fi_errno.h`. 463 464# SEE ALSO 465 466[`fi_getinfo`(3)](fi_getinfo.3.html), 467[`fi_endpoint`(3)](fi_endpoint.3.html), 468[`fi_domain`(3)](fi_domain.3.html), 469[`fi_eq`(3)](fi_eq.3.html) 470