1# ONNX Interface for Framework Integration (ONNXIFI)
2
3ONNXIFI is a cross-platform API for loading and executing ONNX graphs on optimized backends. High-level frameworks and applications can use this API to execute neural network and machine learning models. Hardware vendors can implement this API to expose specialized hardware accelerators and highly optimized software infrastructure to the users.
4
5## Core Features
6
7- Standardized interface for neural network inference on special-purpose accelerators (NPUs), CPUs, GPUs, DSPs, and FPGAs
8- Based on widely supported technologies
9  - C API for function calls
10  - ONNX format for passing model graphs
11  - NCHW tensor layout for passing inputs and outputs
12- Dynamic discovery of available backends for model execution
13  - Multiple backends from different vendors can co-exist on the same system
14- Dynamic discovery of supported ONNX Operators on each backend
15
16### Optional Features:
17
18- Graphs with variable-shape inputs and/or outputs
19- Graphs with data-dependendent output shapes
20
21## How to Use ONNX Interface for Framework Integration
22
230. (Optional) Use `onnxifi_load` to dynamically load the ONNX Interface for Framework Integration library.
241. Call `onnxGetBackendIDs` to get stable identifiers of available backends. Note: it is possible there are no backends installed in the system.
252. Call `onnxGetBackendInfo` to check additional information about any available backend.
263. Call `onnxGetBackendCompatibility` to check which operations within your model can run on the backend.
274. Call `onnxInitBackend` to initialize a backend, then call `onnxInitGraph` to offload one or more model graphs to the backend.
285. Call `onnxSetGraphIO` to set locations and shapes for inputs and outputs of a graph.
296. Initialize an `inputFence` structure of type `onnxMemoryFenceV1`: set `tag` to `ONNXIFI_TAG_MEMORY_FENCE_V1`, `type` to `ONNXIFI_SYNCHRONIZATION_EVENT`, and call `onnxInitEvent` to initiaze the `event` member.
307. Initialize an `outputFence` structure of type `onnxMemoryFenceV1`: set `tag` to `ONNXIFI_TAG_MEMORY_FENCE_V1`, `type` to `ONNXIFI_SYNCHRONIZATION_EVENT`, and `event` to null.
318. Call `onnxRunGraph` with the initialized `inputFence` and `outputFence` structures to enable execution of the graph. The call to `onnxRunGraph` will populate `event` member of the `outputFence` with a newly created event object, asynchronously execute the graph once `inputFence`'s `event` is signalled, and then signal the `outputFence`'s `event`.
329. Call `onnxSignalEvent` with `event` member of `inputFence` to signal to the backend that the inputs are ready to be consumed.
3310. Call `onnxWaitEvent` (alternatively, repeatedly call `onnxGetEventState` in a loop until the event state is `ONNXIFI_EVENT_STATE_SIGNALLED`) with `event` member of `outputFence` to wait until graph outputs are ready to be consumed. Release events for inputs and outputs using `onnxReleaseEvent`.
3411. If your model works with fixed-size inputs and outputs, and shape and location of inputs and outputs does not change, one call to `onnxSetGraphIO` is sufficient for multiple `onnxRunGraph` calls. The previous call to `onnxRunGraph`, however, must have finished before a user calls `onnxRunGraph` again, because concurrent execution with the same input and output locations is not allowed. For models with variable-size inputs or outputs, you'd need to call `onnxSetGraphIO` before each `onnxRunGraph` call.
3512. When done using the model, release the model graph(s) with `onnxReleaseGraph`, then release the backend with `onnxReleaseBackend` and backend ID with `onnxReleaseBackendID`.
36
37## How to Implement ONNX Interface for Framework Integration
38
39The minimum functionality an ONNXIFI implementation must provide is the following:
40
41- Support ONNX 1.0 model format.
42  - There is no minimum list of Operators a backed has to support.
43- Support graph inputs / outputs in CPU memory.
44- Support graph inputs / outputs with fixed shape, specified in GraphProto message.
45
46### Discovery
47Vendor-provided libraries should adhere to some rules to ensure discovery by ONNX-supported frameworks and applications:
48
491. The libraries must be installed in the following directories:
50  - GNU/Linux: user-installed system library directory (typically /usr/lib)
51  - macOS: /opt/onnx/lib
52  - Windows: system directory (typically C:\Windows\System32)
53
542. Filenames of vendor-specific libraries must follow the rule below:
55  - On Windows, library filename must match wildcard `onnxifi-*.dll`
56  - On macOS, library filename must match wildcard `libonnxifi-*.dylib`
57  - On Linux and other OSes, library filename must match wildcard `libonnxifi-*.so`
58
59### Extensions
60
61Hardware vendors are welcome to add their own extensions to ONNX backend interface. The backend interface offers several extension mechanisms:
62- Experimental, exotic, or vendor-specific operators can be supported in a private domain using NodeProto.domain attribute.
63- Vendor-provided ONNXIFI implementation can expose additional functions
64