1PReLU {#dev_guide_prelu} 2============================ 3 4> 5> [API Reference](@ref dnnl_api_prelu) 6> 7 8## General 9 10The PReLU primitive (Leaky ReLU with trainable alpha parameter) performs 11forward or backward operation on data tensor. Weights (alpha) tensor supports 12broadcast-semantics. Broadcast configuration is assumed based on src and 13weights dimensions. 14 15Example broadcasts: 16 17| broadcast type | src dimensions | weights dimensions | 18| --- | --- | --- | 19| Channel-shared | \f$\{n, c, h ,w\}\f$ | \f$\{1, 1, 1 ,1\}\f$ | 20| Channel-wise | \f$\{n, c, h ,w\}\f$ | \f$\{1, c, 1 ,1\}\f$ | 21| Whole-tensor | \f$\{n, c, h ,w\}\f$ | \f$\{n, c, h ,w\}\f$ | 22| Shared-axes | \f$\{n, c, h ,w\}\f$ | \f$\{n, 1, h ,1\}\f$ | 23 24note: Shared-axes indicates broadcast with any combination of shared 25dimensions. 26 27### Forward 28 29The PReLU operation is defined by the following formulas. 30We show formulas only for 2D spatial data which are straightforward to 31generalize to cases of higher and lower dimensions. Variable names follow the 32standard @ref dev_guide_conventions. 33For no broadcast case, results are calculated using formula: 34 35\f[ 36 \dst(n, c, h, w) = 37 \begin{cases} 38 \src(n, c, h, w) & \mbox{if } \src(n, c, h, w) > 0 \\ 39 \src(n, c, h, w) \cdot \weights(n, c, h, w) & \mbox{if } 40 \src(n, c, h, w) \leq 0 41 \end{cases} 42\f] 43 44Depending on broadcast configuration, result is calculated taking into account 45shared dimensions of weights tensor. 46 47#### Difference Between Forward Training and Forward Inference 48 49There is no difference between the #dnnl_forward_training 50and #dnnl_forward_inference propagation kinds. 51 52### Backward 53 54The backward propagation computes \f$\diffsrc\f$ and \f$\diffweights\f$. 55For no broadcast case, results are calculated using formula: 56 57\f[ 58 \begin{align} 59 \mbox{diff_src}(n, c, h, w) &= 60 \begin{cases} 61 \mbox{diff_dst}(n, c, h, w) & \mbox{if } \src(n, c, h, w) > 0 \\ 62 \mbox{diff_dst}(n, c, h, w) \cdot \weights(n, c, h, w) & 63 \mbox{if } \src(n, c, h, w) \leq 0 64 \end{cases}\\\\ 65 \mbox{diff_weights}(n, c, h, w) &= 66 \min(\src(n, c, h, w), 0) \cdot \mbox{diff_dst}(n, c, h, w) 67 \end{align} 68\f] 69 70Similar to forward propagation, result is calculated taking into 71account shared dimensions of weights tensor. 72\f$\diffweights\f$ results are accumulated according to weights tensor shared 73dimensions, since \f$\diffweights\f$ tensor must match \f$\weights\f$ tensor. 74 75 76## Execution Arguments 77When executed, the inputs and outputs should be mapped to an execution 78argument index as specified by the following table. 79 80| Primitive input/output | Execution argument index | 81| --- | --- | 82| \f$\src\f$ | DNNL_ARG_SRC | 83| \f$\dst\f$ | DNNL_ARG_DST | 84| \f$\weights\f$ | DNNL_ARG_WEIGHTS | 85| \f$\diffsrc\f$ | DNNL_ARG_DIFF_SRC | 86| \f$\diffdst\f$ | DNNL_ARG_DIFF_DST | 87| \f$\diffweights\f$ | DNNL_ARG_DIFF_WEIGHTS | 88 89 90## Implementation Details 91 92### General Notes 93 94 * Prelu primitive requires all input/output tensors to have the 95 same number of dimensions. Dimension sizes can differ however. 96 97 * \weights tensor dimensions sizes must follow broadcast semantics. 98 Each dimension can either equal corresponding data dimension or 99 equal 1 - to indicate that dimension is shared. 100 101 * Prelu primitive requires that \diffweights tensor has exact same dimensions 102 sizes as \weights tensor, \diffsrc as src and \diffdst as dst. 103 104 * \weights tensor can be initialized with format_tag::any 105 primitive will match it to data tensor format. 106 107### Data Type Support 108 109The PReLU primitive supports the following combinations of data types: 110 111| Propagation | Source / Destination | 112| :-- | :-- | 113| forward / backward | f32, s32, bf16, s8, u8 | 114 115### Data Representation 116 117The PReLU primitive works with arbitrary data tensors. There is no special 118meaning associated with any logical dimensions. 119 120## Implementation Limitations 121 122Current implementation supports all tensors up to 3D spatial (n, c, d, h, w). 123 124## Performance Tips 125 126Its recommended to allow PReLU primitive to choose the appropriate weights 127memory format by passing weights_md with format_tag::any. 128For best performance, the weights memory format should match 129data memory format. 130 131## Examples 132 133| Engine | Name | Comments 134| :-- | :-- | :-- 135| CPU/GPU | @ref prelu_example_cpp | @copydetails prelu_example_cpp_short 136