1PReLU {#dev_guide_prelu}
2============================
3
4>
5> [API Reference](@ref dnnl_api_prelu)
6>
7
8## General
9
10The PReLU primitive (Leaky ReLU with trainable alpha parameter) performs
11forward or backward operation on data tensor. Weights (alpha) tensor supports
12broadcast-semantics. Broadcast configuration is assumed based on src and
13weights dimensions.
14
15Example broadcasts:
16
17| broadcast type  | src dimensions       | weights dimensions   |
18| ---             | ---                  | ---                  |
19| Channel-shared  | \f$\{n, c, h ,w\}\f$ | \f$\{1, 1, 1 ,1\}\f$ |
20| Channel-wise    | \f$\{n, c, h ,w\}\f$ | \f$\{1, c, 1 ,1\}\f$ |
21| Whole-tensor    | \f$\{n, c, h ,w\}\f$ | \f$\{n, c, h ,w\}\f$ |
22| Shared-axes     | \f$\{n, c, h ,w\}\f$ | \f$\{n, 1, h ,1\}\f$ |
23
24@note
25   Shared-axes indicates broadcast with any combination of shared
26dimensions.
27
28### Forward
29
30The PReLU operation is defined by the following formulas.
31We show formulas only for 2D spatial data which are straightforward to
32generalize to cases of higher and lower dimensions. Variable names follow the
33standard @ref dev_guide_conventions.
34For no broadcast case, results are calculated using formula:
35
36\f[
37    \dst(n, c, h, w) =
38        \begin{cases}
39        \src(n, c, h, w)  & \mbox{if } \src(n, c, h, w) > 0 \\
40        \src(n, c, h, w) \cdot \weights(n, c, h, w) & \mbox{if }
41        \src(n, c, h, w) \leq 0
42        \end{cases}
43\f]
44
45Depending on broadcast configuration, result is calculated taking into account
46shared dimensions of weights tensor.
47
48#### Difference Between Forward Training and Forward Inference
49
50There is no difference between the #dnnl_forward_training
51and #dnnl_forward_inference propagation kinds.
52
53### Backward
54
55The backward propagation computes \f$\diffsrc\f$ and \f$\diffweights\f$.
56For no broadcast case, results are calculated using formula:
57
58\f[
59    \begin{align}
60    \mbox{diff_src}(n, c, h, w) &=
61        \begin{cases}
62        \mbox{diff_dst}(n, c, h, w)  & \mbox{if } \src(n, c, h, w) > 0 \\
63        \mbox{diff_dst}(n, c, h, w) \cdot \weights(n, c, h, w) &
64        \mbox{if } \src(n, c, h, w) \leq 0
65        \end{cases}\\\\
66    \mbox{diff_weights}(n, c, h, w) &=
67        \min(\src(n, c, h, w), 0) \cdot \mbox{diff_dst}(n, c, h, w)
68    \end{align}
69\f]
70
71Similar to forward propagation, result is calculated taking into
72account shared dimensions of weights tensor.
73\f$\diffweights\f$ results are accumulated according to weights tensor shared
74dimensions, since \f$\diffweights\f$ tensor must match \f$\weights\f$ tensor.
75
76
77## Execution Arguments
78
79When executed, the inputs and outputs should be mapped to an execution
80argument index as specified by the following table.
81
82| Primitive input/output | Execution argument index  |
83| ---                    | ---                       |
84| \f$\src\f$             | DNNL_ARG_SRC              |
85| \f$\dst\f$             | DNNL_ARG_DST              |
86| \f$\weights\f$         | DNNL_ARG_WEIGHTS          |
87| \f$\diffsrc\f$         | DNNL_ARG_DIFF_SRC         |
88| \f$\diffdst\f$         | DNNL_ARG_DIFF_DST         |
89| \f$\diffweights\f$     | DNNL_ARG_DIFF_WEIGHTS     |
90
91
92## Implementation Details
93
94### General Notes
95
96 * Prelu primitive requires all input/output tensors to have the
97   same number of dimensions. Dimension sizes can differ however.
98
99 * \weights tensor dimensions sizes must follow broadcast semantics.
100   Each dimension can either equal corresponding data dimension or
101   equal 1 - to indicate that dimension is shared.
102
103 * Prelu primitive requires that \diffweights tensor has exact same dimensions
104   sizes as \weights tensor, \diffsrc as src and \diffdst as dst.
105
106 * \weights tensor can be initialized with format_tag::any
107   primitive will match it to data tensor format.
108
109### Data Type Support
110
111The PReLU primitive supports the following combinations of data types:
112
113| Propagation        | Source / Destination   |
114| :--                | :--                    |
115| forward / backward | f32, s32, bf16, s8, u8 |
116
117### Data Representation
118
119The PReLU primitive works with arbitrary data tensors. There is no special
120meaning associated with any logical dimensions.
121
122## Implementation Limitations
123
124Current implementation supports all tensors up to 3D spatial (n, c, d, h, w).
125
126## Performance Tips
127
128Its recommended to allow PReLU primitive to choose the appropriate weights
129memory format by passing weights_md with format_tag::any.
130For best performance, the weights memory format should match
131data memory format.
132
133## Example
134
135[PReLU Primitive Example](@ref prelu_example_cpp)
136
137@copydetails prelu_example_cpp_short
138