1PReLU {#dev_guide_prelu}
2============================
3
4>
5> [API Reference](@ref dnnl_api_prelu)
6>
7
8## General
9
10The PReLU primitive (Leaky ReLU with trainable alpha parameter) performs
11forward or backward operation on data tensor. Weights (alpha) tensor supports
12broadcast-semantics. Broadcast configuration is assumed based on src and
13weights dimensions.
14
15Example broadcasts:
16
17| broadcast type  | src dimensions       | weights dimensions   |
18| ---             | ---                  | ---                  |
19| Channel-shared  | \f$\{n, c, h ,w\}\f$ | \f$\{1, 1, 1 ,1\}\f$ |
20| Channel-wise    | \f$\{n, c, h ,w\}\f$ | \f$\{1, c, 1 ,1\}\f$ |
21| Whole-tensor    | \f$\{n, c, h ,w\}\f$ | \f$\{n, c, h ,w\}\f$ |
22| Shared-axes     | \f$\{n, c, h ,w\}\f$ | \f$\{n, 1, h ,1\}\f$ |
23
24note: Shared-axes indicates broadcast with any combination of shared
25dimensions.
26
27### Forward
28
29The PReLU operation is defined by the following formulas.
30We show formulas only for 2D spatial data which are straightforward to
31generalize to cases of higher and lower dimensions. Variable names follow the
32standard @ref dev_guide_conventions.
33For no broadcast case, results are calculated using formula:
34
35\f[
36    \dst(n, c, h, w) =
37        \begin{cases}
38        \src(n, c, h, w)  & \mbox{if } \src(n, c, h, w) > 0 \\
39        \src(n, c, h, w) \cdot \weights(n, c, h, w) & \mbox{if }
40        \src(n, c, h, w) \leq 0
41        \end{cases}
42\f]
43
44Depending on broadcast configuration, result is calculated taking into account
45shared dimensions of weights tensor.
46
47#### Difference Between Forward Training and Forward Inference
48
49There is no difference between the #dnnl_forward_training
50and #dnnl_forward_inference propagation kinds.
51
52### Backward
53
54The backward propagation computes \f$\diffsrc\f$ and \f$\diffweights\f$.
55For no broadcast case, results are calculated using formula:
56
57\f[
58    \begin{align}
59    \mbox{diff_src}(n, c, h, w) &=
60        \begin{cases}
61        \mbox{diff_dst}(n, c, h, w)  & \mbox{if } \src(n, c, h, w) > 0 \\
62        \mbox{diff_dst}(n, c, h, w) \cdot \weights(n, c, h, w) &
63        \mbox{if } \src(n, c, h, w) \leq 0
64        \end{cases}\\\\
65    \mbox{diff_weights}(n, c, h, w) &=
66        \min(\src(n, c, h, w), 0) \cdot \mbox{diff_dst}(n, c, h, w)
67    \end{align}
68\f]
69
70Similar to forward propagation, result is calculated taking into
71account shared dimensions of weights tensor.
72\f$\diffweights\f$ results are accumulated according to weights tensor shared
73dimensions, since \f$\diffweights\f$ tensor must match \f$\weights\f$ tensor.
74
75
76## Execution Arguments
77When executed, the inputs and outputs should be mapped to an execution
78argument index as specified by the following table.
79
80| Primitive input/output | Execution argument index  |
81| ---                    | ---                       |
82| \f$\src\f$             | DNNL_ARG_SRC              |
83| \f$\dst\f$             | DNNL_ARG_DST              |
84| \f$\weights\f$         | DNNL_ARG_WEIGHTS          |
85| \f$\diffsrc\f$         | DNNL_ARG_DIFF_SRC         |
86| \f$\diffdst\f$         | DNNL_ARG_DIFF_DST         |
87| \f$\diffweights\f$     | DNNL_ARG_DIFF_WEIGHTS     |
88
89
90## Implementation Details
91
92### General Notes
93
94 * Prelu primitive requires all input/output tensors to have the
95   same number of dimensions. Dimension sizes can differ however.
96
97 * \weights tensor dimensions sizes must follow broadcast semantics.
98   Each dimension can either equal corresponding data dimension or
99   equal 1 - to indicate that dimension is shared.
100
101 * Prelu primitive requires that \diffweights tensor has exact same dimensions
102   sizes as \weights tensor, \diffsrc as src and \diffdst as dst.
103
104 * \weights tensor can be initialized with format_tag::any
105   primitive will match it to data tensor format.
106
107### Data Type Support
108
109The PReLU primitive supports the following combinations of data types:
110
111| Propagation        | Source / Destination   |
112| :--                | :--                    |
113| forward / backward | f32, s32, bf16, s8, u8 |
114
115### Data Representation
116
117The PReLU primitive works with arbitrary data tensors. There is no special
118meaning associated with any logical dimensions.
119
120## Implementation Limitations
121
122Current implementation supports all tensors up to 3D spatial (n, c, d, h, w).
123
124## Performance Tips
125
126Its recommended to allow PReLU primitive to choose the appropriate weights
127memory format by passing weights_md with format_tag::any.
128For best performance, the weights memory format should match
129data memory format.
130
131## Examples
132
133| Engine  | Name                     | Comments
134| :--     | :--                      | :--
135| CPU/GPU | @ref prelu_example_cpp   | @copydetails prelu_example_cpp_short
136