1.. SPDX-License-Identifier: GPL-2.0
2
3.. include:: <isonum.txt>
4
5===============================================================
6Intel Image Processing Unit 3 (IPU3) Imaging Unit (ImgU) driver
7===============================================================
8
9Copyright |copy| 2018 Intel Corporation
10
11Introduction
12============
13
14This file documents the Intel IPU3 (3rd generation Image Processing Unit)
15Imaging Unit drivers located under drivers/media/pci/intel/ipu3 (CIO2) as well
16as under drivers/staging/media/ipu3 (ImgU).
17
18The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake)
19platforms (U/Y processor lines) is made up of two parts namely the Imaging Unit
20(ImgU) and the CIO2 device (MIPI CSI2 receiver).
21
22The CIO2 device receives the raw Bayer data from the sensors and outputs the
23frames in a format that is specific to the IPU3 (for consumption by the IPU3
24ImgU). The CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2*
25and is enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
26
27The Imaging Unit (ImgU) is responsible for processing images captured
28by the IPU3 CIO2 device. The ImgU driver sources can be found under
29drivers/staging/media/ipu3 directory. The driver is enabled through the
30CONFIG_VIDEO_IPU3_IMGU config option.
31
32The two driver modules are named ipu3_csi2 and ipu3_imgu, respectively.
33
34The drivers has been tested on Kaby Lake platforms (U/Y processor lines).
35
36Both of the drivers implement V4L2, Media Controller and V4L2 sub-device
37interfaces. The IPU3 CIO2 driver supports camera sensors connected to the CIO2
38MIPI CSI-2 interfaces through V4L2 sub-device sensor drivers.
39
40CIO2
41====
42
43The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev
44interface to the user space. There is a video node for each CSI-2 receiver,
45with a single media controller interface for the entire device.
46
47The CIO2 contains four independent capture channel, each with its own MIPI CSI-2
48receiver and DMA engine. Each channel is modelled as a V4L2 sub-device exposed
49to userspace as a V4L2 sub-device node and has two pads:
50
51.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
52
53.. flat-table::
54    :header-rows: 1
55
56    * - Pad
57      - Direction
58      - Purpose
59
60    * - 0
61      - sink
62      - MIPI CSI-2 input, connected to the sensor subdev
63
64    * - 1
65      - source
66      - Raw video capture, connected to the V4L2 video interface
67
68The V4L2 video interfaces model the DMA engines. They are exposed to userspace
69as V4L2 video device nodes.
70
71Capturing frames in raw Bayer format
72------------------------------------
73
74CIO2 MIPI CSI2 receiver is used to capture frames (in packed raw Bayer format)
75from the raw sensors connected to the CSI2 ports. The captured frames are used
76as input to the ImgU driver.
77
78Image processing using IPU3 ImgU requires tools such as raw2pnm [#f1]_, and
79yavta [#f2]_ due to the following unique requirements and / or features specific
80to IPU3.
81
82-- The IPU3 CSI2 receiver outputs the captured frames from the sensor in packed
83raw Bayer format that is specific to IPU3.
84
85-- Multiple video nodes have to be operated simultaneously.
86
87Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
882592x1944 image capture.
89
90Using the media controller APIs, the ov5670 sensor is configured to send
91frames in packed raw Bayer format to IPU3 CSI2 receiver.
92
93.. code-block:: none
94
95    # This example assumes /dev/media0 as the CIO2 media device
96    export MDEV=/dev/media0
97
98    # and that ov5670 sensor is connected to i2c bus 10 with address 0x36
99    export SDEV=$(media-ctl -d $MDEV -e "ov5670 10-0036")
100
101    # Establish the link for the media devices using media-ctl [#f3]_
102    media-ctl -d $MDEV -l "ov5670:0 -> ipu3-csi2 0:0[1]"
103
104    # Set the format for the media devices
105    media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG10/2592x1944]"
106    media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt:SGRBG10/2592x1944]"
107    media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt:SGRBG10/2592x1944]"
108
109Once the media pipeline is configured, desired sensor specific settings
110(such as exposure and gain settings) can be set, using the yavta tool.
111
112e.g
113
114.. code-block:: none
115
116    yavta -w 0x009e0903 444 $SDEV
117    yavta -w 0x009e0913 1024 $SDEV
118    yavta -w 0x009e0911 2046 $SDEV
119
120Once the desired sensor settings are set, frame captures can be done as below.
121
122e.g
123
124.. code-block:: none
125
126    yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin \
127          -f IPU3_SGRBG10 $(media-ctl -d $MDEV -e "ipu3-cio2 0")
128
129With the above command, 10 frames are captured at 2592x1944 resolution, with
130sGRBG10 format and output as IPU3_SGRBG10 format.
131
132The captured frames are available as /tmp/frame-#.bin files.
133
134ImgU
135====
136
137The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
138subdev interface to the user space.
139
140Each V4L2 subdev represents a pipe, which can support a maximum of 2 streams.
141This helps to support advanced camera features like Continuous View Finder (CVF)
142and Snapshot During Video(SDV).
143
144The ImgU contains two independent pipes, each modelled as a V4L2 sub-device
145exposed to userspace as a V4L2 sub-device node.
146
147Each pipe has two sink pads and three source pads for the following purpose:
148
149.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
150
151.. flat-table::
152    :header-rows: 1
153
154    * - Pad
155      - Direction
156      - Purpose
157
158    * - 0
159      - sink
160      - Input raw video stream
161
162    * - 1
163      - sink
164      - Processing parameters
165
166    * - 2
167      - source
168      - Output processed video stream
169
170    * - 3
171      - source
172      - Output viewfinder video stream
173
174    * - 4
175      - source
176      - 3A statistics
177
178Each pad is connected to a corresponding V4L2 video interface, exposed to
179userspace as a V4L2 video device node.
180
181Device operation
182----------------
183
184With ImgU, once the input video node ("ipu3-imgu 0/1":0, in
185<entity>:<pad-number> format) is queued with buffer (in packed raw Bayer
186format), ImgU starts processing the buffer and produces the video output in YUV
187format and statistics output on respective output nodes. The driver is expected
188to have buffers ready for all of parameter, output and statistics nodes, when
189input video node is queued with buffer.
190
191At a minimum, all of input, main output, 3A statistics and viewfinder
192video nodes should be enabled for IPU3 to start image processing.
193
194Each ImgU V4L2 subdev has the following set of video nodes.
195
196input, output and viewfinder video nodes
197----------------------------------------
198
199The frames (in packed raw Bayer format specific to the IPU3) received by the
200input video node is processed by the IPU3 Imaging Unit and are output to 2 video
201nodes, with each targeting a different purpose (main output and viewfinder
202output).
203
204Details onand the Bayer format specific to the IPU3 can be found in
205:ref:`v4l2-pix-fmt-ipu3-sbggr10`.
206
207The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`.
208
209Only the multi-planar API is supported. More details can be found at
210:ref:`planar-apis`.
211
212Parameters video node
213---------------------
214
215The parameters video node receives the ImgU algorithm parameters that are used
216to configure how the ImgU algorithms process the image.
217
218Details on processing parameters specific to the IPU3 can be found in
219:ref:`v4l2-meta-fmt-params`.
220
2213A statistics video node
222------------------------
223
2243A statistics video node is used by the ImgU driver to output the 3A (auto
225focus, auto exposure and auto white balance) statistics for the frames that are
226being processed by the ImgU to user space applications. User space applications
227can use this statistics data to compute the desired algorithm parameters for
228the ImgU.
229
230Configuring the Intel IPU3
231==========================
232
233The IPU3 ImgU pipelines can be configured using the Media Controller, defined at
234:ref:`media_controller`.
235
236Running mode and firmware binary selection
237------------------------------------------
238
239ImgU works based on firmware, currently the ImgU firmware support run 2 pipes
240in time-sharing with single input frame data. Each pipe can run at certain mode
241- "VIDEO" or "STILL", "VIDEO" mode is commonly used for video frames capture,
242and "STILL" is used for still frame capture. However, you can also select
243"VIDEO" to capture still frames if you want to capture images with less system
244load and power. For "STILL" mode, ImgU will try to use smaller BDS factor and
245output larger bayer frame for further YUV processing than "VIDEO" mode to get
246high quality images. Besides, "STILL" mode need XNR3 to do noise reduction,
247hence "STILL" mode will need more power and memory bandwidth than "VIDEO" mode.
248TNR will be enabled in "VIDEO" mode and bypassed by "STILL" mode. ImgU is
249running at "VIDEO" mode by default, the user can use v4l2 control
250V4L2_CID_INTEL_IPU3_MODE (currently defined in
251drivers/staging/media/ipu3/include/uapi/intel-ipu3.h) to query and set the
252running mode. For user, there is no difference for buffer queueing between the
253"VIDEO" and "STILL" mode, mandatory input and main output node should be
254enabled and buffers need be queued, the statistics and the view-finder queues
255are optional.
256
257The firmware binary will be selected according to current running mode, such log
258"using binary if_to_osys_striped " or "using binary if_to_osys_primary_striped"
259could be observed if you enable the ImgU dynamic debug, the binary
260if_to_osys_striped is selected for "VIDEO" and the binary
261"if_to_osys_primary_striped" is selected for "STILL".
262
263
264Processing the image in raw Bayer format
265----------------------------------------
266
267Configuring ImgU V4L2 subdev for image processing
268~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
269
270The ImgU V4L2 subdevs have to be configured with media controller APIs to have
271all the video nodes setup correctly.
272
273Let us take "ipu3-imgu 0" subdev as an example.
274
275.. code-block:: none
276
277    media-ctl -d $MDEV -r
278    media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1]
279    media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "ipu3-imgu 0 output":0[1]
280    media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "ipu3-imgu 0 viewfinder":0[1]
281    media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "ipu3-imgu 0 3a stat":0[1]
282
283Also the pipe mode of the corresponding V4L2 subdev should be set as desired
284(e.g 0 for video mode or 1 for still mode) through the control id 0x009819a1 as
285below.
286
287.. code-block:: none
288
289    yavta -w "0x009819A1 1" /dev/v4l-subdev7
290
291Certain hardware blocks in ImgU pipeline can change the frame resolution by
292cropping or scaling, these hardware blocks include Input Feeder(IF), Bayer Down
293Scaler (BDS) and Geometric Distortion Correction (GDC).
294There is also a block which can change the frame resolution - YUV Scaler, it is
295only applicable to the secondary output.
296
297RAW Bayer frames go through these ImgU pipeline hardware blocks and the final
298processed image output to the DDR memory.
299
300.. kernel-figure::  ipu3_rcb.svg
301   :alt: ipu3 resolution blocks image
302
303   IPU3 resolution change hardware blocks
304
305**Input Feeder**
306
307Input Feeder gets the Bayer frame data from the sensor, it can enable cropping
308of lines and columns from the frame and then store pixels into device's internal
309pixel buffer which are ready to readout by following blocks.
310
311**Bayer Down Scaler**
312
313Bayer Down Scaler is capable of performing image scaling in Bayer domain, the
314downscale factor can be configured from 1X to 1/4X in each axis with
315configuration steps of 0.03125 (1/32).
316
317**Geometric Distortion Correction**
318
319Geometric Distortion Correction is used to perform correction of distortions
320and image filtering. It needs some extra filter and envelope padding pixels to
321work, so the input resolution of GDC should be larger than the output
322resolution.
323
324**YUV Scaler**
325
326YUV Scaler which similar with BDS, but it is mainly do image down scaling in
327YUV domain, it can support up to 1/12X down scaling, but it can not be applied
328to the main output.
329
330The ImgU V4L2 subdev has to be configured with the supported resolutions in all
331the above hardware blocks, for a given input resolution.
332For a given supported resolution for an input frame, the Input Feeder, Bayer
333Down Scaler and GDC blocks should be configured with the supported resolutions
334as each hardware block has its own alignment requirement.
335
336You must configure the output resolution of the hardware blocks smartly to meet
337the hardware requirement along with keeping the maximum field of view. The
338intermediate resolutions can be generated by specific tool -
339
340https://github.com/intel/intel-ipu3-pipecfg
341
342This tool can be used to generate intermediate resolutions. More information can
343be obtained by looking at the following IPU3 ImgU configuration table.
344
345https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master
346
347Under baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss
348directory, graph_settings_ov5670.xml can be used as an example.
349
350The following steps prepare the ImgU pipeline for the image processing.
351
3521. The ImgU V4L2 subdev data format should be set by using the
353VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above.
354
3552. The ImgU V4L2 subdev cropping should be set by using the
356VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the target,
357using the input feeder height and width.
358
3593. The ImgU V4L2 subdev composing should be set by using the
360VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the target,
361using the BDS height and width.
362
363For the ov5670 example, for an input frame with a resolution of 2592x1944
364(which is input to the ImgU subdev pad 0), the corresponding resolutions
365for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920
366respectively.
367
368Once this is done, the received raw Bayer frames can be input to the ImgU
369V4L2 subdev as below, using the open source application v4l2n [#f1]_.
370
371For an image captured with 2592x1944 [#f4]_ resolution, with desired output
372resolution as 2560x1920 and viewfinder resolution as 2560x1920, the following
373v4l2n command can be used. This helps process the raw Bayer frames and produces
374the desired results for the main output image and the viewfinder output, in NV12
375format.
376
377.. code-block:: none
378
379    v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
380          --fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X47337069 \
381          --reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1 \
382          --output=/tmp/frames.out --open=/dev/video5 \
383          --fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 \
384          --reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2 \
385          --output=/tmp/frames.vf --open=/dev/video6 \
386          --fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12 \
387          --reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7 \
388          --output=/tmp/frames.3A --fmt=type:META_CAPTURE,? \
389          --reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5
390
391You can also use yavta [#f2]_ command to do same thing as above:
392
393.. code-block:: none
394
395    yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
396          --file=frame-#.out-f NV12 /dev/video5 & \
397    yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
398          --file=frame-#.vf -f NV12 /dev/video6 & \
399    yavta --data-prefix -Bmeta-capture -c10 -n5 -I \
400          --file=frame-#.3a /dev/video7 & \
401    yavta --data-prefix -Boutput-mplane -c10 -n5 -I -s2592x1944 \
402          --file=/tmp/frame-in.cio2 -f IPU3_SGRBG10 /dev/video4
403
404where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point to
405input, output, viewfinder and 3A statistics video nodes respectively.
406
407Converting the raw Bayer image into YUV domain
408----------------------------------------------
409
410The processed images after the above step, can be converted to YUV domain
411as below.
412
413Main output frames
414~~~~~~~~~~~~~~~~~~
415
416.. code-block:: none
417
418    raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.ppm
419
420where 2560x1920 is output resolution, NV12 is the video format, followed
421by input frame and output PNM file.
422
423Viewfinder output frames
424~~~~~~~~~~~~~~~~~~~~~~~~
425
426.. code-block:: none
427
428    raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.ppm
429
430where 2560x1920 is output resolution, NV12 is the video format, followed
431by input frame and output PNM file.
432
433Example user space code for IPU3
434================================
435
436User space code that configures and uses IPU3 is available here.
437
438https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/
439
440The source can be located under hal/intel directory.
441
442Overview of IPU3 pipeline
443=========================
444
445IPU3 pipeline has a number of image processing stages, each of which takes a
446set of parameters as input. The major stages of pipelines are shown here:
447
448.. kernel-render:: DOT
449   :alt: IPU3 ImgU Pipeline
450   :caption: IPU3 ImgU Pipeline Diagram
451
452   digraph "IPU3 ImgU" {
453       node [shape=box]
454       splines="ortho"
455       rankdir="LR"
456
457       a [label="Raw pixels"]
458       b [label="Bayer Downscaling"]
459       c [label="Optical Black Correction"]
460       d [label="Linearization"]
461       e [label="Lens Shading Correction"]
462       f [label="White Balance / Exposure / Focus Apply"]
463       g [label="Bayer Noise Reduction"]
464       h [label="ANR"]
465       i [label="Demosaicing"]
466       j [label="Color Correction Matrix"]
467       k [label="Gamma correction"]
468       l [label="Color Space Conversion"]
469       m [label="Chroma Down Scaling"]
470       n [label="Chromatic Noise Reduction"]
471       o [label="Total Color Correction"]
472       p [label="XNR3"]
473       q [label="TNR"]
474       r [label="DDR", style=filled, fillcolor=yellow, shape=cylinder]
475       s [label="YUV Downscaling"]
476       t [label="DDR", style=filled, fillcolor=yellow, shape=cylinder]
477
478       { rank=same; a -> b -> c -> d -> e -> f -> g -> h -> i }
479       { rank=same; j -> k -> l -> m -> n -> o -> p -> q -> s -> t}
480
481       a -> j [style=invis, weight=10]
482       i -> j
483       q -> r
484   }
485
486The table below presents a description of the above algorithms.
487
488======================== =======================================================
489Name			 Description
490======================== =======================================================
491Optical Black Correction Optical Black Correction block subtracts a pre-defined
492			 value from the respective pixel values to obtain better
493			 image quality.
494			 Defined in struct ipu3_uapi_obgrid_param.
495Linearization		 This algo block uses linearization parameters to
496			 address non-linearity sensor effects. The Lookup table
497			 table is defined in
498			 struct ipu3_uapi_isp_lin_vmem_params.
499SHD			 Lens shading correction is used to correct spatial
500			 non-uniformity of the pixel response due to optical
501			 lens shading. This is done by applying a different gain
502			 for each pixel. The gain, black level etc are
503			 configured in struct ipu3_uapi_shd_config_static.
504BNR			 Bayer noise reduction block removes image noise by
505			 applying a bilateral filter.
506			 See struct ipu3_uapi_bnr_static_config for details.
507ANR			 Advanced Noise Reduction is a block based algorithm
508			 that performs noise reduction in the Bayer domain. The
509			 convolution matrix etc can be found in
510			 struct ipu3_uapi_anr_config.
511DM			 Demosaicing converts raw sensor data in Bayer format
512			 into RGB (Red, Green, Blue) presentation. Then add
513			 outputs of estimation of Y channel for following stream
514			 processing by Firmware. The struct is defined as
515			 struct ipu3_uapi_dm_config.
516Color Correction	 Color Correction algo transforms sensor specific color
517			 space to the standard "sRGB" color space. This is done
518			 by applying 3x3 matrix defined in
519			 struct ipu3_uapi_ccm_mat_config.
520Gamma correction	 Gamma correction struct ipu3_uapi_gamma_config is a
521			 basic non-linear tone mapping correction that is
522			 applied per pixel for each pixel component.
523CSC			 Color space conversion transforms each pixel from the
524			 RGB primary presentation to YUV (Y: brightness,
525			 UV: Luminance) presentation. This is done by applying
526			 a 3x3 matrix defined in
527			 struct ipu3_uapi_csc_mat_config
528CDS			 Chroma down sampling
529			 After the CSC is performed, the Chroma Down Sampling
530			 is applied for a UV plane down sampling by a factor
531			 of 2 in each direction for YUV 4:2:0 using a 4x2
532			 configurable filter struct ipu3_uapi_cds_params.
533CHNR			 Chroma noise reduction
534			 This block processes only the chrominance pixels and
535			 performs noise reduction by cleaning the high
536			 frequency noise.
537			 See struct struct ipu3_uapi_yuvp1_chnr_config.
538TCC			 Total color correction as defined in struct
539			 struct ipu3_uapi_yuvp2_tcc_static_config.
540XNR3			 eXtreme Noise Reduction V3 is the third revision of
541			 noise reduction algorithm used to improve image
542			 quality. This removes the low frequency noise in the
543			 captured image. Two related structs are  being defined,
544			 struct ipu3_uapi_isp_xnr3_params for ISP data memory
545			 and struct ipu3_uapi_isp_xnr3_vmem_params for vector
546			 memory.
547TNR			 Temporal Noise Reduction block compares successive
548			 frames in time to remove anomalies / noise in pixel
549			 values. struct ipu3_uapi_isp_tnr3_vmem_params and
550			 struct ipu3_uapi_isp_tnr3_params are defined for ISP
551			 vector and data memory respectively.
552======================== =======================================================
553
554Other often encountered acronyms not listed in above table:
555
556	ACC
557		Accelerator cluster
558	AWB_FR
559		Auto white balance filter response statistics
560	BDS
561		Bayer downscaler parameters
562	CCM
563		Color correction matrix coefficients
564	IEFd
565		Image enhancement filter directed
566	Obgrid
567		Optical black level compensation
568	OSYS
569		Output system configuration
570	ROI
571		Region of interest
572	YDS
573		Y down sampling
574	YTM
575		Y-tone mapping
576
577A few stages of the pipeline will be executed by firmware running on the ISP
578processor, while many others will use a set of fixed hardware blocks also
579called accelerator cluster (ACC) to crunch pixel data and produce statistics.
580
581ACC parameters of individual algorithms, as defined by
582struct ipu3_uapi_acc_param, can be chosen to be applied by the user
583space through struct struct ipu3_uapi_flags embedded in
584struct ipu3_uapi_params structure. For parameters that are configured as
585not enabled by the user space, the corresponding structs are ignored by the
586driver, in which case the existing configuration of the algorithm will be
587preserved.
588
589References
590==========
591
592.. [#f5] drivers/staging/media/ipu3/include/uapi/intel-ipu3.h
593
594.. [#f1] https://github.com/intel/nvt
595
596.. [#f2] http://git.ideasonboard.org/yavta.git
597
598.. [#f3] http://git.ideasonboard.org/?p=media-ctl.git;a=summary
599
600.. [#f4] ImgU limitation requires an additional 16x16 for all input resolutions
601