Commit b8726aea authored by Yong Zhi's avatar Yong Zhi Committed by Mauro Carvalho Chehab

media: ipu3: update meta format documentation

Language improvements, fix entity naming, make pipeline a graph and move
device usage documentation to device documentation ipu3.rst.
Signed-off-by: default avatarYong Zhi <yong.zhi@intel.com>
Signed-off-by: default avatarSakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: default avatarMauro Carvalho Chehab <mchehab+samsung@kernel.org>
parent 27e2add8
...@@ -19,8 +19,8 @@ These formats are used for the :ref:`metadata` interface only. ...@@ -19,8 +19,8 @@ These formats are used for the :ref:`metadata` interface only.
.. toctree:: .. toctree::
:maxdepth: 1 :maxdepth: 1
pixfmt-meta-intel-ipu3
pixfmt-meta-d4xx pixfmt-meta-d4xx
pixfmt-meta-intel-ipu3
pixfmt-meta-uvc pixfmt-meta-uvc
pixfmt-meta-vsp1-hgo pixfmt-meta-vsp1-hgo
pixfmt-meta-vsp1-hgt pixfmt-meta-vsp1-hgt
...@@ -30,21 +30,22 @@ ...@@ -30,21 +30,22 @@
V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A ('ip3s') V4L2_META_FMT_IPU3_PARAMS ('ip3p'), V4L2_META_FMT_IPU3_3A ('ip3s')
****************************************************************** ******************************************************************
.. c:type:: ipu3_uapi_stats_3a .. ipu3_uapi_stats_3a
3A statistics 3A statistics
============= =============
For IPU3 ImgU, the 3A statistics accelerators collect different statistics over The IPU3 ImgU 3A statistics accelerators collect different statistics over
an input bayer frame. Those statistics, defined in data struct :c:type:`ipu3_uapi_stats_3a`, an input Bayer frame. Those statistics are obtained from the "ipu3-imgu [01] 3a
are obtained from "ipu3-imgu 3a stat" metadata capture video node, which are then stat" metadata capture video nodes, using the :c:type:`v4l2_meta_format`
passed to user space for statistics analysis using :c:type:`v4l2_meta_format` interface. interface. They are formatted as described by the :c:type:`ipu3_uapi_stats_3a`
structure.
The statistics collected are AWB (Auto-white balance) RGBS (Red, Green, Blue and The statistics collected are AWB (Auto-white balance) RGBS (Red, Green, Blue and
Saturation measure) cells, AWB filter response, AF (Auto-focus) filter response, Saturation measure) cells, AWB filter response, AF (Auto-focus) filter response,
and AE (Auto-exposure) histogram. and AE (Auto-exposure) histogram.
struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all above. The struct :c:type:`ipu3_uapi_4a_config` saves all configurable parameters.
.. code-block:: c .. code-block:: c
...@@ -60,105 +61,14 @@ struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all above ...@@ -60,105 +61,14 @@ struct :c:type:`ipu3_uapi_4a_config` saves configurable parameters for all above
struct ipu3_uapi_ff_status stats_3a_status; struct ipu3_uapi_ff_status stats_3a_status;
}; };
.. c:type:: ipu3_uapi_params .. ipu3_uapi_params
Pipeline parameters Pipeline parameters
=================== ===================
IPU3 pipeline has a number of image processing stages, each of which takes a The pipeline parameters are passed to the "ipu3-imgu [01] parameters" metadata
set of parameters as input. The major stages of pipelines are shown here: output video nodes, using the :c:type:`v4l2_meta_format` interface. They are
formatted as described by the :c:type:`ipu3_uapi_params` structure.
Raw pixels -> Bayer Downscaling -> Optical Black Correction ->
Linearization -> Lens Shading Correction -> White Balance / Exposure /
Focus Apply -> Bayer Noise Reduction -> ANR -> Demosaicing -> Color
Correction Matrix -> Gamma correction -> Color Space Conversion ->
Chroma Down Scaling -> Chromatic Noise Reduction -> Total Color
Correction -> XNR3 -> TNR -> DDR
The table below presents a description of the above algorithms.
======================== =======================================================
Name Description
======================== =======================================================
Optical Black Correction Optical Black Correction block subtracts a pre-defined
value from the respective pixel values to obtain better
image quality.
Defined in :c:type:`ipu3_uapi_obgrid_param`.
Linearization This algo block uses linearization parameters to
address non-linearity sensor effects. The Lookup table
table is defined in
:c:type:`ipu3_uapi_isp_lin_vmem_params`.
SHD Lens shading correction is used to correct spatial
non-uniformity of the pixel response due to optical
lens shading. This is done by applying a different gain
for each pixel. The gain, black level etc are
configured in :c:type:`ipu3_uapi_shd_config_static`.
BNR Bayer noise reduction block removes image noise by
applying a bilateral filter.
See :c:type:`ipu3_uapi_bnr_static_config` for details.
ANR Advanced Noise Reduction is a block based algorithm
that performs noise reduction in the Bayer domain. The
convolution matrix etc can be found in
:c:type:`ipu3_uapi_anr_config`.
Demosaicing Demosaicing converts raw sensor data in Bayer format
into RGB (Red, Green, Blue) presentation. Then add
outputs of estimation of Y channel for following stream
processing by Firmware. The struct is defined as
:c:type:`ipu3_uapi_dm_config`. (TODO)
Color Correction Color Correction algo transforms sensor specific color
space to the standard "sRGB" color space. This is done
by applying 3x3 matrix defined in
:c:type:`ipu3_uapi_ccm_mat_config`.
Gamma correction Gamma correction :c:type:`ipu3_uapi_gamma_config` is a
basic non-linear tone mapping correction that is
applied per pixel for each pixel component.
CSC Color space conversion transforms each pixel from the
RGB primary presentation to YUV (Y: brightness,
UV: Luminance) presentation. This is done by applying
a 3x3 matrix defined in
:c:type:`ipu3_uapi_csc_mat_config`
CDS Chroma down sampling
After the CSC is performed, the Chroma Down Sampling
is applied for a UV plane down sampling by a factor
of 2 in each direction for YUV 4:2:0 using a 4x2
configurable filter :c:type:`ipu3_uapi_cds_params`.
CHNR Chroma noise reduction
This block processes only the chrominance pixels and
performs noise reduction by cleaning the high
frequency noise.
See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
TCC Total color correction as defined in struct
:c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
XNR3 eXtreme Noise Reduction V3 is the third revision of
noise reduction algorithm used to improve image
quality. This removes the low frequency noise in the
captured image. Two related structs are being defined,
:c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory
and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector
memory.
TNR Temporal Noise Reduction block compares successive
frames in time to remove anomalies / noise in pixel
values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and
:c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP
vector and data memory respectively.
======================== =======================================================
A few stages of the pipeline will be executed by firmware running on the ISP
processor, while many others will use a set of fixed hardware blocks also
called accelerator cluster (ACC) to crunch pixel data and produce statistics.
ACC parameters of individual algorithms, as defined by
:c:type:`ipu3_uapi_acc_param`, can be chosen to be applied by the user
space through struct :c:type:`ipu3_uapi_flags` embedded in
:c:type:`ipu3_uapi_params` structure. For parameters that are configured as
not enabled by the user space, the corresponding structs are ignored by the
driver, in which case the existing configuration of the algorithm will be
preserved.
Both 3A statistics and pipeline parameters described here are closely tied to Both 3A statistics and pipeline parameters described here are closely tied to
the underlying camera sub-system (CSS) APIs. They are usually consumed and the underlying camera sub-system (CSS) APIs. They are usually consumed and
...@@ -166,13 +76,6 @@ produced by dedicated user space libraries that comprise the important tuning ...@@ -166,13 +76,6 @@ produced by dedicated user space libraries that comprise the important tuning
tools, thus freeing the developers from being bothered with the low level tools, thus freeing the developers from being bothered with the low level
hardware and algorithm details. hardware and algorithm details.
It should be noted that IPU3 DMA operations require the addresses of all data
structures (that includes both input and output) to be aligned on 32 byte
boundaries.
The meta data :c:type:`ipu3_uapi_params` will be sent to "ipu3-imgu parameters"
video node in ``V4L2_BUF_TYPE_META_CAPTURE`` format.
.. code-block:: c .. code-block:: c
struct ipu3_uapi_params { struct ipu3_uapi_params {
......
...@@ -357,6 +357,153 @@ https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/ ...@@ -357,6 +357,153 @@ https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/
The source can be located under hal/intel directory. The source can be located under hal/intel directory.
Overview of IPU3 pipeline
=========================
IPU3 pipeline has a number of image processing stages, each of which takes a
set of parameters as input. The major stages of pipelines are shown here:
.. kernel-render:: DOT
:alt: IPU3 ImgU Pipeline
:caption: IPU3 ImgU Pipeline Diagram
digraph "IPU3 ImgU" {
node [shape=box]
splines="ortho"
rankdir="LR"
a [label="Raw pixels"]
b [label="Bayer Downscaling"]
c [label="Optical Black Correction"]
d [label="Linearization"]
e [label="Lens Shading Correction"]
f [label="White Balance / Exposure / Focus Apply"]
g [label="Bayer Noise Reduction"]
h [label="ANR"]
i [label="Demosaicing"]
j [label="Color Correction Matrix"]
k [label="Gamma correction"]
l [label="Color Space Conversion"]
m [label="Chroma Down Scaling"]
n [label="Chromatic Noise Reduction"]
o [label="Total Color Correction"]
p [label="XNR3"]
q [label="TNR"]
r [label="DDR"]
{ rank=same; a -> b -> c -> d -> e -> f }
{ rank=same; g -> h -> i -> j -> k -> l }
{ rank=same; m -> n -> o -> p -> q -> r }
a -> g -> m [style=invis, weight=10]
f -> g
l -> m
}
The table below presents a description of the above algorithms.
======================== =======================================================
Name Description
======================== =======================================================
Optical Black Correction Optical Black Correction block subtracts a pre-defined
value from the respective pixel values to obtain better
image quality.
Defined in :c:type:`ipu3_uapi_obgrid_param`.
Linearization This algo block uses linearization parameters to
address non-linearity sensor effects. The Lookup table
table is defined in
:c:type:`ipu3_uapi_isp_lin_vmem_params`.
SHD Lens shading correction is used to correct spatial
non-uniformity of the pixel response due to optical
lens shading. This is done by applying a different gain
for each pixel. The gain, black level etc are
configured in :c:type:`ipu3_uapi_shd_config_static`.
BNR Bayer noise reduction block removes image noise by
applying a bilateral filter.
See :c:type:`ipu3_uapi_bnr_static_config` for details.
ANR Advanced Noise Reduction is a block based algorithm
that performs noise reduction in the Bayer domain. The
convolution matrix etc can be found in
:c:type:`ipu3_uapi_anr_config`.
DM Demosaicing converts raw sensor data in Bayer format
into RGB (Red, Green, Blue) presentation. Then add
outputs of estimation of Y channel for following stream
processing by Firmware. The struct is defined as
:c:type:`ipu3_uapi_dm_config`.
Color Correction Color Correction algo transforms sensor specific color
space to the standard "sRGB" color space. This is done
by applying 3x3 matrix defined in
:c:type:`ipu3_uapi_ccm_mat_config`.
Gamma correction Gamma correction :c:type:`ipu3_uapi_gamma_config` is a
basic non-linear tone mapping correction that is
applied per pixel for each pixel component.
CSC Color space conversion transforms each pixel from the
RGB primary presentation to YUV (Y: brightness,
UV: Luminance) presentation. This is done by applying
a 3x3 matrix defined in
:c:type:`ipu3_uapi_csc_mat_config`
CDS Chroma down sampling
After the CSC is performed, the Chroma Down Sampling
is applied for a UV plane down sampling by a factor
of 2 in each direction for YUV 4:2:0 using a 4x2
configurable filter :c:type:`ipu3_uapi_cds_params`.
CHNR Chroma noise reduction
This block processes only the chrominance pixels and
performs noise reduction by cleaning the high
frequency noise.
See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
TCC Total color correction as defined in struct
:c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
XNR3 eXtreme Noise Reduction V3 is the third revision of
noise reduction algorithm used to improve image
quality. This removes the low frequency noise in the
captured image. Two related structs are being defined,
:c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory
and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector
memory.
TNR Temporal Noise Reduction block compares successive
frames in time to remove anomalies / noise in pixel
values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and
:c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP
vector and data memory respectively.
======================== =======================================================
Other often encountered acronyms not listed in above table:
ACC
Accelerator cluster
AWB_FR
Auto white balance filter response statistics
BDS
Bayer downscaler parameters
CCM
Color correction matrix coefficients
IEFd
Image enhancement filter directed
Obgrid
Optical black level compensation
OSYS
Output system configuration
ROI
Region of interest
YDS
Y down sampling
YTM
Y-tone mapping
A few stages of the pipeline will be executed by firmware running on the ISP
processor, while many others will use a set of fixed hardware blocks also
called accelerator cluster (ACC) to crunch pixel data and produce statistics.
ACC parameters of individual algorithms, as defined by
:c:type:`ipu3_uapi_acc_param`, can be chosen to be applied by the user
space through struct :c:type:`ipu3_uapi_flags` embedded in
:c:type:`ipu3_uapi_params` structure. For parameters that are configured as
not enabled by the user space, the corresponding structs are ignored by the
driver, in which case the existing configuration of the algorithm will be
preserved.
References References
========== ==========
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment