1.. _color_map_optimization:
2
3Color Map Optimization
4-------------------------------------
5
6Consider color mapping to the geometry reconstructed from depth cameras. As color and depth frames are not perfectly aligned, the texture mapping using color images is subject to results in blurred color map. Open3D provides color map optimization method proposed by [Zhou2014]_. Before begin, download fountain dataset from `here <https://drive.google.com/open?id=1eT45y8qw3TLED2YY9-K1Ot6dQuF9GDPJ>`_. The following script shows an example of color map optimization.
7
8.. code-block:: python
9
10    from py3d import *
11    from trajectory_io import *
12    import os, sys
13    sys.path.append("../Utility")
14    from common import *
15
16    path = "[set_this_path_to_fountain_dataset]"
17    debug_mode = False
18
19        if __name__ == "__main__":
20            set_verbosity_level(VerbosityLevel.Debug)
21
22            # Read RGBD images
23            rgbd_images = []
24            depth_image_path = get_file_list(
25                    os.path.join(path, "depth/"), extension = ".png")
26            color_image_path = get_file_list(
27                    os.path.join(path, "image/"), extension = ".jpg")
28            assert(len(depth_image_path) == len(color_image_path))
29            for i in range(len(depth_image_path)):
30                depth = read_image(os.path.join(depth_image_path[i]))
31                color = read_image(os.path.join(color_image_path[i]))
32                rgbd_image = create_rgbd_image_from_color_and_depth(color, depth,
33                        convert_rgb_to_intensity = False)
34                if debug_mode:
35                    pcd = create_point_cloud_from_rgbd_image(rgbd_image,
36                            PinholeCameraIntrinsic.get_prime_sense_default())
37                    draw_geometries([pcd])
38                rgbd_images.append(rgbd_image)
39
40            # Read camera pose and mesh
41            camera = read_pinhole_camera_trajectory(os.path.join(path, "scene/key.log"))
42            mesh = read_triangle_mesh(os.path.join(path, "scene", "integrated.ply"))
43
44            # Before full optimization, let's just visualize texture map
45            # with given geometry, RGBD images, and camera poses.
46            option = ColorMapOptmizationOption()
47            option.maximum_iteration = 0
48            color_map_optimization(mesh, rgbd_images, camera, option)
49            draw_geometries([mesh])
50            write_triangle_mesh(os.path.join(path, "scene",
51                "color_map_before_optimization.ply"), mesh)
52
53            # Optimize texture and save the mesh as texture_mapped.ply
54            # This is implementation of following paper
55            # Q.-Y. Zhou and V. Koltun,
56            # Color Map Optimization for 3D Reconstruction with Consumer Depth Cameras,
57            # SIGGRAPH 2014
58            option.maximum_iteration = 500
59            option.non_rigid_camera_coordinate = True
60            color_map_optimization(mesh, rgbd_images, camera, option)
61            draw_geometries([mesh])
62            write_triangle_mesh(os.path.join(path, "scene",
63                "color_map_after_optimization.ply"), mesh)
64
65Input
66````````````````````````
67
68.. code-block:: python
69
70    # read RGBD images
71    rgbd_images = []
72    depth_image_path = get_file_list(
73            os.path.join(path, "depth/"), extension=".png")
74    color_image_path = get_file_list(
75            os.path.join(path, "image/"), extension=".jpg")
76    assert(len(depth_image_path) == len(color_image_path))
77    for i in range(len(depth_image_path)):
78        depth = read_image(os.path.join(depth_image_path[i]))
79        color = read_image(os.path.join(color_image_path[i]))
80        rgbd_image = create_rgbd_image_from_color_and_depth(color, depth,
81                convert_rgb_to_intensity=False)
82        if debug_mode:
83            pcd = create_point_cloud_from_rgbd_image(rgbd_image,
84                    PinholeCameraIntrinsic.get_prime_sense_default())
85            draw_geometries([pcd])
86        rgbd_images.append(rgbd_image)
87
88This script reads color and depth image pairs and makes ``rgbd_image``. Note that ``convert_rgb_to_intensity`` flag is ``False``. This is to preserve 8-bit color channels instead of using single channel float type image.
89
90It is always good practice to visualize RGBD image before applying it to color map optimization. ``debug_mode`` switch is for visualizing RGBD image.
91
92.. code-block:: python
93
94    # read camera pose and mesh
95    camera = read_pinhole_camera_trajectory(os.path.join(path, "scene/key.log"))
96    mesh = read_triangle_mesh(os.path.join(path, "scene", "integrated.ply"))
97
98The script reads camera trajectory and mesh.
99
100.. code-block:: python
101
102    option = ColorMapOptmizationOption()
103    option.maximum_iteration = 0
104    color_map_optimization(mesh, rgbd_images, camera, option)
105    draw_geometries([mesh])
106    write_triangle_mesh(os.path.join(path, "scene",
107        "color_map_before_optimization.ply"), mesh)
108
109To visualize how the camera poses are not good for color mapping, this script intentionally set the iteration number as 0, which means no optimization. ``color_map_optimization`` paints a mesh using corresponding RGBD images and camera poses. Without optimization, the texture map is blurred.
110
111.. image:: ../../_static/Advanced/color_map_optimization/initial.png
112    :width: 300px
113
114.. image:: ../../_static/Advanced/color_map_optimization/initial_zoom.png
115    :width: 300px
116
117Rigid Optimization
118```````````````````````````````
119
120The next step is to optimize camera poses to get a sharp color map.
121
122.. code-block:: python
123
124    option.maximum_iteration = 500
125    option.non_rigid_camera_coordinate = True
126    color_map_optimization(mesh, rgbd_images, camera, option)
127    draw_geometries([mesh])
128    write_triangle_mesh(os.path.join(path, "scene",
129        "color_map_after_optimization.ply"), mesh)
130
131The script sets ``maximum_iteration = 500`` for actual iterations. The optimization displays the following energy profile.
132
133.. code-block:: shell
134
135    [ColorMapOptimization] :: Rigid Optimization
136    [Iteration 0001] Residual error : 25777.372725 (avg : 0.004998)
137    [Iteration 0002] Residual error : 25620.681829 (avg : 0.004967)
138    [Iteration 0003] Residual error : 25463.806101 (avg : 0.004937)
139    :
140    [Iteration 0498] Residual error : 11550.014763 (avg : 0.002255)
141    [Iteration 0499] Residual error : 11549.850827 (avg : 0.002255)
142    [Iteration 0500] Residual error : 11550.062068 (avg : 0.002255)
143
144Residual error implies inconsistency of image intensities. Lower residual leads better color map quality. By default, ``ColorMapOptmizationOption`` enables rigid optimization. It optimizes 6-dimentional pose of every cameras.
145
146.. image:: ../../_static/Advanced/color_map_optimization/rigid.png
147    :width: 300px
148
149.. image:: ../../_static/Advanced/color_map_optimization/rigid_zoom.png
150    :width: 300px
151
152Non-rigid Optimization
153```````````````````````````````````
154
155For better alignment quality, there is an option for non-rigid optimization. To enable, simply add
156
157.. code-block:: python
158
159    option.non_rigid_camera_coordinate = True
160
161before calling ``color_map_optimization``. Besides 6-dimentional camera poses, non-rigid optimization even consider local image warping represented by anchor points. This adds even more flexibility and leads higher quality color mapping. The residual error is smaller than the case of rigid optimization.
162
163.. code-block:: shell
164
165    [ColorMapOptimization] :: Non-Rigid Optimization
166    [Iteration 0001] Residual error : 25777.372725, reg : 0.000000
167    [Iteration 0002] Residual error : 25330.445704, reg : 13.005639
168    [Iteration 0003] Residual error : 24885.912182, reg : 40.000765
169    :
170    [Iteration 0498] Residual error : 7585.606850, reg : 3294.124184
171    [Iteration 0499] Residual error : 7585.274846, reg : 3294.887659
172    [Iteration 0500] Residual error : 7583.972930, reg : 3294.634065
173
174Results of non-rigid optimization follow.
175
176.. image:: ../../_static/Advanced/color_map_optimization/non_rigid.png
177    :width: 300px
178
179.. image:: ../../_static/Advanced/color_map_optimization/non_rigid_zoom.png
180    :width: 300px
181