1.. _color_map_optimization: 2 3Color Map Optimization 4------------------------------------- 5 6Consider color mapping to the geometry reconstructed from depth cameras. As color and depth frames are not perfectly aligned, the texture mapping using color images is subject to results in blurred color map. Open3D provides color map optimization method proposed by [Zhou2014]_. Before begin, download fountain dataset from `here <https://drive.google.com/open?id=1eT45y8qw3TLED2YY9-K1Ot6dQuF9GDPJ>`_. The following script shows an example of color map optimization. 7 8.. code-block:: python 9 10 from py3d import * 11 from trajectory_io import * 12 import os, sys 13 sys.path.append("../Utility") 14 from common import * 15 16 path = "[set_this_path_to_fountain_dataset]" 17 debug_mode = False 18 19 if __name__ == "__main__": 20 set_verbosity_level(VerbosityLevel.Debug) 21 22 # Read RGBD images 23 rgbd_images = [] 24 depth_image_path = get_file_list( 25 os.path.join(path, "depth/"), extension = ".png") 26 color_image_path = get_file_list( 27 os.path.join(path, "image/"), extension = ".jpg") 28 assert(len(depth_image_path) == len(color_image_path)) 29 for i in range(len(depth_image_path)): 30 depth = read_image(os.path.join(depth_image_path[i])) 31 color = read_image(os.path.join(color_image_path[i])) 32 rgbd_image = create_rgbd_image_from_color_and_depth(color, depth, 33 convert_rgb_to_intensity = False) 34 if debug_mode: 35 pcd = create_point_cloud_from_rgbd_image(rgbd_image, 36 PinholeCameraIntrinsic.get_prime_sense_default()) 37 draw_geometries([pcd]) 38 rgbd_images.append(rgbd_image) 39 40 # Read camera pose and mesh 41 camera = read_pinhole_camera_trajectory(os.path.join(path, "scene/key.log")) 42 mesh = read_triangle_mesh(os.path.join(path, "scene", "integrated.ply")) 43 44 # Before full optimization, let's just visualize texture map 45 # with given geometry, RGBD images, and camera poses. 46 option = ColorMapOptmizationOption() 47 option.maximum_iteration = 0 48 color_map_optimization(mesh, rgbd_images, camera, option) 49 draw_geometries([mesh]) 50 write_triangle_mesh(os.path.join(path, "scene", 51 "color_map_before_optimization.ply"), mesh) 52 53 # Optimize texture and save the mesh as texture_mapped.ply 54 # This is implementation of following paper 55 # Q.-Y. Zhou and V. Koltun, 56 # Color Map Optimization for 3D Reconstruction with Consumer Depth Cameras, 57 # SIGGRAPH 2014 58 option.maximum_iteration = 500 59 option.non_rigid_camera_coordinate = True 60 color_map_optimization(mesh, rgbd_images, camera, option) 61 draw_geometries([mesh]) 62 write_triangle_mesh(os.path.join(path, "scene", 63 "color_map_after_optimization.ply"), mesh) 64 65Input 66```````````````````````` 67 68.. code-block:: python 69 70 # read RGBD images 71 rgbd_images = [] 72 depth_image_path = get_file_list( 73 os.path.join(path, "depth/"), extension=".png") 74 color_image_path = get_file_list( 75 os.path.join(path, "image/"), extension=".jpg") 76 assert(len(depth_image_path) == len(color_image_path)) 77 for i in range(len(depth_image_path)): 78 depth = read_image(os.path.join(depth_image_path[i])) 79 color = read_image(os.path.join(color_image_path[i])) 80 rgbd_image = create_rgbd_image_from_color_and_depth(color, depth, 81 convert_rgb_to_intensity=False) 82 if debug_mode: 83 pcd = create_point_cloud_from_rgbd_image(rgbd_image, 84 PinholeCameraIntrinsic.get_prime_sense_default()) 85 draw_geometries([pcd]) 86 rgbd_images.append(rgbd_image) 87 88This script reads color and depth image pairs and makes ``rgbd_image``. Note that ``convert_rgb_to_intensity`` flag is ``False``. This is to preserve 8-bit color channels instead of using single channel float type image. 89 90It is always good practice to visualize RGBD image before applying it to color map optimization. ``debug_mode`` switch is for visualizing RGBD image. 91 92.. code-block:: python 93 94 # read camera pose and mesh 95 camera = read_pinhole_camera_trajectory(os.path.join(path, "scene/key.log")) 96 mesh = read_triangle_mesh(os.path.join(path, "scene", "integrated.ply")) 97 98The script reads camera trajectory and mesh. 99 100.. code-block:: python 101 102 option = ColorMapOptmizationOption() 103 option.maximum_iteration = 0 104 color_map_optimization(mesh, rgbd_images, camera, option) 105 draw_geometries([mesh]) 106 write_triangle_mesh(os.path.join(path, "scene", 107 "color_map_before_optimization.ply"), mesh) 108 109To visualize how the camera poses are not good for color mapping, this script intentionally set the iteration number as 0, which means no optimization. ``color_map_optimization`` paints a mesh using corresponding RGBD images and camera poses. Without optimization, the texture map is blurred. 110 111.. image:: ../../_static/Advanced/color_map_optimization/initial.png 112 :width: 300px 113 114.. image:: ../../_static/Advanced/color_map_optimization/initial_zoom.png 115 :width: 300px 116 117Rigid Optimization 118``````````````````````````````` 119 120The next step is to optimize camera poses to get a sharp color map. 121 122.. code-block:: python 123 124 option.maximum_iteration = 500 125 option.non_rigid_camera_coordinate = True 126 color_map_optimization(mesh, rgbd_images, camera, option) 127 draw_geometries([mesh]) 128 write_triangle_mesh(os.path.join(path, "scene", 129 "color_map_after_optimization.ply"), mesh) 130 131The script sets ``maximum_iteration = 500`` for actual iterations. The optimization displays the following energy profile. 132 133.. code-block:: shell 134 135 [ColorMapOptimization] :: Rigid Optimization 136 [Iteration 0001] Residual error : 25777.372725 (avg : 0.004998) 137 [Iteration 0002] Residual error : 25620.681829 (avg : 0.004967) 138 [Iteration 0003] Residual error : 25463.806101 (avg : 0.004937) 139 : 140 [Iteration 0498] Residual error : 11550.014763 (avg : 0.002255) 141 [Iteration 0499] Residual error : 11549.850827 (avg : 0.002255) 142 [Iteration 0500] Residual error : 11550.062068 (avg : 0.002255) 143 144Residual error implies inconsistency of image intensities. Lower residual leads better color map quality. By default, ``ColorMapOptmizationOption`` enables rigid optimization. It optimizes 6-dimentional pose of every cameras. 145 146.. image:: ../../_static/Advanced/color_map_optimization/rigid.png 147 :width: 300px 148 149.. image:: ../../_static/Advanced/color_map_optimization/rigid_zoom.png 150 :width: 300px 151 152Non-rigid Optimization 153``````````````````````````````````` 154 155For better alignment quality, there is an option for non-rigid optimization. To enable, simply add 156 157.. code-block:: python 158 159 option.non_rigid_camera_coordinate = True 160 161before calling ``color_map_optimization``. Besides 6-dimentional camera poses, non-rigid optimization even consider local image warping represented by anchor points. This adds even more flexibility and leads higher quality color mapping. The residual error is smaller than the case of rigid optimization. 162 163.. code-block:: shell 164 165 [ColorMapOptimization] :: Non-Rigid Optimization 166 [Iteration 0001] Residual error : 25777.372725, reg : 0.000000 167 [Iteration 0002] Residual error : 25330.445704, reg : 13.005639 168 [Iteration 0003] Residual error : 24885.912182, reg : 40.000765 169 : 170 [Iteration 0498] Residual error : 7585.606850, reg : 3294.124184 171 [Iteration 0499] Residual error : 7585.274846, reg : 3294.887659 172 [Iteration 0500] Residual error : 7583.972930, reg : 3294.634065 173 174Results of non-rigid optimization follow. 175 176.. image:: ../../_static/Advanced/color_map_optimization/non_rigid.png 177 :width: 300px 178 179.. image:: ../../_static/Advanced/color_map_optimization/non_rigid_zoom.png 180 :width: 300px 181