Testing spatial patterns for acquiring shape and subsurface scattering properties

Models of both the shape and material properties of physical objects are needed in many computer graphics applications. In many design applications, even if shape is not needed, it is desirable to start with the material properties of existing non-planar objects. We consider the design of a system to capture both shape and appearance of objects. We focus particularly on objects that exhibit significant subsurface scattering and inter-reflection effects. We present preliminary results from a system that uses coded light from a set of small, inexpensive projectors coupled with commodity digital cameras. Introduction We consider the problem of capturing digital models of the shape and material properties of physical objects. These models are needed in a variety of applications. A digital model with both shape and material is needed to visualize an object in a virtual scene in computer graphics. Material properties are needed in design tasks in which a new object is to be created with a material “look” similar to existing objects. Even if the shape is not of interest, if the object is not flat the shape of the object must be estimated to extract the material properties. Many methods have been developed to scan both shape and appearance. However these methods produce errors in both the shape and material properties when there is subsurface scattering in the material. In this project we build on recent work in separating light paths using spatially coded patterns of light to estimate both shape and material. Our goal is to have an inexpensive capture system that produces models that can be rendered in a computer graphics system with visually acceptable results. Background and Previous Work An image of an object is a function of the object itself, the incident lighting and the properties of the imaging system, as shown in Figure 1. Systems for capturing shape and material properties fundamentally depend on following light paths from a light source, to the object, and then to an camera sensor. The paths depend on the object shape and material scattering properties. Most systems assume simple paths, such as the one shown in Figure 1. Knowing the light source and camera positions, and the direction from each to a point on the object allows the calculation of the point position using triangulation. Knowing the magnitude of the reflected light can be used to estimate the surface reflectance property. The accuracy of the 3D point position depends on accurately locating the 2D point, shown in bright red in Figure 1, in the camera image. Locating the point in the image however can be difficult when the object diffuses the light into a region around the point of incidence as a result of subsurface scattering. Rather than a precise point on the image there is a blurred spot, as represented in pink in the figure. The appearance of the blurred spot also indicates that the material property can not be represented by a simple surface reflectance. Figure 1. Shape and material capture systems depend on estimating/controlling the parameters of object illumination and the imaging device. With the availability of inexpensive digital cameras (in particular the high resolution cameras on smart phones) and scanners (such as the NextEngineTM and KinectTM there has been a great deal of interest in assembling systems for capturing digital content for 3D graphics applications, rather than modeling content from scratch. Early work in shape and material properties used laser scanners, cameras and lights [1]. Recent work includes limited acquisition (large, flat samples with statistically stationary textures) using just a smart phone camera [2] , simultaneous shape and surface reflectance capture with an RGB-D sensor [3], and sophisticated separation of light paths using an interferometer [4]. With the exception of [4] these systems do not account for the errors in shape introduced by subsurface scattering, and do not attempt to estimate subsurface scattering properties. In our project we seek a simple, inexpensive system that accounts for subsurface scattering. In previous work [5] we considered the use of spatial patterns for estimating material properties for near-flat surfaces. We made use of the seminal work presented by Nayar et al. [6] for separating direct and indirection illumination effects in images by using projected spatial patterns. In addition to verifying the effects originally demonstrated in [6] we showed that indirect effects can be further separated by using direct/indirect separations from multiple illumination directions. Further we showed that while subsurface scattering effects are never spatially uniform, direct/indirect separations can be used to produce spatially maps on the material surface for where subsurface scattering is significant. Finally we proposed that subsurface scattering parameters could be estimated by iteratively comparing images formed with light patterns and synthetic images of the material generated with varying scattering parameters. We seek to extend this work to arbitrarily shaped objects. Since we use projected patterns for examining the material properties, we wish to use projected patterns to estimate the shape as well. The use of light patterns for shape acquisition has been studied extensively [7], and techniques such as temporal sequences of binary patterns have been used for decades [8]. In work following [6], Gu et al. [9] demonstrated that direct/indirect separation using spatial patterns could improve a wide range of shape measurement techniques developed in the field of computer vision. Arguing that separation techniques require large numbers of images and are vulnerable to noise, some subsequent research on shape acquisition has focused on creating robust patterns for shape that do not depend on explicit path separation [10, 11]. These projects however do not consider the simultaneous recovery of material properties, in particular subsurface scatting properties. We return to the idea of direct/indirect separation for sequences of binary patterns, with the goal of acquire both shape and subsurface scattering properties. System Overview The system we use for our experiments is an upgraded version of the setup we described in [5]. We have upgraded components of the system used in that system based on our experience with the captured results, and extend our processing pipeline to the estimation of arbitrary shapes.