A light field of Michelangelo's statue of Night

Photographs and text by Marc Levoy and Jonathan Shade
May 25, 1999

Although we came to Italy mainly to digitize statues, we also had in mind acquiring at least one really big light field (RBLF) during our year abroad. After spending several weeks in the Medici Chapel struggling to scan Michelangelo's four allegories - they are shiny, full of narrow crevices, and and very close to the wall behind them, we decided it was time to try something different.

What is a light field?

Light fields, sometimes called Lumigraphs, are an example of image-based rendering (IBR), a relatively new idea in the graphics community. They are based on the notion of generating (i.e. rendering) new images of a virtual scene from existing images rather than from 3D models. A light field can be thought of as an array of perspective images of a scene taken from viewpoints spaced closely together on a 2D plane. If the spacing between the viewpoints is sufficiently dense, and the views themselves are sufficiently wide-angle, then the array contains within it a measurement of the light leaving every point in the scene and traveling in every direction, literally the field of light surrounding the scene. By extracting pixels from these images - sometimes only a few pixels from each, it is possible to construct correct perspective images for viewpoints other than those present in the original array.

For viewpoints spaced around a circle, a "flatland" light field can be visualized using this simple diagram. The blue blob represents the scene. Each red dot is the viewpoint corresponding to one image of this scene. Each black line segment emanating from a red dot is a pixel in one of these images. It represents a unique line of sight. The yellow dot shows a new viewpoint. Its image can be constructed by borrowing pixels from the existing images, as the diagram shows. If the red dots are sufficiently dense, then new images can be constructed for any viewpoint lying outside the convex hull of the scene, even viewpoints lying outside the circle.

The advantages of light fields over 3D computer models are that rendering is cheap - just shuffling pixels, its cost is independent of object complexity, and the resulting images are completely photorealistic. The disadvantages of light fields are that the scene geometry and its lighting cannot change. Light fields also require a lot of memory, although in some cases no more than a canned video sequence, and they offer interactivity, which a canned video lacks. For more information on light fields and light field rendering, look at Levoy and Hanrahan, Proc. Siggraph '96, or click here.

Acquiring a light field of a statue

There are many technical choices to be made when designing a light field. In our case the first choice to be made was: which statue? We looked for one whose light field was intrinsically interesting and hard to capture with a 3D model. Michelangelo's allegorical statue of Night (shown at left) is both highly polished and slightly transparent, leading to a variety of subtle, hard-to-render visual effects. We also looked for a statue whose lighting we could control and, more importantly, freeze for several days. We could do this on Night, but to avoid changes in natural lighting and the inevitable flashbulbs of tourists, we had to work only at night. Finally, we looked for a statue that we were also planning to digitize with our laser scanner. We have done this on Night. The resulting combination of a high-resolution 3D model and a dense light field will provide a unique dataset for exploring a variety of image-based rendering algorithms.
The next choice to make was: where should we stand? It wasn't practical to completely encircle the statue with viewpoints. Instead, we decided to make a 90-degree arc around the statue while standing about 8 feet away. In this photograph the arc can be seen taped out on the floor. To acquire the light field, we removed the 3D scanner head from our motorized gantry and installed in its place a high-quality digital still camera. The camera is positioned roughly at eye level, which is also the height Michelangelo designed the statue to be seen from.

We divided this arc into 7 "light field slabs", shown in this plan view by yellow line segments. Each slab is an array of images taken from viewpoints closely spaced on a vertical "camera plane" that coincides with the line segment. A conventional optical system with fixed field of view and fixed focus was used, and the directions of view, shown by orange arrows on the figure, were chosen by hand to keep the statue nicely framed. Thus, unlike the light field slabs described in our 1996 paper, the perspective views were conventional rather than off-axis, the focal plane rotated instead of being fixed, and the center of attention moved instead of being stationary. It is possible to generate canonical two-plane light field slabs from this data, but it would require reprojection and resampling of each acquired image.

The camera plane measured 775mm wide x 700mm high, overlapping its neighbor 75mm. This is large enough to allow a reasonable range of observer motion during light field rendering: up, down, forward (i.e. towards the statue) backward, as well as left and right along the arc. Within this plane, we moved the camera through a grid of 62 x 56 viewpoints spaced 12.5mm apart. The total number of images acquired was thus 3,472 per light field slab, or 24,304 for the entire light field. Since the physical aperture of our camera was considerably smaller than 12.5mm, the light field is not correctly "anti-aliased", but we could not avoid this without acquiring far more images than we did.

Although the exposure time per frame was short, 1/12 second, additional time was required to download and store the image and move the camera to the next position. Since the direction of view varied with each image, all four motion axes (horizontal, vertical, pan, and pitch, shown here by green arrows) were in play at all times. Fortunately, our gantry is stiff, so we didn't have to wait for vibrations to die down before shooting the next frame. Nevertheless, each light slab took almost 5 hours to acquire. Shooting 2 slabs per night (working from dusk to dawn), the entire light field took 4 nights to acquire. Image resolution was 1300 x 1030 RGB pixels, or 95GB uncompressed. We used JPEG compression at 6:1, reducing our storage requirements to 16GB. We expect that vector quantization (VQ) can do much better, but it will more seriously compromise image quality.

Results

Actual acquisition of the light field took place during the nights of March 26 - 29, 1999. Like our other acquisition projects, this one was more challenging than we expected. In particular, controlling the lighting was hard. To avoid changes in natural lighting, we worked only at night. To reduce the harsh shadows of the existing electric lighting, we installed additional spotlights. To keep the moving gantry from changing the lighting, we kept it far back from the statue. Nevertheless, spotlights aged and blew out, the gantry cast shadows on the floor and tomb near the statue, and the sun rose inconveniently early each morning. We also found that we could not position the gantry relative to our taped layout more accurately than about 1cm. Even if we could, the floor of the chapel is not level. This forced us to enlarge the overlap between slabs, which in turn increased acquisition time, which in turn pushed us closer to dawn each night.

Nevertheless, we managed to acquire a reasonably nice dataset. Here are representative images, slightly reduced in size, from each of the 7 light field slabs. Histogram matching has been used to eliminate changes in lighting (see above), but otherwise the images appear exactly as acquired.

Click here for a full-size (1300 x 1030) version of the third image. Click here for a movie of one row of images from the fourth slab, at half-size (.avi (6.5MB) or .rm (900KB)). We are currently assembling these images into a light field that will be viewable using our downloadable light field viewer software. When we finish the light field, we'll put it here.

webmaster@graphics.stanford.edu

	There are many technical choices to be made when designing a light field. In our case the first choice to be made was: which statue? We looked for one whose light field was intrinsically interesting and hard to capture with a 3D model. Michelangelo's allegorical statue of Night (shown at left) is both highly polished and slightly transparent, leading to a variety of subtle, hard-to-render visual effects. We also looked for a statue whose lighting we could control and, more importantly, freeze for several days. We could do this on Night, but to avoid changes in natural lighting and the inevitable flashbulbs of tourists, we had to work only at night. Finally, we looked for a statue that we were also planning to digitize with our laser scanner. We have done this on Night. The resulting combination of a high-resolution 3D model and a dense light field will provide a unique dataset for exploring a variety of image-based rendering algorithms.
	The next choice to make was: where should we stand? It wasn't practical to completely encircle the statue with viewpoints. Instead, we decided to make a 90-degree arc around the statue while standing about 8 feet away. In this photograph the arc can be seen taped out on the floor. To acquire the light field, we removed the 3D scanner head from our motorized gantry and installed in its place a high-quality digital still camera. The camera is positioned roughly at eye level, which is also the height Michelangelo designed the statue to be seen from.
	We divided this arc into 7 "light field slabs", shown in this plan view by yellow line segments. Each slab is an array of images taken from viewpoints closely spaced on a vertical "camera plane" that coincides with the line segment. A conventional optical system with fixed field of view and fixed focus was used, and the directions of view, shown by orange arrows on the figure, were chosen by hand to keep the statue nicely framed. Thus, unlike the light field slabs described in our 1996 paper, the perspective views were conventional rather than off-axis, the focal plane rotated instead of being fixed, and the center of attention moved instead of being stationary. It is possible to generate canonical two-plane light field slabs from this data, but it would require reprojection and resampling of each acquired image.
	The camera plane measured 775mm wide x 700mm high, overlapping its neighbor 75mm. This is large enough to allow a reasonable range of observer motion during light field rendering: up, down, forward (i.e. towards the statue) backward, as well as left and right along the arc. Within this plane, we moved the camera through a grid of 62 x 56 viewpoints spaced 12.5mm apart. The total number of images acquired was thus 3,472 per light field slab, or 24,304 for the entire light field. Since the physical aperture of our camera was considerably smaller than 12.5mm, the light field is not correctly "anti-aliased", but we could not avoid this without acquiring far more images than we did.
	Although the exposure time per frame was short, 1/12 second, additional time was required to download and store the image and move the camera to the next position. Since the direction of view varied with each image, all four motion axes (horizontal, vertical, pan, and pitch, shown here by green arrows) were in play at all times. Fortunately, our gantry is stiff, so we didn't have to wait for vibrations to die down before shooting the next frame. Nevertheless, each light slab took almost 5 hours to acquire. Shooting 2 slabs per night (working from dusk to dawn), the entire light field took 4 nights to acquire. Image resolution was 1300 x 1030 RGB pixels, or 95GB uncompressed. We used JPEG compression at 6:1, reducing our storage requirements to 16GB. We expect that vector quantization (VQ) can do much better, but it will more seriously compromise image quality.