One of the most fundamental challenges in computer vision is to reliably establish correspondence - how to match a location in one image to its counterpart in another. It lies at the heart of numerous important problems, for example stereo, optical flow, tracking and the reconstruction of scene geometry from several photographs. The most popular approaches to solve these problems are based on the simplification that a scene point looks the same from wherever and whenever it is observed. However, this is fundamentally wrong, since its color changes with viewing direction and illumination. This invariably leads to failures when dealing with reflecting or transparent surfaces or changes in lighting, which commonly occur in natural scenes.We therefore propose to radically rethink the underlying assumptions and work with light fields to describe the visual appearance of a scene. Compared to a traditional image, a light field offers information not only about the amount of incident light, but also the direction where it is coming from. In effect, the light field implicitly captures scene geometry and reflectance properties. In the following, we will argue that variational algorithms based on light field data have the potential to considerably advance the state-of-the-art in all image analysis applications related to lighting-invariant robust matching, geometry reconstruction or reflectance estimation.Since computational cameras are currently making rapid progress, we believe that light fields will soon become a focus of computer vision research. Already, commercial plenoptic cameras allow to easily capture the light field of a scene and are suitable for real-world applications, while a recent survey even predicted that in about 20 years time, every consumer camera will be a light field camera. Our research will investigate fundamental mathematical tools and algorithms which will substantially contribute to drive this development.