1.5 A General Model of Color Processing

Digital color management follows a clear general path tightly coupled with digital imaging. The high-level diagram below covers this path, from reflectance, image formation, storage and processing to display.

The ’Computer Graphics’ box hides a complex pipeline that largely mimics the physical processes that precede it in this diagram, with the goals of creating content that will blend seamlessly with the captured image. Key: Purple - physical light, Blue - stored light, Green - Synthetic light, Orange - perceived light

While there are many imaging considerations along the full imaging pipeline, such as spatial and temporal considerations, this discussion restricted itself to the propagation and transformation of color in the imaging pipeline.

The simplest example of the digital color pipeline for cinema is exemplified by a digital camera, covering the full imaging pipeline from lens to sensor to the display on the back of the camera. Even this simplest case involves processing and transforms that are a microcosm of more complex pipelines. From this example, it is possible to expand to cover pipelines and workflows that have processing steps in-camera, on-set, in a scanning house, with editors, in a renderer, visual effects, animation and games teams, colorists, distribution processing, cinemas, streaming services and everywhere in between. In this section, the paper will present simple models for thinking about color processing and understanding which aspects are important in each phase of production.

In the simple case mentioned above, an image captured by a digital camera and displayed on the screen on the back of the camera, there are a series of states and transforms to consider: Light reflected off a surface, into camera’s lens and onto an image sensor Sensor photosite values encoded and stored in memory A transform to combine adjacent Red, Green and Blue photosites into single pixel values A "working space" image in memory A transform to remap the intensities of the combined pixels for display A "displayable" image ready to be sent to the display hardware The display hardware showing the image

Optical diagram of a typical mirrorless digital camera. Unknown. (n.d.). DSLR Optical Diagram. Retrieved October 14, 2018, from https://www.dpreview.com/forums/post/61693925

The aspects of the example above that will show up repeatedly throughout this document are:

Input, Working, Output: The example includes the stages that are common to most modern color pipelines, namely an Input, Capture or Generated image state, a Working image state used for editing and production and an Output image state used for display. There may be other steps along the way, but virtually all modern color pipeline designs contain these three distinct states. The Input image for a visual effects or animated feature or game may be the product of a renderer.

Scene-referred, Output-referred images: The sensor image in Step 3 represents the intensity of light in the scene, filtered through the camera’s lens and specific red, green and blue filters over each sensor photosite. The "working space" image from Step 5 also represents the intensity of light in the scene, but may be device-independent, at least slightly abstracted from the camera sensor and processing hardware. After being transformed for display, the image in Step 6 has values representing the activations necessary for each of the target display’s pixels.

Transforms - Input, Output, Look: Going between each image state is a transformation that has a specific, meaningful effect. The most widely used transforms will typically include a remapping of intensities and some degree of remapping of the color gamut: the underlying meaning of the red, green and blue of each pixel. There are many more exotic transforms, and often a number of these simpler transforms are concatenated. Transforms going from Input to the Working space are usually strictly defined based on the capabilities of the capture device and the attributes of the working space. Transforms from the Working space to the Output space usually combine an element of artistry, commonly referred to as "the Look", with an explicit mapping based on the capabilities of the output device. Later sections of the document contain many examples of transforms used in practice.

Staged Processing: The process to go from light hitting the sensor to a display emitting light is defined by a specific, staged processing pipeline. Transforms take color from one image state to the next, constrained by the processing needs of each successive stage and the limitations and capabilities of the capture, processing and display hardware. Images may be written to disk or passed along an IO line representing the state of color at any of the points in the processing chain, as different applications or departments within production may be the source or the destination for the data.

Meaning and Placement - Understanding how to work with an image or color data set is more natural when you can be clear about what the data is supposed to represent and where it lives in the production or product’s processing chain. These two elements, meaning and placement, provide the context needed to apply transformations and make aesthetic judgements in production.

1.5.1 ACES

The Academy Color Encoding System (ACES) is a color management system that embodied the general model of color processing described above. It codifies and standardizes many of the elements described in the Workflow section with an eye towards addressing many of the difficulties listed in the Challenges section below. The basic system diagram for ACES is presented below.

ACES system diagram

This diagram contains scene-referred imagery, transforms converting from one state or color space to the next, working spaces, output-referred imagery and an ordered, staged processing pipeline.

ACES is one of many ways to define a color pipeline, but it presents a nice embodiment of the general model of color processing presented above, mirrored in most of the examples discussed in this paper. ACES will be referenced throughout this document because it is open, well documented, widely available and has been used successfully in a wide variety of projects of all types and complexities. It provides a great starting point for exploration. This should not be taken as a suggestion that ACES is the only or necessarily the best option. Decisions about which color transforms and spaces to use should be base on the requirements and constraints of a given project.