Publications and Data

HDR Guidelines

HDR GUIDELINES FOR PRODUCTION, POST, AND DISTRIBUTION

1. Introduction

The HDR4EU project was launched as a means for representatives ranging the motion picture ecosystem to collaboratively identify and develop solutions for technical problems associated with the emerging high dynamic range (HDR) image format. To accomplish this, regular communication channels and specific collaborative development objectives were established and carried out over the course of three years. The project included high level technical representatives from ARRI (camera manufacturer), Filmlight (post-production software developer), Smoke & Mirrors (post-production company), Brainstorm (broadcast software developer), Barco (display manufacturer), and Universitat Pompeu Fabra (computer graphics and image processing academic group). The following document is a collection of the practical takeaways encountered by each of the project partners that may be helpful for artists and practitioners working with HDR.

HDR is in the process of top to bottom adoption by technology groups, standards bodies, artists, and viewers. In essence, the format allows for the display of images at a larger range of light levels (a larger dynamic range) than has been possible with legacy displays and projectors. This increase in range allows for images to contain darker, more detailed shadow regions as well as highlights that are brighter and more accentuated, resulting in a visually significant increase in contrast. This impacts not only the global perception of contrast when the image is viewed as a whole, but also increases the perception of contrast on a local scale, which in turn increases the perceived sharpness of image detail. In the same vein, a long list of interdependent perceptual factors are impacted by this seemingly simple adjustment, the management of which we will discuss throughout this document.

In addition, we address a number of common mistakes, misconceptions, and confusion points which we encountered while developing HDR technology, testing tools with artists, and producing content. We hope that these insights will ease the transition for filmmakers who are new to HDR, although we acknowledge the best way to learn to work with the format is through hands-on experimentation. As has been the case with new imaging formats from the past, HDR opens the playing field for artists to create novel styles and looks. By this virtue, once we collectively have a stable and consistent technology ecosystem in which to create and enjoy HDR content, artists will demonstrate the full potential of the medium through their images.

2. Image Capture (Production)

The most important general recommendation that can be offered for shooting HDR content is to monitor the images at the dynamic range they will be distributed. While this may seem obvious, there always exists the temptation to use SDR monitoring equipment to more or less imagine how the images will translate in HDR. This approach can lead to unhappy surprises in the color grading process. To name a few, noise will increase visually as the dynamic range of images increases, motion judder will become more noticeable, and any over-exposed or sensor-clipped regions (e.g. practical lights) in the image become far more distracting in HDR, as they are pinned to the maximum light value. These artifacts can lead to great difficulty in the grading of HDR masters and can go unnoticed when monitoring in SDR. In addition to ensuring that acceptable imagery is being captured, monitoring in HDR can allow all production departments to do the best version of their work. For example, due to the increased local contrast (sharpness) that comes along with increasing global dynamic range, the appearance of skin, hair, pores, and makeup textures can be affected significantly. For this reason, it would be important to allow hair & makeup and art departments to monitor an HDR representation of the image as creative decisions are being made on set.

Shooting scenes which push the limits of the camera dynamic range on both ends (shadows and highlights) simultaneously can be particularly hard to balance in post. With the majority of grading tools currently available, it can be difficult to correct one end of the dynamic range without losing detail in the other. Care should be taken in lighting different regions (foreground and background, for example) as the relationship between these regions can have a powerful influence on where the viewer will focus on the image (as is the case with SDR, but like the other effects listed, is magnified in HDR). All this being said and considering the fact that clipped highlights are more problematic when displaying HDR, Directors of Photography should consider lighting scenes more sympathetically, making use of diffusion techniques (silks, filters, etc.) When shot properly, sparse specular highlights allow for particularly good HDR effects. Reflections of light sources on water (in daylight or nighttime), eyes, glitter or other specular (shiny) surfaces add new dimensionality to characters and scenes in HDR.

As an additional note, one must remember that while monitoring HDR, it is important to pay attention to the viewing environment the display is located (as with SDR). Bright lights and glare on the monitor can cause the appearance of contrast to decrease and for shadow detail to be obscured. While bright HDR images may appear to be less affected by changes in the viewing environment compared to images on dim SDR screens, its influence still cannot be ignored. A general recommendation would be to simulate the intended grading environment to the best of one’s ability while on set.

One of the major questions that came up during the course of the project was the following: Since HDR and SDR are fundamentally different formats, artists will use them differently. However, in order to share the content with the largest amount of people, both HDR and SDR final versions will need to be produced. So the question is, which version should the DP intend to shoot the content for? To try to do something “in between” is impractical – it would result in a picture which is simultaneously compromising for both formats. Ultimately, the response is up to the director of photography (DP) who can choose what version should be prioritized. It is then the responsibility of the colorist to translate the image as best as possible to the secondary format.

3. Image Manipulation (Post-Production)

In order to have a discussion on how to go about mastering HDR images, which can include accentuated and detailed areas of brightness and shadow to create a more immersive film viewing experience, one should first address how the perceptual processes of the viewer will react to this new range. While exploring the effects of varying global contrast and the average light level (ALL) of images, one will find that the eye easily grows accustomed to any level of constant brightness, as long as there is time to adapt to it. By this token, any images which do not use highlights sparingly risk looking more or less like SDR as the viewer will quickly adapt to the new contrast range. One should also remember that HDR highlights have the potential to be painful to viewers. Large transitions between ALL should be done conservatively and slowly to avoid making an uncomfortable and fatiguing visual experience. In our daily life we seamlessly transition between regions with vastly different absolute dynamic ranges. When the transition is truly intense it can catch our eye for a moment (e.g. opening a door or a curtain to the outdoors in a dimly lit room), but the level of discomfort this causes is quite temporary and goes mostly unnoticed. However, in real life we do not experience 'cuts' like we do in motion pictures, where our vision transitions instantaneously between different scenes or points of view. For this reason, colorists should keep an awareness of their state of adaptation in relation to the images, as to not master content which viewers find physically uncomfortable to watch.

Since the light levels in HDR images are distributed differently than SDR images, it stands to reason that the lift, gamma, gain controls which have been traditionally used in motion picture color correction will not react in the same way between formats. With SDR images, these controls could be used to roughly adjust highlight, mid-tone, and shadow regions of images. In essence, the feeling reported by many colorists in using them for HDR grading is that they are too ham-fisted, and do not allow for the same range-specific correction. In the event that colorists feel they run into limits too quickly with their standard color correction workflow, they should look into alternative grading tools and image encoding formats which have been developed with HDR images in mind. These new tools should allow colorists to adjust images along consistent perceptual axes (e.g. brightness, hue, saturation, sharpness) and make HDR images appear in the way they direct, rather than the other way around – where colorists feel trapped into some “look” which they deem inherent to HDR because they cannot control the images in the way they are accustomed to.

With regard to one perceptual axis, that of saturation, HDR images pose an interesting new challenge. As the dynamic range of imagery is increased, certain surface colors may start to "pop" to the point where they appear to be brighter than the light source in the scene, giving the viewer the impression that the object is fluorescent (appears as if it is emitting light), which can be a jarring effect (e.g. a brightly colored bird should not appear as if it is emitting light, as we have an understanding that birds do not emit light). Colorists should be careful to control the saturation of object colors in a way that does not break these perceptual barriers and should be generally aware that HDR images may react with legacy tools to create unnatural visual experiences.

Another perceptual axis, sharpness, is intimately related to dynamic range. By increasing the global contrast of an image, one also increases the local contrast, or sharpness. Large increases in sharpness can result in unpleasant artifacts (blocky clipped highlights which are particularly obtrusive in the case of practical lights, noise, unnatural or overdetailed textures [e.g. skin, hair], etc.) To control these elements, colorists will need to familiarize themselves with new tools for selectively blurring and sharpening image regions, and equalizing textures throughout the images. Along a similar line, any glow/glare that either is artificially added in post-production to emphasize the brightness effect of an object or captured physically with the help of an optical filter can reduce the HDR effect by lowering the local contrast and increasing the average brightness level of the image, making highlights less pronounced.

4. Image Viewing, Distribution (Delivery)

The different display technology options available at the moment (LCD, OLED, QLED, etc..) have widely varying performance. Even among displays marketed to meet the same standards, different technologies can vary significantly in how they render images. Highlight and shadow rendering is often perceptibly different between the various technologies. Different display types have different properties of sharpness, or are normally observed at different viewing angles, which also make a large impact on the appearance of images. Display manufacturers include various image enhancement functions which operate “under the hood” as a means to differentiate their products from the competition, increasing the variability in the rendering behavior of consumer displays. Finally, the environment surrounding the display will have an impact on the degree to which image renderings are consistently conveyed throughout the production process and in delivering the final piece. These sources of variability are important to keep in mind when mastering content for delivery.

With regard to producing an SDR version of an HDR master or vice-versa, the average light level of an HDR image should be roughly the same as its SDR equivalent, just expanding the highlights and the shadows. Otherwise, if the average light level is scaled proportionally with the peak highlight, the HDR effect will be lost completely. The viewer will adapt to the image and it will appear roughly the same as it would on an SDR display. Care should be taken when using automatic up-conversion methods (SDR->HDR) as simply increasing the global dynamic range can result in accentuated compression artifacts and generally over-sharp images. Although artifacts are likely to be less noticeable in the opposite case (HDR->SDR), some skepticism should always be taken with automatic transformation methods as a satisfactory preservation of artistic intentions. A proper solution proposed by Filmlight for this problem is to grade an SDR and HDR version separately and then to use a simple blending scheme to produce images for screens which have a dynamic range that lies in between that of the SDR and HDR reference monitors. In this case, there is a specific, pixel-wise mapping transitioning between the two creative-approved image states which can be used to produce masters at varying dynamic range levels. In effect, this blending could be accomplished by taking a weighted average between the image representations, where the weights are determined depending on the destination dynamic range.

5. Summary of Guidelines

We present a collection of practical recommendations that filmmakers can use to improve their understanding of the HDR format and avoid some common workflow pitfalls. In summary these recommendations are the following:

Always monitor what you are shooting for the intended distribution dynamic range on set (with some degree of error allowed, of course.) Details may go unseen which could make the captured images unusable for an HDR version.

Choose a single hero format to light and shoot for - don't bake a compromise between the two into the source material. The material can be often be adapted appropriately for all other delivery formats by a skilled colorist.

Over-use of image highlights (occupying too much screen time or area) will dull the viewer’s sensitivity and lose the HDR effect. Properly shot, sparse specular highlights allow for a visually stunning use of the upper range.

Tools which are not developed with the format in mind can often have their functionality broken by HDR material. Consult with technology developers on whether their products were significantly tested with HDR workflows.

Proof content on a range of display technology types as their varying capabilities may impact creatively relevant image details.

Perform individual grades for HDR and SDR to maximize benefits of each, then masters for intermediate ranges can be generated by blending between these approved image states. Automatic conversion techniques implemented by content distributors or within display devices should not be trusted to preserve your artistic intentions.

More detailed descriptions of the work conducted for the project, as well as contact info for sourcing the HDR footage shot throughout the project can be found here: https://www.upf.edu/web/hdr4eu/publications. For the case of many challenges listed throughout the document, project partners already offer developed solutions. Their respective websites can be found here: https://www.upf.edu/web/hdr4eu/participants.

While we hope readers find these guidelines helpful, we acknowledge that the best way to learn how to create excellent HDR imagery is through hands-on experimentation. We strongly encourage the reader to produce their own HDR content and play a part in defining the medium.

Final Dissemination Showcase

Due to the COVID-19 pandemic, the final dissemination plan to showcase the project results and prototypes at an industry international event could not go ahead. Instead, partners agreed to create a series of videos that would highlight their contribution to the project in a form that would be attractive and understandable to a wider audience.

*Click on each image to play video

ARRI

ARRI’s video demonstrates clearly the effect HDR has on steering the attention of the viewer, and how HDR and SDR may require different treatment so not to distract from the main focus of the story. Various examples show the considerations for practical lighting, for composition and camera movement when shooting for HDR.

While the visual impact and the heightened realism of HDR imagery work spectacularly well if used carefully, it also shows the dangers of ignoring the need for multiple versions for distribution. Critical HDR monitoring during the shoot is important for taking full advantage of the camera’s high dynamic range.

The video addresses the Directors of Photography and other content creators and helps building confidence in shooting for HDR by being open about the potential difficulties.

UPF-GTI

UPF-GTI demonstrate how the collaborative open webglstudio platform has been extended to incorporate HDR capabilities. They include a better way of assembling HDR images from a series of SDR images and improved collaboration through an improved toolset and the newly specified HDRE repository, which is also supported in Brainstorm’s eStudio product. The combination of a chromakeyer with environmental lighting and improved webgl based photorealistic rendering complete the demonstration.

The video addresses the large community of not-yet-professionals and shows how a new generation of content creators can get familiar with HDR, explore its advantages and build confidence in the underlying technology.

UPF-IP4EC

UPF-IP4EC give a detailed demonstration of why a good understanding of our visual system is an essential element in developing effective HDR technology. Many different aspects are covered – how we can minimise colour appearance errors when interchanging materials between colour critical observers in a wide gamut environment; how colour adaptation is not dependent on screen size but on environmental illumination; how tone mapping and adaptation to different viewing environments can interact with each other and what that means for an optimised tone mapping and inverse tone mapping algorithm; how induction effects –which describe how colour perception can vary with our field of view– can be compensated for; and how the same model can be used for gamut mapping to produce visually equivalent results on displays with smaller display gamut.

The video presents this genuinely groundbreaking academic research with clear visual examples in an understandable form, showing both the relevance of this research for the practical implementation of HDR tools as well as the further understanding of how we perceive and decode colour information in response to different viewing environments.

The research has been submitted for publication in the relevant academic vision science publications and is bound to find interest well beyond the specific field of HDR content creation and consumption.

BRAINSTORM

Brainstorm demonstrate how their well-established and industry-leading virtual studio product eStudio was updated to include HDR capability. To maintain maximum flexibility, this required the introduction of a colour management layer with support for external LUTs, so SDR and HDR content could be used simultaneously, as well as producing SDR and HDR content in various industry standard formats.

In turn this required a re-write of the internal rendering engine, introducing more photorealism and physics based rendering even for SDR workflows. The support of the HDRE file format developed by UPF for 3D assets shows close integration between partners. The inclusion of an automatic exposure control tool as well as an eye-adaptation tool show the application of the project academic research in the commercial product.

The video addresses the broadcast market, which is pushing for practical and productive solutions in the area of live-action virtual studio production. With Brainstorm working in close cooperation with its clients, industry acceptance and adoption will follow.

Smoke&Mirrors

Smoke&Mirrors have contributed a document about the practicalities of HDR production and post-production, specifically for broadcast commercials. It contains a large number of practical considerations when considering an HDR production. Clearly aimed at their commercials clients, it encourages them to take full advantage of the creative possibilities.

FilmLight

FilmLight structures the developments for its Baselight mastering and finishing system around colour tools, spatial operations and temporal considerations. Like Brainstorm, FilmLight saw the need to introduce a comprehensive colour management framework, covering all input materials and output formats, whether producing content in SDR or HDR or both. At the same time, it introduced a new colour grading paradigm based on a “scene referred” workflow, providing more natural, photographic controls to image manipulation and making the simultaneous mastering of multiple deliverables more efficient and consistent.

The video addresses the post-production community and demonstrates that multiple deliverables for any range of viewing environments can be handled with confidence and high productivity. This reflects the feedback from within the industry.

BARCO

The video from Barco starts with a close look at what HDR actually means in relation to existing display and cinema technology. It goes on to explain the underlying considerations that lead to the choice of technology for their HDR cinema projector, highlighting practical considerations like power consumption and conversion costs for existing cinemas, but also taking into account our visual perception and the special immersive cinema environment.

It looks at the steps required for a complete HDR eco-system, and explains Barco’s involvement in international standardisation efforts and its collaboration with the project partners to bring it about. Barco’s role as the leading cinema projector supplier will be critical in the successful deployment of HDR in the cinema.

Technical Shootings

During the course of the project 3 technical shootings were performed with the objectives of:

Capturing technical content for testing and validation of the HDR tools and pipeline.
Providing suitable solutions for the ‘on-set’ visualisation of the captured material.

1. Initial HDR Technical Shooting: S&M and all partners suggested to ARRI a list of content to be captured that would later be used for testing techniques, creating exemplary content and evaluation of the HDR tools and pipeline. The shooting was done with provided equipment and resulted in test material for various applications of HDR, such as: Scenic productions, Documentary, Sports, Stage shows and concerts.

2. Enhanced HDR Technical Shooting: The second technical shooting allowed partners to shoot content using their enhanced HDR tools already developed by the project. The shot footage was a combination of:

Similar content to the initial shoot so that qualitative validation could be performed as part of the enhanced HDR pipeline.
New content that would allow demonstration/validation of the new tools.

S&M and all partners suggested to ARRI a complete list to shoot during this second phase of the project and the footage allowed S&M to do their benchmark evaluation.

3. Production Shooting: The final shoot was an actual production in which S&M engaged with one of its clients to arrange on-set access so that the tools would be used in a proper production environment. This allowed the footage to subsequently be used for further validation and demonstration of the HDR pipeline and tools. S&M used the captured footage as part of their Global Professional Validation.

*To download data from the technical shootings please contact [email protected]