“In the cinema, the process of modifying, enhancing and correcting colour images for defining the style or look of a film is known as ‘colour grading’. The process is traditionally carried out manually in post-production, and requires time, specialized equipment and also has a high cost”, explains Itziar Zabaleta, a researcher at the Image Processing and Enhanced Cinematography (IP4EC) research group at UPF.

Now, a team of scientists from UPF and the Institute of Optics at the Spanish National Research Council (CSIC) has developed an algorithm that is able to transfer the style of a realistic image to a video, to adopt the tone, colour palette and the contrast of the image. In addition, the algorithm enables real-time processing of the raw video output from a camera and, therefore, the resulting video can be seen at the same time as recording.

Zabaleta assures that “the proposed algorithm achieves automatic colour grading, which copies the style of a chosen reference image, and works in real time so that it can be used when shooting to experiment with different looks”.

The resulting videos provide an excellent approximation to the intended look, bringing savings in pre-production, shooting and post-production time. “The results of the method have been validated through psychophysical experiments that have shown that, despite being a simple and inexpensive computational method, it produces better results than other existing cutting-edge  systems”, the researcher asserts.

It is an improvement that may ease the workload of cinematographers and colourists during shooting and post-production. In many cases, the director wants to emulate the style and look present in a reference image, e.g., a still from an existing movie, a photograph..., and this algorithm is applied directly on the unprocessed images and it generates a display-ready result that matches the style of the reference image.

“The results of the method have been validated through psychophysical experiments that have shown that, despite being a simple and inexpensive computational method, it produces better results than other existing cutting-edge  systems”

In the future, the researchers wish to incorporate a whole series of extensions to the methodology to allow, for example, the separation of foreground and background and using keyframes, in order to overcome some limitations that have been observed.

Reference work:

Itziar Zabaleta, Marcelo Bertalmío, “Photorealistic style transfer for video” Signal Processing: Image Communication, July 2021. DOI: 10.1016/j.image.2021.116240