Aesthetics of Higher Frame Rates in Cinematic Motion Pictures
The following presentation was held in the conference REACHING OUT of the journal Cinematography in Progress CITO, in Lisbon at Lusófona University on November 13th, 2021.
The presentation gives insights into current research of Higher Frame Rates at HdM.
The full PDF with all figures can be seen via the link to full program of the conference (https://cinematographyinprogress.com/index.php/cito/catalog/category/reachingout).
Aesthetics of Higher Frame Rates in Cinematic Motion Pictures
Prof. Stefan Grandinetti, BVK
Institute Visual Media Lab
Hochschule der Medien Stuttgart, Germany
Abstract:
Motion pictures can be improved by applying higher frame rates (HFR) offering significantly more temporal visual information compared to the standard 24 frames per second in both, image acquisition and reproduction. Fluid motion renderings are achieved and unwanted motion artifacts like judder or strobing are reduced. On the other hand the specific aesthetics of HFR can influence the perception of cinematic storytelling fundamentally, especially in fiction films. In the conference “Reaching Out” in Lisbon (11-2021, Cinematography in Progress CITO) cinematic test footage of three research projects was presented, that illustrates and compares essential aspects of HFR aesthetics for cinematic motion pictures and the influence on the perception of motion. This footage represents the current state of research in this matter at Hochschule der Medien, Stuttgart. It provides an insight into the methodology of its creation and illustrates the significance of HFR aesthetics to the art of cinematography and storytelling.
Keywords:
higher frame rates, temporal resolution, cinematic storytelling, visual literacy, film look
Introduction
Digital cinematography imagery was constantly developed and improved over the past few years through engineering: Spatial resolution, dynamic range in acquisition, postproduction and reproduction, color rendering through intensive postproduction and digital projection and display technologies, are the most important aspects responsible for an enhanced quality level. Hence, these improvements show more and more deficiencies of the temporal resolution coming from the traditional frame rate of film (24 frames per second, fps), that was established as an industrial standard in the late twenties of the last century.
Higher Frame Rate (HFR)1 aesthetics and technologies are now regarded as a possible future solution for new deficiencies in cinematography. High Dynamic Range (HDR) imagery is becoming more and more common nowadays and emphasizes higher contrast and peak luminance through the increasingly disruptive perception of judder or strobing. Since HFR were introduced in cinematography to improve deficiencies of stereo-3D-imagery2, the aesthetics were perceived contrastive by an audience, especially with 48 fps HFR3: One part appreciated improvements in rendering of motion pictures while the other part disapproved the new aesthetic that reminds them visually of tv-reportages (“soap opera look”).
With reference to the work of Ang Lee in his films “Billy Lynn’s Long Halftime Walk” (2016) and “Gemini Man” (2019), which the Oscar winning director produced applying HFR (120 fps) in combination with HDR and stereoscopic 3D, we stated that many questions should be researched in a cinematic context.
What are interesting, convincing or displeasing aspects in relation to storytelling, especially in concern of authenticity or truthfulness in fictional films if we improve temporal resolution?
What do we gain in relation to visual storytelling?
Can we counteract the perception of displeasing realism with cinematic abstraction?
Are the generated differences significant and are they leading to satisfying results? Are they pleasing?
Can we adapt to HFR aesthetics through habituation?
In order to research questions like these, various scenes were specifically designed and produced as high quality cinematic motion pictures at Hochschule der Medien Stuttgart, University of Applied Sciences, Germany (HdM). They are displayed in Standard Frame Rates (SFR: 24/25 fps) in comparison to High Frame Rates (96/100/120 fps) on a standard HFR- (120Hz) and HDR- (≈ 900 cd/m2) capable display4 via the internal player of the display. The entire tests are produced with High Dynamic Range (HDR) imaging to point out the significant correlation between both aesthetics.
This paper introduces test footage from three case studies, designed and created under my supervision in collaboration with colleagues and students from Hochschule der Medien, Stuttgart in research-, Bachelor or Master degree projects. In a previous research on HDR-imagery the HdM-HDR-2014 data set5 (Jan Fröhlich, Stefan Grandinetti) has shown a significant increase of the perception of motion judder6 or strobing in motion pictures resulting in increased contrast and peak luminance. This is a motivation to counteract these phenomena through higher temporal resolution and synthetic shutter shapes in postproduction.
Note: The simultaneous comparisons of this demonstration are never full objective to the viewer because of the missing ability of adaption by the human visual system on each single version, but helps in this specific case to differentiate different particular aspects.
Test Footages and Examples:
I.
1. Perception of Judder at higher Frame Rates and higher Luminance Levels
(Aaron Kuder, Florian Martin, Jan Fröhlich, Stefan Grandinetti)
A film can be recorded and played back in standard frame rate (SFR) at 24 frames per second or in high frame rate (HFR) with higher temporal resolution. As part of this thesis, cinematic test footage is created and compared in the light of perception of motion. Higher temporal resolution reduces motion artefacts. In HFR motion is perceived in higher detail, at slower speed and unnatural under certain circumstances as. By synthesizing the shutter in post-production, the HFR footage can be modified in a creative way. (I. Beitzel, et al., 2020)
The source material for this case study with HFR and HDR was produced7 with 192 fps and elaborated in postproduction by frame blending in Nuke8 from 192 fps at 356° shutter to SFR (24 fps) and HFR (96 fps) each with 180° shutter. The Staging, the lighting style, the camera movements and the framings were designed to point out and to evaluate the application of HFR in correlation with HDR in typical and critical situations in cinematic motion pictures.
The test footage was created by Aaron Kuder et al. as groundwork for his Bachelor Thesis8.
Fig.1: SFR 24 fps (180° shutter) generated from 192 fps source footage (Aaron Kuder)
Fig.2: HFR 96 fps (180° shutter) generated from 192 fps source footage (Aaron Kuder)
Both versions are displayed in Standard Dynamic Range (SDR: 100 cd/m2) and High Dynamic Range (HDR: 650 cd/m2). Three different scenes are designed to demonstrate specific differences and are graded in DaVinci Resolve in HLG10, displayed in a HEVC container (mp4) with HD-resolution (1920x1080) - first as a single clip on full screen and afterwards as a split-screen with all four versions simultaneous.
1. 96 fpsHFR, 650 cd/m2 (180° shutter) 3. 24 fps, 650 cd/m2 (180° shutter)
2. 96 fpsHFR, 100 cd/m2 (180° shutter) 4. 24 fps, 100 cd/m2 (180° shutter)
2. “Nature” Scene: Campfire with sparks in comparison of 24 fps and 96 fps HFR
In this test footage prooves that there is no difference between 24 fps acquisition and 192
synthesized to 24 in postproduction visible. In comparison of 24 fps and 96 fps HFR it
demonstrates a perception of smoother flickering of the flames with HFR and the
fundamental change in reproduction of fast moving objects (sparks) that are displayed with
almost no motion blur and are looking significantly more fluid by applying a synthetic shutter
(s. chapter 2, Fig.3 and Fig.6).
Fig.3: Comparison of campfire from 96 fpsHFR and 24 fps (both from 192 fps source) with applied synthetic shutter shape (Aaron Kuder et al.)
3. “Drama” Scene: Barman is serving drinks to customers
This test footage was designed for the evaluation of typical motion in fiction films. The actors’ movements are typical for drama, a barman carries a tablet through the room, shakes a cocktail in a close up, works naturally behind the bar and serves drinks to his customers. The camera movements focus on going with the actors’ movements and are mostly guided pans. As a challenge for motion judder on the actors and the backgrounds a contrasty lighting style with bright skin tones in a daylight mood was chosen. In the production design supports the HDR aesthetic with practicals and windows and challenges the perception of judder and motion blur with many vertical objects.
As a result the HFR aesthetic reminds to reportage-aesthetics and appears as profane, the movements of the barman are perceived as slower while shaking a cocktail. HFR can reduce motion blur and judder, but can also disturb the acceptance or truthfulness in fictional stories and the credibility of acting by revealing weaknesses.
Fig.4a: Bar scene, guided pan comparison: 96 fpsHFR (obove), still from 24 fps (below) (Aaron Kuder et al.)
Fig.4b: Bar scene, splitscreen comparison of 24 fps HDR (obove), still from 24 fps SDR (below) (Aaron Kuder et al.)
4. „Action“ Scene: Artistic BMX driving outdoor
In this test footage the typical motion of action scenes with a fast driving and jumping by a BMX artist in a outdoor location in bright sunlight is presented, shot in a documentary style with fast pans and tracks following the BMX artist. In the background is a building with hard vertical lines in high contrast with reflections of light on windows. The demonstrative use of longer and wider lenses accentuates the correlation of HFR and HDR guided pans and trackings on the fast moving object, or in opposite fast motion of objects crossing the frame in fixed camera positions.
The results are similar to the drama scene, but the improvement of fluid reproduction of motion with HFR is obvious and movements are perceived as slower in general.
For all scenes of this test footage it can be stated that HDR imaging is stressing out the perception of judder.
Fig.5a: BMX scene splitscreen-comparison of all four aspects: SDR/SFR/HDR/HFR (Aaron Kuder et al.)
Fig.5b: Comparison of fast moving BMX - 24 fps and 120 fpsHFR. (Aaron Kuder et al.)
II.
Cinematic HFR? – Demonstration of Synthetic Shutter in “Ship” with 120 fps
(David Kovacs, Felicitas Schmelz, Bianca Thielen, Gabriel Veiz, Jan Fröhlich, Stefan Grandinetti)
For the evaluation of the aesthetic influence of synthetic shutter shapes in order to reduce the perception of judder and a fluid perception of motion a source footage in demonstrative cinematic high quality in Ultra High Definition was created. The scene has a typical staging of motion in the acting for a drama scene. The use of lenses (Zeiss CP.3) with a high accurate reproduction quality (contrast and abberations), a soft, but contrasty lighting style as well as a high contrast in skin tone, costume and background, the perception of judder was increased. Therefore the footage was designed and produced on a Phantom Flex 4K with 4K resolution with 480 fps in 16:9 aspect ratio. A synthetic shutter shape was introduced in postproduction in Nuke9.
Fig.6: Beitzel et al., 2020
For the specific differentiation of the influence on the image appearance in general, one synthetic shutter shape was applied to SFR and HFR with 120 fps versions and in addition displayed without the application. Each version was displayed as a single clip and in a split screen with all four versions.
Fig.7: Typical movements of actress and camera in drama scene “Ship” (David Kovacs et al.)
Fig.8: Split screen comparison “Ship”: SFR/HFR versions with or without synthetic shutter are only visible in moving images. (David Kovacs et al.)
III.
Cinematic HFR? – Demonstration of two different optical approaches on “Good Vibrations”: “Hyperrealism” vs. “Cinematic Abstraction”
(Daniel Grootz, Martin Hübsch, Jan Fröhlich, Stefan Grandinetti)
Fig.9: Still of “Good Vibrations” of version 2, shot on anamorphic Hawk Vintage’74 lenses with optical streak in 100/25 fps HFR and HDR (Martin Hübsch et al.)
To research the image appearance in relation to realism and cinematic abstraction in general, a trailer of a fictive fiction film “Good Vibrations” was produced in a cinematic way: A young woman lost her hearing and compensates her loss through dancing… The story takes place in three different scenes on three locations and takes place in various lighting moods (lowlight tungsten, high contrasty daylight). This test footage is designed to evaluate if the perception of an unwanted reportage aesthetic of HFR in a fiction film setting with typical motions of movies and the slower feeling of movements in the scenes can be countersteered through abstraction by applying a characterful cinematic design.
In order to understand if enhanced temporal resolution can have or keep a cinematic appearance through abstraction, each scene was shot twice with consistant aspects in staging, camera movements, lighting,… however, two resolutely different versions of cinematographic optical design (and aspect ratios) were applied. On one hand a “most realistic” optical version should be created (“window to the world”): “Hyperrealistic” follows the idea to create a potential naturalistic and realistic approach of cinematographic language. On the other hand the approach of “Cinematic Abstraction” follows the idea to apply as much aspects as possible, that are regarded as typical cinematic: Aesthetics with sophisticated and favored “artifacts” like spherical abberations, streaks, or the specific aesthetic of (front-) anamorphic cinelenses and a specifically historical coating of the lenses, that are popular and regarded as most cinematic optical look are aiming to compensate aesthetical weaknesses in HFR-perception.
The source footage was shot on a ARRI AMIRA with 100 fps (356° shutter) that is displayed in HDR with HFR 100 fps and in comparison to standard 25 fps. The contrast of the two different approaches was applied by the diverse choice of cinelenses and filters for each version.
“Hyperrealistic”: Zeiss CP.3 lenses, no optical filters, (16:9 aspect ratio)
Well known for a accurate and contrasty image reproduction with minimum distortions and abberations, used at F-stop 8 for deep focus
“Cinematic Abstraction”: Anamorphic Vantage Hawk Vintage’74 lenses 2:1, Tiffen Black Promist11 filters 1/8, ¼ (and 1: 2,39 CinemaScope aspect ratio), used at F-stop 2,8 to achieve a shallow depth of field.
Well known for a very characteristic anamorphic lens design with “tasteful” spherical abberations, streaks and a specific historic coating. The “smokey” aesthetic of the lenses and filters is graded to more contrast and saturation in postproduction.
Fig.10: Still of “Good Vibrations”, version 1, Zeiss CP.3 lenses, 100/25 fps
(Martin Hübsch et al.)
Fig.11: Still of “Good Vibrations”, version 2, anamorphic Hawk Vintage’74 lenses, 100/25 fps emphasize typical look characteristics regarded as cinematic (Martin Hübsch et al.)
Fig.12: Still of “Good Vibrations”, version 1, Zeiss CP.3 lenses, 100/25 fps
(Martin Hübsch et al.)
Fig.13: Still of “Good Vibrations”, version 2, anamorphic Hawk Vintage’74 lenses, 100/25 fps (Martin Hübsch et al.)
This test footage is currently still under examination, but was the groundwork for several bachelor and master thesis at HdM (s. appendix)
Conclusion
The increase of visual information per second related to motion is a relevant aspect in the perception of cinematic storytelling. In many motion picture scenes the HFR images can change the attention of viewers in comparison to classic 24/25 fps playbacks. Since the judder is reduced HFR can reveal more visual information in an image in sharp rendering because moving objects are displayed not rendered blurred any more. If these should be of lower priority in relation to the story, a shallow depth of field i.e., it could be appropriate to avoid, that the viewers attention is attracted to parts or objects in images but are not relevant for the story, they are displayed with more detail and sharpness because of a higher temporal resolution.
In summary it is stated that HFR aesthetics improve motion rendering but change the aesthetics in perception of motion, speed, focus, rhythm and realism in cinematic motion pictures. Since the aesthetics of HFR can be perceived as attention steering, brilliant or pleasing, or in opposite perceived as slower and less exciting, it is still to describe how many frames per second HFR are suitable for certain motion pictures.
Watching HFR footage over longer periods can lead to familiarization HFR aesthetics and to a less extraordinary perception of it.
The film industry and cinematographers might embrace the HFR-technology in the future because of the obvious improvements in motion rendering through the higher temporal resolution and the fact, that this will be technically possible and economically affordable. Technical workflow problems like the amount of produced data, editing in real time, timecode for sound recordings, handling of variable frame rates in postproduction, etc. might already be solved in a few years already. Innovative TV channels like the BBC discuss to produce their content in 100 fps HFR in near future. The use of variable frame rates (VFR) could be applied as intraframe technology or as a sequential technique and is not jet well researched. First tests in documentary filming at HdM with 100 fps HFR are very promising.
On the other hand the application of HFR will lead to fundamental changes in the habituations of cinematography and staging in order to achieve the right aesthetic for the individual storytelling of each story or scene. Using HFR is a complex challenge for filmmakers, since many artistic practices need to be adapted and still have to be researched more in depth.
Footer:
1 Higher Frame Rates (HFR): more frames than 24 fps are used in acquisition and replay
2 (i.e. 48 fps HFR “The Hobbit: An unexpected Journey”, Peter Jackson 2012)
3 48 fpsHFR: in acquisition and display of each camera
4 LG OLED 48C14LB UHD/120Hz/HDR TV
5 https://doi.org/10.1117/12.2040003, IS&T/SPIE Electronic Imaging, 2014, San Francisco
6 Judder: Stuttering image artifact
7 ARRI-AMIRA and Ultra Primes, QuickTime, 2k, 12 bit, LogC Wide Gamut, 330 Mbit/s intra-
frame
8 Creation and Comparison of Cinematic Reference Footage filmed at Standard Frame Rate
and High Frame Rate and its Impact on the Perception of Motion, Bachelor Thesis Aaron
Kuder.
9 Nuke: Software for compositing and visual effects for professional film- and video
Postproduction, The Foundry
10Hybrid log-gamma: Transfer function, HDR format
11 Tiffen “Black Promist” optical filters distribute light into the shades of the images
Acknowledgements:
I. Beitzel, A. Kuder and J. Fröhlich, "The Effect of Synthetic Shutter on Judder Perception—an HFR & HDR Data Set and User Study," in SMPTE Motion Imaging Journal, vol. 129, no. 1, pp. 42-50, Jan.-Feb. 2020, doi: 10.5594/JMI.2019.2951267.
Jan Froehlich, Stefan Grandinetti, Bernd Eberhardt, Simon Walter, Andreas Schilling, and Harald Brendel "Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays", Proc. SPIE 9023, Digital Photography X, 90230X (7 March 2014); https://doi.org/10.1117/12.2040003
Appendix Thesis:
Locally Varying Frame Rates for Judder Reduction in High-Dynamic-Range Content
Master Thesis, Jan Hoydem
Abstract:
Higher Frame Rates (HFR) have been available to the film industry for a couple of years, but its use for cinematic productions has been received with skepticism. Nevertheless Higher Frame Rates offer the potential to solve technical challenges as well as providing new creative ways to express stories in film. The research of this thesis explores the application of Locally Varying Frame Rates (LVFR) which emulate frame rates per pixel based on frame rate maps. This technique could provide a solution to reduce unwanted judder in High-Dynamic- Range (HDR) film, while keeping parts of the frame within a familiar 24 fps look. To that end a new frame rate map based on an existing perceptual judder model is introduced and evaluated in a subsequent study with three High-Dynamic- Range sequences. The results show a significant reduction in judder, while still sustaining a "film look".
Wirkung von High Frame Raten auf kreative Aspekte der Filmmontage
Bachelor Thesis, Jamie Lewis
Abstract:
The following work highlights the new film technique of high frame rates (HFR) and its effects on creative aspects of film editing.
Based on the editing process of the HFR short film Good Vibrations, that was produced at the Stuttgart Media University in 2019/2020, the effects of HFR are being observed, studied, and discussed. The resulting self-elaborated hypotheses are put into context with recent studies and the current state of research of high frame rates, motion, film editing and film perception. The work demonstrates that high frame rates not only affect the editing process itself but also the impression of creative aspects such as visual activity, shot size, rhythm, staging and slow-motion. As a new technique, HFR follows the ongoing trends of filmmaking and film perception.
Der Einsatz von Variable Framerates in der cinematischen Filmpostproduktion und seine Auswirkungen auf das visuelle Empfinden des Rezipienten
Bachelor Thesis, Emily Strunk
Abstract:
Film material can be acquired at Standard Framerate or Higher Framerate. The use of Higher Framerates provides many advantages, such as a more realistic motion reproduction, increased detail sharpness and an immersive film experience. However, the resulting aesthetics are in contradiction to the established cinematic look, achieved by acquisition at Standard Framerate. The Higher Framerate Look is therefore controversially discussed by viewers and filmmakers alike. To combine the advantages of both formats, one option to the solution could be to use Variable Framerates for film productions. This thesis takes up this approach and examines the implementation of Variable Framerates Ramp within a short film production. Subsequently I am analyzing and discussing the effects of the Variable Framerate Ramp in a post-production process. In addition, the structure and function of this Variable Framerate Ramp is defined. Finally, the generated variable frame rate version of the short film is compared with a standard frame rate and higher frame rate version by means of a visual study. The effects of the cinematic look on the visual perception of the recipient are examined and the results are evaluated.
Veranstaltungsort: Lissabon, Universidade Lusófona
Weiterführende Links:
Link zum Konferenzprogramm "Reaching Out"- Cinematography in Progress CITO
Autoren
Eingetragen von
- Name:
-
Prof.
Stefan Grandinetti
Mehr zu diesem Autor
