AV1@Scale: Film Grain Synthesis, The Awakening

The noise does not contain a signal, does not dance over it, and is not detail. It is purely random fluctuations that are added to a signal.

If you have a few static frames and average them, you improve SNR by retaining the unchanged signal and having the purely random noise cancel itself out. Retaining noise itself is not useful.

I suspect the effect you might be seeing is either just an aesthetic preference for the original grain behavior, or that you are comparing low bandwidth content with heavy compression artifacts like smoothing/low pass filtering (not storing fine detail saves significant bandwidth) to high bandwidth versions that maintain full detail, entirely unrelated to the grain overlaid on top.

Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely). So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.

Regarding aesthetics, I don't think AV1 synthesized grain takes into account the size of the grains in the source video, so chunky grain from an old film source, with its big silver halide crystals, will appear as fine grain in the synthesis, which looks wrong (this might be mitigated by a good film denoiser). It also doesn't model film's separate color components properly, but supposedly that doesn't matter because Netflix's video sources are often chroma subsampled to begin with: https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf

Disclaimer: I just read about this stuff casually so I could be wrong.

> both it and the synthesized grain image look noticeably less sharp than the source

That's true, but at a given bitrate (until you get to very high bitrates), the compressed original will usually look worse and less sharp because so many bits are spent trying to encode the original grain. As a result, that original grain tends to get "smeared" over larger areas, making it look muddy. You lose sharpness in areas of the actual scene because it's trying (and often failing) to encode sharp grains.

Film Grain Synthesis makes sense for streaming where bandwidth is limited, but I'll agree that in the examples, the synthesized grain doesn't look very grain-like. And, depending on the amount and method of denoising, it can definitely blur details from the scene.

Since the beginning of film editors have added tricks in post.

I would love for them to provide an option to view it with film simulation vs without.

One of my favorite movies of all time, The Holdovers, did film simulation extremely well. It's set in the '70s so it attempts to look like a movie of that era.

It looked great to me, but if you're an actual film nerd you're going to notice a lot of things aren't exactly accurate.

Maybe in the near future we'll see Netflix being able to process some post effects on the client. So if you're color blind, you get a mode for that. If you don't want fake grain you can turn it off.

AV1 has tunable FGS levels, and to my eye they went very slightly higher than they should have (though there are tradeoffs; at some bitrates the blurring+renoising is so much better than the other visual artifacts you will otherwise get, that you do want it that high).

A few things to note:

- still-frames are also a mediocre way to evaluate video quality.

- a theoretically perfect[1] noise-removal filter will always look less detailed than the original source, since your brain/eye system will invent more detail for a noisy image than for a blurry image.

1: By which I mean a filter that preserves 100% of the non-grain detail present, not one that magically recovers detail lost due to noise.

A film that explores this is Antonioni’s Blowup: https://en.wikipedia.org/wiki/Blowup

Agree, as someone who has spent way too much time studying the way motion picture film looks up close, this isn’t very realistic looking. It’s really just a form of dithering.

Get the retro gaming nerds on this, they'll make a film grain shader that's indistinguishable from reality in no time flat.

> and not really grain like

that's an understatement. it just looks like RGB noise effect was added. film grain does not look like RGB noise. to me, film grain is only one part of what gave film the film look. the way the highlights bloom rather than clip. it also was more natural/organic/some descriptive other than the ultrasharp of modern digital acquisition. using some SoftFX or Black Mist type filters help, but it's just not the same as it is a digital vs analog type of acquisition. all of these attempts at making something look like it's not just keep falling down in the same ways. but hey, there's a cool tech blog about it this time. film grain filters have been around for a long time, yet people just don't care for them. even in Blu-ray time frame, there was attempts at removing the grain in the encode and applying it in playback. Netflix isn't coming up with anything new, and apparently nothing exciting either based on the results.

These days, when we see noise/grain in an end product it has likely been added in post-production. So, ideally, studios would provide distributors with a noiseless source plus grain synthesis parameters. Bonus: many viewers would welcome an option to turn it off.

My issue is that grain is good based on the creative decisions of the creators of the content. It is not something that a group of nerds compressing 1s and 0s should be making

The article points out the masking effect of grain, which hides the fake-looking compression artifacts, and also the familiarity/nostalgia aspect. But I will offer an additional explanation.

Look around you: nearly all surfaces have some kind of fine texture and are not visually uniform. When this is recorded as video, the fine texture is diminished due to things like camera optics, limited resolution, and compression smoothing. Film grain supplies some of the high frequency visual stimulus that was lost.

Our eyes and brains like that high frequency stimulation and aren't choosy about whether the exact noise pattern from the original scene is reproduced. That's why the x265 video encoder (which doesn't have grain synthesis since it produces H.265 video) has a psy-rd parameter that basically says, "try to keep the compressed video as 'energetic' as the original, even if the energy isn't in the exact same spot", and even a psy-rdoq parameter that says, "prefer higher 'energy' in general". These parameters can be adjusted to make a compressed video look better without needing to store more data.

People are always trying to rationalize and justify aesthetic preferences. The depth and nuance of your understanding of a thing will change how you perceive variations of that thing, whether it's guitar tonewoods, style of music, types of paint, flavor of beer, or the grain in film. If you know a lot about a subject, you can tell a lot about the history of a thing, and that's going to change how you feel about a thing.

A child watching a Buster Keaton skit and gasping and giggling and enjoying it is going to have a different subjective aesthetic experience of the media than a film critic who knows exactly what type of film and camera were used, and what the meaning of all the different abstractions imply about the scene, and the fabric of Keaton's costume, and so on, and so forth.

Subjective aesthetic preferences are in the realm of cognition - we need a formal theory of intelligence mapped to the human brain, and all of these subjective phenomena collapse into individualized data processing and initial conditions.

There's something about film grain contrasted against clean cel animation which might make it easier for people to suspend disbelief. They are conditioned to think that absence of grain is associated with unreal animation, particular types of media, and CGI. Home video and news and so forth had grain and low quality, so grain gets correlated with "real". In my view, there's nothing deeper than that - we're the product of our times. In 40 years, media will have changed, and it may be that film grain is associated with surrealism, or edited out completely, as it's fundamentally noise.

The way I see it is that grain makes the film look more detailed than it really is, it can also hide compression artefacts and blurriness.

I don't know the psychovisuals behind that. Maybe it adds some high frequencies that compression often washes out, or maybe acts like some kind of dithering.

As for your eyes, I am pretty sure that they have grain, that's how quantum physics work, you just don't perceive it because your brain filters it out. But again, I don't know how it interacts with film grain.

> my real eyes don’t have grain.

They definitely do at night when it's dark out. There's a kind of "sparkling" or "static" that comes in faint light.

Fortunately, our eyes have way better sensitivity than cameras. But the "realism" just comes from how it was captured using the technology of the day. It's no different from phonograph hiss or the way a CRT signal blurs. The idea is to be "real" to the technology that the filmmaker used, and the way they knew their movie would be seen.

It's the same way Van Gogh's brush strokes were real to his paintings. You wouldn't want his oil paintings sanded down to become flat. It's the reality of the original medium. And so even when we have a digital print of the film, we want to retain as much of the reality of the original as we can.

It used to be a bigger deal (when digital cameras started being used) since people felt like digital video didn't look real/as good - movies shot on film were generally better looking (as crews were used shooting with it and digital video wasn't as sophisticated as today) and HAD grain.

It might be that there is a large part of the population that still has that association.

Cinephiles are also more likely to watch older (i.e. with grain) movies that ARE well shot and beautiful (which is why they are classics and watched by cinephiles) and not see bad film movies, only the cream of the crop, while being exposed to the whole gamut of quality when watching todays movies shot digitally. Would reinforce that grain = good while not being necessarily the case - and their opinion might be heard more than gen pop.

At any rate, it can be a neat tool to lower sharpness!

Mine do, at least when it's very dark

This reminds me of modern windows having fake panes. They’re just strips that are applied to give the impressions that there are multiple smaller panes because people are used to that and it feels “correct”.

I have to imagine past glassmakers would have been absolutely enthralled by the ability we now have to make uniform, large sheets of glass, but here we are emulating the compromises they had to make because we are used to how it looks.

My vision is grainy because of visual snow. Which is why I turn off film grain in games, it stacks on my vision and makes a visual mess.

This sent me down a very interesting rabbit hole, thanks!

Documentaries might care about accurately representing reality. For every other cinematic genre, "authenticity" is not an inherent goal. If film grain is part of the director's vision, then that's just as valid as a choice to have dramatic non-diagetic music playing in the background of a scene (which is highly inauthentic, but also highly effective at evoking emotion, which is the point of art).

> But there is no grain in reality, so it does the opposite

Well ackchually -- illumination is inherently random, so all time-bounded captures of a scene (including what your eyes do) are subject to shot noise: https://en.wikipedia.org/wiki/Shot_noise

> It just adds visual noise that obscures details of the authentic scene

Does it add any more than modern video compression techniques? What constitutes noise in cinema, is somewhat subjective.

This is just a fact of lossy compression: you want to throw away information that contributes less to the perception of the video so that you can describe it with fewer bits of information.

There are two possible advantages for this kind of grain synthesis. For Netflix, they could produce the same perceived quality at lower bitrates, which reduces costs per view and allows customers with marginally slow connections to get a higher quality version. For a consumer, the advantage would be getting more non-grain detail for a fixed bitrate.

You are right that if you subtract the dentists frame from the raw one, showing only the estimated noise, you would get some impression of the scene. I think there’s two reasons for this. Firstly, the places where the denoiser produced a blurry line that should be sharp may show up as faint lines. I don’t think this is ‘hidden information’ so much as it is information lost to lossy compression. In the same way, if you look at the difference between a raw image and one with compression, you may see some emphasized edges due to compression artefacts. Secondly, the less exposed regions of the film will have more noise so noisiness becomes a proxy for darkness, allowing some reproduction of the scene. I would expect this detail to be lost after adjusting for the piecewise linear function for grain intensity at different brightness levels.

Perhaps a third thing is the level of noise in the blacks and the ‘grain size’ or other statistical properties tell you about the kind of film being used, but I think those things are captured in the film grain simulation model.

Possibly there are some other artefacts like evidence of special effects, post processing, etc.

If you extract the grain (e.g. by subtracting a blurred version of the image), the result contains mostly noise, not meaningful scene information outside of some variation according to image brightness. Film grain is random, so the extracted "grain layer" doesn’t encode original image detail but film grain itself encodes relative lightness in its _density_.

You worry too much, it's all fake. What you think is the "footage" will be many layers from various sources composited together.

Fake lights, fake shadows, fake sky, ...

Only if you are aware of it, which 99.9% of people consuming video content are not. It’s simply an unimportant implementation detail (from a viewer’s perspective who doesn’t really care about bitrate-as-cost).