AMC Theatres will screen a Swedish movie 'visually dubbed' with the help of AI

You can see a lot of open mouth/tongue stuff being skipped. Dealing with the tongue and the inside of the mouth is a huge problem with this sort of visual dubbing. Using traditional techniques, you can model teeth and gums as rigid bodies, and faces as rubber sheets (to first approximation), but tongues, for which you typically have no visual reference in any given shot, are much more difficult to model, and continuously, subtly, on the move. "AI" is the general answer to this problem nowadays, but even ML-based systems struggle to deal with the tongue issue while trying to reconcile visual appearance with animation fidelity.

Yes, Kung Pow is a great example of how jarring it can be! :-D

It was quite jarring in Amazon's Citadel Diana. The voices were ever so slightly out of sync with the lip movements, and the audio sounded like the studio recording hadn't been processed to match the environment.

[dead]

I want to be on board with the “bring more audience” but the truth is knowing how this industry works, any success with AI on this front will lead almost immediately to a total drying up of funding for professionals engaging in this work currently. I hope I’m wrong, I really do.

Imo the real situation is that US distributors suck at dubbing and/or rely too much on celebrity voices rather than good voice actors that can dub properly.

I have seen some US dubs that would have been improved by a polish-style voiceover dialog

Context: I think most recent use cases of AI is slop, generally people who identify as pro AI think I’m an AI doomer. But there are some good use cases of AI.

This sounds like a good use. It reminds me of the early days of CG, it took a while for it to look good, and now it’s a tool in the toolbox alongside practical effects. This is gonna look bad for years, but I think ultimately will be a good thing. Anything that helps shrink the world and understand other cultures is a plus for me.

But man, as a film lover I know this gonna annoy me over the next 5 years.

If it looks good it's fine, but I'm not as optimistic as you.

I would argue that pan-and-scan failed long-term, perhaps even short term. I remember widescreen releases being available on VHS because a significant amount of people didn't like it.

Colorizing black and white movies...I can't think of any time I've seen that done in a commercially successful way.

I also think that a dub with lips that don't match is something people barely notice in the first place.

I would argue that pan-and-scan solved a practical problem, and while it wasn't ideal, it was also just flat out necessary for a while. Your average TV in the '80s would have been around 20", and VHS only gave you 240 lines of vertical resolution. Letterboxing a 2.39:1 movie meant cutting both of those metrics in half. Preserving the original framing doesn't mean much when a sizable portion of your audience is stuck squinting at the screen.

Some filmmakers kept this in mind when making their movies in the '80s and '90s. Films like Terminator 2 were shot open matte, allowing each shot to be framed for 4:3 and the target theatrical widescreen aspect ratio simultaneously.

I think these facts got lost on a lot of people who were die-hard widescreen LaserDisc fans. LaserDisc had twice the vertical resolution of VHS. And if you were a big enough moviebuff to have a LaserDisc collection, you probably also had a bigger than average TV.

The rest of these techniques, like colorization, seem more about making older content more palatable to newer viewers, and don't really do anything to solve a technical problem.

I'm learning a language, and I'd love to be able to filter Netflix titles by language, but they absolutely refuse to do it (I assume because allowing users to filter content would expose how bare the offerings are).

You might enjoy https://easterneuropeanmovies.com/, https://sovietmoviesonline.com/ or https://asian-movies-online.com/. They have both movies that are popular hits and that have been important to the craft and history of the art. Subtitles are usually from open sub-style sources so not professional quality, but I find it sometimes help with learning the language when it's more literally translated than idiomatically.

As someone who doesn't speak Swedish, what makes this particular passage interesting, and what is it about the Norrköping dialect makes it interesting?

(I have attempted to study Swedish briefly, but I feel like I need to do more sustain discovery of content to consume in order to keep me engaged long term)

I feel like Clutch Cargo was only the most successful (however limited you define that) of range of cartoons that used that technique.

???

https://youtu.be/PTngv5MmtXo?t=105

They're admittedly pretty cagey with the parts they're willing to show, but the parts they did show look fine to me. Are you referring to the parts when they showed the 3d mask overlay on top of the characters?

A 3 minute previewing the technology that shows actual ~4s of "dubbed" footage.

Seems legit lol, nothing to brush under the rug definitely.

Man, this is so much better than many Bollywood movies that are re-dubbed. When the actors go back to the sound studio to re-record their lines, they don't match up with their lips exactly, and it can be off-putting.

Why wasn't the interview done in Swedish and processed with this technology?

Note that for this movie, the actors are recording the translated lines manually. The AI is just there to change the lips.

TFA:

> Notably, the original actors recorded their own dialogues in English in a sound booth — Flawless AI's technology merely altered the movements of their lips in the movie.

the article has the answer