Facebook Artificial intelligence extracts video characters so we can play with

Network USA

Facebook researchers are developing a technology based on artificial intelligence that can extract a character from a video, then control it in another environment.

Facebook Artificial intelligence extracts video characters so we can play with

Facebook Artificial intelligence extracts video characters so we can play with

Facebook has developed a technology that, if it continues to be refined, could offer interesting perspectives in the world of video games, and why not in virtual worlds. Vid2Game is an artificial intelligence that analyzes videos of characters performing specific actions (a tennis player, for example) to recreate them and to control them elsewhere, in another environment.

Technology to perfect

To accomplish this task, the team used two neural networks. The first, Pose2Pose, takes care of the original video, it isolates not only the character but also its different poses. The second system, Pose2Frame, will embed the person in a new scene and manage the interactions with the places (shadows, light, etc.) while avoiding the artifacts as much as possible.

The user can then control the player's movements with a joystick or keyboard. For the moment, the result looks a bit like the first inlays on a green background, and then we still need to improve some animations (the character gives the impression of "float" on the ground).

But it's an interesting start, which if the search goes to the end, will insert his favorite sportsman, actor or YouTuber into a scene and play with it.