Viggle Creates Controllable AI Characters for Memes and Idea Visualization

You might not be aware of Viggle AI, but you’ve got probably seen the viral memes it created. The Canadian AI startup is responsible for dozens of remix videos of rapper Lil Yachty bouncing around on stage at a summer music festival. In one among the videos, Lil Yachty was replaced by Joaquin Phoenix is ​​the Joker. In one other, Jesus gave the impression to be stirring up the emotions of the group.. Users have created countless versions of this video, but one AI startup fueled the memes. And Viggle’s CEO says YouTube videos are fueling his AI models.

Viggle trained a 3D video foundation model, JST-1, to attain a “true understanding of physics,” the corporate says in its press release. Viggle CEO Hang Chu says a key difference between Viggle and other AI video models is that Viggle lets users specify the movement they need the characters to make. Other AI video models often create unrealistic character movements that don’t obey the laws of physics, but Chu says Viggle’s models are different.

“We’re basically building a new type of graphics engine, but only using neural networks,” Chu said in an interview. “The model itself is completely different from existing video generators, which are mostly pixel-based and don’t really understand the structure and properties of physics. Our model is designed to have that understanding, and so it’s much better in terms of controllability and generation efficiency.”

For example, to create a video of the Joker as Lil Yachty, simply upload the unique video (Lil Yachty dancing on stage) and a picture of the character (the Joker) to perform the move. Alternatively, users can upload images of the character together with text prompts with instructions on how one can animate them. As a 3rd option, Viggle lets users create animated characters from scratch using just text prompts.

But memes only make up a small percentage of Viggle’s users; Chu says the model has gained widespread adoption as a visualization tool for creators. The videos are removed from perfect—they’re shaky, the faces expressionless—but Chu says it’s proven effective for filmmakers, animators, and video game designers to show their ideas into something visual. Right now, Viggle’s models only create characters, but Chu hopes they’ll eventually enable more complex videos.

Viggle currently offers a free, limited version of its AI model on Discord and its web app. The company also offers a $9.99 subscription for more storage and provides some creators with special access through its Creator Program. The CEO says Viggle is in talks with film and video game studios about licensing the technology, but he also sees it being adopted by independent animators and content creators.

On Monday, Viggle announced that it has raised $19 million in a Series A round led by Andreessen Horowitz, with participation from Two Small Fish. The startup says the round will help Viggle scale, speed up product development, and expand its team. Viggle tells TechCrunch that it has partnered with Google Cloud, amongst other cloud providers, to coach and run its AI models. These partnerships with Google Cloud often include access to GPU and TPU clusters, but typically don’t include YouTube videos on which to coach the AI ​​models.

Training data

During a TechCrunch interview with Chu, we asked what data Viggle’s AI video models were trained on.

“Until now, we have relied on publicly available data,” Chu said, echoing an identical line, OpenAI CTO Mira Murati responds to query about Sora’s training data.

When asked if Viggle’s training dataset included YouTube videos, Chu replied bluntly, “Yes.”

That might be an issue. In April, YouTube CEO Neal Mohan told Bloomberg that using YouTube videos to coach an AI video text generator could be “obvious violation” terms of use. The comments focused on OpenAI’s potential use of YouTube videos to coach Sora.

Mohan explained that Google, which owns YouTube, could have agreements with some creators to make use of their videos in training data sets for Google DeepMind’s Gemini. However, collecting videos from the platform is just not permitted, based on Mohan and YouTube Terms of Servicewithout obtaining prior consent from the corporate.

Following TechCrunch’s interview with Viggle’s CEO, a Viggle spokesperson emailed to retract Chu’s statement, telling TechCrunch that the CEO “spoke too soon about whether Viggle uses YouTube data for training. In fact, Hang/Viggle is not in a position to share the details of its training data.”

However, we noted that Chu has already done this officially and asked for a transparent statement on the matter. A Viggle spokesperson confirmed in his response that the AI ​​startup trains on YouTube videos:

Viggle uses various public sources, including YouTube, to generate AI content. Our training data has been fastidiously curated and refined, ensuring compliance with all terms of service throughout the method. We prioritize maintaining strong relationships with platforms like YouTube and are committed to adhering to their terms, avoiding mass downloads and every other activities that will involve unauthorized video downloads.

This approach to compliance seems to contradict Mohan’s comments in April that YouTube’s video corpus is just not a public source. We reached out to YouTube and Google spokespeople but have yet to listen to back.

The startup joins others within the gray area of ​​using YouTube as training data. It has been reported that multiple AI model developers – including OpenAI, Nvidia, Apple and Anthropic – everyone uses video transcripts or YouTube clips for training purposes. It’s a grimy secret in Silicon Valley that is not so secret: Everyone probably does it. What’s really rare is when people say it out loud.