The AI-generated video revolution is here. Much like how text-to-image AI generators took the world by storm in the summer of 2022, 2023 is shaping up to be the year of text-to-video experimentation and discovery. If you’ve had the pleasure of watching the myriad uncanny AI videos that have been popping up online in recent months — occasionaly parodying food commercials, like this unofficial AI-generated Pepsi ad — then you’ve likely seen the work of tools like Runway’s Gen-2 model or the more recent Zeroscope.
Runway’s official tagline for its text-to-video product offerings — “No Lights. No camera. All action.” — is at once trite and exhilarating, underscoring the potential for tools like Gen-2 to completely upend the film industry, from indie productions to high-budget Hollywood affairs. If that sounds hyperbolic, just take a look at some of the entries to Claire Silver’s fifth AI art contest, which revolves around short-form AI video and animation.
More sophisticated AI video tools and productions are going to arrive on the scene quicker than you might realize. Before they do, they’re likely to play a small but growing role in studio productions, just as text-to-image AI models did in the opening credits to Marvel’s Secret Invasion last month.
Whether you’re interested in playing around with AI video for fun or want to employ these tools for something more serious, knowing how to use them will make all the difference. To get a better understanding of AI video and how to make it, we sat down with Roope Rainisto, the man behind the ground-breaking Life In West America and Reworld NFT collections and one of the leading figures of the AI post-photography and post-video movement.
Just five of Rainisto’s AI video creations have garnered more than 3 million views on TikTok in recent weeks, and few know their way around these programs as well as he does. In this week’s edition of /Promptly, our series that breaks down AI tools from the experts who use them, Rainisto tells us what to do (and what to avoid doing) to bring your video ideas to life with Runway’s Gen-2 text-to-video model and the newly released Zeroscope.
Tip 1: Use Runways’ Gen-2 model for consistent results
Runway is an applied AI research company that provides a host of tools to its users from text-to-image, image-to-image, inpainting functionality, and more. But it’s most well-known for its Gen-1 and Gen-2 text-to-video AI models, which Rainisto has used to create some wonderfully absurd short story creations in recent months, including Duck Story.
One of the benefits of using Gen-2, Rainisto says, is its reliability and consistency in its outputs. If you’re looking for aesthetically pleasing that are going to give you pretty much what you’ve described in your prompts, Runway is the way to go.
“Runway is the more mature option in a way,” Rainisto said while speaking to nft now. “It has an extensive set of tools, and I think the quality of the video output from an individual prompt is more mature compared to others. There’s less of a chance that you’ll get something completely freaky. It’s similar to Midjourney. It almost doesn’t matter what you write in Midjourney, you get a result that is at least slightly pleasing. Runway has that compared to Zeroscope.”
“If you’re creating something that you want to look beautiful and aesthetically pleasing, Runway’s Gen-2 is the tool to use right now.”
Roope Rainisto
Rainisto contrasts Runway with Zeroscope, the free and open-source software for AI video creation available on HuggingFace, the data science platform that hosts tools for users to create, train, and implement machine learning models. Fully utilizing the former’s model is easier for the average user with a laptop, though it comes with a price — Runway currently has four tiers of plans available for people to use its software, starting at free and moving up to $76 a month.
“If you’re creating something that you want to look beautiful and aesthetically pleasing, Runway’s Gen-2 is the tool to use right now,” Rainisto emphasized.
Tip 2: Use Runway’s Gen-1 and Gen-2 differently (and together)
Rainisto advises that Runway users play around with combining Gen-1 and Gen-2 outputs to obtain a desired result, as both are good at slightly different things.
“It’s worth mentioning Gen-1 and Gen-2 at the same time,” Rainisto elaborated. “If you have a certain thing in mind that you want to tell a story about, it’s actually a good idea to use them both. In those cases where Gen-2 fails, use Gen-1. Gen-2’s results, at this moment in time, there’s not so much movement, it’s almost like a picture that is slightly moving. But in many cases, you want a lot of movement, someone running or something. Gen-2 is not as capable of doing that. It often produces something like slow-motion shots.”
Utilizing a workflow that pulls in other tools, like Midjourney, is a great way to achieve better results. Rainisto often creates images in text-to-image generators to use as visual guides for Runway’s Gen-2. He will then take a screenshot of the video output to use as a seed for Midjourney, repeating the process as many times as needed.
Tip 3: Use Zeroscope for movement (and surprises)
Pointing to one of his most recent AI video creations, Agent Story, as evidence of what Zeroscope’s model can do, Rainisto says that it’s the best tool to use if you’re looking for unexpected results or a model that can handle lots of (literal) moving parts.
“The cons with Zeroscope are that the base-level quality is much rougher, but the pro is that there’s much more movement on screen,” Rainisto said. “Sometimes it even moves around too much, but that’s an easier problem. You can do a cut in the editing process.”
Tip 4: When marketing your video content, know your social platforms
While not directly connected to the creation of AI video, Rainisto underlined the difference in audience interaction with his AI video content from platform to platform. Apart from TikTok’s ability to bring content views and comments in numbers that he just doesn’t see on Instagram, Twitter, or YouTube, Rainisto says the kind of engagement he gets on the short-form video platform for his AI videos is unique.
“My first guess was that TikTok was full of bots, but it’s actually a lot of real people, and it’s really interesting conversation,” the artist said. “Your first guess might be that all the comments would be like, ‘Hey, this looks bad.’ That’s like five percent of the commentary. Most people actually say things like, ‘Hey, this reminds me of my dreams,’ or, ‘This is absolutely horrifying,’ ‘I’m scared, or, ‘I’m excited.’ It’s really not so much about the tech side. When I post something on Twitter, most of the engagements is, ‘Hey, how did you do that?’
When asked why he thinks people, regardless of social platform, are drawn to the perceived “wrongness” or “strangeness” of current text-to-video AI models, Rainisto says the answer boils down to what entertainment really is at its core.
“Would those stories be better if they were 100 percent photorealistic? Would you be more entertained? The answer is not instantly ‘yes.’”
Roope Rainisto
“The first question is this: Would those stories be better if they were 100 percent photorealistic? Would you be more entertained? The answer is not instantly ‘yes.’ When an art piece leaves something for interpretation, this engagement really attracts the mind. This technology will certainly get better and more realistic. But now it’s capable of doing something that no expensive camera is able to do.”
Tip 5: Don’t wait to experiment (especially if you’re a film student)
Lastly, Rainisto emphasizes the importance of capturing these tools as they exist at this point in time. AI models get updated all the time — Midjourney has gone through more than five versions of its model in the span of a year — and that means what users create with these tools now will forever be unique to this particular moment in the development of AI technology.
“It’s really interesting to try to capture the tools as they are now in this month in 2023 because, in a few years, they will be far different. That’s also why these tools are really fascinating right now.”
What’s more, the wider societal benefits of having AI video generators have yet to be fully realized, according to Rainisto. Breaking into the film industry without connections and proper funding used to be something of an impossibility, he notes. Instead of needing to write a and submit a screenplay and hope it gets in front of the right people to be approved and then sent into production on even the most basic level, text-to-video AI tools can help creatives get their vision out into the world in record time.
“These tools will make it so that a writer can write a script, make a short version of that script using AI, and put the video on YouTube instead of spending their time trying to contact 100 producers and hoping somebody will read their material.”
Roope Rainisto
“If you’re a film student, you would certainly want to use this in school just to practice and learn,” Rainisto encouraged. “Writing scripts, submitting them to producers, to competitions, hoping to find an agent, and then breaking into the industry — the traditional routes are extremely hard if you’re not known. These tools will make it so that a writer can write a script, make a short version of that script using AI, and put the video on YouTube instead of spending their time trying to contact 100 producers and hoping somebody will read their material.”
The post /Promptly: Roope Rainisto on How to Create Great AI Video appeared first on nft now.
Web Source: nft now