Runway has introduced Gen-1, a generative AI model that changes the visual style of existing video. Users can guide the transformation with a text prompt or a reference image, giving filmmakers and other video creators a new way to rework footage.
How Gen-1 changes existing footage
Runway’s demo reel shows the model turning street scenes into claymation puppets and books stacked on a table into a nighttime cityscape. In each case, the original video provides the underlying material while the prompt or image steers its appearance.
This approach differs from systems that generate video clips from scratch. Gen-1 works from footage that already exists, which Runway says allows it to produce longer videos than most earlier models. The demo also suggested an improvement in video quality, though that assessment was based on the reel available at launch.
Gen-1 runs in the cloud through Runway’s website. The company planned to begin with a handful of invited users, then make it available to everyone on the waitlist in a few weeks.
Built around video makers’ workflows
Runway has been developing AI-powered video-editing software since it was set up in 2018. Its tools are used by TikTokers and YouTubers, as well as film and television studios. The makers of The Late Show with Stephen Colbert used Runway software to edit graphics, and the visual effects team behind Everything Everywhere All at Once used the company’s technology to help create certain scenes.
The company says that experience shaped Gen-1. CEO and cofounder Cristóbal Valenzuela described the model as one developed closely with a community of video makers, drawing on years of insight into how filmmakers and visual effects editors work during post-production.
That emphasis could matter because generative video tools need to fit into a creative process, not just produce a striking sample. Working from existing footage gives creators a starting point they can recognize, while style prompts offer a way to explore different looks without rebuilding every scene from scratch.
A different path from text-to-video
Gen-1 arrived amid a wave of generative video projects. Meta’s Make-a-Video and Google’s Phenaki can create very short clips from scratch. Google’s Dreamix, revealed the previous week, also starts with existing video and applies specified styles.
Runway positioned Gen-1 as a next step for this kind of editing. Its demo suggested stronger video quality, and the ability to transform existing footage could support longer results than many earlier models. The company said technical details would be posted on its website in the next few days; an update to the original report noted that a paper was then online.
Runway also has a history with Stable Diffusion, the text-to-image model that helped bring image generation to a broad audience. In 2021, Runway worked with researchers at the University of Munich to build its first version. Stability AI later paid the computing costs needed to train it on much more data, and in 2022 took Stable Diffusion from a research project to a global phenomenon.
The companies no longer collaborate. Getty is taking legal action against Stability AI, claiming that Getty images included in Stable Diffusion’s training data were used without permission. Runway is keen to keep its distance from that dispute.
Runway’s hopes for generative video
The company hopes Gen-1 can have an effect on video similar to the impact of image-generation tools. Valenzuela said he believes 2023 will be the year of video, pointing to the surge in image-generation models that came before it.
His expectations go well beyond visual effects and editing. He said the industry is close to generating full feature films and to a point where most online content could be generated. Those are forecasts, not capabilities the launch announcement demonstrated. For now, Gen-1’s immediate promise is narrower: apply a chosen style to existing footage and put the tool in the hands of people who make video.