Editing an image could be as simple as describing the change you want. InstructPix2Pix is a method from researchers at the University of California, Berkeley that uses natural language instructions to modify images. It can change an image’s style or setting, replace objects, or make it look like it was created in a different artistic medium.
Teaching a model to follow edit instructions
For an image-editing model to respond to a request, it needs examples connecting an instruction to the change it describes. The researchers built those examples largely with synthetic data generated using GPT-3 and Stable Diffusion.
GPT-3 produced a description of an initial image, an instruction for changing some details, and a description of the intended result. The researchers then used the two image descriptions to generate about 100 images with Stable Diffusion and the Prompt-to-Prompt image modification method. CLIP helped narrow those candidates to two similar variants that matched the requested changes.
That process produced a large training set. The team trained InstructPix2Pix on more than 450,000 Stable Diffusion image pairs, along with the corresponding GPT-3 modification instructions. In practical terms, the model learned from examples of an image, a written request, and a version changed to reflect that request.
What text-guided editing can do
The approach brings instruction-following into image processing. Instead of describing every technical adjustment, a user can state the intended change in ordinary language. The source article gives examples such as replacing an object, changing a scene’s setting, or altering its style.
The researchers say the model can process user input and images and make changes in seconds, despite relying on synthetically generated training material. That makes it a demonstration of how generated examples can support a model designed to respond to open-ended requests.
There is a meaningful difference between producing an edited image and reliably understanding every instruction. A request may involve changing how many objects appear or specifying where something should be. Those tasks require the model to handle object counts or spatial relationships, areas the researchers say it struggles with.
Where the method still falls short
InstructPix2Pix is not perfect, and its limits matter for anyone relying on it to make a precise change. The researchers specifically identify instructions that alter the number of objects and instructions requiring spatial understanding as difficult cases.
Those shortcomings also point to a possible next step: human feedback. The researchers identify it as important future work for improving the model. The article does not describe a completed feedback process, so the current results should be understood as a demonstration of the method’s capabilities and limits.
From research to image-editing tools
The model was made available on Hugging Face, and early implementations appeared in Stable Diffusion interfaces including NMKD and Auto1111. The article also reported that Playground AI seemed to have made the model available, with access after free registration.
The work may matter beyond standalone experiments. The article points to Adobe’s use of machine learning in Photoshop, including Neural Filters added in 2021 that can change the season of a landscape with a click. With InstructPix2Pix and Stable Diffusion integrations for Photoshop already available, the article suggests that workflows in the graphics industry could change quickly.
The broader idea is straightforward: image editing can become more conversational, with a written request serving as the starting point for a visual change. InstructPix2Pix shows how that can work, while its trouble with counts and spatial understanding makes clear that describing an edit does not guarantee the model will interpret it correctly.