ChatGPT is gaining ways to work with images and spoken conversation. OpenAI says users will be able to share pictures for discussion, while its mobile app will add synthetic voices that can respond aloud. The changes expand how people can interact with the assistant, but OpenAI also cautions that its answers can be inaccurate.
Questions can start with a picture
The image feature will let users upload one or more pictures and discuss them with either GPT-3.5 or GPT-4. OpenAI says it is intended for everyday tasks, such as looking at a fridge and pantry to consider dinner options or asking for help when a grill will not start.
Users will also be able to mark a specific part of an image on a device’s touch screen, directing ChatGPT’s attention to that area. That gives people a way to provide visual context and indicate what they want the assistant to consider.
OpenAI’s promotional example shows someone asking how to raise a bicycle seat, then sharing photos, an instruction manual and a picture of a toolbox. ChatGPT responds with guidance for the task. The example illustrates the intended interaction, but the feature’s real-world effectiveness had not been tested in the source article.
Voice brings spoken replies to the app
The mobile app’s speech synthesis is designed to work alongside speech recognition already available in ChatGPT. With both, users can have a spoken back-and-forth conversation instead of relying only on typed messages and written replies.
OpenAI says the voice feature will come to iOS and Android. Users will be able to opt in through the app’s settings and choose among five synthetic voices: “Juniper,” “Sky,” “Cove,” “Ember,” and “Breeze.” The company says professional voice actors helped create them.
For incoming speech, OpenAI’s Whisper system will continue to transcribe what users say. Whisper has been part of the ChatGPT iOS app since it launched in May. The ChatGPT Android app arrived in July.
Availability and how the features may work
OpenAI planned to release the features to Plus and Enterprise subscribers “over the next two weeks.” Image recognition was slated for the web interface and mobile apps, while speech synthesis was limited to the iOS and Android apps.
OpenAI had not shared technical details about how GPT-4 or its multimodal version, GPT-4V, works internally. The article describes one possible explanation based on research by others, including OpenAI partner Microsoft: multimodal systems can represent images and text in a shared encoding space, allowing a neural network to process both kinds of input.
It also raises the possibility that CLIP could help align visual and text representations in the same latent space. That might support contextual reasoning across a picture and a written question, but the article presents this as speculation rather than a confirmed description of OpenAI’s system.
Useful inputs still need careful interpretation
OpenAI acknowledges that the vision feature may misidentify things and may not recognize non-English languages reliably. The company says it conducted risk assessments “in domains such as extremism and scientific proficiency” and gathered input from alpha testers. It nevertheless advises caution, especially for specialized or high-stakes uses such as scientific research.
The company also says it has taken “technical measures to significantly limit ChatGPT’s ability to analyze and make direct statements about people” to address privacy concerns. The limits matter because a visual answer can sound confident while still being incorrect, and images can contain information about individuals.
OpenAI describes the assistant as able to “see, hear, and speak,” while AI researcher Dr. Sasha Luccioni argues that this language makes the system sound too human. Her point is that the model receives information through different kinds of input; that does not make it human. The new features broaden the ways people can communicate with ChatGPT, but their usefulness and limits depend on how well they perform in practice.