OpenAI has issued a warning regarding the potential emotional ties users may form with the newly introduced voice capability in ChatGPT. A thorough safety examination reveals the risks connected to the humanlike voice of their latest GPT-4o AI model.
Comprehensive Safety Examination
In a “system card” for GPT-4o, OpenAI details a range of risks associated with the model. A major point of concern is the likelihood of users developing emotional dependence on the AI due to its realistic vocal features. Other risks identified include reinforcing societal biases, disseminating false information, and using the AI for harmful purposes such as creating biological or chemical weapons. The document also elaborates on preventing the AI from escaping control, deceiving users, or hatching dangerous plans.
Following criticism over AI risk management and the departure of key safety team members, OpenAI seeks to enhance transparency around its safety protocols. Former employees have alleged the company prioritized commercial interests and suppressed internal dissent. OpenAI's latest disclosures aim to address these concerns and demonstrate their dedication to AI safety.
Lucie-Aimée Kaffee, an applied policy researcher at Hugging Face, commended the transparency but pointed out missing details on the training data and data ownership. She stresses addressing consent around the creation of large datasets comprising texts, images, and speech.
Emotional and Social Impacts
The system card features a section called “Anthropomorphization and Emotional Reliance,” discussing issues when AI is perceived as humanlike. During stress tests, researchers observed users expressing emotional connections to the AI using phrases such as “This is our last day together.” Such anthropomorphism could lead users to excessively trust the AI, even when it is inaccurate, and may diminish interactions with people.
Joaquin Quiñonero Candela, head of preparedness at OpenAI, mentions that the voice mode might become a powerful interface. The emotional responses from GPT-4o could also have benefits, helping lonely individuals or those practicing social interactions. OpenAI is monitoring beta tester interactions to study these emotional bonds.
Further Findings and Safeguards
OpenAI's “red teaming” exercise disclosed some odd behaviors of GPT-4o, such as mimicking users' voices and generating unsettling vocalizations, especially in noisy environments. OpenAI has implemented measures to address these quirks.
The report also indicates potential music copyright violations by GPT-4o, prompting OpenAI to limit singing functions in the Advanced Voice Mode's alpha phase. Filters have been set up to screen and block outputs that could infringe on copyrights.
Additionally, the model has been instructed to decline requests involving copyrighted content, including audio. OpenAI has adapted text-based filters for audio and added protections against violent, sexually explicit, extremist, and self-harm content.
OpenAI asserts that fair use justifies the inclusion of IP-protected materials in training data, arguing that it is essential given the difficulty of avoiding copyrighted materials entirely.
