A recent study by emlyon business school and Toulouse School of Management reveals that consumers find it increasingly difficult to tell whether product reviews are written by humans or artificial intelligence, such as ChatGPT. The findings highlight the growing role of generative AI in marketing and raise important questions about the effectiveness and authenticity of AI-generated content.
AI-Generated Reviews: More Positive, But Less Detailed
According to the research, AI-generated reviews are often more overwhelmingly positive compared to their human-written counterparts. However, these reviews are typically less detailed and more likely to contain inaccuracies. This pattern suggests that while AI is capable of producing convincing content, it still struggles with the nuances and depth that human creators bring to their work.
The study, led by Yingting Wen, Professor of Marketing at emlyon business school, and Sandra Laporte, Professor of Marketing at Toulouse School of Management, sought to explore whether generative AI tools could create content that resonates with consumers at a similar level to that of human marketers. The research specifically focused on marketing content such as product reviews, social media posts, and advertising copy.
The Study Methodology
The researchers conducted three separate studies to compare AI-generated content to human-generated content. Two products, whisky and chocolate, were chosen to focus on sensory experiences and highlight how AI might perform in these contexts.
In the first study, a text analysis tool was used to compare AI-generated product reviews with human-generated ones. The results showed that human-written content appeared more genuine, with a less positive tone than the AI-generated reviews. The second study confirmed these findings, as human raters evaluated the content and agreed that AI reviews lacked the authenticity and depth of human-written reviews.
The third study examined social media posts, with some posts created by ChatGPT and others by human social media marketers. Human raters reviewed the content, assessed the sentiment, and evaluated engagement levels. The study concluded that while AI struggled with generating rich, varied, and detailed content, it was highly effective at creating posts with positive emotions and persuasive appeal.
AI’s Strengths and Limitations
Despite the positive emotions and persuasive effectiveness, the study found that AI struggles to create detailed and diverse content like human marketers. However, generative AI tools like ChatGPT are highly effective at engaging consumers and increasing purchase intent, particularly on social media. Notably, the study also revealed that ChatGPT 4 performed significantly better than its predecessor, ChatGPT 3.5, in generating persuasive content.
“As generative AI tools like ChatGPT are increasingly integrated into marketing, they streamline tasks such as creating social media posts and handling customer interactions, leading to greater engagement and boosting purchase intent,” explains Professor Wen.
The Need for Human Input in AI Marketing
Although AI-generated content can be effective, the study highlights the importance of human input. AI lacks the nuanced understanding and authentic voice that human marketers provide, making human involvement critical in the creation of impactful content. “While AI can be hugely beneficial in marketing, particularly in automating tasks and enhancing efficiency, it still requires human oversight to ensure accuracy and authenticity,” Professor Wen explains.
Addressing Misinformation
The researchers also pointed out the challenge of misinformation, particularly in cases where AI tools like ChatGPT generate false or fabricated information. This issue underlines the need for careful integration of AI-generated content into marketing strategies, ensuring that inaccuracies are avoided.
Conclusion
The study’s findings underline the growing potential of AI in marketing, especially for tasks like creating social media posts and responding to customer interactions. However, the research clearly indicates that human marketers are still essential in maintaining the authenticity, accuracy, and depth of content. As AI continues to evolve, its role in marketing will grow, but collaboration between AI tools and human input will be key to creating successful, engaging, and trustworthy marketing content.