Man or machine: who can write the best content? To investigate this question, Peak Ace CEO Bastian Grimm conducted an experiment at SEOkomm 2022. To begin, the 500+ participants in the audience were divided into four different, random groups, each consisting of 116-46 people. The participants reached under their seats, where they each found an envelope containing two texts on the same topic. The texts were labeled with the letters A and B. One had been written by a human, while the other was machine-produced by the AI writing platform GPT-3.
The texts were about four different topics – one for each group of participants – but text A was always penned by a real person and text B was always created by a machine. The participants didn’t know which text had been written by whom ahead of time; they just had three minutes to read their texts before voting for their favourite in an online survey.
The results of the vote line up with those of the Flesch Index
The results of the experiment quickly made it clear that the participants couldn’t tell which text was created by a human and which by a machine. Almost all of the four groups rated both texts as similarly good. You can see the results below. Remember: text A was created by a human, while text B was generated by a machine.
The Flesch index uses a formula to judge how readable a text is. It considers factors like the number of words and sentences within a text, the average sentence length and number of syllables, but it doesn’t look at the content. Ultimately, it provides you with a value – the Flesch Reading Ease Score – which is between 0 and 100 and helps to classify the text. The easier the text is to read, the higher the score.
The SEOkomm texts were also put through the Flesch Index. It came to a very similar conclusion as the participants. Take a look:
The AI-produced content scored slightly higher on the Flesch index than the content written by humans. This means it’s also considered “better” by this metric. This makes sense when you consider that according to Flesch, shorter sentences mean improved readability. GPT-3 tends to take this into account, so complex sentence constructions are unlikely to be found in the B texts.
The study finds that AI creates better marketing texts
The fact that machine-generated content can keep up with texts written by humans was also proven by the recent study conducted by the Institute for Conversational Business at Aalen University. As part of the study “Human or Machine: Who Writes the Better Content?” published in September 2022, landing pages, blog posts and social media posts of international top brands such as Vodafone, Telekom, M&Ms and Starbucks were compared with AI-generated alternatives. The relevant keywords were determined in advance with the help of the AI content platform rellify and GPT-3 produced the new content on the given topics. Then, the AI texts and the originals were presented to 100 randomly selected people for review.
Meanwhile, the AI versions and the original texts were also rated using the Flesch Index. The AI content consistently performed better with the Flesch Index. While the machine-generated landing page from Vodafone led by just two percentage points, the difference for social media posts was much more marked at 38 points. This clearly shows that short texts can be well written using AI in its current state.
AI-assisted content creation could become mainstream in the long term
“According to both the subjective evaluations of our test group of around 100 people and the objective results of the Flesch metric, the AI-written variants of the existing content were better than their originals,” says study leader and AI researcher Prof. Dr. Peter Gentsch. “From the subjects’ point of view, the AI texts were not only easier to read and more appealing, but also surprisingly more personal,” explains the AI researcher.
However, these results don’t necessarily make AI a better author than trained copywriters at an agency – at least not yet. “It still doesn’t work without human input,” says Gentsch. For one thing, the machine-generated input from the AI ideation phase must be proofread and possibly edited. Additionally, fine-tuning the prompts and parameters of the GPT-3 platform are absolutely critical to success.
Professor Dr. Gentsch’s overall conclusion is as follows: “Human expertise is still needed for setting up AI and, above all, for quality assurance and fact checking”. Of course, the study also shows that when this human expertise is provided, better results can be produced in less time – and thus at a lower cost. Ultimately it’s certainly possible that, as Gentsch puts it, “AI-assisted content creation will quickly become established.”
The Peak Ace AI Survival Guide
