On November 30, 2022, OpenAI unveiled the ‘ChatGPT‘ AI system (https://openai.com/blog/chatgpt/), which is a universal writer’s helper that can create a range of output, including school assignments. ChatGPT’s output (e.g., essays, blog posts) and capabilities are outstanding, truly mind-blowing – especially if you think how fast this technology is evolving.
If I were a student, I would use ChatGPT for sure to finish most of my school assignments with minor edits.
This creates a problem for teachers and instructors since it is difficult to tell if an essay was written by the student or ChatGPT. They would need some sort of instrument to check this.
So the Question is:
How to Detect OpenAI ChatGPT Output?
GPT Output Detector
The good news is that Hugging Face has built a tool known as GPT-2 Output Detector Demo, which was created in collaboration with OpenAI and individuals from Harvard University (for more information, please visit this link: https://huggingface.co/roberta-base-openai-detector).
I requested ChatGPT to write us a blog post about the well-known WordPress Caching Plugin WP Rocket and an essay of 500 words about William Shakespeare so that I could illustrate the use of the program.
It took ChatGPT around 10 seconds to produce a brief entire blog post that included 375 words (!), and it took approximately 30 seconds to compose the essay that was over 500 words. Without making any other adjustments, we have simply cut and pasted the results into the following format:
The tool has determined that there is a 99.98% probability the blog post was generated using OpenAI GPT.
Interesting enough, it also identified correctly that the Essay was AI generated with a 99.98% probability.
It is important to keep in mind that this tool, just like anything else related to AI, has a high possibility of recognizing ChatGPT output, but that probability is not one hundred percent.
Here come the bad news.
See the prognosis shown below after we made specific adjustments to the blog article, which were produced automatically by a system that rephrases sentences. The necessary amount of time is two minutes.
As you can see, it is still not too hard to make use of AI-generated content. It will be very fascinating to see how this will develop in the future.
Try out the tool! If you are interested in learning more about how to detect OpenAI ChatGPT, have a look at the research paper that was referenced earlier in this blog post.
Update 01.02.2023:
OpenAI themselves released an own AI Classifier Tool to detect AI generated content – with rather modest results, only 26% true positives detections in their studies. (“New AI classifier for
indicating AI-written text”). You can try it here.
Resources
The following is a list of resources used or referenced in this tutorial:
- Hugging Face (https://huggingface.co/)
- OpenAI own AI Classifier Tool
- GPT-2 Output Detector Demo (https://huggingface.co/openai-detector)
- Details on GPT-2 Output Detector (https://huggingface.co/roberta-base-openai-detector)
- OpenAI chatGPT (https://chat.openai.com/chat)
- LearnGPT, dozens of ChatGPT Examples (https://www.learngpt.com/)
Demo Video How to detect ChatGPT Output
Frequently Asked Questions
As we have shown, there are tools which can detect AI-generated content like GPT-3 with a high probability. Yet there is not a 100% reliability. OpenAI Researchers showed the possibility of being able to watermark invisibly output generated by AI.
Update: OpenAI released a tool (“Open AI Classifier“) of their own who could identify AI generated Content – although with only 26% true positives rate.
It was trained on vast amounts of data from the internet, including conversations, and it was also trained using a machine learning technique called Reinforcement Learning from Human Feedback (RLHF), in which human trainers provided the model with conversations in which they played both the role of the AI chatbot and the user. The model was then trained on both types of data. In total, engineers said the whole data took 45 Terabytes.
If you already have an account at labs.openai.com or beta.openai.com, you can use the same login information at chat.openai.com. If you don’t already have an account, you’ll need to create one at chat.openai.com.
Repetition is one of the most obvious signs that a piece of content was written by an algorithm. Until now a bot just doesn’t have the same store of knowledge that a human writer does, and even when it combines data from outside sources. It doesn’t have the intuition to sort through text and find original ideas. The result is text that keeps being rewritten.
It is thought that even the most advanced AI writer can write about 100 words on a given topic before it starts repeating itself or making up new words (as of year 2022). Studies indicate that the best blog length for SEO is at least 2,000 words, so it’s unlikely that the current generation of writing bots will be able to write long-form articles that look like they were written by a human. The emphasis here is on current.
Our other Blog posts
6 Top Places to Hire WordPress Developers
Sometimes you need the help of a WordPress Professional. As WordPress is so widely used, there is no shortage of
Top 10 Mistakes Small Business Owners Make When Using SEO
Know Which Common SEO Mistakes to Avoid to Grow Your Local Business In spite of the importance of search engine