By now, most of you have probably tried using artificial intelligence (A.I.) GPT to generate text, and have seen text generated by A.I.'s such as ChatGPT, Google Bard, or Microsoft Bing.
Increasingly, it is becoming challenging to distinguish between text generated by A.I. and text generated by humans. College admission offices and educators are particularly concerned about this issue, as are news agencies. Reporters worry that they may receive fake news and report on it based on false information.
So, is it really possible to tell the difference between A.I.-generated text and human-generated text?
Example
Let's do a thought experiment. GPT A.I. is designed to write like a human. In fact, it was trained on billions of text samples, mostly written by humans. So, on the one hand, we have the technology to create A.I.'s that can write like humans. On the other hand, humans want to be able to tell the difference between AI-generated text and human-generated text.
Before we go any further, let's try using a tool called GPTZero (https://gptzero.me/). GPTZero is a new tool developed by a 22-year-old Princeton student named Edward Tian. It has been used by over 1.2 million people since it was launched in December 2022.
Just for fun, we will use ChatGPT to generate a biography about Jack.
Using GPT-3.5, it generates something like this:
I then use something written, for sure by a human --- my own personal web page at https://seng.hkust.edu.hk/about/people/faculty/jack-lau as a comparison.
First, we enter the text generated by ChatGPT and see what the "fake text" detector says.
20 out of 20 of the sentences are "A.I. generated"; 87% likelihood that the text is A.I. generated. Ooops.
What about the "ground zero" truth? Based on the text that for sure was written by human, here is the result.
OK. I am convinced. The machine says this is indeed a a human written text. 0 out of the 31 sentences are written by A.I., the machine says. Less than 50% chance (46% to be exact), the text is generated by A.I.
How does it Work?
Under the hood, there are several observations by A.I. engineers: perplexities and burstiness. What does it mean?
Perplexities: Say, a text is given, then you chop up the text into 2 parts. You show A.I. the first part and ask it what is the probability then it will generate something like the given 2nd part. Intuitively, you would draw the conclusion if it is written by a human, the text will be more creative (or you can say, all over the map), than one that is generated by an A.I.
Burstiness: For the time being, an A.I. generated text, words are more mechanically (evenly) spaced. (*Not sure as technologies continue to evolve, this will be the case though.)
While we obviously do not have source code of the commercial GPTZero, we found open source version that takes advantages of similar properties. One is provided via GitHub at https://github.com/BurhanUlTayyab/GPTZero
We ran the software. And, here is the result:
| GPT Generated Biography | Human Written Biography |
ZeroGPT.me | Probability of A.I.: 87%l Sentences likely to be A.I. Generated: 20/20 | Probability of A.I.: 46%; Sentences likely to be A.I. Generated: 0/31 |
OpenSource ZeroGPT | Perplexity: 12; Perplexity per Line: 34 | Perplexity: 25; Perplexity Per Line: 136 |
| Burstiness: 71 | Burstiness: 1393 |
There we go. Indeed A.I. detects quite well theses 2 examples, telling one from the other.
P.S. Looking at the concept of Perplexity using a daily life example.
A few of our friends said that the concept of "perplexity" is actually too perplexing. Okay. Then, let's try looking at it differently.
Suppose this. It is lunch time now, and you decide to go to a fancy restaurant to have a really nice meal. You take a look at the menu.
You will probably see a menu that starts with an appetizer, a soup, a main course, maybe a second course, followed by a dessert and a coffee.
Now, if you go to an Italian restaurant, as you scroll down the menu, you will see Italian delicacies one after another. And, at the dessert section, you will probably be torn between a tiramisu and a panna cotta.
Imagine that, what if, you see on the Italian menu, after all the fancy Italian risotto, the dessert is actually a Chinese red bean soup or a Chinese herbal tea?
That would be unusual, right? Though not improbable, but unless the chef tells you that you should be expecting a fusion meal, you would come to conclude that given all the fancy Italian dishes ahead of the dessert section, the "probability" of seeing a Chinese-style dessert should be quite low.
Bingo, that's it. Perplexity is just that in a normal writing, given a text with different words and passages, the text that follows a certain initial section should appear with different probabilities.
In an AI-generated text, the words following different words are quite predictable (less perplexing). In a human text, words can go in a very creative direction.
For instance, in my own biography, towards the end, I mention that I am a huge Star Wars fan and enjoy doing yoga. Well, in terms of probability, it is rather unusual (though not impossible) that the biography of a silly geek would say something like that. Correct? So, in my own human-written biography, the ending is more perplexing (to the computer) and more likely it is written by a human.
I hope the above explanation makes sense. See you all again soon.