The goal of this study is to develop and validate a model capable of accurately classifying text as either AI-generated or human-written. This model not only serves to enhance security measures but also aids businesses in managing the quality and authenticity of the content associated with their services. The outcomes of this research will provide foundational knowledge and tools for stakeholders across multiple industries to make informed decisions about content management and fraud prevention.
The project uses two models:
- Logistic Regression which is trained from the features extracted after feature engineering
- Long Short Term Memory-LSTM
The dataset is extracted from kaggle- "https://www.kaggle.com/datasets/shanegerami/ai-vs-human-text"
The dataset employed in this study consists of approximately 500,000 essays, meticulously compiled to ensure a balanced representation of both AI-generated and human-written texts.
Columns: The dataset is structured into two columns:
• Text: Contains the full text of each essay, representing a diverse range of topics, styles, and complexities, which provides a comprehensive basis for the analysis.
• Generated: A binary indicator where ’0’ denotes text written by humans and ’1’ signifies text generated by AI.
The blue bar, labeled ’Human-generated’ on the x-axis, represents content authored by humans. It illustrates a count of 305,797, indicating a substantial volume of content produced by human authors.
The orange bar, labeled ’LLM-generated’ on the x-axis, represents content generated by the Language Model. It shows a count of 181,438, denoting a comparatively lower volume of content produced by the LLM.
The following features are created to train a logistic Regression model:
- Stop-Word Ratio
- Number of Words
- Complex-Word Ratio
- Avg-Tf-IDF
- Number of Sentences
- Polarity Score
- Subjectivity Score
The stop word ratio is a measure that represents the proportion of stop words within the text. It is calculated by dividing the number of stop words by the total number of words in the text segment. A higher ratio indicates a larger proportion of stop words. This metric can give insight into the complexity or richness of the content; texts with a lower stop word ratio might be more information-dense. In our analysis, for each segment, we count the stop words and divided that number by the total number of tokens to get the stop word ratio. This provides a normalized value that makes it easier to compare across different text segments, regardless of their length.
Complex word means words with more than 2 syllables.
• Class 0 (Human): The left box (blue), which represents human-written text, shows a lower median complex-word ratio compared to the LLM. This indicates that, on average, the human-written texts have fewer complex words relative to their total word count.
• Class 1 (LLM): The right box (orange), representing texts generated by a large language model, shows a higher median complex-word ratio. This suggests that the LLM tends to use more complex words in its text generation.
• Interquartile Range (IQR): The IQR for the human-written texts is narrower, which implies that human writers are more consistent in their use of complex words compared to the LLM, which shows a wider IQR indicating greater variability.
• Outliers: The human-written texts have fewer outliers, while the LLM texts have several, indicating instances where the model’s use of complex words is significantly higher or lower than the typical output.
The interpretation of this is that the language model’s outputs vary more in terms of lexical complexity, possibly due to attempts to match a wide range of writing styles or contexts. In contrast, human writing, as represented in this sample, tends to be more uniform in its complexity.
This model was constructed using seven features derived from the feature engineering process, all meticulously crafted to enhance predictive accuracy. To gauge the model’s performance, the dataset is partitioned allocating 70% for training and 30% for testing. During the training phase, the logistic regression model is subjected to an array of C values, ranging from 0.01 to 100. C serves as a crucial hyperparameter signifying the inverse regularization strength. A higher C value implies lower regularization, while a lower C value indicates higher regularization. This parameter is pivotal in averting overfitting, a common pitfall in machine learning models.
Accuracy: 0.7745333333333333, C: 0.01
Accuracy: 0.8390666666666666, C: 0.1
Accuracy: 0.8429333333333333, C: 1
Accuracy: 0.8424, C: 10
Accuracy: 0.8441333333333333, C: 100
Best accuracy: 0.8441333333333333, Best C: 100Intercept and Coefficients of Logistic Regression Model
Intercept: 0.6510411238324886
stop_word_ratio: -14.59608627611113
Number_of_words: -0.005853481591525254
complex_word_ratio: 31.780465417577332
Avg_Tf-Idf: -1.0219420725587296
Polarity_Score: 1.9025795335847135
Subjective_Score: 4.606149477679758
Number_of_sentences: 0.01078394883874536Logistic Regression Model Equation
Further insights into the logistic regression model were gleaned from the intercept and coefficients. Of notable significance are two key observations derived from the logit model equation. Firstly, the coefficient associated with the stop word ratio (-14.6) is noteworthy for its negative value. This indicates that a higher prevalence of stop words in the text diminishes the log-odds of the outcome being 1. Such findings suggest that human-generated texts tend to exhibit a greater abundance of stop words compared to machine-generated texts.
Similarly, the coefficient attributed to the complex word ratio (31.78) is of particular interest. Its positive value implies that texts characterized by a higher ratio of complex words significantly augment the likelihood of the outcome being 1. This observation underscores the notion that AI-generated texts often feature a more intricate vocabulary compared to human-generated counterparts, shedding light on the nuances between the two text categories.
The second model employed in this classification analysis is the Long Short-Term Memory (LSTM) model, a sophisticated neural network architecture renowned for its prowess in handling sequential data. Prior to feeding the data into the LSTM architecture, an essen�tial preprocessing step was undertaken. Here, each word within the texts was transformed into a 100-dimensional vector using the GloVe word embedding technique. This embedding process imbued the model with a rich understanding of the semantic relationships between words, enhancing its ability to discern intricate patterns within the textual data.
The initial layer, is a masking layer, serves the pivotal function of disregarding zero paddings in the input sequences, ensuring that only relevant data was processed by subsequent layers. Following this, two LSTM layers were incorporated, each comprising 64 units, strategically designed to capture intricate temporal dependencies inherent in sequential data. Finally, the architecture culminated in a dense layer equipped with a sigmoid activation function, facilitating binary classification by outputting probabilities associated with the two distinct classes.
The model is trained with the batch size of 200 and the model acheives great results just for one epoch.
Training Loss: [0.21300531923770905]
Training Accuracy: [0.9246857166290283]
Validation Loss: [0.1302489936351776]
Validation Accuracy: [0.9583333134651184]
Logistic Regression Model Performance:
• Validation Accuracy: 84.6%
• Misclassified Essays:
– Human Essays Misclassified as AI Generated: 532
– AI Generated Essays Misclassified as Human: 617
LSTM Model Performance:
• Validation Accuracy: 95.8%
• Misclassified Essays:
– Human Essays Misclassified as AI Generated: 243
– AI Generated Essays Misclassified as Human: 382
Logistic regression is computationally efficient, but it heavily relies on feature engineering, which can be challenging. Some times the quality of the features might not match the effectiveness of generative large language models (LLMs), which excel at producing human-like text.
On the other hand, deep learning approaches are resource-intensive. The Lstm model which was developed is trained on an Azure compute instance with 32GB of RAM and it took approximately 12.11 minutes, though it simplifies the process by handling feature engineering automatically.
- Sai Harsha Vardhan Reddy, Kolan- skolan@horizon.csueastbay.edu, harsha62334@gmail.com
Thanks for reading!