Do Artificial Intelligence Chatbots Have a Writing Style? An Investigation into the Stylistic Features of ChatGPT-4

I.INTRODUCTION

From the time of their introduction, artificial intelligence (AI) Chatbots have received enormous interest from academicians, professionals, and students. Researchers examined opportunities and challenges and provided recommendations for the use of AI for academic purposes (i.e., [1]). For opportunities, they highlighted that the use of ChatGPT can be beneficial for students as it provides them with answers to theory-based questions and ideas for application-based questions [1]. For challenges, they highlighted that ChatGPT can be used by students to generate human-like responses that cannot be detected by plagiarism detection software. For recommendations, they highlighted that ChatGPT can be used in classes to generate examples that can be used in classes for practice purposes. These responses can provide authentic material to examine and analyze for the benefit of students. AlAfnan et al. [1] also provided a recommendation to plagiarism detection software providers to upgrade their software to allow the detection of AI-generated material to, as a result, maintain their status.

Recently, Turnitin announced that their software can detect AI-generated material [2]. They declared that they are working for years on an AI-powered solution lab to detect material that is generated by AI. They announced that their model has been carefully trained on academic material that has been drawn from a large database, as opposed to just information that is freely accessible. As a result, Turnitin is better suited to identifying instances of possible plagiarism in student work, Chechitelli [2] claims. Conroy [3] stated that as AI writing tools develop rapidly, so are plagiarism detectors. He declared that Turnitin can detect material that is written by AI writing tools, in general, and ChatGPT, in particular. This means that even with the development of sophisticated AI writing tools, instructors can rest easy knowing that the work of their students will be carefully vetted for plagiarism, according to Conroy [3]. This, if true, is considered a huge advancement for the similarity detection software provider as it is also considered a huge relief for academicians.

On the 14^th of March, 2023, Open AI announced the release of their ChatGPT 4. According to Open AI [4], GPT-4 is 40% more likely than GPT-3.5 to produce factual responses and 82% less likely to react to requests for content that is not allowed. Open AI [4] believes that GPT-4 is more innovative and team-oriented than ever. While working with users on creative and technical writing activities like songwriting, screenwriting, or figuring out a user’s writing style, it can generate, edit, and iterate with them. Metz and Collins [5] reported that GPT-4 passed the bar exam, and many standardized examinations had the ability to summarize and comment on visuals and texts and showed significant accuracy increases over GPT-3.5. As Open AI did not announce any facts about the development of ChatGPT 4, some scientists contended that the model’s restrictive design prevented others from extending GPT-4’s advancements, rendering it a “dead end” for the scientific community [6]. This study is not interested in the creation of ChatGPT from the scientific perspective; it is concerned about the outcome that ChatGPT-4 can generate from the academic perspective.

In particular, this study strives to provide answers to the below-listed research questions.

• 1.Can Turnitin detect ChatGPT 4-generated responses?
• 2.Are Turnitin AI reports accurate?
• 3.What are the stylistic features of ChatGPT-4?

II.LITERATURE REVIEW

According to Sadiku et al. [7], AI is the capacity of a computer system to carry out human functions (such as thinking and learning) that are often only possible through human intelligence. AI technology in education offers a level of adaptability and flexibility that was previously unattainable. Technology is transforming education and easing the work of teachers in classrooms and schools. Joshi et al. [8] note that practically every aspect of our life today exhibits the use of AI. Modern technology like AI is revolutionizing all facets of how we interact with one another. It creates innovative educational strategies that are put to the test in various settings. New educational technologies can help manage and accomplish educational objectives more effectively. Huang et al. [9] stress that the development of novel technologies affects instructional strategies. The usage of AI in education is becoming more and more clear as a result of the technology’s recent rapid progress. This article examines the use of AI, ChatGPT 4 in particular, for educational purposes.

Baker and Smith [10] examined educational AI technologies from three perspectives, which are learner-facing, teacher-facing, and system-facing AI for Education. Learner-facing Software such as adaptive or personalized learning management systems, or ITS, are examples of AI tools that students can utilize to learn a subject. By automating functions like administration, evaluation, feedback, and plagiarism detection, teacher-facing technologies assist teachers and lessen their workload. AI for education systems also gives teachers visibility into their students’ learning progress so that they can proactively help and direction as needed. AI for education with system-facing instruments can offer data to managers and administrators at the institutional level, for example, to track attrition trends among faculties or colleges.

One-on-one private tutoring can be simulated with the help of intelligent tutoring systems (ITS). They can choose a student’s learning path and the content to teach them based on learner models, algorithms, and neural networks. They can also help the student engage in conversation by offering cognitive scaffolding. In large-scale distance learning institutions that conduct modules with thousands of students, where one-on-one coaching from humans is not practical, ITS offers enormous promise. Many studies demonstrate that learning is a social activity, with interaction and collaboration at its core [11]. However, it is necessary to facilitate and police online collaboration [12]. AI for education can assist adaptive group creation based on learner models, facilitate online group interaction, or summarize discussions that can be utilized by a human tutor to direct students toward the course’s goals and objectives. Any of these methods can contribute to collaborative learning.

Recently, AI has developed in a way that can provide human-like responses to questions. These responses can be used as parts of academic submissions for students as they can be used by professionals. This brought a lot of scare to many professionals and academicians. AlAfnan et al. [1] examined the use of one of these technologies (ChatGPT) in communication, business writing, and composition courses. They found that the material generated by ChatGPT is accurate and, if used by students, it can receive a relatively high grade. After checking the similarity of the submissions, it was apparent that the submissions were mainly original, but included elements of plagiarism. AlAfnan et al. [1] provided a number of recommendations to the instructors teaching these courses as they provided a recommendation to similarity detection software providers to upgrade their software to detect AI-generated responses.

Turnitin has announced that they have been working on AI detection software for years. They mentioned that their tool can detect these AI-generated responses. However, as Turnitin developed its software, Open AI has also developed a new version of its ChatGPT Chatbot (ChatGPT 4). ChatGPT 4 is viewed as a very powerful AI tool that can, according to the Open AI, generate 40% more likely than GPT-3.5 factual responses and 82% less likely to react to requests for content that isn’t allowed. Open AI also announced that their new Chatbot tool is more innovative and team-oriented than ever. In this study, the examination of ChatGPT 4-generated responses is carried out from two different perspectives. Firstly, the generated responses are tested on Turnitin for similarity detection to identify ChatGPT 4’s ability to generate original material, on the one hand, and Turnitin’s ability to detect ChatGPT 4’s generated responses, on the other hand. Secondly, ChatGPT4 responses are examined manually by the researchers to identify any patterns in the ChatGPT 4-generated responses. The examination is carried out through the investigation of the stylistic features of texts.

Analyzing a text’s style is the emphasis of stylistics. According to [13], stylistics studies language expression as well as how it is described in terms of its purpose and impact. Ganakumaran and Edwin [14], who defines stylistics as combining literary texts with linguistic approaches and goals, emphasize this further. So, it fills a need in linguistics since stylistics is the only branch of linguistics that permits the use of linguistic methods in the examination of texts and their interpretations. At various levels, stylistics allows for the observation of grammatical, phonological, lexical, and graphological aspects of language. According to Simpson [15], stylistics is a method of textual explanation where language is prioritized. The different forms, patterns, and levels that make up linguistic structure serve as a significant index of the function of the text, which is why statisticians place such a high value on language. According to Short [16], stylistics explains how language and esthetic function are related. Since each text should be conscious of the artistic effect of the whole and the way that details fit into the overall composition, the study of style is an attempt to discover or extract the artistic elements based on the writer’s choice of language. This study will analyze ChatGPT 4-generated texts using stylistics to identify the style of these texts.

III.METHODOLOGY

This study strives to answer these research questions: (1) Can Turnitin detect ChatGPT-4-generated responses? (2) Are Turnitin AI reports accurate? (3) What are the stylistic features of ChatGPT-4? The answers to these research questions shall assist academicians in (1) verifying claims of Turnitin that their plagiarism detection software can detect AI-generated responses, (2) manually identifying writing patterns in ChatGPT-4-generated responses, and (3) assisting academicians in finding methods to discourage students from using ChatGPT-4-generated responses in their academic submissions.

The data used in this study are created by ChatGPT-4. The researchers conducted 20 ChatGPT-4 tests that relate to case studies, business correspondence, and academic writing. ChatGPT-4 is given prompts and was asked to provide answers based on some criteria. The responses for every single prompt were regenerated 4 times. The responses were saved on a Microsoft Word document and were uploaded on Turnitin to check the similarity. After that, researchers, firstly, added sentences to the ChatGPT-4-generated responses and checked their similarity and AI reports. After that, they added a paragraph, later on added a couple of paragraphs to the ChatGPT-4 responses, and checked their similarity and AI reports. These responses and similarity reports are used to answer the first and second research questions on Turnitin’s ability to detect ChatGPT-4-generated responses and differentiate between human-written and AI-written material.

To answer the third research question, the 20 ChatGPT-4 responses were manually analyzed by the researchers to identify stylistic patterns in the ChatGPT-4-generated responses. The analysis included a number of elements that relate to sentence length, paragraph structure, word choice, mood, tense, voice, pronouns, keyword density, lexical density, lexical diversity, and reading ease.

The analysis is carried out on the quantitative and qualitative levels. Quantitative analysis is carried out to report frequencies in terms of numbers and percentages. Quantitative analysis is carried out to provide insights into the use of these features and elements in their context. This shall provide an in-depth analysis and a comprehensive understanding of the stylistic features used and the purpose of their use. The mixed research method is an established research method that is used in genre analysis [17], textual analysis [18,19], and communication studies [20].

IV.ANALYSIS AND FINDINGS

20 ChatGPT-4 tests were carried out on case studies, business correspondence, and academic writing-based prompts. Case study and business correspondence-based prompts were regenerated 5 times each by ChatGPT-4. The academic writing prompt was regenerated 10 times as academic writing ChatGPT-4 responses were reported to have a relatively high similarity [1]. The 20 ChatGPT-4 responses were uploaded on Turnitin for plagiarism check. The outcome of this initial analysis was surprising as the outcome of the Turnitin similarity check was either a zero or close to zero in the overwhelming majority of the conducted tests (see Table I). Luckily, Turnitin’s AI report managed to detect that the entire submission is generated by AI.

Table I. Turnitin similarity and AI reports index for the 20 ChatGPT-4 tests

In a strive to comprehend Turnitin’s hesitation to consider AI reports for misconduct or punitive purposes, the researchers amended the AI-generated responses by adding human-written paragraph to the submission. Turnitin’s AI report returned a 100% AI index. This test was repeated by adding a couple of human-written paragraphs to every ChatGPT-4 generated material; AI report also returned with a 100% AI index. This test was returned a third time in which more than 50% of the submissions were human-generated submissions, AI reports also returned with a 100% AI index. This explains Turnitin’s hesitation to definitely declare that AI reports can be used for misconduct and punitive measures. This makes the task of academicians challenging as they might either find themselves in situations grading ChatGPT-4 material or possibly accusing student for making AI submissions while the submissions are partly to mainly human-generated. To avoid these situations, the researchers carried out the analysis on the writing/generating style of ChatGPT-4-generated responses.

The stylistic analysis of the 20 ChatGPT-4-generated responses showed a number of elements as Table II shows. The number of paragraphs varied a great deal, as ChatGPT-4 was not given instructions on the number of expected paragraphs per response. However, it is noticed that the number of words per paragraph, the average number of sentences per paragraph, and the average number of words per sentence were comparable in a way that created a strong pattern or style. As Table II shows, the average number of words per sentence in the 20 ChatGPT-4 tests ranged between 16.1 and 20.46 words per sentence. Less than 4-word difference in 20 tests shows a pattern in sentence length in ChatGPT-4. In addition, as Table II shows, the average number of sentences per paragraph has a margin of between 2 and 4.3 sentences per paragraph. This shows that ChatGPT-4 constructs relatively long sentences in relatively short paragraphs.

Table II. Sentences and paragraphs in ChatGPT-4 responses

In regard to the average number of words per paragraph, as Table II shows, the average number of words per paragraph in case study responses ranged between 29.6 and 38.25 words. This shows that the case study-based paragraphs are relatively short. These short paragraphs are constructed in 2 to 3 sentences. For business correspondence responses, the word count per paragraph ranged between 37.7 and 50. 6 words. The paragraphs are constructed in 2 to 2.6 sentences. This shows the word count per sentence is more than case study responses, but, as Table II shows, the number of paragraphs per response is less. For academic writing, the number of words per paragraph ranged from 53.3 to 84.3, which shows that the number of words is more than business correspondence and case study-based responses.

After investigating the overview of words, sentences, and paragraphs, the focus is directed to the structure of sentences. As part of the lexicogrammatical analysis, the focus will be on the mood (i.e., declarative, imperative, interrogative), the use of tenses, the use of voice (i.e., passive voice, active voice), and referencing. In addition to these, the lexicogrammatical analysis also looks into some possible issues in the ChatGPT-4 use of language in the provided responses. As we examine ChatGPT-4 responses for case studies, business correspondence, and academic writing, these nuclei are examined separately.

For case studies, ChatGPT-4 was given a scenario and asked to provide a solution. As Table III shows, ChatGPT-4 responses were mainly provided using the imperative mood and the use of the interrogative mood was also popular. As example 1 shows, ChatGPT-4 provided a solution using imperative mood and elaborated on the methods of implementing the solution by asking a rhetorical question. The use of the imperative mood reflects directness in presenting ideas. It is also noticed, as example 2 shows, that the use of the simple present tenses in its active mood is very popular in the case studies based on ChatGPT-4 responses.

• Ex 1 (Test 1): Consider delegating some of your responsibilities if possible. Are there any tasks that can be assigned to others on your team?
• EX 2 (Test 3): It’s also important to communicate your situation to your managers and colleagues.
• EX 3 (Test 5): Can you work with your superiors to reassign some of your workload?

Table III. Case Studies – mood, tenses, voice, referencing, and possible issues

For business correspondence responses, ChatGPT-4 responses were mainly declarative. The only imperative mood sentence was reported in the pre-closing [21] move as you can see in example 4. In example 4, ChatGPT-4 used the formulaic expression of pre-closing business letters and emails [21]. As example 5 shows, the sentences in the ChatGPT-4 created business letter were mainly created using declarative sentences. The letters were written using the first person, which led to an increased percentage of using the first-person singular pronoun “I.” The reference to the student (ChatGPT-4 was asked to write a recommendation letter) was carried out in the third person, which also led to an increased usage of the third-person singular objective pronouns of his/her as you can see in example 6. The business letter was mainly written in an active voice. As Table IV shows, the passive voice was not used at all.

• EX 4 (Test 10): If you require any further information or clarification, please do not hesitate to contact me.
• EX 5 (Test 7): I am writing this letter of recommendation for [Student Name], who was a student of mine in [Course Name] at [University Name] during the [Semester/Year].
• EX 6 (Test 9): Moreover, [he/she] is a great communicator and possesses excellent interpersonal skills. [He/She] is always willing to lend a helping hand to [his/her] classmates, often tutoring those who are struggling with their studies.
• EX 7 (Test 6): I was particularly impressed with [Student Name]’s ability to think critically and solve complex problems. They possess strong analytical and research skills, which were evident in their written assignments and exams.
• EX 8 (Test 8): It is my pleasure to write this recommendation letter for my student, John Doe.

Table IV. Business writing – mood, tenses, voice, referencing, and possible issues

In addition to the use of third-person singular and plural pronouns, it is noticed that ChatGPT-4 came up with a first name and last name for the student in the recommendation letter. This happened in one out of the 5 business correspondence tests as you can see in example 8. In example 8, ChatGPT-4 came up with an arbitrary name for the student, which adds to the list of options that ChatGPT-4 can do.

In academic writing, the sentences are mainly declarative sentences. As example 9 shows, the compound sentence is written in active voice and the tense is simple present tense as the intended meaning is providing information. In addition to the use of the simple present tense, it is noticed that the use of the present perfect continuous is a relatively popular tense in academic writing responses. Present perfect continuous is a tense that is used to talk about a finished activity in the recent past or single activity that began at a point in the past and is still continuing or talk about repeated activities that started at a particular time in the past and are still continuing up until now [22]. The present perfect continuous tense is mainly used as a method to reflect on a past activity that is still ongoing now, as you can see in example 10. In example 10, the improvement started earlier and still continues with every new version or release.

• EX 9 (Test 11): It offers a lot of flexibility in terms of customization, but it may not be as user-friendly as iOS.
• EX 10 (Test 14): Samsung’s Exynos and Snapdragon chips have also been improving with every iteration, offering excellent performance and energy efficiency.
• EX 11 (Test 18): It is also known for its tight integration with Apple’s ecosystem, including its popular services like iCloud and iMessage.
• EX 12 (Test 19): They are known for their raw power and efficiency, making them ideal for heavy multitasking and gaming.
• EX 13 (Test 20): The latest iPhones have an OLED screen, which matches Samsung’s AMOLED in terms of color and contrast.

As Tables 4–6 show, the simple present tense is the most common tense used by ChatGPT-4. The simple present tense is used for a number of functions as you can see in examples 14, 15, 16, and 17. In example 14, the simple present tense is used to provide a general statement about the state of the person. This example is taken from a case study-based question. This is similar to the functional use of the simple present in example 15. ChatGPT-4 uses the same style in introducing the topic using a very general simple present statement. The main function of this statement is to grab the attention of readers to the topic of the essay. This statement is provided in a simple sentence. In example 16, ChatGPT-4 also starts the correspondence using a simple present sentence to provide a general overview of the topic and grab the attention of the readers.

• EX 14 (test 4): It sounds like you are facing a lot of challenges in your personal and professional life.
• EX 15 (test 13): Apple’s iPhones and Samsung Galaxy are two of the most popular smartphones in the world.
• EX 16 (test 7): I am pleased to recommend [Student Name] for admission to your esteemed university.
• Ex 17 (test 16): Ultimately, the choice between Apple’s iPhones and Samsung Galaxy smartphones comes down to personal preference and priorities, whether it be design, features, operating systems, or price.
• EX 18 (test 10): I am happy to provide additional information as needed.
• EX 19 (test 5): Your family commitments nor your siblings understand your professional commitments.

This shows the special importance of the simple present tense in the ChatGPT-4-generated responses. It is the grammatical tense that is mainly used to introduce a write-up or correspondence as it is also the grammatical tense used to conclude the generated responses or the correspondence. This adds to the stylistic features used by ChatGPT-4-generated responses. The above shows that ChatGPT-4 has stylistic characteristics that can define its style. This style is categorized through reoccurring patterns in terms of word choice, sentence length, paragraph length, the use of tenses, use of mood, the use of voice, and the use of pronouns. In addition to these, it is also noticed that ChatGPT-4 has some characteristics that can define it through its usage.

To provide a holistic overview of the corpus of ChatGPT-4-generated case studies, business correspondence, and academic writing responses, Table 6 compares keyword density, lexical density, lexical diversity, reading ease, and average syllables/word. As Table 6 shows, keyword density is the highest in the business correspondence corpus with an average of 3.8% for the most popular keyword. This means that repetition is one of the key elements in business writing correspondence. This is followed by academic writing. The case study corpus has the lowest average keyword density (1.9%), which means that ChatGPT-4 replaced the keyword with alternatives or synonyms. For lexical density, the academic writing corpus provides the highest lexical density (52%), which also shows that the repetition of keywords was very popular in the academic writing corpus. Repetition was not one of the features in the case studies-based corpus as lexical density is the lowest (37%). Interestingly, in terms of lexical diversity, the business correspondence corpus is the highest in diversity (58%) and this is followed by the case study corpus (45%) and the lowest is the academic writing corpus (39%). This means that the recommendation letter used a big variety of terms in comparison to the essay writing and the case study analysis. For reading ease, the case studies analysis is the easiest to read and this is followed by academic writing and business correspondence. The diversity of words in business correspondence makes it relatively difficult to understand, and the lexical density in the academic writing corpus also makes it a little bit more difficult to read than case studies as they have a low keyword and lexical density.

Table V. Academic writing – mood, tenses, voice, referencing, and possible issues

Table VI. Keyword density, lexical density, lexical diversity, and reading ease

V.DISCUSSION

This study examined the stylistic features of ChatGPT-4. It investigates ChatGPT-4 reoccurring patterns in terms of word choice, sentence length, paragraph length, the use of tenses, the use of mood, the use of voice, and the use of pronouns. In addition, this study also examines Turnitin’s ability to identify ChatGPT-4-generated responses. In a recent study, AlAfnan et al. [1] found that Turnitin was not able to identify responses that are generated by ChatGPT-4. AlAfnan et al. [1] also reported that ChatGPT-4 was able to skillfully paraphrase responses to generate new responses. These paraphrased responses were not identified as similar by plagiarism identification software (i.e., Turnitin). AlAfnan et al. [1] also found that ChatGPT, at times and especially when generating composition responses, copies material from universities’ or colleges’ papers. Recently, Turnitin announced that its similarity detection software will be able to detect AI-generated responses. On the other hand, OpenAI very recently released its new version of ChatGPT (ChatGPT-4) which is way more powerful than the previous version. The examination in this study investigates recent progress.

This study found that even though Turnitin’s AI report can detect AI-generated responses, their accuracy cannot be taken into consideration for punitive purposes for a couple of reason. Firstly, Turnitin mentions that these AI reports shall not be used for misconduct purposes. Secondly, after testing the accuracy of these reports, this study found an inconsistency in their outcome. The researchers gradually added human-generated material to the AI (ChatGPT-4)-generated material but the AI report remained 100%. This proves Turnitin’s claim that these reports shall not be used for misconduct purposes as their accuracy is way below 98%, the percentage claimed by Turnitin, accurate.

For the stylistic features of the case study-based ChatGPT-4 responses, the study revealed that the paragraphs are relatively short as they mainly include two to three sentences. The sentences are relatively average in length as the approximate word count per sentence is from 16 to 18 words. The sentences are mainly imperative that start with a verb without a mitigation device. This means that ChatGPT has a direct communication style. ChatGPT-4 also produces declarative and interrogative mood sentences but they are less common. The interrogative mood sentences provide rhetorical questions to outline or provide a point. Declarative mode sentences are mainly in simple present form. They are used at the beginning of the generated responses to grab the attention and at the end of the generated response to either finish with impact or give a sense of conclusion. Case study-based-generated responses mainly use the second-person pronoun “you” and the second-person possessive determiner “your.” The sentences are mainly constructed using the active voice, which means that the focus in these sentences and generated responses are on “doer” of the action, who is mainly “you” (the addressee). In regard to word choice, keyword density, and lexical density in ChatGPT-4 generated responses, they are relatively low, which means that repetitions are not popular. The lexical diversity is relatively average, which makes the reading ease percentage high. This means that ChatGPT-4-generated responses are easy to read.

For the stylistic features of the business correspondence ChatGPT-4-generated style, this study showed that ChatGPT-4 generates short paragraphs of two to three sentences each. The sentences are mainly combined sentences that include 16 to 20 words each. Unlike case study-based responses, business writing-based correspondence mainly included declarative sentences that are written using the simple present and simple past tenses. The simple present tense is used at the beginning of the business write-up to provide a general overview of the content and at the end of the letter to provide a sense of close-up or what AlAfnan [21] called a pre-closing move. The simple past tense is used to provide statements about the student to outline his/her abilities and skills. The sentences are mainly written in active voice. ChatGPT-4 business correspondence is written in the third person. The most common pronouns that are used are the third-person singular subjective pronouns (i.e., he, she) and the third-person singular objective pronouns (i.e., his, her). In addition to these third-person subjective and objective pronouns, it is also noticed that ChatGPT-4 supports the transgender movement by using the third-person plural subjective pronoun “they” to refer to a singular. The use of “they” to refer to a singular, based on the text and context analysis, is not a grammatical mistake; it is intentional. In reference to word choice, it is noticed that ChatGPT-4 uses technical subject-specific words without providing their definitions or explanations. The keywords density, lexical density, and lexical diversity are high and reading ease is low.

For the stylistic features of the academic writing ChatGPT-4-generated style, this study shows that ChatGPT-4-generated paragraphs of three to four sentences each. The word limit per sentence ranged between 16 and 19 words. This shows that the paragraphs of the academic writing style are the biggest in relation to the number of sentences and the number of words. Unlike case studies’ codependence which included imperative sentences, the sentences in the academic writing responses are mainly written in a declarative mood. The sentences were mainly structured in simple present, simple past, present perfect, and present perfect continuous tenses. The simple present is used to express a general statement of fact. Simple past is used to refer to past events. The present perfect is used to refer to an action that happen and was completed in the past. Present perfect continuous refers to an action that happened in the past but still continues. It is noticed that ChatGPT-4 used the present perfect tense together with the adverb “always” to extend the use of present perfect tense to include the present perfect continuous. ChatGPT-4 mainly generated active voice sentences. Passive voice sentences were always agentless as the “doer” was not important. The use of pronouns was limited to “they” as it compared and contrasted and “their” as an indefinite third-person singular antecedent. In relation to word choice, it is noticed that ChatGPT-4 used abbreviations and technical words without defining them or providing explanations. It is also noticed that the style of academic writing in ChatGPT-4 generated responses included relatively high keyword density, high lexical density, and low lexical diversity. This led to an average reading ease.

VI.CONCLUSION

This study examined Turnitin’s ability to detect ChatGPT-4-generated responses and the stylistic features of ChatGPT-4. This study showed that ChatGPT-4 widened the gap with plagiarism detection software providers (i.e., Turnitin). In comparison to AlAfnan et al. [1], ChatGPT-4 generated responses that are high in originality and low in similarity according to Turnitin’s similarity reports. Turnitin’s AI report did not provide definitive and accurate report on the use of AI as it also reported human-generated work as an AI-generated response. In investigating the stylistic features of the ChatGPT-4-generated responses, the focus was given to sentence length, paragraph length, word choice, mood, tenses, voice, pronouns, lexical density, keyword density, lexical diversity, and reading ease. The study found that ChatGPT-4 generated 2- to 4-sentence-long paragraphs. The word count of sentences ranges between 16 and 20 words. ChatGPT-4 started with an attention grabber or by providing an overview using the simple present tense. It also concluded the response by summarizing or indicting closure using simple present tenses as well. It is also noticed that ChatGPT-4 used declarative, imperative, and interrogative moods. The declarative mood is common in academic writing responses while the imperative mood is common in case study-related responses. ChatGPT-4 mainly used active voice and, at times, agentless passive voice. In general, ChatGPT-4-generated responses provided an average level of keyword density, lexical density, lexical diversity, and reading ease. It used abbreviations and jargon without providing any information on what they stand for. Noticeably, ChatGPT-4 supported the transgender movement by intentionally using the plural third-person pronoun “they” to refer to a singular.