I.INTRODUCTION

Artificial intelligence (AI) has come a long way in cognitive abilities, including understanding language and cultural context, with advances in machine learning techniques such as the GPT-4 large language model allowing AI to analyze literary texts, including short stories, in a way that approximates human thought. AI can understand texts’ context, themes, and messages and identify language patterns, metaphors, and symbolism used to convey local wisdom [1]. However, AI still faces challenges in understanding complex and implicit cultural nuances and contexts, as local wisdom values are often rooted in oral traditions, long histories, and collective experiences that are difficult to translate into textual data [2]. AI’s statistical and predictive understanding is also not equivalent to human lived experience and emotional understanding, so while AI can provide plausible interpretations of literary texts, it may struggle to capture the deep meanings derived from rich and contextualized human experience [35].

Relevant research related to the comparison of human and AI understanding in literacy education has been conducted by several studies [3,69]. However, although many studies have explored various aspects of AI in education, there still needs to be a research gap in understanding how AI can improve the understanding of local wisdom in Indonesian literature. This study was conducted to fill the gap by evaluating the comparison of local wisdom understanding between university students and AI and the factors that influence it.

Geertz [10] emphasized understanding culture through thick description, culture as text, symbolism, and cultural interpretation [1113]. Anthropology, differing from hermeneutic historicism, draws on naturalistic aspects of philosophy [1416]. Local wisdom in short stories reflects cultural and traditional elements, depicted in characters, dialog, and conflicts resolved through traditional norms [1720]. These stories serve as cultural learning materials and social criticism, enriching cultural identity [2125]. AI in cultural literacy education enhances learning by interpreting literary works, identifying symbolism, developing adaptive learning aids, and providing curriculum feedback [2629], deepening cultural heritage understanding [4].

AI chatbots are sophisticated software applications that simulate human conversation through natural language processing (NLP). These chatbots utilize advanced machine learning algorithms to understand, interpret, and generate human language, enabling them to engage in meaningful user interactions. In education, AI chatbots serve as innovative tools that can significantly enhance the teaching and learning of language and literature, particularly when embedding local wisdom values. AI chatbots can be programmed with extensive databases containing cultural knowledge, local idioms, folklore, proverbs, and other elements of local wisdom, making them effective mediums for imparting this knowledge to students. As AI becomes more advanced and its use in education expands, it is important to assess the extent to which AI can match or even surpass human understanding in ecological literacy [30].

By engaging students in interactive dialogs, AI chatbots can present literary texts and their underlying cultural contexts in a more accessible and engaging manner. For example, when teaching a piece of literature incorporating local wisdom, a chatbot can provide background information on cultural references, explain the significance of particular metaphors, and offer insights into the historical and social context in which the text was written. It helps students understand the text at a superficial level and appreciate the deeper cultural and philosophical meanings embedded within.

Furthermore, AI chatbots can tailor their responses to individual students’ learning needs, offering personalized explanations and additional resources to help them grasp complex concepts. This adaptability is particularly valuable in a diverse classroom setting where students may have varying levels of familiarity with local wisdom and cultural nuances. Moreover, AI chatbots can facilitate continuous learning beyond the classroom by being available 24/7, allowing students to engage with educational content at their own pace and convenience. It can be particularly beneficial for reinforcing learning, addressing questions outside of class, and providing practice opportunities for understanding and applying local wisdom in various contexts. Additionally, AI chatbots can help assess students’ comprehension by posing questions, providing feedback, and tracking their progress. It not only aids teachers in identifying areas where students may need additional support but also encourages students to engage in reflective and critical thinking. Integrating AI chatbots in language and literature education also aligns with the increasing digital literacy among students, making learning more relevant and engaging in today’s technology-driven world.

By leveraging the capabilities of AI chatbots, educators can create a more dynamic and interactive learning environment that enhances students’ understanding of literary texts and fosters a deeper appreciation of the cultural heritage and local wisdom that these texts convey. Ultimately, using AI chatbots to teach local wisdom through literature represents a convergence of technology and tradition, where modern AI tools are used to preserve and propagate age-old cultural knowledge. This innovative approach addresses the limitations of traditional teaching methods and opens up new possibilities for engaging students in meaningful and culturally enriching educational experiences. As AI technology evolves, its potential to transform language and literature education by integrating local wisdom will only grow, offering exciting opportunities for educators and students alike.

This study aims to compare the level of understanding of local wisdom values in short stories between students and AI systems. To achieve this goal, this research formulates the following hypothesis.

H1a: There is a significant difference in understanding local wisdom values in short stories based on gender groups.

H1b: There is a significant difference in understanding local wisdom values in short stories based on domiciles.

H1c: There is a significant difference in understanding the value of local wisdom in short stories based on universities.

H2: The ability of AI exceeds students in understanding the value of local wisdom in short stories.

This research provides insights into the strengths and weaknesses of AI understanding of local wisdom values compared to university students, which can help design curricula that are more adaptive and responsive to learning needs in the digital age. The results of this study can be used to develop more effective teaching strategies, utilizing AI technology as a learning tool that can complement and enrich the educational process. In addition, a better understanding of how AI and students process and understand cultural values can assist educators in emphasizing certain aspects of the curriculum that technology may poorly understand. As such, this research could encourage more intelligent and more culturally context-sensitive integration of technology in education.

II.METHODS

A.RESEARCH DESIGN

This study used a comparative design with a quantitative approach to evaluate the understanding of local wisdom between university students from different backgrounds and several AI systems. The study began by identifying and recruiting a sample of university students to participate in the local wisdom comprehension test. Before answering the test questions, students first read four short stories from ruangsastra.com, namely Ngaben Sederhana (A Simple Ngaben) by Sugianto [31], Pulang Tanpa Kampung Halaman (Going Home Without a Hometown) by Ahmad [32], Laut Mengambil Cintaku (The Sea Took My Love) by Piran [33], and Ikan Sungai Bejoe (Fish of Bejoe River) by Rahman [34]. Furthermore, to obtain comparative data, the same local wisdom understanding test was given to several AI systems: ChatGPT, Google Gemini, Perplexity AI, Claude 3.5 Sonnet, and Chatsonic.

B.PARTICIPANT CHARACTERISTICS

The study involved 474 students from diverse ages, domiciles, universities, and ethnic backgrounds across Indonesia, ensuring a balanced gender distribution and representation of urban and rural areas. Participants from various universities provided variation in curriculum and teaching methods, as listed in Table I. Based on Table II, five AI systems, including ChatGPT and Google Gemini, were included to compare local wisdom comprehension.

C.INSTRUMENTATION AND DATA COLLECTION

The instrument used in this study is a test containing 32 multiple-choice questions, as in Table III. It was designed based on Geertz’s concept of cultural interpretation, which included thick description, cultural textual, symbolism, and cultural interpretation. The test was designed to measure students’ and AIs’ understanding of the local wisdom values contained in selected short stories. The list of selected short stories is shown in Table IV.

This test instrument was distributed through a Google Form distributed to students. In addition, this instrument was also used to assess how AI interpreted the values of local wisdom in the selected short stories. Data collection was conducted on June 12, 2024.

D.DATA DIAGNOSTICS

To improve data credibility, measures such as screening duplicate data and outliers, and conducting homogeneity and normality tests were taken. Screening identified and removed 38 duplicate entries and 5 outliers. Based on Table V, the homogeneity test showed no significant variance differences in gender (P = 0.083), domicile (P = 0.287), and student vs. AI comparison (P = 0.299). In addition, Table VI shows that for one-way ANOVA analysis, Levene’s (P = 0.1) and Bartlett’s (P = 0.616) tests indicated no significant variance differences between universities. Figure 1 shows that the normality test using Q-Q plots demonstrated that the Local_Wisdom data distribution for gender, domicile, university, and student-to-AI ratio is close to the diagonal line, indicating normal distribution.

E.ANALYTIC STRATEGY

The study’s analytic strategy involved several steps to compare university students’ and AIs’ understanding of local wisdom in short stories. An independent sample t-test assessed variations in students’ scores based on gender and domicile, while Fisher’s one-way ANOVA compared scores by university. Finally, independent sample t-test and Mann–Whitney U tests compared mean scores between student and AI groups, ensuring robust findings.

III.RESULTS

A.DESCRIPTIVE STATISTICS

The descriptive statistics in Table VII show that for the Local_Wisdom variable, based on 479 samples (including five AI programs) with no missing data, the mean is 49.8, with a 95% confidence interval between 48.0 and 51.5. The mean standard error (SE) is 0.872, and the standard deviation (SD) is 19.1, indicating significant variation. The range is from 6.25 to 93.8. Skewness is –0.229 with a SE of 0.112, indicating a slight left skew, and kurtosis is –0.966 with a SE of 0.223, showing a flatter than normal distribution. Overall, there’s significant variation in understanding local wisdom values.

Table I. Demographics data of human participants (students)

DemographicsnPercentageCumulative
GenderMale7616.0%16.0%
Female39884.0%100.0%
Age< 19 years10321.7%21.7%
19–21 years27457.8%79.5%
22–24 years7415.6%95.1%
25–27 years143.0%98.1%
> 27 years91.9%100.0%
DomicileUrban12927.2%27.2%
Rural34572.8%100.0%
UniversityMakassar State University16534.8%34.8%
University of Timor14931.4%66.2%
Manado State University459.5%75.7%
Al-Asyariah Mandar University408.4%84.2%
University of West Sulawesi7515.8%100.0%
EthnicTimorese14330.2%30.2%
Mandarese8618.1%48.3%
Makassarese7415.6%63.9%
Buginese7315.4%79.3%
Minahasan214.4%83.8%
Torajan204.2%88.0%
Javanese112.3%90.3%
Mamasa51.1%91.4%
Mongondow; Pattae; Sangir (n=4; 0.8%)122.5%93.9%
Sundanese; Bima; Balinese; Sangihe; Siau (n=2; 0.4%)102.1%96.0%
Kaili; Selayar; Ende; Butonese; Tetun; Kain Leon; Kajong; Dani; Banggai; Ternate; Sumbanese; Karo; Ulumanda; Minsel; Manadonese; Talaud; Batak Toba; Nias; Luyo (n=1; 0.2%)194.0%100.0%

Table II. Involved AI and language models

AILanguage modelDeveloper
ChatGPT-4oGPT-4OpenAI
Google GeminiGeminiGoogle DeepMind
Perplexity AIGPT-3.5Perplexity
Claude 3.5 SonnetGPTAnthropic
ChatsonicGPT-3.5Writesonic

Table III. Distribution of questions by aspects

AspectsQuestion number
Thick DescriptionQ1, Q5, Q10, Q15, Q16, Q24, Q25
Culture TextualQ2, Q6, Q9, Q11, Q14, Q17, Q18, Q23, Q26, Q27, Q32
SymbolismQ3, Q7, Q12, Q19, Q20, Q28, Q29
Cultural InterpretationQ4, Q8, Q13, Q21, Q22, Q30, Q31

Table IV. Selected short stories

Short storyAuthorPublisher
Ngaben Sederhana (A Simple Ngaben)I Made SugiantoKompas
Pulang Tanpa Kampung Halaman (Going Home Without a Hometown)Artie AhmadJawa Pos
Laut Mengambil Cintaku (The Sea Took My Love)Jemmy PiranJawa Pos
Ikan Sungai Bejoe (Fish of Bejoe River)Dul Abdul RahmanFajar Makassar

Table V. Homogeneity test for independent samples t-test analysis

Statistics (Levene’s | Variance ratio)
Grouping variableFdfdf2P
Gender (male and female)3.02 | 0.8441 | 397472 | 750.083 | 0.312
Domicile (rural and urban)1.14 | 1.091 | 128472 | 3440.287 | 0.541
Students vs. AI1.08 | 1.291 | 473477 | 40.299 | 0.919

Table VI. Homogeneity test for one-way ANOVA analysis

Statisticdfdf2P
Levene’s1.9544690.1
Barlett’s2.6640.616

Table VII. Descriptive statistics

Measures of data centering and dispersionLocal_Wisdom
N479
Missing0
Mean [95% CI lower, 95% CI upper]49.8 [48.0, 51.5]
SE0.872
SD19.1
Minimum6.25
Maximum93.8
Skewness (Std. error skewness)–0.229 (0.112)
Kurtosis (Std. error kurtosis)–0.966 (0.223)

Note. The CI of the mean assumes sample means follow a t-distribution with N – 1 degrees of freedom

B.COMPARISON BETWEEN STUDENTS

An independent sample t-test was applied to test the hypothesis that there is a significant difference in the understanding of local wisdom based on gender and domicile group. The results of the analysis are presented in Table VIII.

Table VIII. Comparison of understanding of local wisdom between gender and domicile using independent sample t-test

StatisticsGrouping variable
GenderDomicile
Student’s t2.36–0.699
df472472
P-value0.0190.485
Mean44.8 | 50.3(Male | Female)48.5 | 49.8(Urban | Rural)
SD20.1 | 18.5(Male | Female)19.5 | 18.6(Urban | Rural)
Mean difference5.55–1.36
SE difference2.351.95
Effect size (Cohen’s d)0.296–0.0721

Note. H1a μ Female ≠ μ Male; H1b μ Urban ≠ μ Rural

Based on Table VIII, the gender-based t-test shows a t-value of 2.36, df of 472, and a P-value of 0.019, indicating a significant difference in local wisdom understanding between men and women. The mean difference was 5.55 with a SE of 2.35 and an effect size (Cohen’s d) of 0.296. In contrast, the domicile comparison showed a t-value of –0.699, df of 472, and a P-value of 0.485, indicating no significant difference between urban and rural students. The mean difference was –1.36 with a SE of 1.95 and an effect size of –0.0721.

Figure 2 shows the average scores of students’ understanding of local wisdom across universities, with Al-Asyariah Mandar University scoring highest (56.09) and West Sulawesi University lowest (42.67). Fisher’s one-way ANOVA test in Table IX indicates a significant difference in understanding between universities (F=5.23, df1=4, df2=146, P<0.001). This suggests that the university significantly influences students’ understanding of local wisdom.

Fig. 1. Normality test result using Q-Q plots.

Fig. 2. Average score of understanding local wisdom of students at several universities.

Table IX. Comparison of local wisdom understanding between universities using Fisher’s one-way ANOVA

StatisticsValue
F5.23
df14
df2146
P-value<.001

Note. H1c μ Makassar State University ≠ μ Timor University ≠ μ Manado State University ≠ μ Al-Asyariah Mandar University ≠ μ West Sulawesi University.

C.COMPARISON BETWEEN STUDENTS AND AI

This study compares the understanding of local wisdom between university students and several AI systems. This analysis evaluates how AI can understand and interpret local wisdom values in short stories compared to students. The AI systems used in this study include ChatGPT-4, Google Gemini, Perplexity AI, Claude 3.5 Sonnet, and Chatsonic. Understanding the value of local wisdom consists of several parts, namely thick description, cultural text, symbolism, and cultural interpretation. The test results are presented in Table X.

Table X. Comparison of understanding of local wisdom between students and AI using independent sample t-test

StatisticsTests
Student’s tMann–Whitney U
Value–3.61268
df477
P-value<.0010.001
Mean difference–30.6–31.2
SE difference8.47
Effect size–1.62 (Cohen’s d)0.774 (Rank biserial correlation)

Note. H2 μ students < μ AI.

Table X compares the understanding of local wisdom between students and AIs using the independent t-test and Mann–Whitney U-test. The t-test results show a t-value of –3.61 with degrees of freedom (df) 477 and a P-value <0.001, indicating that this difference is highly statistically significant. The mean difference between college students and AIs was –30.6 with a standard error (SE) of 8.47 and an effect size (Cohen’s d) of –1.62, signifying a large and significant impact. To confirm the robustness of the findings, the Mann–Whitney U test was employed, yielding a U value of 268 with a P-value of 0.001 and an effect size (biserial rank correlation) of 0.774, indicating a significant effect. This finding indicates that AIs have a significantly higher understanding of local wisdom than university students. The detailed assessment results are presented in Fig. 3.

Fig. 3. Comparison of understanding of local wisdom between AIs and students in various assessment aspects.

Figure 3 compares the understanding of local wisdom between AIs and students in various aspects of assessment, namely Thick Description, Culture Textual, Symbolism, Culture Interpretation, and Local Wisdom (Overall). In all aspects of assessment, AI consistently showed higher understanding than students.

In the Thick Description, AI scored 65.71, while students only scored 44.27. This difference in scores suggests that the AI can identify and convey cultural details more thoroughly and structure, which may be difficult for students’ intuitive understanding to reach. AI’s ability to systematically parse information allows for capturing implicit meanings of local context, often in thick descriptions. It is essential because thick descriptions require a holistic understanding of cultural details. AI can provide a more accurate and consistent picture than human interpretation, which may be limited by individual understanding or personal cultural experience.

For the Culture Textual aspect, AI recorded the highest score of 81.82, while students obtained 47.60. AI’s strong ability to recognize and understand culture-related textual elements, such as symbols, idioms, and typical narrative contexts, is evident in its high score. Its proficiency in analyzing linguistic patterns and text structures in depth allows it to pick up on nuances that students might miss, especially if they lack exposure to the culture. With this advantage in textual aspects, AI can offer a more comprehensive understanding of the text, thereby strengthening its educational value in cultural and linguistic literacy.

In the Symbolism aspect, AI’s score was 100.00, while the student far exceeded the student score of 62.03. The AI’s perfect score demonstrates its ability to identify and interpret cultural symbols consistently and precisely without being influenced by individual perceptual biases. It indicates that AI has access to extensive data regarding the meaning of symbols in various cultural contexts, allowing for a more objective and thorough analysis. While students may have varying interpretations depending on their backgrounds and experiences, AI offers a more stable approach to understanding symbolism, making it an effective tool in cross-cultural teaching and understanding.

In the cultural interpretation aspect, the AI score was 71.43, while the student score was 44.91. These results suggest that AI has a better capacity to understand the cultural context that envelops the text, allowing for a more accurate and thorough interpretation of the values and norms contained in the story. AI’s ability to draw connections between different cultural elements gives it an edge in interpreting meaning more comprehensively, especially in cultural contexts unfamiliar to students. It is significant for teaching literature and culture, as AI can help bridge the gap of cultural understanding by providing near-original interpretations and making it easier for students to absorb complex cultural contexts.

The overall score for understanding local wisdom was 80.00 for AI and 49.44 for students. This data shows that AIs have a deeper and broader understanding of local wisdom than students in all aspects assessed. This score indicates that AI can process various elements of local wisdom in an integrated manner, enabling a more complete understanding of the cultural values contained in the text. AI’s ability to systematically identify patterns and contexts makes it excellent at capturing implicit meanings students may not know. As such, AI has the potential to be a practical learning resource for enriching students’ knowledge of local wisdom while providing an objective approach that helps minimize cultural misinterpretation.

In addition, a comparison of the local wisdom scores between different AIs shows that ChatGPT and Claude AI have the highest score (93.75), followed by Chatsonic (78.13), Gemini (81.25), and Perplexity with the lowest score (53.13). The radar diagram in Fig. 4 illustrates that ChatGPT and Claude AI significantly excel in local wisdom understanding compared to the other AIs, while Perplexity lags far behind.

Fig. 4. Comparison of local wisdom score between AIs.

IV.DISCUSSION

The findings show that there is a significant difference in the understanding of local wisdom between male and female students (t = 2.36, P = 0.019), with females having a better understanding (mean = 50.3) than males (mean = 44.8). In contrast, there is no significant difference based on domicile (t = –0.699, P = 0.485), with almost the same mean between urban (48.5) and rural (49.8) students. It may be due to the fact that gender influences an individual’s perspective and understanding of local wisdom values more than domicile. In addition, the findings show that there are significant differences in the understanding of local wisdom between students from different universities. Variations in curriculum, teaching methods, and institutional focus on local wisdom at each university cause this difference. This finding is consistent with Juanda and Afandi’s [7] research, which found that women were superior in understanding texts compared to men, and there were significant differences in text understanding based on university groups. The interaction between cultural-historical contingencies and anthropological constants significantly influences the formation of assumptions about humans within the realm of digital humanism [12,13]. In the context of literature, the values of local wisdom not only serve as the background of the story but also as social criticism and reflection on the cultural identity of the community being told, thus enriching the meaning and providing a deeper dimension for readers in understanding the cultural context raised in the short story [2125].

Additionally, the findings demonstrate that AI significantly outperforms university students in comprehending local wisdom values, with a mean difference of 30.6 and a substantial effect size (Cohen’s d = –1.62). This result is supported by the Mann–Whitney U test, which shows that this difference is robust, with a U value of 268 and a biserial correlation effect size of 0.774. This difference occurs because AIs can access more information and have more sophisticated data processing capabilities than students. Meanwhile, ChatGPT and Claude AI significantly outperformed Gemini, Chatsonic, and Perplexity in understanding local wisdom. This finding is in line with Juanda and Afandi’s [7] research showing that ChatGPT is superior in text reprocessing compared to university students, indicating a gap in text comprehension between university students and AI. In addition, the findings of Dai et al. [8] revealed that a pedagogical approach to AI literacy in upper primary education improved students’ learning performance and attitude. Wang and Xue’s study [3] evaluated the impact of AI-based chatbots on enhancing academic engagement among EFL students in China and discovered a positive effect of AI on student participation. However, Rashidi et al. [9] evaluated the ability of AI text detection tools to distinguish manufactured text and found significant limitations in text detection. Young people have four orientations toward digital literary participation: literary kinship, literary intuition, literary intimacy, and literary activism [2628]. Chatbots possess the capability to proactively acquire new conversation topics [1].

Integrating local wisdom education through AI in literature is imperative for several reasons. First, local wisdom represents a rich cultural heritage that embodies the values, beliefs, and practices of a community. Its preservation and dissemination are vital for maintaining cultural identity and fostering a sense of belonging among students. AI’s ability to analyze and present these values within literary contexts can enhance students’ appreciation and understanding of their cultural heritage.

Second, the current educational landscape needs help effectively convey local wisdom through traditional methods. Students often need more engagement and contextual understanding to grasp these values’ depth fully. With its interactive and adaptive learning capabilities, AI can bridge this gap by providing personalized and contextually relevant learning experiences.

Moreover, AI’s capacity to continuously learn and adapt allows it to stay updated with new cultural developments and integrate them into educational materials. This dynamic approach ensures that local wisdom teaching remains relevant and reflects contemporary societal changes.

Additionally, AI can facilitate a more inclusive learning environment by catering to diverse learning styles and needs. It can provide alternative explanations, examples, and interactive activities that resonate with students from different backgrounds, enhancing their learning experience.

The rapid technological advancements and the increasing digital literacy among young people further highlight the urgency of integrating AI into local wisdom education. As the study by Rashidi et al. [9] indicates, young individuals engage in digital literary participation through various orientations such as literary kinship, intuition, intimacy, and activism. AI’s proactive capability to acquire and integrate new conversation topics can align with these orientations, making learning more engaging and relevant to student’s digital lives.

Integrating AI in teaching local wisdom through literature is a technological advancement and a necessary evolution in educational practices. It addresses the current gaps in comprehension and engagement, ensures the preservation and relevance of cultural values, and caters to the diverse needs of students. As AI continues to evolve, its role in education will undoubtedly expand, making it a crucial tool for fostering a deeper understanding and appreciation of local wisdom among future generations.

V.CONCLUSION

There was a significant difference in the understanding of local wisdom between male and female students, with females showing a better understanding. At the same time, there was no significant difference based on domiciles. This gender-based difference is due to different perspectives and understandings of local wisdom values. In addition, there were significant differences between students from different universities due to variations in curriculum, teaching methods, and institutional focus on local wisdom. This research also showed that AIs were far superior to university students in understanding local wisdom values, thanks to their broader and deeper data processing capabilities and access to information in the context of short story texts. Meanwhile, ChatGPT and Claude AI scored significantly better than Gemini, Chatsonic, and Perplexity.

This research could impact Indonesian language and literature education, culture, AI technology, and policy. It could encourage inclusive curricula sensitive to gender differences and emphasize integrating local wisdom to enrich cultural understanding. The findings highlighted AI’s potential in interpreting cultural values, suggesting opportunities for more contextual AI applications. Policymakers may formulate adaptive educational strategies and adopt AI technologies for more effective learning.

Future research could explore factors influencing gender and university differences in local wisdom understanding, including qualitative analysis of teaching methods and curricula. Additionally, examining AI technology’s role in supporting local wisdom learning and comparing different AI models to understand their strengths and weaknesses in cultural contexts is crucial. Expanding the research to more universities and regions, considering variables like socioeconomic background and previous learning experiences, is essential. Finally, studying how AI integration in education affects students’ long-term learning outcomes and cultural understanding, and adapting educational policies to support effective use of these technologies, is vital.