Retrieval Augmented Generation-Based Chatbot for Prospective and Current University Students

Luluk Setiawati Hartono, Esther Irawati Setiawan, Vrijraj Singh

Abstract


Universities utilize chatbots as assistants for users, especially prospective and current students, to access information and answer questions with relevant answers. This study introduces a new approach to an open-source model-based Q&A system using Gemma2-2b-it by combining Retrieval Augmented Generation (RAG) and Fine-tuning (FT) techniques. Previously, some studies have focused on only one approach, but this study will combine and compare both methods separately. Raw conversation data from WhatsApp, the main university website, and university PDF documents are used. The Retrieval Augmented Generation Assessment (RAGAS) framework will be used to evaluate the performance of the RAG model. In contrast, precision, recall, and similarity are used to assess the comparative performance of RAG and fine-tuning. The results of the RAGAS show that RAG using the base model is better than RAG using a fine-tuned model, which has 0.78 faithfulness, 0.64 answer relevancy, 0.81 context precision, and 0.68 context recall, so the overall RAGAS Score is 0.72. The comparison of precision and recall of fine-tuning are higher than those of using RAG, but the similarity score is not much different. Furthermore, the potential improvement for RAG of this study can be increased by adding a reranking process in the retrieved context, and fine-tuning of the embedding model can also be added to increase the retrieval process's performance. In addition, further experiments on various datasets and the challenge of overfitting in fine-tuning must be overcome so that the model can also perform better generalization.


Keywords


Natural Language Processing, Retrieval Augmented Generation, RAGAS, Fine-tuning, Chatbot

Full Text:

PDF

References


G. F. Febrian and G. Figueredo, “KemenkeuGPT: Leveraging a Large Language Model on Indonesia’s Government Financial Data and Regulations to Enhance Decision Making,” Jul. 2024, [Online]. Available: http://arxiv.org/abs/2407.21459

B. A. T. de Freitas and R. de Alencar Lotufo, “Retail-GPT: leveraging Retrieval Augmented Generation (RAG) for building E-commerce Chat Assistants,” 2024. [Online]. Available: https://arxiv.org/abs/2408.08925

M. Quidwai and A. Lagana, A RAG Chatbot for Precision Medicine of Multiple Myeloma. 2024. doi: 10.1101/2024.03.14.24304293.

A. Sen, D. R. Satya Ranjan Dash, D. R. Manas Ranjan Pradhan, and S. Parida, “HEALTHCARE QUESTION ANSWERING SYSTEM IN BENGALI-A PROPOSED MODEL,” J Theor Appl Inf Technol, vol. 30, no. 12, 2024.

H. Dinh and T. K. Tran, “EduChat: An AI-Based Chatbot for University-Related Information Using a Hybrid Approach,” Applied Sciences, vol. 13, no. 22, p. 12446, Nov. 2023, doi: 10.3390/app132212446.

U. H. Khan, M. H. Khan, and R. Ali, “Large Language Model based Educational Virtual Assistant using RAG Framework,” Procedia Comput Sci, vol. 252, pp. 905–911, 2025, doi: 10.1016/j.procs.2025.01.051.

T. A. I. T. Baha, M. E. L. Hajji, Y. Es-Saady, and H. Fadili, “Towards highly adaptive Edu-Chatbot,” in Procedia Computer Science, Elsevier B.V., 2021, pp. 397–403. doi: 10.1016/j.procs.2021.12.260.

S. Neupane et al., “From Questions to Insightful Answers: Building an Informed Chatbot for University Resources,” May 2024, [Online]. Available: http://arxiv.org/abs/2405.08120

Z. Jiang et al., “Active Retrieval Augmented Generation,” May 2023, [Online]. Available: http://arxiv.org/abs/2305.06983

Q. Zaman, S. Safwandi, and F. Fajriana, “Supporting Application Fast Learning of Kitab Kuning for Santri’ Ula Using Natural Language Processing Methods,” International Journal of Engineering, Science and Information Technology, vol. 5, no. 1, pp. 278–289, Jan. 2025, doi: 10.52088/ijesty.v5i1.713.

S. Sharma et al., “Retrieval Augmented Generation for Domain-specific Question Answering,” Apr. 2024, [Online]. Available: http://arxiv.org/abs/2404.14760

P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” May 2020, [Online]. Available: http://arxiv.org/abs/2005.11401

H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu, “Evaluation of Retrieval-Augmented Generation: A Survey,” May 2024, [Online]. Available: http://arxiv.org/abs/2405.07437

J. Swacha and M. Gracel, “Retrieval-Augmented Generation (RAG) Chatbots for Education: A Survey of Applications,” Apr. 01, 2025, Multidisciplinary Digital Publishing Institute (MDPI). doi: 10.3390/app15084234.

Y. Huang and J. Huang, “The Survey of Retrieval-Augmented Text Generation in Large Language Models,” Apr. 2024, [Online]. Available: http://arxiv.org/abs/2404.10981

Gemma Team et al., “Gemma 2: Improving Open Language Models at a Practical Size,” Jul. 2024, [Online]. Available: http://arxiv.org/abs/2408.00118

S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, “RAGAS: Automated Evaluation of Retrieval Augmented Generation,” Sep. 2023, [Online]. Available: http://arxiv.org/abs/2309.15217

M. R. J, K. VM, H. Warrier, and Y. Gupta, “Fine Tuning LLM for Enterprise: Practical Guidelines and Recommendations,” 2024. [Online]. Available: https://arxiv.org/abs/2404.10779

E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” Jun. 2021, [Online]. Available: http://arxiv.org/abs/2106.09685

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient Finetuning of Quantized LLMs,” May 2023, [Online]. Available: http://arxiv.org/abs/2305.14314

E. I. Setiawan, J. Santoso, and Gunawan, “Answer Ranking with Weighted Scores in Indonesian Hybrid Restricted Domain Question Answering System,” in 3rd 2021 East Indonesia Conference on Computer and Information Technology, EIConCIT 2021, Institute of Electrical and Electronics Engineers Inc., Apr. 2021, pp. 456–461. doi: 10.1109/EIConCIT50028.2021.9431915.

S. Minaee et al., “Large Language Models: A Survey,” Feb. 2024, [Online]. Available: http://arxiv.org/abs/2402.06196

Z. Fitra Ramadhan and A. Benny Mutiara, “Sentiment Analysis of Honkai: Star Rail Indonesian Language Reviews on Google Play Store Using Bidirectional Encoder Representations from Transformers Method,” International Journal of Engineering, Science and Information Technology, vol. 3, no. 3, pp. 1–6, Sep. 2023, doi: 10.52088/ijesty.v3i3.462.

A. Balaguer et al., “RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture,” Jan. 2024, [Online]. Available: http://arxiv.org/abs/2401.08406

H. Soudani, E. Kanoulas, and F. Hasibi, “Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge,” Mar. 2024, doi: 10.1145/3673791.3698415.

P. Sutanto, J. Santoso, E. I. Setiawan, and A. P. Wibawa, “LLM Distillation for Efficient Few-Shot Multiple Choice Question Answering,” 2024. [Online]. Available: https://arxiv.org/abs/2412.09807

C.-Y. Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” in Text Summarization Branches Out, Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013/

S. Roychowdhury, S. Soman, H. G. Ranjani, N. Gunda, V. Chhabra, and S. K. Bala, “Evaluation of RAG Metrics for Question Answering in the Telecom Domain,” Jul. 2024, [Online]. Available: http://arxiv.org/abs/2407.12873




DOI: https://doi.org/10.52088/ijesty.v5i3.951

Article Metrics

Abstract view : 0 times
PDF - 0 times

Refbacks

  • There are currently no refbacks.


Copyright (c) 2025 Luluk Setiawati Hartono, Esther Irawati Setiawan, Vrijraj Singh

International Journal of Engineering, Science, and Information Technology (IJESTY) eISSN 2775-2674