Applying Retrieval Augmented Generation(Rag) With Llm-based Chatbot to Enhance Patient-facing Health Information Tools
Abstract
Large Language Models(LLMs) run on frozen parametric knowledge which limits
their out-of-the box application for specific domains with constraints such as privacy
regulations, proprietary data, costs, hallucination et cetera. They can be adapted to
specific domains via prompt engineering, model fine tuning, alignment, Retrieval
Augmented Generation(RAG) or a hybrid of either. Each of the techniques have their
pros and cons; however, RAG has recently proven to be more cost effective for
many general purpose applications. This study investigates the applicability of RAG
for telemedicine chatbots by conducting experiments for information needs in
telemedicine and creating a prototype chatbot based on RAG. We try to optimise
various aspects of RAG such as combination of multiple retrieval sources with
structured and semi-structured data formats like PDFs, website scrubs, Whatsapp
chat exports as well as parquet datasets from Hugging Face. We also apply index
optimisation approaches such as metadata enhancement and chunk size sizes. We
evaluate the RAG pipeline using the RAGAS framework for core metrics including
answer relevancy, context recall and faithfulness. The prototype achieves a high
context recall rate of 56% and more than 75% in answer faithfulness and relevancy.
The scarcity and quality of data proved to be a major limitation for this study; and a
room for improvement for successful adoption of RAG in telemedicine.
Publisher
University of Nairobi
Rights
Attribution-NonCommercial-NoDerivs 3.0 United StatesUsage Rights
http://creativecommons.org/licenses/by-nc-nd/3.0/us/Collections
The following license files are associated with this item: