Applying Retrieval Augmented Generation(Rag) With Llm-based Chatbot to Enhance Patient-facing Health Information Tools

Abstract

Large Language Models(LLMs) run on frozen parametric knowledge which limits their out-of-the box application for specific domains with constraints such as privacy regulations, proprietary data, costs, hallucination et cetera. They can be adapted to specific domains via prompt engineering, model fine tuning, alignment, Retrieval Augmented Generation(RAG) or a hybrid of either. Each of the techniques have their pros and cons; however, RAG has recently proven to be more cost effective for many general purpose applications. This study investigates the applicability of RAG for telemedicine chatbots by conducting experiments for information needs in telemedicine and creating a prototype chatbot based on RAG. We try to optimise various aspects of RAG such as combination of multiple retrieval sources with structured and semi-structured data formats like PDFs, website scrubs, Whatsapp chat exports as well as parquet datasets from Hugging Face. We also apply index optimisation approaches such as metadata enhancement and chunk size sizes. We evaluate the RAG pipeline using the RAGAS framework for core metrics including answer relevancy, context recall and faithfulness. The prototype achieves a high context recall rate of 56% and more than 75% in answer faithfulness and relevancy. The scarcity and quality of data proved to be a major limitation for this study; and a room for improvement for successful adoption of RAG in telemedicine.