Evaluating Nepali Sexual, Reproductive, and Maternal Health (SRMH) Chat-bot with Large Language Models (LLMs)

Problem: 
Most people in Nepal don’t have easy access to clear information about Sexual, Reproductive, and Maternal Health (SRMH) because they feel embarrassed to talk about it openly, and most of the information online is in English and not easy to interact with. This lack of timely and accurate SRMH information can lead to problems like unsafe pregnancies, STDs, HIV, and unintended pregnancies. ChatGPT and similar tools could be a helpful way to get answers about SRMH, but there are four main issues:  

1) Sometimes, it can provide wrong or made-up answers. 

2) It doesn’t work as well in languages like Nepali as it does in English. 

3) It mostly learns from the internet, which might not include Nepal’s local cultural context. 

4) It needs a lot of computer power and can be expensive, making it hard for people in places with limited resources to use. 

Research Aim: 

The main aim of our research/solution is to help the general public get reliable, accurate, and localized SRMH information on different stages of life such as adolescence and pregnancy, via a user-friendly accessible chatbot. We also want the Female Community Health Volunteers (FCHV) to use it for counselling purposes.  

Current Stage of Our Research 

We are at the first stages of this research. Currently we have one ongoing project on this theme where we intend to identity the ability, gaps, and risks of ChatGPT and similar open-source Large Language Models. 5000 people will anonymously engage with our LLM-based chatbots. The interactions with the bot will be analysed by an expert using questionnaires that relate to safety, utility, key gaps. 

Outcomes 

The project was recently started. We were awarded “Grand Challenges Catalyzing Equitable Artificial Intelligence (AI) Use Award” from the Bill and Melinda Gates Foundation, selected as one of the 51 awardees out of 1331 applications from across the world; and is the only one from Nepal. Outcomes will be updated as project comes towards the end. 

Funding Agencies: 

Bill and Melinda Gates Foundation for the project, “Evaluating Nepali Sexual, Reproductive, and Maternal Health (SRMH) Chat-bot with Large Language Models (LLMs) ” 

Team 

Principle Investigator: Bishesh Khanal, PhD, Director/Research Scientist at NAAMII 

Research Assistants: Saugat Kafle, Neha Verma 

Partner Organizations: Vision Impact, Diyo.AI, PSD Nepal