Ragas: Open-source evaluation and testing Infrastructure for LLM applications
Deploy your LLM application with confidence.
TL;DR
We are building Ragas — an open-source evaluation and testing infrastructure for LLM application developers to deploy their applications in production with confidence.
About us
We’re Jithin and Shahul! Having met in college, we’ve collaborated on various projects for almost a decade now.
Jithin takes care of building the software and infrastructure. He was an early employee at Bento ML, where he built and maintained tools like Bentoctl, Bentoml, and Yatai. Shahul is responsible for AI research and engineering. He is a Kaggle Grandmaster and a lead contributor to different open-source AI projects, including Open-Assistant AI.
Problem
Before 2023 software used to be written in code but with the emergence of foundational models software and applications are going to be compound systems containing code, prompts, and other components. This introduces several new problems
- How do you select the best model or component suitable for your application from the abundance of available resources?
- How do you test these systems and ensure continuous quality?
- How do you derive insights from production to measure and improve your system?
As early adopters of this technology to build applications, we faced this problem while we were building RAG systems early last year.
Solution
We at Ragas make use of model-graded evaluations and testing techniques to ensure quality. This includes automated synthesis of test data points, explainable metrics, and adversarial testing.
We started by building this for RAGs, which is the most popular application of LLM as of today. Ragas is now the default open-source standard for evaluating RAG applications, processing over 4.7 million responses last month and used by engineers from enterprises like AWS, Microsoft, Databricks, Moody’s, UHG, and Tencent.
