Skip to main content
Retrieval Augmented Generation (RAG) is a common way to build Generative AI applications that have access to custom knowledge bases. This tutorial walks you through building a RAG application and using Weave to track its retrieval steps and evaluate its responses with an LLM judge, so you can measure and improve the quality of the answers your application returns. This guide is for developers building RAG applications who want to add observability and systematic evaluation to their pipelines. Evals hero

​
What you’ll learn

This guide shows you how to:
  • Build a knowledge base.
  • Create a RAG application with a retrieval step that finds relevant documents.
  • Track retrieval steps with Weave.
  • Evaluate RAG applications using an LLM judge to measure context precision.
  • Define custom scoring functions.

​
Prerequisites

  • A W&B account
  • Python 3.10+ or Node.js 18+
  • Required packages installed:
    • Python: pip install weave openai
    • TypeScript: npm install weave openai
  • An OpenAI API key set as an environment variable.

​
Build a knowledge base

The knowledge base is the corpus your RAG application searches at query time. In this section, you compute vector embeddings for a small set of articles so that you can later retrieve the most relevant article for a given question. First, compute the embeddings for the articles. You typically do this once with your articles and put the embeddings and metadata in a database, but here it’s done every time the script runs for simplicity.

​
Create a RAG app

With the knowledge base in place, you can now build the RAG application itself. This section combines the retrieval step with an LLM call and wraps both with Weave so that every input and output is tracked automatically. Next, wrap the retrieval function get_most_relevant_document with a weave.op() decorator and create a Model class. Wrapping the retrieval function with weave.op() lets Weave capture its inputs and outputs for every call, which is what makes the retrieval step inspectable later. Call weave.init('<team-name>/rag-quickstart') to begin tracking all the inputs and outputs of your functions for later inspection. If you don’t specify a team name, Weave records the output to your W&B default team or entity.

​
Evaluate with an LLM judge

Now that your RAG application is running and being tracked by Weave, the next step is to evaluate how well it answers questions. This section shows how to use an LLM as an automatic judge so you can score your application’s responses without writing labels by hand. When there aren’t simple ways to evaluate your application, one approach is to use an LLM to evaluate aspects of it. Here’s an example of using an LLM judge to measure the context precision by prompting it to verify whether the context was useful in arriving at the given answer. This prompt was augmented from the popular RAGAS framework.

​
Define a scoring function

As in the Build an Evaluation pipeline tutorial, define a set of example rows to test your app against and a scoring function. The scoring function takes one row and evaluates it. The input arguments should match the corresponding keys in your row, so question here is taken from the row dictionary. output is the output of the model. The input to the model is taken from the example based on its input argument, so question here too. This example uses async functions so they run in parallel. For a quick introduction to async, see the Python asyncio documentation.

​
Optional: Define a Scorer class

The scoring function in the previous section works well for simple cases, but a Scorer class is useful when you want to reuse the same judge across evaluations or customize how scores are aggregated. The following steps show when and how to define one. In some applications you may want to create custom evaluation classes (for example, a standardized LLMJudge class with parameters such as chat model and prompt), custom scoring of each row, and custom calculation of an aggregate score. Weave defines a list of ready-to-use Scorer classes and also makes it easy to create a custom Scorer. The following example shows how to create a custom class CorrectnessLLMJudge(Scorer). At a high level, the steps to create a custom Scorer are:
  1. Define a custom class that inherits from weave.flow.scorer.Scorer.
  2. Overwrite the score function and add a @weave.op() if you want to track each call of the function.
    • This function has to define an output argument where the prediction of the model is passed. Define it as type Optional[dict] in case the model returns “None”.
    • The rest of the arguments can either be a general Any or dict or can select specific columns from the dataset that is used to evaluate the model using the weave.Evaluate class. They have to have the exact same names as the column names or keys of a single row after being passed to preprocess_model_input if that is used.
  3. Optional: Overwrite the summarize function to customize the calculation of the aggregate score. By default, Weave uses the weave.flow.scorer.auto_summarize function if you don’t define a custom function.
    • This function has to have a @weave.op() decorator.
To use this as a scorer, initialize it and pass it to the scorers argument in your Evaluation like this:

​
Put it all together

This section consolidates everything from the previous steps into a single end-to-end example so you can see how the pieces fit together and adapt them to your own RAG application. To get the same result for your RAG apps:
  • Wrap LLM calls and retrieval step functions with weave.op().
  • Optional: Create a Model subclass with predict function and app details.
  • Collect examples to evaluate.
  • Create scoring functions that score one example.
  • Use Evaluation class to run evaluations on your examples.
NOTE: Sometimes the async execution of Evaluations triggers a rate limit on the models of OpenAI, Anthropic, and others. To prevent that, you can set an environment variable to limit the number of parallel workers, for example, WEAVE_PARALLELISM=3. Here’s the code in its entirety.

​
Conclusion

This tutorial showed how to build observability into different steps of your applications, like the retrieval step in this example. You also learned how to build more complex scoring functions, such as an LLM judge, for automatic evaluation of application responses.

​
Next steps

See the RAG++ course for a more advanced dive into practical RAG techniques for engineers, where you learn production-ready solutions from W&B, Cohere, and Weaviate to optimize performance, cut costs, and enhance the accuracy and relevance of your applications.