Sign In

Dataster Documentation

Dataster helps you build Generative AI applications with better accuracy and lower latency.
Back to documentation

Run a Human Evaluation Test

Dataster provides a robust human evaluation framework that empowers builders to rigorously assess the quality of their GenAI applications' outputs across their entire use case. This framework can handle hundreds of prompts, sending them to various Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems.

Prerequisites

  1. A Dataster account.
  2. One or more user prompts grouped in a use case.
  3. One or more system prompts part of the same use case as the user prompts.
  4. One or more LLMs. Dataster provides off-the-shelf LLMs that can be used for performance testing.
  5. Optionally, one or more RAGs.

Step 1: Navigate to the Human Evaluation Page

  1. Navigate to the Human Evaluation page by clicking "Human Evaluation" in the left navigation pane.

Step 2: Select User Prompts

  1. Select the use case to use for testing.
  2. The interface indicates how many user prompts have been created for this use case.
  3. One use case must be selected.


Select User Prompts

Step 3: Select LLMs and RAGs

  1. Select the LLMs to use for testing.
  2. Select the RAGs to use for testing.
  3. At least one RAG or one LLM must be selected.
  4. LLMs and RAGs are indicated by different icons.


Select LLMs and RAGs

Step 4: Select System Prompts

  1. Select one or more system prompts for the use case.
  2. At least one system prompt must be selected.


Select System Prompts

Step 5: Select Job Options

  1. Optionally select one or more option to reduce bias.
  2. Select the type of evaluation between binary (good/bad) and multi (5 star rating).


Select System Prompts

Step 6: Run the Human Evaluation Job

  1. The user interface indicates how many tests will be run.
  2. Click Run.


Run the Human Evaluation job

Step 7: Human Evaluation Job Execution

  1. The user interface displays each test execution.
  2. Based on the type of evaluation selected, the human in the loop will see either a thumb up and a thumb down or five stars.
  3. The human in the loop evaluates the outputs.


Human Evaluation job execution

Step 8: Observe the Results

  1. After all the tests are complete, the consolidated results are displayed.
  2. For each model and RAG, the average score is displayed.
  3. For each system prompt, the average score is displayed.
  4. For each combination of model, RAG, and system prompt, the average score is displayed.
  5. Optionally, save the job results.


Human Evaluation job results

Conclusion

You have successfully run a human evaluation test in Dataster. This allows you to measure the performance of your use case and make informed decisions to optimize output quality.


If you encounter any issues or need further assistance, please contact our support team at support@dataster.com.