Sign In

Dataster Documentation

Dataster helps you build Generative AI applications with better accuracy and lower latency.
Back to documentation

Run a Latency Test

Dataster provides a robust latency testing framework that allows builders to measure the actual latency for their entire use case, potentially consisting of hundreds of prompts, by sending them to any number of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems.

Prerequisites

  1. A Dataster account.
  2. One or more user prompts grouped in a use case.
  3. One or more system prompts part of the same use case as the user prompts.
  4. One or more LLMs. Dataster provides off-the-shelf LLMs that can be used for latency testing.
  5. Optionally, one or more RAGs.

Step 1: Navigate to the Latency Page

  1. Navigate to the Latency testing page by clicking "Latency" in the left navigation pane.

Step 2: Select User Prompts

  1. Select the use case to use for testing.
  2. The interface indicates how many user prompts have been created for this use case.
  3. One use case must be selected.


Select User Prompts

Step 3: Select LLMs and RAGs

  1. Select the LLMs to use for testing.
  2. Select the RAGs to use for testing.
  3. At least one RAG or one LLM must be selected.
  4. LLMs and RAGs are indicated by different icons.


Select LLMs and RAGs

Step 4: Select System Prompts

  1. Select one or more system prompts for the use case.
  2. At least one system prompt must be selected.


Select System Prompts

Step 5: Run the Latency Job

  1. The user interface indicates how many tests will be run.
  2. Click Run.


Run the Latency job

Step 6: Latency Job Execution

  1. The user interface displays each test execution.
  2. Upon test completion, the latency is displayed.


Latency job execution

Step 7: Observe the Results

  1. After all the tests are complete, the consolidated results are displayed.
  2. For each model and RAG, the average latency and standard deviation are displayed.
  3. For each system prompt, the average latency and standard deviation are displayed.
  4. For each combination of model, RAG, and system prompt, the average latency and standard deviation are displayed.
  5. Optionally, save the job results.


Latency job results

Conclusion

You have successfully run a latency test in Dataster. This allows you to measure the performance of your use case and make informed decisions to optimize latency.


If you encounter any issues or need further assistance, please contact our support team at support@dataster.com.