Edit a test set

[This article is prerelease documentation and is subject to change.]

During or after creating a test set, you can edit the test cases by changing the wording of questions, choosing different test methods, or modifying the expected responses as needed. You can select multiple test cases to edit them in bulk by selecting the checkboxes beside each test case.

You have a choice of three test methods, also referred to as graders, to evaluate agent responses: quality, similarity, text match. For more information about the different test methods, see Test methods.

Edit the details of a test set

You can change the basic details of a test set and which account it uses for authentication.

1. In your agent's Evaluations page, select the test set.

2. In the test set, select the test case you want to edit.

3. Optionally, change the name of the test set.

4. Select Manage profile to add an account, or select the current profile to manage connections or choose a different profile. Automated testing uses the authentication of the selected test account. If your agent has knowledge sources or tools that require specific authentication, you can select the appropriate account for your testing.

Manage user profiles and connections

You can run test sets by using the authentication of a user's account. This authentication method allows your agent to access the knowledge sources and tools it needs to complete its goals.

You can also use profiles to simulate specific user experiences. For example, a director's user profile can access different knowledge sources than an intern's user profile, and the agent should return different results.

Test results show which user profile was used.

To add or change a user profile in a test set:

1. Open a test set, then select Manage profile.

1. Select the User dropdown, then:

• Select the user account you want to use, or
• Select Add an account, then sign in to the account.

1. For connections with a warning, select the connection. Change the existing connection to a working connection, select Reconnect to try to reestablish the existing connection, or select Create new connection to try a new connection.

For new connections, enter the requested connection details, then select Connect.

1. When all connections are working, select Save.

Edit test cases within a test set

1. In your agent's Evaluations page, select the test set.

2. In the test set, select the test case you want to edit.

1. In the right pane, change the wording of a question by editing the text in the Question field.

1. Select the test method that you want to use.

• Quality:

  • Select General quality to evaluate the answer based on relevance, groundedness, and completeness.

  • Select Compare meaning to evaluate the answer based on how well it captures the meaning of the expected response. Under Passing score, you can set the threshold for what constitutes a passing score for an answer.

    In the Expected response box, provide the response against which the test method evaluates the agent's answer.

• Similarity: uses a Cosine similarity metric to assess how similar the agent's answer is to the wording and meaning of the expected response. It determines a score between 0 and 1, where 1 means it matches closely and 0 means it doesn't match at all. Under Passing score, you can set the threshold for what constitutes a passing score for an answer.

  In the Expected response box, provide the response against which the test method evaluates the agent's answer.

• Text Match:

  • Select Exact match to evaluate the agent's answer against the expected response, where a passing score means the agent's answer exactly matched the defined expected response.

    In the Expected response box, provide the response against which the test method evaluates the agent's answer.

  • Select Partial match to evaluate the agent's answer against the expected response, where a passing score means the agent's answer contained some of the words or phrases from the defined expected response.

    In the Expected response box, provide a phrase or keyword against which the test method evaluates the agent's answer. To add multiple keywords or phrases, select Add, select the operator and or or between the boxes, and provide the keyword or phrase.

    

1. Select Apply.

2. When you're finished with your changes, select Save to save your test set or Evaluate to run the test.

Add more test cases to a test set

While creating or editing a test set, you can add more test cases.

1. In your agent's Evaluations page, select the test set.

2. Generate or manually add more test cases.

1. Select and edit any new test cases to add details.

Delete a test case or test set

To delete a test case, hover over a test case, then select the delete button.

You can also delete test sets:

1. On your agent's Analytics page, for a test set, select the More icon (…) > Delete.