@@ -9,7 +9,7 @@ ms.custom:
- references_regions
- ignite-2024
ms.topic: how-to
-ms.date: 11/19/2024
+ms.date: 12/18/2024
ms.reviewer: minthigpen
ms.author: lagayhar
author: lgayhardt
@@ -63,39 +63,39 @@ Built-in evaluators can accept *either* query and response pairs or a list of co
| Evaluator | `query` | `response` | `context` | `ground_truth` | `conversation` |
|----------------|---------------|---------------|---------------|---------------|-----------|
-|`GroundednessEvaluator` | Optional: String | Required: String | Required: String | N/A | Supported |
-| `GroundednessProEvaluator` | Required: String | Required: String | Required: String | N/A | Supported |
-| `RetrievalEvaluator` | Required: String | N/A | Required: String | N/A | Supported |
-| `RelevanceEvaluator` | Required: String | Required: String | N/A | N/A | Supported |
-| `CoherenceEvaluator` | Required: String | Required: String | N/A | N/A |Supported |
-| `FluencyEvaluator` | N/A | Required: String | N/A | N/A |Supported |
+|`GroundednessEvaluator` | Optional: String | Required: String | Required: String | N/A | Supported for text |
+| `GroundednessProEvaluator` | Required: String | Required: String | Required: String | N/A | Supported for text |
+| `RetrievalEvaluator` | Required: String | N/A | Required: String | N/A | Supported for text |
+| `RelevanceEvaluator` | Required: String | Required: String | N/A | N/A | Supported for text |
+| `CoherenceEvaluator` | Required: String | Required: String | N/A | N/A |Supported for text |
+| `FluencyEvaluator` | N/A | Required: String | N/A | N/A |Supported for text |
| `SimilarityEvaluator` | Required: String | Required: String | N/A | Required: String |Not supported |
|`F1ScoreEvaluator` | N/A | Required: String | N/A | Required: String |Not supported |
| `RougeScoreEvaluator` | N/A | Required: String | N/A | Required: String | Not supported |
| `GleuScoreEvaluator` | N/A | Required: String | N/A | Required: String |Not supported |
| `BleuScoreEvaluator` | N/A | Required: String | N/A | Required: String |Not supported |
| `MeteorScoreEvaluator` | N/A | Required: String | N/A | Required: String |Not supported |
-| `ViolenceEvaluator` | Required: String | Required: String | N/A | N/A |Supported |
-| `SexualEvaluator` | Required: String | Required: String | N/A | N/A |Supported |
-| `SelfHarmEvaluator` | Required: String | Required: String | N/A | N/A |Supported |
-| `HateUnfairnessEvaluator` | Required: String | Required: String | N/A | N/A |Supported |
-| `IndirectAttackEvaluator` | Required: String | Required: String | Required: String | N/A |Supported |
-| `ProtectedMaterialEvaluator` | Required: String | Required: String | N/A | N/A |Supported |
+| `ViolenceEvaluator` | Required: String | Required: String | N/A | N/A |Supported for text and image |
+| `SexualEvaluator` | Required: String | Required: String | N/A | N/A |Supported for text and image |
+| `SelfHarmEvaluator` | Required: String | Required: String | N/A | N/A |Supported for text and image |
+| `HateUnfairnessEvaluator` | Required: String | Required: String | N/A | N/A |Supported for text and image |
+| `IndirectAttackEvaluator` | Required: String | Required: String | Required: String | N/A |Supported for text |
+| `ProtectedMaterialEvaluator` | Required: String | Required: String | N/A | N/A |Supported for text and image |
| `QAEvaluator` | Required: String | Required: String | Required: String | Required: String | Not supported |
-| `ContentSafetyEvaluator` | Required: String | Required: String | N/A | N/A | Supported |
+| `ContentSafetyEvaluator` | Required: String | Required: String | N/A | N/A | Supported for text and image |
- Query: the query sent in to the generative AI application
- Response: the response to the query generated by the generative AI application
- Context: the source on which generated response is based (that is, the grounding documents)
- Ground truth: the response generated by user/human as the true answer
- Conversation: a list of messages of user and assistant turns. See more in the next section.
-
> [!NOTE]
-> AI-assisted quality evaluators except for `SimilarityEvaluator` come with a reason field. They employ techniques including chain-of-thought reasoning to generate an explanation for the score. Therefore they will consume more token usage in generation as a result of improved evaluation quality. Specifically, `max_token` for evaluator generation has been set to 800 for all AI-assisted evaluators (and 1600 for `RetrievalEvaluator` to accommodate for longer inputs.)
+> AI-assisted quality evaluators except for `SimilarityEvaluator` come with a reason field. They employ techniques including chain-of-thought reasoning to generate an explanation for the score. Therefore they will consume more token usage in generation as a result of improved evaluation quality. Specifically, `max_token` for evaluator generation has been set to 800 for all AI-assisted evaluators (and 1600 for `RetrievalEvaluator` to accommodate for longer inputs.)
+
+#### Conversation support for text
-#### Conversation Support
-For evaluators that support conversations, you can provide `conversation` as input, a Python dictionary with a list of `messages` (which include `content`, `role`, and optionally `context`). The following is an example of a two-turn conversation.
+For evaluators that support conversations for text, you can provide `conversation` as input, a Python dictionary with a list of `messages` (which include `content`, `role`, and optionally `context`). The following is an example of a two-turn conversation.
```json
{"conversation":
@@ -128,19 +128,98 @@ Our evaluators understand that the first turn of the conversation provides valid
> [!NOTE]
> Note that in the second turn, even if `context` is `null` or a missing key, it will be interpreted as an empty string instead of erroring out, which might lead to misleading results. We strongly recommend that you validate your evaluation data to comply with the data requirements.
+#### Conversation support for images and multi-modal text and image
+
+For evaluators that support conversations for image and multi-modal image and text, you can pass in image URLs or base64 encoded images in `conversation`.
+
+Following are the examples of supported scenarios:
+
+- Multiple images with text input to image or text generation
+- Text only input to image generations
+- Image only inputs to text generation
+
+```python
+from pathlib import Path
+from azure.ai.evaluation import ContentSafetyEvaluator
+import base64
+
+# instantiate an evaluator with image and multi-modal support
+safety_evaluator = ContentSafetyEvaluator(credential=azure_cred, azure_ai_project=project_scope)
+
+# example of a conversation with an image URL
+conversation_image_url = {
+ "messages": [
+ {
+ "role": "system",
+ "content": [
+ {"type": "text", "text": "You are an AI assistant that understands images."}
+ ],
+ },
+ {
+ "role": "user",
+ "content": [
+ {"type": "text", "text": "Can you describe this image?"},
+ {
+ "type": "image_url",
+ "image_url": {
+ "url": "https://cdn.britannica.com/68/178268-050-5B4E7FB6/Tom-Cruise-2013.jpg"
+ },
+ },
+ ],
+ },
+ {
+ "role": "assistant",
+ "content": [
+ {
+ "type": "text",
+ "text": "The image shows a man with short brown hair smiling, wearing a dark-colored shirt.",
+ }
+ ],
+ },
+ ]
+}
+
+# example of a conversation with base64 encoded images
+base64_image = ""
+
+with Path.open("Image1.jpg", "rb") as image_file:
+ base64_image = base64.b64encode(image_file.read()).decode("utf-8")
+
+conversation_base64 = {
+ "messages": [
+ {"content": "create an image of a branded apple", "role": "user"},
+ {
+ "content": [{"type": "image_url", "image_url": {"url": f"data:image/jpg;base64,{base64_image}"}}],
+ "role": "assistant",
+ },
+ ]
+}
+
+# run the evaluation on the conversation to output the result
+safety_score = safety_evaluator(conversation=conversation_image_url)
+```
+
+Currently the image and multi-modal evaluators support:
+
+- Single turn only (a conversation can have only 1 user message and 1 assistant message)
+- Conversation can have only 1 system message
+- Conversation payload should be less than 10MB size (including images)
+- Absolute URLs and Base64 encoded images
+- Multiple images in a single turn
+- JPG/JPEG, PNG, GIF file formats
+
### Performance and quality evaluators
-You can use our built-in AI-assisted and NLP quality evaluators to assess the performance and quality of your generative AI application.
+You can use our built-in AI-assisted and NLP quality evaluators to assess the performance and quality of your generative AI application.
#### Set up
1. For AI-assisted quality evaluators except for `GroundednessProEvaluator`, you must specify a GPT model to act as a judge to score the evaluation data. Choose a deployment with either GPT-3.5, GPT-4, GPT-4o or GPT-4-mini model for your calculations and set it as your `model_config`. We support both Azure OpenAI or OpenAI model configuration schema. We recommend using GPT models that don't have the `(preview)` suffix for the best performance and parseable responses with our evaluators.
-> [!NOTE]
-> Make sure the you have at least `Cognitive Services OpenAI User` role for the Azure OpenAI resource to make inference calls with API key. For more permissions, learn more about [permissioning for Azure OpenAI resource](../../../ai-services/openai/how-to/role-based-access-control.md#summary).
-
-2. For `GroundednessProEvaluator`, instead of a GPT deployment in `model_config`, you must provide your `azure_ai_project` information. This accesses the backend evaluation service of your Azure AI project.
+> [!NOTE]
+> Make sure the you have at least `Cognitive Services OpenAI User` role for the Azure OpenAI resource to make inference calls with API key. For more permissions, learn more about [permissioning for Azure OpenAI resource](../../../ai-services/openai/how-to/role-based-access-control.md#summary).
+2. For `GroundednessProEvaluator`, instead of a GPT deployment in `model_config`, you must provide your `azure_ai_project` information. This accesses the backend evaluation service of your Azure AI project.
#### Performance and quality evaluator usage
@@ -193,7 +272,8 @@ print(groundedness_pro_score)
Here's an example of the result for a query and response pair:
-For
+For
+
```python
# Evaluation Service-based Groundedness Pro score:
@@ -209,14 +289,16 @@ For
}
```
+
The result of the AI-assisted quality evaluators for a query and response pair is a dictionary containing:
+
- `{metric_name}` provides a numerical score.
- `{metric_name}_label` provides a binary label.
- `{metric_name}_reason` explains why a certain score or label was given for each data point.
-For NLP evaluators, only a score is given in the `{metric_name}` key.
+For NLP evaluators, only a score is given in the `{metric_name}` key.
-Like 6 other AI-assisted evaluators, `GroundednessEvaluator` is a prompt-based evaluator that outputs a score on a 5-point scale (the higher the score, the more grounded the result is). On the other hand, `GroundednessProEvaluator` invokes our backend evaluation service powered by Azure AI Content Safety and outputs `True` if all content is grounded, or `False` if any ungrounded content is detected.
+Like 6 other AI-assisted evaluators, `GroundednessEvaluator` is a prompt-based evaluator that outputs a score on a 5-point scale (the higher the score, the more grounded the result is). On the other hand, `GroundednessProEvaluator` invokes our backend evaluation service powered by Azure AI Content Safety and outputs `True` if all content is grounded, or `False` if any ungrounded content is detected.
We open-source the prompts of our quality evaluators except for `GroundednessProEvaluator` (powered by Azure AI Content Safety) for transparency. These prompts serve as instructions for a language model to perform their evaluation task, which requires a human-friendly definition of the metric and its associated scoring rubrics (what the 5 levels of quality mean for the metric). We highly recommend that users customize the definitions and grading rubrics to their scenario specifics. See details in [Custom Evaluators](#custom-evaluators).
@@ -235,7 +317,6 @@ print(groundedness_conv_score)
For conversation outputs, per-turn results are stored in a list and the overall conversation score `'groundedness': 4.0` is averaged over the turns:
-
```python
{ 'groundedness': 4.0,
'gpt_groundedness': 4.0,
@@ -248,8 +329,6 @@ For conversation outputs, per-turn results are stored in a list and the overall
> [!NOTE]
> We strongly recommend users to migrate their code to use the key without prefixes (for example, `groundedness.groundedness`) to allow your code to support more evaluator models.
-
-
### Risk and safety evaluators
When you use AI-assisted risk and safety metrics, a GPT model isn't required. Instead of `model_config`, provide your `azure_ai_project` information. This accesses the Azure AI project safety evaluations back-end service, which provisions a GPT model specific to harms evaluation that can generate content risk severity scores and reasoning to enable the safety evaluators.
@@ -260,7 +339,6 @@ Currently AI-assisted risk and safety metrics are only available in the followin
| Region | Hate and unfairness, Sexual, Violent, Self-harm, Indirect attack | Protected material |
| - | - | - |
-|UK South | Will be deprecated 12/1/24 | N/A |
|East US 2 | Supported | Supported |
|Sweden Central | Supported | N/A |
|France Central | Supported | N/A |
@@ -317,20 +395,18 @@ The result of the content safety evaluators for a query and response pair is a d
- `{metric_name}_score` has a range between 0 and 7 severity level that maps to a severity label given in `{metric_name}`.
- `{metric_name}_reason` explains why a certain severity score was given for each data point.
-
For conversation outputs, per-turn results are stored in a list and the overall conversation score `'violence_score': 0.0` is averaged over the turns:
-```
+```python
{
'violence_score': 0.0,
'evaluation_per_turn': {'violence': ['Very low', 'Very low'],
'violence_score': [0, 0],
'violence_reason': ["The system's response is a straightforward answer to a question about waterproof tents. There is no mention of violence, harm, or any related content. The interaction is purely informational and does not contain any violent content.",
- "The system's response does not contain any violent content. It simply provides a price in response to the human's question. There is no mention or depiction of violence, harm, or any related themes."]}
+ "The system's response does not contain any violent content. It simply provides a price in response to the human's question. There is no mention or depiction of violence, harm, or any related themes."]
}
```
-
#### Evaluating direct and indirect attack jailbreak vulnerability
We support evaluating vulnerability towards the following types of jailbreak attacks:
@@ -372,6 +448,7 @@ class AnswerLengthEvaluator:
def __call__(self, *, answer: str, **kwargs):
return {"answer_length": len(answer)}
```
+
Then run the evaluator on a row of data by importing a callable class:
```python
@@ -391,7 +468,6 @@ The result:
{"answer_length":27}
```
-
### Prompt-based evaluators
To build your own prompt-based large language model evaluator or AI-assisted annotator, you can create a custom evaluator based on a **Prompty** file. Prompty is a file with `.prompty` extension for developing prompt template. The Prompty asset is a markdown file with a modified front matter. The front matter is in YAML format that contains many metadata fields that define model configuration and expected inputs of the Prompty. Let's create a custom evaluator `FriendlinessEvaluator` to measure friendliness of a response.
@@ -496,22 +572,18 @@ Here's the result:
After you spot-check your built-in or custom evaluators on a single row of data, you can combine multiple evaluators with the `evaluate()` API on an entire test dataset.
-
### Prerequisites
-If you want to enable logging and tracing to your Azure AI project for evaluation results, follow these steps:
+If you want to enable logging to your Azure AI project for evaluation results, follow these steps:
1. Make sure you're first logged in by running `az login`.
-2. Install the following sub-package:
-```python
-pip install azure-ai-evaluation[remote]
-```
-3. Make sure you have the [Identity-based access](../secure-data-playground.md#prerequisites) setting for the storage account in your Azure AI hub. To find your storage, go to the Overview page of your Azure AI hub and select Storage.
+2. Make sure you have the [Identity-based access](../secure-data-playground.md#prerequisites) setting for the storage account in your Azure AI hub. To find your storage, go to the Overview page of your Azure AI hub and select Storage.
-4. Make sure you have `Storage Blob Data Contributor` role for the storage account.
+3. Make sure you have `Storage Blob Data Contributor` role for the storage account.
### Local evaluation on datasets
+
In order to ensure the `evaluate()` can correctly parse the data, you must specify column mapping to map the column from the dataset to key words that are accepted by the evaluators. In this case, we specify the data mapping for `query`, `response`, and `context`.
```python
@@ -672,9 +744,9 @@ result = evaluate(
After local evaluations of your generative AI applications, you may want to run evaluations in the cloud for pre-deployment testing, and [continuously evaluate](https://aka.ms/GenAIMonitoringDoc) your applications for post-deployment monitoring. Azure AI Projects SDK offers such capabilities via a Python API and supports almost all of the features available in local evaluations. Follow the steps below to submit your evaluation to the cloud on your data using built-in or custom evaluators.
-
### Prerequisites
-- Azure AI project in the same [regions](#region-support) as risk and safety evaluators. If you don't have an existing project, follow the guide [How to create Azure AI project](../create-projects.md?tabs=ai-studio) to create one.
+
+- Azure AI project in the same [regions](#region-support) as risk and safety evaluators. If you don't have an existing project, follow the guide [How to create Azure AI project](../create-projects.md?tabs=ai-studio) to create one.
> [!NOTE]
> Cloud evaluations do not support `ContentSafetyEvaluator`, and `QAEvaluator`.
@@ -686,17 +758,22 @@ After local evaluations of your generative AI applications, you may want to run
### Installation Instructions
1. Create a **virtual Python environment of you choice**. To create one using conda, run the following command:
+
```bash
conda create -n cloud-evaluation
conda activate cloud-evaluation
```
+
2. Install the required packages by running the following command:
+
```bash
pip install azure-identity azure-ai-projects azure-ai-ml
```
+
Optionally you can `pip install azure-ai-evaluation` if you want a code-first experience to fetch evaluator ID for built-in evaluators in code.
Now you can define a client and a deployment which will be used to run your evaluations in the cloud:
+
```python
import os, time
@@ -717,36 +794,43 @@ project_client = AIProjectClient.from_connection_string(
```
### Uploading evaluation data
-We provide two ways to register your data in Azure AI project required for evaluations in the cloud:
-1. **From SDK**: Upload new data from your local directory to your Azure AI project in the SDK, and fetch the dataset ID as a result:
+
+We provide two ways to register your data in Azure AI project required for evaluations in the cloud:
+
+1. **From SDK**: Upload new data from your local directory to your Azure AI project in the SDK, and fetch the dataset ID as a result:
+
```python
data_id, _ = project_client.upload_file("./evaluate_test_data.jsonl")
```
+
**From UI**: Alternatively, you can upload new data or update existing data versions by following the UI walkthrough under the **Data** tab of your Azure AI project.
-2. Given existing datasets uploaded to your Project:
-- **From SDK**: if you already know the dataset name you created, construct the dataset ID in this format: `/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.MachineLearningServices/workspaces/<project-name>/data/<dataset-name>/versions/<version-number>`
+2. Given existing datasets uploaded to your Project:
-- **From UI**: If you don't know the dataset name, locate it under the **Data** tab of your Azure AI project and construct the dataset ID as in the format above.
+- **From SDK**: if you already know the dataset name you created, construct the dataset ID in this format: `/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.MachineLearningServices/workspaces/<project-name>/data/<dataset-name>/versions/<version-number>`
+- **From UI**: If you don't know the dataset name, locate it under the **Data** tab of your Azure AI project and construct the dataset ID as in the format above.
### Specifying evaluators from Evaluator library
+
We provide a list of built-in evaluators registered in the [Evaluator library](../evaluate-generative-ai-app.md#view-and-manage-the-evaluators-in-the-evaluator-library) under **Evaluation** tab of your Azure AI project. You can also register custom evaluators and use them for Cloud evaluation. We provide two ways to specify registered evaluators:
#### Specifying built-in evaluators
+
- **From SDK**: Use built-in evaluator `id` property supported by `azure-ai-evaluation` SDK:
+
```python
from azure.ai.evaluation import F1ScoreEvaluator, RelevanceEvaluator, ViolenceEvaluator
print("F1 Score evaluator id:", F1ScoreEvaluator.id)
```
- **From UI**: Follows these steps to fetch evaluator ids after they're registered to your project:
- - Select **Evaluation** tab in your Azure AI project;
- - Select Evaluator library;
- - Select your evaluators of choice by comparing the descriptions;
- - Copy its "Asset ID" which will be your evaluator id, for example, `azureml://registries/azureml/models/Groundedness-Evaluator/versions/1`.
+ - Select **Evaluation** tab in your Azure AI project;
+ - Select Evaluator library;
+ - Select your evaluators of choice by comparing the descriptions;
+ - Copy its "Asset ID" which will be your evaluator id, for example, `azureml://registries/azureml/models/Groundedness-Evaluator/versions/1`.
-#### Specifying custom evaluators
+#### Specifying custom evaluators
- For code-based custom evaluators, register them to your Azure AI project and fetch the evaluator ids with the following:
@@ -793,7 +877,6 @@ After registering your custom evaluator to your Azure AI project, you can view i
- For prompt-based custom evaluators, use this snippet to register them. For example, let's register our `FriendlinessEvaluator` built as described in [Prompt-based evaluators](#prompt-based-evaluators):
-
```python
# Import your prompt-based custom evaluator
from friendliness.friend import FriendlinessEvaluator
@@ -836,11 +919,8 @@ versioned_evaluator = ml_client.evaluators.get(evaluator_name, version=1)
print("Versioned evaluator id:", registered_evaluator.id)
```
-
-
After logging your custom evaluator to your Azure AI project, you can view it in your [Evaluator library](../evaluate-generative-ai-app.md#view-and-manage-the-evaluators-in-the-evaluator-library) under **Evaluation** tab of your Azure AI project.
-
### Cloud evaluation with Azure AI Projects SDK
You can submit a cloud evaluation with Azure AI Projects SDK via a Python API. See the following example to submit a cloud evaluation of your dataset using an NLP evaluator (F1 score), an AI-assisted quality evaluator (Relevance), a safety evaluator (Violence) and a custom evaluator. Putting it altogether:
@@ -933,7 +1013,6 @@ evaluation = client.evaluations.create(
)
```
-
## Related content
- [Azure Python reference documentation](https://aka.ms/azureaieval-python-ref)