Automations

• 1: Create an automation
• 1.1: Create a Slack automation
• 1.2: Create a webhook automation
• 2: Automation events and scopes

1 - Create an automation

This page gives an overview of creating and managing W&B automations. For more detailed instructions, refer to Create a Slack automation or Create a webhook automation.

Requirements

• A team admin can create and manage automations for the team’s projects, as well as components of their automations, such as webhooks, secrets, or Slack instance integrations. Refer to Team settings.
• To create a registry automation, you must have access to the registry. Refer to Configure Registry access.
• To create a Slack automation, you must have permission to post to the Slack instance and channel you select.

Create an automation

Create an automation from the project or registry’s Automations tab. At a high level, to create an automation, follow these steps:

1. If necessary, create a W&B secret for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your team settings. Secrets are most commonly used in webhook automation.
2. Configure the webhook or Slack integration to authorize W&B to post to Slack or run the webhook on your behalf. A single integration can be used by multiple automations. When you create the integration, you grant it access to any secrets it requires for authentication. Automation integrations are defined in your team settings.
3. In the project or registry, create the Slack or webhook automation. When you create a webhook automation, you configure the payload it sends and grant it access to any secrets it requires for authorization or to compose the payload.

View and manage automations

View and manage automations from a project or registry’s Automations tab.

• To view an automation’s details, click its name.
• To edit an automation, click its action ... menu, then click Edit automation.
• To delete an automation, click its action ... menu, then click Delete automation.

Next steps

• Create a Slack automation.
• Create a webhook automation.
• Create a secret.

1.1 - Create a Slack automation

This page shows how to create a Slack automation. To create a webhook automation, refer to Create a webhook automation instead.

At a high level, to create a Slack automation, you take these steps:

1. Add a Slack integration, which authorizes W&B to post to the Slack instance and channel.
2. Create the Slack automation, which defines the event to watch for and the channel to post to.

Add a Slack integration

A team admin can add integrations to the team.

1. Log in to W&B and go to Team Settings page.

2. In the Slack channel integrations section, click Connect Slack to add a new Slack instance. To add another channel in an existing Slack instance, click New integration.

   If necessary, sign in to your Slack instance. A browser window appears, asking that you grant Weights and Biases permission to post to the Slack channel you select. Read the page, then click Search for a channel and begin typing the channel name. Select the channel from the list, then click Allow.

3. In Slack, go to the channel you selected. If you see a post like [Your Slack handle] added an integration to this channel: Weights & Biases, the integration is configured correctly.

Now you can create a Slack automation.

View and manage Slack integration

A team admin can view and manage the team’s integrations.

1. Log in to W&B and go to Team Settings.
2. View each Slack destination in the Slack channel integrations section.
3. Delete a destination by clicking its trash icon.

Create a Slack automation

After you configure a Slack integration, select Registry or Project, then follow these steps to create a Slack automation.

• Registry
• Project

View and manage automations

• Registry
• Project

1.2 - Create a webhook automation

This page shows how to create a webhook automation. To create a Slack automation, refer to Create a Slack automation instead.

At a high level, to create a webhook automation, you take these steps:

1. If necessary, create a W&B secret for each sensitive string required by the automation, such as an access token, password, or SSH key. Secrets are defined in your team settings.
2. Add a webhook integration to define the webhook’s endpoint and authorization details and grant the integration access to any secrets it needs.
3. Create the webhook automation to define the event to watch for and the payload W&B will send. Grant the automation access to any secrets it needs for the payload.

Add a webhook integration

A team admin can add a webhook integrations to the team.

1. Log in to W&B and go to Team Settings page.

2. In the Webhooks section, click New webhook.

3. Provide a name for the webhook.

4. Provide the endpoint URL for the webhook.

5. If the webhook requires an access token or any other secrets for authentication, grant the webhook access to each secret by setting Secret to the secret’s name. When you configure the automation that uses the webhook, you can access the secret as a payload variable by prefixing its name with $.

   If the webhook’s access token is stored in a secret, you must also complete the next step to specify the secret as the access token.

6. If required, set Access token to the name of the secret that contains the token. When you configure an automation that uses the webhook, you can access the token in the ${ACCESS_TOKEN} payload variable, and the HTTP header sets Authorization: Bearer to the access token. Your webhook’s authentication and authorization requirements are determined by the webhook’s service.

7. Click Test. W&B attempts to authenticate to the webhook’s endpoint using the accesss token you configured.

   Optionally, provide a payload to test. To refer to a secret the webhook has access to in the payload, prefix its name with $. This payload is only used for testing and is not saved. You configure an automation’s payload when you create the automation. See Troubleshoot your webhook to view where the secret and access token are specified in the POST request.

Now you can create a webhook automation.

Create a webhook automation

After you configure a webhook integration, select Registry or Project, then follow these steps to create a webhook automation.

• Registry
• Project

View and manage automations

• Registry
• Project

Payload reference

Use these sections to construct your webhoook’s payload. For details about testing your webhook and its payload, refer to Troubleshoot your webhook.

Payload variables

This section describes the variables you can use to construct your webhook’s payload.

Payload examples

This section includes examples of webhook payloads for some common use cases. The examples demonstrate how to use payload variables.

• GitHub repository dispatch
• Microsoft Teams notification
• Slack notifications

on:
repository_dispatch:
  types: BUILD_AND_DEPLOY

{
  "event_type": "BUILD_AND_DEPLOY",
  "client_payload": 
  {
    "event_author": "${event_author}",
    "artifact_version": "${artifact_version}",
    "artifact_version_string": "${artifact_version_string}",
    "artifact_collection_name": "${artifact_collection_name}",
    "project_name": "${project_name}",
    "entity_name": "${entity_name}"
    }
}

${event_type} --> "LINK_ARTIFACT" or "ADD_ARTIFACT_ALIAS"
${event_author} --> "<wandb-user>"
${artifact_version} --> "wandb-artifact://_id/QXJ0aWZhY3Q6NTE3ODg5ODg3""
${artifact_version_string} --> "<entity>/model-registry/<registered_model_name>:<alias>"
${artifact_collection_name} --> "<registered_model_name>"
${project_name} --> "model-registry"
${entity_name} --> "<entity>"

{
"@type": "MessageCard",
"@context": "http://schema.org/extensions",
"summary": "New Notification",
"sections": [
  {
    "activityTitle": "Notification from WANDB",
    "text": "This is an example message sent via Teams webhook.",
    "facts": [
      {
        "name": "Author",
        "value": "${event_author}"
      },
      {
        "name": "Event Type",
        "value": "${event_type}"
      }
    ],
    "markdown": true
  }
]
}

{
    "text": "New alert from WANDB!",
"blocks": [
    {
            "type": "section",
        "text": {
            "type": "mrkdwn",
            "text": "Registry event: ${event_type}"
        }
    },
        {
            "type":"section",
            "text": {
            "type": "mrkdwn",
            "text": "New version: ${artifact_version_string}"
        }
        },
        {
        "type": "divider"
    },
        {
            "type": "section",
        "text": {
            "type": "mrkdwn",
            "text": "Author: ${event_author}"
        }
        }
    ]
}

Troubleshoot your webhook

Interactively troubleshoot your webhook with the W&B App UI or programmatically with a Bash script. You can troubleshoot a webhook when you create a new webhook or edit an existing webhook.

• W&B App UI
• Bash script

2 - Automation events and scopes

An automation can run when a specific event occurs at a given scope, either a registry or a project. This page lists the events that can trigger an automation at each scope.

Registry

This section describes the scopes and events for an automation in a Registry.

Scopes

You create a Registry automation in these scopes:

• A Registry and all of its collections (the default).
• A single collection.

Events

These events can trigger a Registry automation:

• Linking a new artifact to a collection: Test and validate new models or datasets when they are registered.
• Adding a new alias to a version of an artifact: Trigger a special step of your workflow when a new artifact version has a specific alias applied. For example, deploy a model when it has the deploy alias applied.

Project

This section describes the scopes and events for an automation in a project.

Scopes

You can create a project automation in these scopes:

• The project and all of its collections (the default).
• All collections that match the filter you supply.

Artifact events

This section describes the events related to an artifact that can trigger an automation.

• Linking a new artifact: Test and validate new models, datasets or dataset automatically.
• Creating a new version of an artifact: Apply recurring actions to each version of an artifact. For example, start a training job when a new dataset artifact version is created.
• Adding a new alias to a version of an artifact: Trigger a special step of your workflow when a new artifact version in a project or collection has a specific label or alias applied. For example, run a series of downstream processing steps when an artifact has the test-set-quality-check alias applied.

Run metrics events

This section describes the run metrics events that can trigger an automation.

• Run metrics threshold met: Trigger a workflow when, for a given metric, a run or the average of a number of runs meets the threshold you specify for a given metric.
• Run metrics change threshold met: Trigger a workflow when, for a given metric, a run or the average of a number of runs increases or decreases by the threshold you specify for a given metric.

To set up the metric for a run metrics event, you specify:

• An optional run name filter. Only runs matching this filter can trigger the automation.
• The metric.
• The threshold for evaluation.
• The comparison to make. Choices differ for each type of event. Refer to Run metrics comparison choices.

Run metrics comparison choices

When configuring an automation for a run metric event, you can configure how to compare the run metric value with the threshold you specify. Your choices depend on the event type.

For Run metrics threshold met events, you can configure:

1. The number of logged values to average across (defaults to 5).
2. How to compare the values with the threshold:
   • Above
   • Above or equal to
   • Below
   • Below or equal to
   • Not equal to
   • Equal to

For Run metrics change threshold met, you can configure:

1. The current window of logged values to average across (defaults to 10).
2. The prior window of logged values to average across (defaults to 50).
3. Whether to evaluate the values as relative or absolute (defaults to Relative).
4. How to compare the values with the threshold:
   • Increases by at least
   • Decreases by at least
   • Increases or decreases by at least

Next steps

• Create a Slack automation
• Create a webhook automation