Automate workflows with ArcGIS Notebooks

ArcGIS Notebook Server allows you to implement multiple methods of workflow automation, including automation through scheduled notebooks, webhooks, and your own custom scripts using the Execute Notebook administrative API.

Note:

To automate a notebook using scheduled tasks, webhooks, or the Execute Notebook API, the notebook must use a runtime of version 3.0 or later.

Schedule notebook tasks

Starting at 10.8.1, notebook authors can schedule ArcGIS Notebooks to run automatically at a fixed time in the future, either once or on a recurring basis. Creating tasks to schedule notebooks allows you to automate routine workflows, run data-intensive processes during off-peak usage hours, and regularly update datasets. For example, you can schedule a notebook to do the following:

• Import data from an online source that updates monthly, automatically clean the data and apply necessary transformations, and move the data to your workspace.
• Run a big data analysis workflow that requires heavy processing power overnight, when your machine resources are otherwise unused.
• Manage users who have created accounts in your portal during the past week and send the list to you via email.

You can create one or more tasks for a notebook. By default, the notebook author or administrator can create a maximum of 20 tasks. If the ownership of notebook is changed, any tasks associated with that notebook will be deactivated and assigned to the new owner.

Scheduled tasks allow you to parameterize notebooks. When a notebook is parameterized, it allows you to write generic code that can be adapted to varying inputs without your interaction. The chosen parameters are inserted into the notebook when a task is run and can optionally be saved to the notebook. For example, a parameterized notebook can be used to generate region-wide air pollution reports on a recurring basis. The notebook can have multiple scheduled tasks, one for each region to be studied, and for each task, parameterized input such as city name and pollution type can be fed into the notebook.

Note:

The number of concurrent runs of notebook tasks is limited by the maxExecuteNotebookContainersPerNode system property in Notebook Server.

You can configure the task to save the state of the notebook to the original notebook item after completion.

A static HTML view of the notebook will be saved for each scheduled task that is run. By default, only the results of the last 30 runs will be saved.

Using the Tasks page in ArcGIS Notebook Server Manager, administrators can view details, edit, pause and resume, or delete each active notebook task in the site. Administrators and notebook authors with the schedule notebook privilege can view details, edit, pause and resume, or delete a notebook task in the details page of the notebook or in the Task pane of the notebook editor.

To prevent repeated failed runs of a notebook, a scheduled task will be disabled after five consecutive failures. The task owner must ensure that the notebook can be run successfully without any user interaction before reactivating the task.

If a previous run of a task is still running, a new scheduled task run will be skipped. For example, if a task is scheduled to run every 15 minutes, but an instance of that task runs for 20 minutes, the next scheduled run will be skipped. If this occurs regularly, the task owner should adjust the scheduled time interval so that there is no overlap between runs.

For more information on scheduled notebook tasks, see the Schedule a notebook task help topic.

Automate a notebook using webhooks

A webhook is a mechanism that allows an application to provide other applications with event-driven information. As an ArcGIS Enterprise Portal administrator, you can create, manage, and configure webhooks. You can configure webhooks to automatically run a notebook you own when events associated with your portal items, groups, or users occur. Once a webhook is triggered, an HTTP request is made to the Notebook Server to run a chosen notebook without any user interaction.

Note:

The number of concurrent runs of notebooks via webhooks is limited by the maxExecuteNotebookContainersPerNode system property in Notebook Server.

Create a webhook

Webhooks can only be administered through the ArcGIS Portal Directory (Sharing API). Use the following steps to create a webhook:

1. Browse to the ArcGIS Portal Directory.
   https://machine.domain.com/webadaptor/sharing/rest
2. Sign in as an administrator.

   Webhooks can only be created and managed by an administrator.

   The administrator user page appears.

3. Click the Org ID hyperlink, or make a request of the form below, to go to the Portal Self resource page.
   https://machine.domain.com/webadaptor/sharing/rest/portals/<orgID>
4. Scroll to the bottom of the page to Webhooks, under Child Resources.
   https://machine.domain/com/webadaptor/sharing/rest/portals/<orgID>/webhooks
5. Under Supported Operation, select Create Webhook.
6. Specify the parameters for your webhook. To create a webhook automating a notebook, provide the following information:

   Parameter	Details
   name (Required)	Specifies the webhook's name. Example: name=mynotebook_webhook
   url (Optional)	The payload URL is not required when configuring a webhook for notebooks, as the system will automatically derive it. You do not have to provide this information.
   config (Required)	Sets the configuration properties for your webhook. Note: You must be an administrator and own the notebook item to be run via webhook. Example: { "deactivationPolicy": { "numberOfFailures": 5, "daysInPast": 5 }, "properties": { "federatedServer": { "itemId": "<Notebook item id to be executed>", "tokenTypeToSend": "owner", "tokenExpirationTimeMinutes": 10 } } }
   events (Optional)	The URI representation of an event that will trigger the webhook. This parameter is required if changes is specified as manual. For more information, see the Supported trigger events section. For additional examples of trigger events, see the Trigger Events section of Create webhooks in the REST API documentation. Example: events=/items,/users

   See the webhook REST API documentation for details about these parameters.

   Your webhook is now listed under webhooks: https://machine.domain.com/webadaptor/sharing/rest/portals/<orgID>/webhooks.

For more information on automation using webhooks, see the Webhooks in ArcGIS Enterprise help topic.

Manage webhooks

You can manage your webhooks through the ArcGIS Portal Directory by making a request of the following form:

The supported operations for managing your webhooks are as follows:

• Update Webhook—Update your webhook's parameters. You can update the name, payload URL, configuration, or trigger events for the specified webhook.
• Delete Webhook—Remove the webhook from your portal.
• Deactivate Webhook and Activate Webhook—Deactivate your webhook, which stops payloads from being delivered when the webhook is triggered. When the webhook is deactivated, the Activate Webhook operation is available to resume delivery of payloads.

The Notification Status page displays information relating to trigger events associated with the specific webhook. You can use this table to monitor your webhook as well as details of delivered payloads, such as the time the webhook was triggered and the responses received from the payload URL and the delivered payload. Records indicating the successful delivery of a payload are removed after one day. Records indicating a failed delivery attempt are stored for seven days.

See the Webhooks API for examples of these operations.

For more information on automation using webhooks, see the Webhooks in ArcGIS Enterprise help topic.

Webhook payload

When a notebook is run via a webhook, the webhook's payload information is automatically injected into the notebook in an initial cell. Once injected, the information provided in the webhook payload can be used in the notebook for further automation workflows.

Webhook payload information included in a notebook:

{
  "properties": {
    "itemId": "<itemId of the current notebook>"
  },
  "events": [
    {
      "source": "items",
      "id": "<itemId of the item that resulted in webhook trigger>",
      "userId": "<webhook user>",
      "when": \<epoch time in ms when webhook was triggered\>,
      "operation": "<type of operation that resulted in webhook trigger>",
      "properties": {},
      "username": "<webhook username>"
    }
  ],
  "info": {
    "webhookId": "<webhookId>",
    "portalURL": "<public portal url>",
    "webhookName": "<webhook name>",
    "when": \<epoch time in ms when webhook was triggered\>,
  }
}

Execute Notebook API

Administrators and notebook authors can also automate a notebook to run without user interaction by using the Execute Notebook operation in the ArcGIS Notebook Server Administrator Directory. This operation will automatically run a notebook when called, but by using your own custom script, you can schedule it to run automatically at a set time or on a regular schedule. A cron job or Windows scheduler can also be used to schedule the Execute Notebook operation to run at a set time or on a recurring interval.

To learn more, see the Execute Notebook topic in the Administrator Directory reference guide.

Maximum concurrent automated notebook runs

When a notebook is run via a scheduled task, webhook, or the Execute Notebook API, ArcGIS Notebook Server will open a new container and run that notebook automatically without any user interaction. By default, ArcGIS Notebook Server is configured to run a maximum of 10 concurrent notebook runs per each Notebook Server machine. Considering the resources (CPU and RAM) available on each machine in the Notebook Server site, an administrator can adjust this limit by modifying the maxExecuteNotebookContainersPerNode system property in the Notebook Server.