Automate workflows with ArcGIS Notebooks

ArcGIS Notebook Server allows you to implement multiple methods of workflow automation, including automation through scheduled notebooks, webhooks, and your own custom scripts using the Execute Notebook administrative API.

Note:

To automate a notebook using scheduled tasks, webhooks, or the Run Notebook API, the notebook must use a runtime of version 3.0 or later.

Schedule notebook tasks

Starting at 10.8.1, notebook authors can schedule ArcGIS Notebooks for automated running at a fixed time in the future, either once or on a recurring basis. Creating tasks to schedule notebooks allows you to automate routine workflows, run data-intensive processes during off-peak usage hours, and regularly update datasets. For example, you can schedule a notebook to do the following:

• Import data from an online source that updates monthly, automatically clean the data and apply necessary transformations, and move the data to your workspace.
• Run a big data analysis workflow that requires heavy processing power overnight, when your machine resources are otherwise unused.
• Manage users who have created accounts in your portal during the past week and send the list to you in an email.

You can create one or more tasks for a notebook. By default, the notebook author or administrator can create a maximum of 20 tasks. If the ownership of notebook is changed, any tasks associated with that notebook will be deactivated and assigned to the new owner.

Scheduled tasks allow you to parameterize notebooks. When a notebook is parameterized, it allows you to write generic code that can be adapted to varying inputs without your interaction. The chosen parameters are inserted into the notebook when a task is run and can optionally be saved to the notebook. For example, a parameterized notebook can be used to generate region-wide air pollution reports on a recurring basis. The notebook can have multiple scheduled tasks, one for each region to be studied, and for each task, parameterized input such as city name and pollution type can be fed into the notebook.

Note:

The number of concurrently running notebook tasks is limited by the maxExecuteNotebookContainersPerNode system property in Notebook Server.

You can configure the task to save the state of the notebook to the original notebook item after completion.

A static HTML view of the notebook will be saved for each scheduled task that is run.

Using the Manage tasks window from the ArcGIS Notebook Server home page or the Tasks page in ArcGIS Notebook Server Manager, administrators can view details, edit, pause and resume, or delete each active notebook task in the site. Administrators and notebook authors with the schedule notebook privilege can view details, edit, pause and resume, or delete a notebook task in the details page of the notebook or in the task pane of the notebook editor.

If a previous run of a task is still running, a new scheduled task run will be skipped. For example, if a task is scheduled to run every 15 minutes, but an instance of that task runs for 20 minutes, the next scheduled run will be skipped. If this occurs regularly, the task owner should adjust the scheduled time interval so that there is no overlap between runs.

For more information on scheduled notebook tasks, see the Schedule a notebook task help topic.

Scheduled task limits

There are certain user, organizational, and ArcGIS Notebook Server site (machine) level limits related to scheduled notebook tasks.

Change maximum concurrent automated notebook runs (maxExecuteNotebookContainersPerNode)

When a notebook is run by a scheduled task, webhook, or the Execute Notebook API, ArcGIS Notebook Server automatically opens a new container and runs the notebook without user interaction. By default, ArcGIS Notebook Server is configured to run a maximum of 10 concurrent notebook runs for each Notebook Server machine. Considering the resources (CPU and RAM) available on each machine in the Notebook Server site, an administrator can adust this limit by modifying the maxExecuteNotebookContainersPerNode system property in Notebook Server. Any automated notebook request submitted after this limit is exceeded will be added to a queue and run once the number of automated task runs falls below this limit. Queued tasks will fail if the wait time exceeds the timeout time.

Note:

This does not limit the number of notebooks that are run interactively from the notebook editor.

You can change the maxExecuteNotebookContainersPerNode limit by following the steps below.

Note:

Increasing the limit can result in additional resources used on the Notebook Server machine.

1. Sign in to your ArcGIS Enterprise portal as an administrator.
2. Open ArcGIS Notebook Server Manager.
3. Click Settings > Site
4. Change the value for the maxExecuteNotebookContainersPerNode property.
5. Click Save.
6. Review the warning regarding restarting your Notebook Server and click Save and Restart to save the changes.

Maximum active scheduled notebook tasks per user

Each notebook author with the privilege to schedule notebooks can create up to a maximum of 20 active notebook tasks. Once this limit is reached, the user cannot create new scheduled tasks. A new task can be created once an existing task changes from Active to Complete, Failed, or Inactive. This limit can be changed by updating the ExecuteNotebooksUserLimit property using the Update System Limits operation in the ArcGIS Portal Administrator API.

Maximum active scheduled notebook tasks per organization

The maximum active scheduled notebook tasks for an organization is limited to 200. This limit represents the total number of active tasks that can be owned by all users across a Portal organization. Once this limit is reached, users cannot create new scheduled notebook tasks. This limit can be changed by updating the ExecuteNotebooksOrgLimit property using the Update System Limits operation in the ArcGIS Portal Administrator API.

Number of results reported for a scheduled task

The results of a task are reported and maintained for 30 runs. Any task runs prior to the most recent 30 runs of a task are permanently deleted. This limit can be changed by updating the TaskRunHistoryCount property using the Update System Limits operation in the ArcGIS Portal Administrator API.

Automatic failure of a task

Any tasks that fail five consecutive times will automatically be switched to a failed state and will no longer run. The task owner must ensure that the notebook can be run successfully without any user interaction before reactivating the task. To ensure that the tasks continue to run, the owner of the task must identify and rectify the failure and change the task to the Active state. This limit can be changed by updating the FailedRunsDisableTask property using the Update System Limits operation in the ArcGIS Portal Administrator API.

Automate a notebook using organization webhooks

A webhook is a mechanism that allows an application to provide other applications with event-driven information. As an ArcGIS Enterprise Portal administrator or a user with the Organization webhooks privilege, you can create, manage, and configure webhooks. You can configure webhooks to automatically run a notebook you own when events associated with your portal items, groups, or users occur. Notebook Server acts as the receiver of the webhook payload. Once a webhook is triggered, an HTTP request is made to Notebook Server to run a chosen notebook without any user interaction.

Note:

The number of concurrently running notebooks from webhooks is limited by the maxExecuteNotebookContainersPerNode system property in Notebook Server.

Create a webhook

Webhooks can be administered from your portal's Organization settings page. Use the following steps to create a webhook:

1. Sign in as a default administrator or custom role with the Organization webhooks privilege.
2. Browse to Organization > Settings > Webhooks.
3. Click Create webhook.
4. Specify a webhook name.

   For example, you could type Name=mynotebook_webhook.

5. Select the type of event that will trigger the webhook.

   By default, all event types (Items, Users, Groups, and Roles) are considered.

6. Use Trigger to either accept the default value (/) or specify one or more values as trigger events.

   An example is Events=/items, /items/<itemId>,/users.

   See Supported trigger events for more information.

   For additional examples, see Trigger events.

7. Select Notebook as the payload type.

   Note:

   If the Notebook option is not available, Notebook Server has not been configured with your portal. If the Notebook option is available but cannot be selected, you do not own any notebooks that are available for configuring webhooks. You must own the notebook item that will be used for the webhook.

8. Select the notebook that will run when the webhook is triggered.
9. Optionally, enter a Secret key that will be included in the webhook payload.

   You can also choose to use random, pregenerated values for the Secret key.

10. Click Create webhook to save the webhook settings and create a webhook.

Manage webhooks

Webhooks can be administered from your portal's Organization settings page. Use the following steps to manage existing webhooks:

1. Sign in as a default administrator or as a custom role with the Organization webhooks privilege.
2. Browse to Organization > Settings > Webhooks to see a list of webhooks created by you and other webhook administrators.

You can manage your existing webhooks with the following operations.

• Deactivate webhook—When a webhook is in an active state, click Active. You will be prompted to confirm the change in status. Click Confirm to deactivate the webhook. Deactivating the webhook stops payloads from being delivered when the webhook is triggered.
• Activate webhook—When a webhook is in an inactive state, click Inactive. You will be prompted to confirm the change in status. Click Confirm to activate the webhook. Once a webhook is activated, it will resume delivery of payloads.
• Edit webhook—Click the options button on the webhook and click Edit to update your webhook's trigger events or the notebook specified for the webhook.
• Delete webhook—Click the options button on the webhook and click Delete. Click Confirm to delete the webhook.

Advanced webhook settings

Use Advanced webhook settings to change properties such as the number of delivery attempts that can be made, notifications for elapsed time, and notification timeout properties for webhooks. These advanced settings will apply to all portal webhooks. To modify the Advanced webhook settings, complete the following steps:

1. Sign in as a default administrator or as a custom role with the Organization webhooks privilege.
2. Browse to Organization > Settings > Webhooks > Advanced webhook settings to modify the default values of the Advanced webhook properties.

Webhook payload

When a notebook is run from a webhook, the webhook's payload information is automatically injected into the notebook in an initial cell. Once injected, the information provided in the webhook payload can be used in the notebook for further automation workflows.

Webhook payload information included in a notebook:

{
  "properties": {
    "itemId": "<itemId of the current notebook>"
  },
  "events": [
    {
      "source": "items",
      "id": "<itemId of the item that resulted in webhook trigger>",
      "userId": "<webhook user>",
      "when": \<epoch time in ms when webhook was triggered\>,
      "operation": "<type of operation that resulted in webhook trigger>",
      "properties": {},
      "username": "<webhook username>"
    }
  ],
  "info": {
    "webhookId": "<webhookId>",
    "portalURL": "<public portal url>",
    "webhookName": "<webhook name>",
    "when": \<epoch time in ms when webhook was triggered\>,
  }
}

Automate a notebook using service webhooks

Introduced as a beta feature in ArcGIS Notebook Server 11.4, webhooks—including those for asynchronous geoprocessing services, feature services, and other generic webhooks—can trigger a notebook run in response to specific events. Administrators can register a webhook receiver URL as the hook or payload URL for these webhooks. When a specific event associated with a webhook is triggered, it will automatically call the Notebook Webhook Receiver URL, initiating the run of the corresponding notebook.

The following sections describe how to configure a notebook webhook receiver in ArcGIS Notebook Server.

Create an API Key Application

An API key is a long-lived token that authorizes a service webhook to run an associated notebook item and securely access the ArcGIS Enterprise capabilities and functionality required for the notebook to run.

API keys are generated using API key credentials, a type of item hosted in your portal. API key credentials contain settings that allow you to generate API keys and manage properties such as their privileges and expiration date. The process of creating API key credentials and using API keys in your application is known as API key authentication.

See the Create an API key tutorial for steps on how to generate an API key.

Note:

Before performing the final step of copying the cache, follow the steps in the sections below to apply any additional privileges necessary for running the notebook.

Apply privileges to your API key application

To apply additional privileges to your API key application, complete the following steps:

1. Sign in to the Portal directory using the administrative user credentials that were used to create the API key.

   The URL for the Portal directory is https://portal.domain.com/webadaptorname/sharing/rest.

   Once signed in, you will be directed to the User resource page for the signed in user.

   The User resource page URL is https://portal.domain.com/webadaptorname/sharing/rest/community/users/<username>

2. Click User Content.
3. Click the link of the item corresponding to your newly created API key.

   The item details API page for the API key opens: https://portal.domain.com/webadaptorname/sharing/rest/community/users/<username>/items/<itemId>

4. Click Update App
5. Append the following privileges in the Privileges section.
   1. Grant permission to run a notebook: "premium:publisher:createNotebooks"
   2. Optionally, allow the import and use of ArcPy in the associated notebook: "premium:publisher:createAdvancedNotebooks"
   3. Include any other necessary privileges for your notebook code to successfully run.

   The following API key example has the privilege to access the notebook item referenced by its itemId, privileges for standard and advanced notebooks, and privileges for publishing feature layers.

   [
     "portal:app:access:item:bfebec55911a4cd2a85ff9eae51dcf68",
     "premium:publisher:createAdvancedNotebooks",
     "premium:publisher:createNotebooks",
     "portal:user:createItem",
     "portal:publisher:publishFeatures"
   ]

6. Click Update App to save your changes.
7. Sign out of the Portal directory.

Generate and copy an API key

Follow the steps below to generate and copy an API key:

1. Sign in to your portal as an administrative user.
2. Click the Item Details page of the API key item.
3. Follow the steps in the Copy the API key section of the Create an API key tutorial.

Register a notebook webhook receiver

To register a notebook webhook receiver, complete the following steps:

1. Sign in to the ArcGIS Notebook Server Administrator directory using the portal administrative user credentials.

   The URL is https://notebookserverwebadaptor.domain.com/webadaptor/admin.

2. Click Notebooks > Webhook Receivers > Register Webhook Receiver.

   The URL is https://notebookserverwebadaptor.domain.com/webadaptor/admin/notebooks/webhookReceivers/create.

3. Register the webhook receiver using the information in the Create Webhook Receivers topic in the Administrative API.

   Upon the successful creation of a notebook webhook receiver, a webhook receiver URL will be returned in the response. This URL can be used as the hook URL when registering asynchronous geoprocessing service webhooks, feature service webhooks, and other generic webhooks.

Create an asynchronous geoprocessing service or feature service webhook

To create an asynchronous geoprocessing service or feature service webhook, complete the following steps:

1. Use the webhook receiver URL, secret headers, and signature headers to create an asynchronous service webhook.
2. Validate that the notebook associated with the notebook webhook receiver runs each time a specific service webhook event is triggered.

Execute Notebook API

Administrators and notebook authors can also automate a notebook to run without user interaction by using the Execute Notebook operation in the ArcGIS Notebook Server Administrator Directory. This operation will automatically run a notebook when called, but by using your own custom script, you can schedule it to run automatically to occur at a set time or on a regular schedule. A cron job or Windows scheduler can also be used to schedule the Execute Notebook operation to run at a set time or on a recurring interval.

To learn more, see the Execute Notebook operation topic in the Administrator Directory reference guide.