Extend a notebook runtime

ArcGIS Notebook Server includes two notebook runtimes (collections of Python modules). These runtimes are available to notebook users through container images, which are applied to each container that runs in a server site.

Both the Standard and Advanced notebook runtimes include ArcGIS API for Python and various Python modules that can be used for data science, spatial analysis, content management, and administration. The Advanced runtime also includes ArcPy.

See the list of available Python libraries

In some cases, notebook authors may need modules or specific versions of modules that are not included in these two runtimes. You can access additional Python modules in ArcGIS Notebook Server in the following ways:

• Install a Python module during a notebook session. This makes the module available for use in that notebook.
• Extend an Esri container image to include additional Python libraries in a custom runtime. The container image you create will make the extended runtime available to all notebook authors in your site.

Install modules during a session

Notebook authors can make Python modules, or versions of modules, that are not included in either runtime available for use during a notebook session using the conda package management system. It is recommended that you use conda, because ArcGIS Notebook Server uses it to manage modules.

This approach installs the desired module for the duration of the notebook session. Once the notebook is closed or the container belonging to the user is terminated, the module is no longer available. To install modules using this method, you must have internet connectivity to the conda channel.

Nota:

conda is included in the two notebook runtimes. You do not need to install it.

To install the scrapy package during a session, for example, run a notebook cell with one of the following commands, depending on the package management system you're using:

Using the conda package management system

!conda install --yes scrapy

To use a different version of a Python module in the notebook runtime you're working in, specify the version number in the command.

You must restart the kernel after running a conda command for your new package to be available.

For example, a certain package is available in both notebook runtimes at version 1.5, but you want to use a tool introduced in 1.6. To make version 1.6 of the package available during a notebook session, run a notebook cell with one of the following commands, depending on the package management system you're using:

Using the conda package management system

!conda install --yes <package name>==1.6

Build a custom container image

To make persistent changes to a notebook runtime, you can build a custom container image in a site. The following workflow, which requires administrative privileges, creates a custom image and configures ArcGIS Notebook Server to use it.

Use either of the two notebook runtimes (Standard or Advanced) as a starting point. Once you have built your custom runtime, you can register it as a new runtime in your site.

1. Install and configure ArcGIS Notebook Server.
2. Sign in to the ArcGIS Notebook Server Administrator Directory as an administrator.

   The URL is formatted as https://notebookserver.domain.com:11443/arcgis/admin/.

3. Click notebooks > runtimes and select the notebook runtime you want to extend for your custom image.

   On the runtime's resource page, note the imageId value for this runtime. You will include this value in the next step.

4. Create a Dockerfile that adds your desired Python modules to the custom image.

   A Dockerfile is a simple text file of instructions to build a Docker container image. Follow the Dockerfile documentation for information and best practices.

   In this example, you'll add the data visualization Python package graphviz to the Advanced notebook runtime. The following code block in your Dockerfile instructs Docker to access the Advanced runtime using its repository name and tag and to install graphviz in the runtime. You can obtain the repository name and tag using the command docker images on an ArcGIS Notebook Server machine where this runtime is installed. The conda clean command is recommended to reduce file size.

   Use the docker images command to get the image name (repository) and tag for an existing notebook runtime.

   
   docker images
   
   REPOSITORY                        TAG                 IMAGE ID            CREATED             SIZE
   arcgis-notebook-python-advanced   10.8.1.14362        e46a81f69ea0        12 months ago       12.4GB
   arcgis-notebook-python-standard   10.8.1.14362        106a660a4d0b        12 months ago       7.51GB

   Create a custom runtime by adding the data visualization Python package graphviz to the Advanced notebook runtime.

   # Specify the existing notebook runtime imageId as FROM
   FROM arcgis-notebook-python-advanced:10.8.1.14362
   
   # Use RUN to issue a command to install, then clean
   RUN conda install graphviz \
     && conda clean -y -a

5. Save your Dockerfile on the machine in a new folder that only contains the Dockerfile.
6. Build a custom container image using a Docker command.

   This image will contain a new runtime as specified in your Dockerfile. Open a terminal and run a command with the following syntax:

   docker build -t arcgis-notebook-python-advanced-custom:10.8.1.14362 -f /data/DockerfileDir/Dockerfile .

   Nota:

   Be sure to include the period at the end of the command.

   When the command runs, Docker builds the new container image. The last line in the command output will be Successfully built <imageId> with the abridged ID value of the new image. You must provide your ArcGIS Notebook Server site with its full imageId value.

7. Run the following command in your terminal to get the image ID value:

   docker inspect <imageId>

   If you have a single-machine Notebook Server site, you can skip steps 8 through 10. If you have a multimachine Notebook Server site, complete steps 8 through 10.

8. Save the custom Docker image as a *.tar file using docker save.

   Open a command window on the Notebook Server machine where the custom runtime image is installed and run the following command to save the custom runtime image as a *.tar file.

    docker save arcgis-notebook-python-advanced-custom:10.8.1.14362 > arcgis-notebook-python-advanced-custom.tar

9. On each machine where Notebook Server is installed, load the custom runtime image using the ArcGIS Notebook Server post installation utility.
   1. Open a command prompt as an administrator.
   2. Change directories to the ArcGIS Notebook Server post installation utility folder.

      The default path is ${HOME}/arcgis/notebookserver/tools/postInstallUtility.

   3. Install your custom Docker image by specifying its file path.

       ./PostInstallUtility.sh -l <path of .tar Custom Docker image>

10. Repeat step 9 on each machine participating in your ArcGIS Notebook Server site.

Your custom image is now built and you have its full imageId value, which you will use to register it with the ArcGIS Notebook Server site in the next section.

Register a new runtime in Manager

Starting at 10.8, you can use ArcGIS Notebook Server Manager to register a custom runtime you've created.

1. Sign in to your ArcGIS Enterprise portal as an administrator and open ArcGIS Notebook Server Manager.
2. Open the Settings page and click Runtimes.
3. Click Register runtime to open the registration dialog box.
4. Specify the appropriate values to complete the registration.

   Parameter	Details
   Name	A string to represent the new runtime, such as "Notebook Custom Runtime Python 3".
   Container type	The only valid value is docker.
   Max CPU	The maximum number of CPU units to be used for each container.
   Advanced privileges required	Switch on the toggle button if the runtime requires the Advanced notebooks privilege.
   Image ID	Full (nontruncated) ID for the new Docker image when it was built.
   Docker runtime	Only applies to GPU runtimes in Linux environments. Learn more about ArcGIS Notebook Server and GPUs
   Max memory	The maximum amount of memory to be used for each container.
   Max memory unit	The unit specified by the maxMemory parameter.
   Max swap memory	The maximum amount of swap memory available for each container if it reaches the maxMemory limit.
   Max swap memory unit	The unit specified by the maxSwapMemory parameter.
   Shared memory	The amount of shared memory available for each container.
   Shared memory unit	The unit specified by the sharedMemory parameter.
   Library manifest	A *.json file (in UTF-8 or ANSI encoding) containing the list of Python libraries in the runtime.

   Nota:

   To create a manifest.json file, open a terminal on the ArcGIS Notebook Server machine where the custom notebook runtime image is installed and run the following command:

   ID=$(docker container run -d --rm -it -v /:/host <custom_runtime_image_ID> /bin/bash) && docker exec -it $ID /opt/conda/bin/conda list --json >> ~/manifest.json && docker kill $ID

5. When you're ready, click Save to register the new runtime.
6. Open a new notebook to test this update. Verify that the modules or versions of modules you added to your custom image are available in the notebook.

Create a custom runtime image with the ArcGIS Deep Learning backbones package for disconnected environments

ArcGIS API for Python supports the ability to train multiple deep learning models that each support their own base-models (backbones). The arcgis_dl_backbones conda metapackage provides support to offline users by automatically placing the backbones at the necessary locations, eliminating the need to download backbones from the internet when training deep learning models using ArcGIS. Follow the steps below to create a custom runtime image that includes the arcgis_dl_backbones conda metapackage.

1. Create a folder on your ArcGIS Notebook Server machine.

   An example location is /data/custom

2. Download the arcgis_dl_backbones package.

   See Additional Installation for Disconnected Environment for more information.

3. Extract the package arcgis_dl_backbones-2.0.1-py_0.tar.bz2 to the folder you created.
4. Create a Docker file with the following content in the folder you created:
   FROM arcgis-notebook-python-standard:11.3.0.51575
   USER root
   RUN mkdir -p /home/user/DUMP/arcgis_dl_backbones-2.0.1
   USER $NB_UID
   WORKDIR /home/user/DUMP/arcgis_dl_backbones-2.0.1
   COPY arcgis_dl_backbones-2.0.1-py_0.tar.bz2 /home/user/DUMP/arcgis_dl_backbones-2.0.1
   RUN conda install --offline arcgis_dl_backbones-2.0.1-py_0.tar.bz2 -y
   RUN conda clean -y -a
   WORKDIR $HOME

5. Go to the folder you created and build a custom container image using the following Docker command:
   docker build -t arcgis-notebook-python-standard-custom:11.3.0.51575 -f /data/custom/Dockerfile .

6. Follow the steps in the Register a new runtime in Manager section above to install this custom runtime image on your Notebook Server machines.

Update a custom runtime after upgrading ArcGIS Notebook Server

Follow the steps below to update a custom runtime after you have upgraded ArcGIS Notebook Server.

1. Build a custom runtime image using the new version of the notebook runtime image.
2. Use the notebook server post installation utility to install the new runtime image on each notebook server machine.
3. Register the new runtime using the ArcGIS Notebook Server manager.
4. On the items details page of each notebook using the custom runtime, click Settings.
5. Verify and update the runtime to the newly registered custom runtime for each machine using the custom runtime.
6. Once you have confirmed that the new runtime image works, deprecate the previous version of the custom runtime by unregistering the runtime and removing the runtime image from each notebook server machine.

To return the runtimes in your ArcGIS Notebook Server site to the original settings, use the Restore Factory Runtimes operation in the Administrator Directory. This is available at https://notebookserver.domain.com:11443/arcgis/admin/notebooks/runtimes/restore and will only restore the factory runtimes for your site's current version.