Configure and deploy raster analytics

Raster analytics using ArcGIS Enterprise is a flexible raster processing, storage, and sharing system that employs distributed computing and storage technology based on ArcGIS Image Server. Use raster analytics to apply raster analysis tools and raster functions offered in ArcGIS, build your own custom functions and tools, or combine multiple tools and functions into raster processing chains to execute your custom algorithms on large collections of raster data. Source data and processed results are stored, published, and shared across your enterprise according to your needs and priorities.

This extensive capability can be further expanded by leveraging cloud computing capabilities and resources. The net result is that image processing and analysis jobs that used to take days or weeks can now be done in minutes or hours, and jobs that were too large or extensive can now be handled.

Introduction

Raster analytics is available with an ArcGIS Image Server deployment configured for raster analysis. Built-in tools and functions provide preprocessing, orthorectification and mosaicking, remote sensing analysis, and an extensive range of math and trigonometry operators. Your custom functions can extend the platform's analytical capabilities even further.

Use your existing ArcGIS Image Server on-site or configure your deployment in a cloud environment such as ArcGIS Enterprise on Amazon Web Services or ArcGIS Enterprise on Microsoft Azure. The scalable environment of raster analytics allows you to implement computationally intensive image processing that was previously unavailable or cost prohibitive.

Raster analytics is also designed to streamline and simplify collaboration and sharing. Users across your enterprise can contribute data, processing models, and expertise to your imagery project, and share results with individuals, departments, and organizations in your enterprise.

Configure a raster analytics deployment

Raster analytics is composed of three main components: ArcGIS Enterprise; raster analysis, based on ArcGIS Image Server; and an image hosting site, including raster data store, based on ArcGIS Image Server.

Raster analysis requires an ArcGIS Enterprise base deployment, which provides software and user interfaces to organize and manage processing, storage, and sharing of raster and feature data, maps, and other geographic information on a variety of devices. This integrated system manages the dissemination of processing and storage of results on-premises and behind the firewall, in cloud processing and storage environments, or a combination of both environments.

The ideal deployment of raster analytics is composed of three server sites to perform the primary roles of the image hosting site, raster analysis site, and dynamic image site. Two licenses are required for raster analytics, ArcGIS Enterprise and ArcGIS Image Server.

The advantage of configuring raster analysis with three servers having distinct roles is to maximize efficiency and productivity. Separating distributed raster analysis processing and image service hosting roles enables resources to be dedicated to specific tasks without interruption. In this way, multiple users requesting access to various raster products and services do not compete for and impact raster analysis and image processing resources.

Set up ArcGIS Enterprise

ArcGIS Enterprise integrates the components of the raster analytics system to support scalable raster processing and storage workflows.

You can install an ArcGIS Enterprise base deployment across one or more machines, any of which can be physical, virtual, or cloud machines. For details about the required machine specifications for the ArcGIS Enterprise components, see the system requirements.

The process of installing and configuring an ArcGIS Enterprise base deployment can be done manually or automatically with the use of the ArcGIS Enterprise Builder.

Esri also provides templates to enable you to deploy ArcGIS Enterprise in the cloud. These templates set up ArcGIS Enterprise in a configuration of your choice. For a single-machine deployment, use the following template:

• Single-machine deployment using CloudFormation templates on Amazon Web Services.

For a multiple-machine deployment, use the following templates:

• Multiple-machine deployment on-premises—See the highly available system configuration template.
• Provision a highly available ArcGIS Enterprisedeployment on Amazon Web Services.

Configure ArcGIS Image Server for raster analysis

Distributed raster processing is performed by the ArcGIS Image Server assigned to the role of raster analysis. It executes the raster function processing chains on any supported raster data types. Source imagery and raster data can come from the distributed raster data store, a local client such as ArcGIS Pro, or other servers not configured in your raster analysis deployment. Ideally, your source data is staged on the distributed raster data store for optimum performance and efficiency.

Members of your portal will not interact directly with ArcGIS Image Server; they will use ArcGIS Pro, Map Viewer Classic, or clients built from APIs such as ArcGIS REST API, ArcGIS API for Python, or ArcGIS Maps SDK for JavaScript to perform raster analysis. ArcGIS Image Server will perform the actual processing, storage, and publishing roles.

You need to set up and federate an ArcGIS Image Server in the role of a raster analysis server to perform distributed raster analysis and processing workflows. The raster analysis server automatically manages and distributes image processing and raster analysis tasks at the block level or the scene level depending on the tools and functions used. The raster analysis server manages the distribution of processing results to either ArcGIS Data Store on the hosting server for feature data products, or to the raster data store for imagery and raster data products.

The steps to set up and deploy an on-premises ArcGIS Image Server for raster analysis are outlined below.

Set up an ArcGIS Image Server raster analysis site

The following instructions may require changes to the way you've deployed ArcGIS in your organization; review them carefully before proceeding.

1. Ensure that you have configured a base ArcGIS Enterprise deployment.
2. Install an additional ArcGIS Server site to function as the ArcGIS Image Server site.
3. Authorize ArcGIS Server with an ArcGIS Image Server license.
4. Register the raster data store with ArcGIS Image Server. Raster analysis services store processed rasters in a data store. This can be either a file share or cloud store type of data store.

   Note:

   If an image hosting server is specified in your raster analytics deployment, and you want to publish an image service, the same raster stores need to be registered in both your image hosting server and raster analysis server.

   If the input data is accessed from your data stores, those data stores should be registered in both the raster analysis and image hosting servers.

   1. To use a folder, use ArcGIS Server Manager, register a UNC share in the data store, and give it a name.

      Note:

      If there are multiple instances of _raster_store defined, for example, raster_store1, raster_store2, and so on, raster analysis will randomly pick one for a task request.

   2. To use a cloud store, provide the access key. The cloud store settings include subfolder support.

      

      Note:

      To use a cloud store as a raster store, the ArcGIS Server account user must have read, write, and delete permission on the cloud bucket to properly manage the data.

Federate and configure the server for raster analysis

Next, you will designate the ArcGIS Image Server site to perform raster analysis.

1. Sign in to the portal as an administrator. The URL is in the format https://webadaptorhost.domain.com/<webadaptorname>/home.
2. Browse to Organization > Settings > Servers.
3. Federate your ArcGIS Image Server site with your portal.
4. On the Servers page, locate the ArcGIS Image Server site that you federated with the portal.
5. Open the Configure server role dialog by doing one of the following:
   1. Next to Server role, click the current server role or Configure server role if no role is currently assigned.
   2. Click the More options button and select Configure server role.
6. On the Configure server role dialog box, enable Raster Analysis Server.
7. Click Save.

Assign permissions

Next, you will assign the minimum permissions needed to perform raster analysis.

1. Browse to the Organization > Settings > Member roles page.
2. Next to the role that will be performing raster analysis, click the More options button and select Edit.
3. In the Edit role window, expand Content.
4. Enable the options for Create, update, and delete and Publish hosted feature layers.
5. Expand Content and Analysis.
6. Enable Standard Feature Analysis.
7. Enable Imagery Analysis.

Configure raster analysis services

It is recommended that you configure raster analysis services and increase the SOC maximum heap size.

ArcGIS Image Server raster analysis services have a default processing capacity configuration. You may need to customize this configuration based on your hardware configuration. Sign in to ArcGIS Server Manager, click Services, and open the System folder, where you can perform the following configurations:

• To increase the processing speed and capacity, increase the maximum instance of the RasterProcessing service.
• To increase availability of the analytical results, increase the maximum instance of the RasterRendering service.
• To increase the number of processing tasks that can be executed concurrently, increase the maximum instance of the RasterAnalysisTools service.

1. Sign in to the ArcGIS Server Administrator Directory using the URL format http://machine.domain.com:6443/arcgis/admin.
2. From the directory home page, click machines.
3. From the list of machines, select the first machine licensed with ArcGIS Image Server, whether it is used as an image hosting server or as a raster analytics server.
4. In the Server Machine Properties menu, check the value for SOC maximum heap size. If it is set to 64MB, click Edit in the Supported Operations list.

   The Edit Machine page appears.

5. Change the value for SOC maximum heap size from 64MB to 128MB.
6. Click Save Edits.

   The server restarts.

7. Repeat these steps for each machine in your deployment that is licensed with ArcGIS Image Server.

Note:

If you want to deploy your image hosting server in the cloud with ArcGIS Enterprise on Amazon Web Services, see ArcGIS Enterprise Cloud Builder CLI for AWS and AWS CloudFormation and ArcGIS. If you want to deploy your raster analysis server in the cloud with Microsoft Azure, see Deploy ArcGIS Enterprise on Microsoft Azure.

Configure a temporary directory for cloud raster format

Raster analysis tools can use a temporary directory to speed up reading and writing. The directory has two primary purposes:

• Caching cloud stored .crf datasets
  • Accommodates all raster analysis tools when reading and writing cloud-stored .crf files.
  • The disk space usage is controlled by a maximum of 30 bundles cached per opened .crf dataset. Using this number, the average size of a bundle (greater than 50 MB), and estimated number of opened .crf datasets, the temporary space required for each SOC machine can be determined.
• Storing temporary processing block data for the distributed processing global tools, including the Hydrology, Distance, and Generalization tools
  • The disk space usage is proportional to the size of the primary input. Depending on the tool, it can be 3 to 5 times the size of the input, distributed across multiple SOC machines.

A temporary folder can be a shared folder or a local folder on each SOC machine. If not specifically configured, the default temporary folder is located under your user’s profile, for example, C:\Users\username\AppData\Local\ESRI\LocalCaches\MapCacheV1.

There are two options to configure the temporary directory for the raster analytics image server.

1. A single temporary folder path, "localTempFolder", in the server administration system properties, for example, https://servername:6443/arcgis/admin/system/properties.

   It is also used by the server caching tools, for example, {"localTempFolder":"E:/Temp/RA"}.

2. A property defining a list of temporary folder paths (semicolon separated), “localTempFolders”, in the server administration system properties, for example, https://servername:6443/arcgis/admin/system/properties. Raster analysis tools randomly choose a temporary folder from the list. This allows the use of multiple local disks on SOC machines, for example, {"localTempFolders":"E:/Temp/RA;F:/Temp/RA"}, assuming all SOC machines have E: and F: drives.

The list of temporary folder paths, “localTempFolders”, takes higher priority over the single temporary folder path, "localTempFolder", if both are configured.

Configure a temporary location for raster analysis tools

The raster analysis geoprocessing tools, Summarize Raster Within and Convert Raster To Feature, can execute in parallel on multiple machines. For these tools to share partial results between those machines, specially named entries must be added to the raster data store for the raster analytics server cluster. A fast, local temporary directory on each machine must be identified, and a UNC share containing the string machinename_tmp must be defined for that directory. That share must be added as a file share to the raster data store.

For example, assume the raster analytics cluster has two machines named workerA and workerB. Create a temporary directory on each machine, on a fast local disk with lots of space, and define these UNC shares: \workerA_raster_store_workerA_tmp and \workerB_raster_store_workerB_tmp. The share names can have any name, so long as the text string machinename_tmp is present. When a SummarizeRasterWithin or ConverRasterToFeature job is executing, worker machines will use those locations to write their partial results in parallel. The results will then be read across the local network and collated as needed by the specific operation.

Set up the image hosting site

The image hosting site hosts the raster data store and manages the file share storage as well as the cloud storage, and distributes results according to portal member requests. The raster data store is not part of ArcGIS Data Store but rather a registered file share or cloud share store federated with ArcGIS Enterprise. The raster data store is used by the raster analysis services to store output imagery optimized for reading, writing, and storage in a distributed format on-premises or in the cloud.

The advantage of configuring raster analysis with three servers having distinct roles is to maximize efficiency and productivity. Separating distributed raster analysis processing and image service hosting roles enables resources to be dedicated to specific tasks without interruption. In this way, multiple users requesting access to various raster products and services do not compete for and impact raster analysis and image processing resources.

You need to set up and federate a second ArcGIS Image Server site to function as the image hosting server, which hosts all the distributed image services generated by the raster analysis server. It includes the raster data store configured with ArcGIS Image Server Manager, which manages distributed file share storage and cloud storage such as Amazon Simple Storage Service (S3) or Microsoft Azure Blob storage of image services. The image hosting server stores and returns distributed results requested by members in the Enterprise portal. The image hosting server requires an ArcGIS Image Server license.

The instructions to set up and configure the image hosting server are detailed below.

The following instructions may require changes to the way you've deployed ArcGIS in your organization; review them carefully before proceeding. To configure the portal to perform image hosting services, follow the steps below.

1. Install ArcGIS Server.
2. Authorize ArcGIS Server with an ArcGIS Image Server license.
3. Sign in to ArcGIS Server Manager and browse to Site > Data Stores. Register a new raster store.

   The raster store is a location for output from raster analysis and should not be used as an input data directory. File shares can be registered directly as a raster data store. For cloud-based stores, first register a cloud store. Then create a raster store to use this cloud store.

   Note:

   If multiple raster stores are defined, the raster analysis service will randomly pick one as the output location.

4. Ensure that you have increased the SOC maximum heap size to 128 MB for ArcGIS Image Server.

Note:

To deploy your image hosting server in the cloud with Amazon Web Services, see ArcGIS Enterprise Cloud Builder CLI for AWS. To deploy your raster analysis server in the cloud with Microsoft Azure, see Deploy ArcGIS Enterprise on Microsoft Azure.

Federate and configure the server for image hosting

Next, you will designate the ArcGIS Image Server site to host your image services created from imagery layers.

1. Sign in to the portal as a member with administrative privileges. The URL is in the format https://webadaptorhost.domain.com/<webadaptorname>/home.
2. Browse to Organization > Settings > Servers.
3. Federate your ArcGIS Server site.
4. On the Servers page, locate the ArcGIS Image Server site that you federated with the portal.
5. Open the Configure server role dialog by doing one of the following:
   1. Next to Server role, click the current server role or Configure server role if no role is currently assigned.
   2. Click the More options button and select Configure server role.
6. On the Configure server role dialog box, enable Image Hosting Server.
7. Click Save.

Minimum raster analysis system

If an organization has resources and limited conflicts in terms of competing raster analysis processing and image service distribution resources, the raster analysis server and image hosting server functionality can be combined and hosted on one server. In this implementation, the raster analysis and image processing operations are executed on an ArcGIS Image Server site configured as the raster analysis server, while the results are managed and published through the same ArcGIS Image Server site. The ArcGIS Image Server site does the work of processing analytics requests, storing the results, and returning results to members in the Portal for ArcGIS site.

A diagram of the minimum raster analysis system is shown below.

The raster analysis and output image hosting server requires an ArcGIS Image Server license.

The instructions to set up and configure the minimum raster analysis system are detailed below.

Setup a minimum raster analysis system

The following instructions may require changes to the way you've deployed ArcGIS in your organization; review them carefully before proceeding. To configure the portal to perform image hosting services, follow the steps below.

1. Install ArcGIS Server.
2. Authorize ArcGIS Server with an ArcGIS Image Server license.
3. Sign in to ArcGIS Server Manager and browse to Site > Data Stores. Register a new raster store.

   The raster store is a location for output from raster analysis and should not be used as an input data directory.

   Note:

   If multiple raster stores are defined, the raster analysis service will randomly pick one as the output location.

4. Federate your ArcGIS Server site with your portal.
5. On the Servers page, locate the ArcGIS Image Server site that you federated with the portal.
6. Open the Configure server role dialog by doing one of the following:
   1. Next to Server role, click the current server role or Configure server role if no role is currently assigned.
   2. Click the More options button and select Configure server role.
7. On the Configure server role dialog box, enable Raster Analysis Server.
8. Click Save.
9. Assign the minimum permissions needed to perform raster analysis.
10. Ensure that you have increased the SOC maximum heap size to 128 MB for ArcGIS Image Server.