High availability is a technique to ensure system uptime and to minimize or prevent data loss in the event of a machine failure. ArcGIS Server, like other ArcGIS Enterprise components, can be deployed in a highly available configuration with a third-party network load balancer.
This configuration is a variation of the single-machine high-availability (active-passive) deployment, in which the load balancer is configured to spread the load across all sites at all times. In this configuration, there are no standby sites.
In this architecture, two or more ArcGIS Server sites are configured behind a third-party load balancer to increase the capacity of your ArcGIS Server deployment. You can use this technique to accommodate some of the high-availability limitations described in the single-machine deployment and single-machine deployment with reverse proxy server deployment scenarios, or to scale up by adding more machines.
While you can scale up and achieve high availability by using multiple-machine sites, there are advantages and limitations to active-active deployments, which are discussed below.
The active-active architecture works to clone a single-machine site and place independent instances of it behind a load balancer. Technically, this configuration cannot be described as a multiple-machine site, because each of the sites is independent of each other, composed of a single ArcGIS Server machine, and has its own local configuration store and server directories.
Multiple-machine ArcGIS Server site deployments greatly simplify server administration. However, the active-active architecture can be used in scenarios in which the number and settings of services are well defined and remain static. In these cases, such a configuration can provide significant performance advantages over multiple-machine sites, especially for cached map services.
This architecture can make it easy to replace stale or faulty machines, apply upgrades, or add and remove machines from the site as needed without interrupting services or aborting requests. However, with an active-active architecture, it is your responsibility to keep all sites in sync. This adds an administrative overhead that can make this deployment pattern impractical for cases in which you have many machines or services or caches that change frequently. You or your IT staff must also have a good understanding of third-party load balancers.
ArcGIS Server machines, server directories, and configuration store
Each ArcGIS Server machine must have its own local configuration store and cache, jobs, and system directories. This ensures maximum performance and minimizes interdependency. Conversely, the output directory (or directories) must be shared between each site. For additional details, see other considerations below.
If using file-based data sources in your GIS services, it is recommended that these be stored locally on every ArcGIS Server machine instead of network shares to maximize the performance of your services. In some cases, such as when handling large amounts of imagery, sharing your files over the network may be the only practical choice. If using files in shared resources over the network, it is very important that you choose a storage device configured for high availability.
If using databases, you can dedicate a database to each of your sites: one database for your primary site and a separate one for your standby site. To keep the databases in sync, you can leverage database replication, or if needed, geodatabase replication techniques. Alternatively, you can leverage other high-availability techniques from your database provider.
Reverse proxy server
In this configuration, a third-party load balancer is required. At a minimum, this component distributes the load across all the sites. Load balancers have different configurations for distributing the load, such as round robin and least connections. Selecting the correct load distribution depends on the web services you have running on the ArcGIS Server sites, and their patterns of use.
Load balancers also typically leverage different options for handling failures. For example, you may apply rules in your load balancer that prevent it from forwarding requests to a machine that is unavailable due to a hardware or network failure, or to a particular ArcGIS Server service that is not available. If using single-machine sites as in this pattern, requests sent to a particular machine are guaranteed to be managed by that machine.
The use of ArcGIS Web Adaptor is optional and typically only necessary for this scenario if you want to take advantage of web-tier authentication. You can choose to configure it on the same machine as your ArcGIS Server, or on a dedicated machine. In either case, if using ArcGIS Web Adaptor, you must configure a separate ArcGIS Web Adaptor for both sites in the active-active configuration.
Typically, the load balancer itself also fulfills the role of the reverse proxy server. In some scenarios, you may have already configured a reverse proxy server independently of the load balancer.
If your network load balancer supports a health check function, you can use the ArcGIS Server health check endpoint to determine if the site is available to receive requests. This is useful to quickly determine if there's a software or hardware failure in the site. For more information, see Health Check in the ArcGIS REST API.
Considerations for active-active configurations
The following should be kept in mind when planning an ArcGIS Server site with a highly available active-active configuration.
Unlike a true multiple-machine site, this configuration requires that all of the sites behind the load balancer host exactly the same content and that they follow the same security model. You are responsible for ensuring that all the sites look exactly the same to the load balancer.
Here are a few techniques that can help you keep ArcGIS Server services in sync across the primary and failover sites:
- Scripting—ArcGIS Server includes a REST API to script administrative tasks, such as publishing services and changing their
security settings. You can create your own scripts to apply changes
to all the ArcGIS Server machines involved in your deployment consistently.
Scripting is especially useful when you need to make small
adjustments, such as changing the security of a service or
Each site can be created through scripting. After all sites are created, create a backup from one of the sites and restore the backup to each other site. This ensures that all machines use the same encryption key.
- Virtualization—Create a virtual machine template and use it to launch new sites. As mentioned above, this ensures all machines use the same encryption key. In addition, each template will have a copy of the data needed for GIS services (unless a database is used) and all published and configured services. If changes are necessary, such as adding or updating existing services, create a template to launch subsequent virtual machines that would replace the existing pool of ArcGIS Server machines in use under the load balancer. You can also use virtual machine templates to recover stale ArcGIS Server machines.
The recommended procedure to apply changes to your sites in this deployment pattern is as follows:
- Make administrative changes against a site that is in standby mode first. For example, add a new service and change the security of another in a site that is not actively handling requests. This ensures there is no impact to applications using your primary site.
- Manually configure your load balancer to hand over all requests to the standby site on which the changes have been made.
- Apply the same changes to the idle site.
- Revert the load balancer so requests are directed back to the original primary site and leave the standby site in standby mode.
Changes to your site in the procedure described above can be applied manually through ArcGIS Server Manager, scripts, or virtual images.
Output directory sharing
Some ArcGIS Server service operations reference resources in one or more output directories. For example, map services can write images to an output directory and reference these images through a URL in the request response. For clients to successfully obtain the image, all sites in your active-active configuration must reference the same output directory. This can be accomplished by placing the output directories on a network resource and sharing them with your sites.
The following is a list of service operations that use output directories:
- Creating a feature service replica (feature service)
- Downloading a raster image (image service)
- Exporting a map image (map service)
- Exporting a schematic diagram (schematics capability in a map service)
- Exporting a web map (geoprocessing service)
Asynchronous execution of geoprocessing services
ArcGIS Server geoprocessing services support two execution modes: synchronous and asynchronous. Synchronous execution follows a stateless request-response pattern and is completely supported in an active-active configuration. Asynchronous execution follows a stateful request-response pattern and is the default. To use asynchronous execution in an active-active configuration, you'll need to consider the following:
- When submitting an asynchronous geoprocessing job, you're returned a job ID that refers to the submitted job and its outputs. Only the ArcGIS Server site that receives the original can recognize this ID. For this reason, the active-active configuration requires you to define affinity in your load balancer (also known as sticky sessions) if you want to use asynchronous execution. This helps provide high availability for asynchronous geoprocessing and map service outputs. Refer to your load balancer vendor to understand the implications of enabling sticky sessions.
- If your geoprocessing service does not use map services to render outputs and no outputs of type File have been defined, you can select synchronous execution for your geoprocessing services. No sticky sessions in your load balancer are required.
Using token-based security
If using token-based authentication, also referred to as server-tier authentication, it is important that all sites in this configuration use exactly the same shared token key. Otherwise, tokens generated for one machine will not be valid when used against the other machine. To duplicate the shared token keys across multiple sites, you can edit token settings in Manager.
To ensure the least amount of downtime, you can upgrade your ArcGIS Server sites in sequence. When upgrading a site, you can configure your load balancer to prevent it from forwarding requests to the unavailable site and distribute the load across other independent sites in your active-active architecture.
The other approach, upgrading all ArcGIS Server sites in parallel, is practical if your organization can tolerate some amount of downtime and data loss. Since all sites in an active-active architecture are independent of each other, you can upgrade them simultaneously with no interdependency issues.
Since there are distinct configuration stores for each site, single-machine active-active ArcGIS Server deployments cannot be federated with Portal for ArcGIS. To federate a highly available server site with your portal, configure a multiple-machine site.