Skip To Content

Deploy ArcGIS Data Store in cluster mode using CloudFormation

The Esri arcgis-datastore-cluster.template.json Amazon Web Services (AWS) CloudFormation template creates an ArcGIS Data Store tile cache or spatiotemporal big data store on multiple Amazon Elastic Compute Cloud (EC2) instances. Using a cluster of instances allows you to scale the data store to meet storage demands.

This template configures the ArcGIS Data Store instances in an AWS Auto Scaling group, which offers advantages such as capacity planning based on certain criteria, better fault tolerance, better availability, and better cost management.

When you create a spatiotemporal big data store cluster with this template, all machines are assigned the data and coordinator roles. To deploy a cluster that contains separate machines for the data and coordinator roles instead, see Deploy an ArcGIS Data Store spatiotemporal big data store cluster with separate roles using CloudFormation.

Tip:
Use this template as part of creating an ArcGIS Enterprise deployment as described in Deploy ArcGIS Enterprise components on separate machines using CloudFormation.

You have the following backup storage options when using this template:

  • Store the backup files in Amazon Simple Storage Service (S3).
  • Store the backup files on a separate file server (EC2 instance).

This template creates the following architecture in Amazon Web Services:

Tile cache or spatiotemporal big data store with machines in an Auto Scaling group

In the diagram above, backups are shown to be in an S3 bucket. Automatic backups can be created in an S3 bucket or an EC2 file server.

License:

Certain icons in the diagram are used with permission from Amazon Web Services.

Prerequisites

Prerequisites can be grouped by the items—such as files and accounts—that you must obtain and the tasks you must perform before running the CloudFormation template.

Required items

You need the following before running this template:

  • An Amazon Web Services account.

    The account must have access to basic AWS services such as CloudFormation, Amazon Elastic Compute Cloud (EC2), Amazon Virtual Private Cloud (VPC), Amazon Simple Storage Service (S3), Systems Manager, Amazon CloudWatch, Lambda, AWS Identity and Access Management (IAM), Amazon DynamoDB, Secrets Manager, AWS Certificate Manager, and Amazon Relational Database Service (RDS).

  • If you create a deployment on Ubuntu EC2 instances in AWS GovCloud, you need an AMI ID. If you want to use the base canonical Ubuntu AMI, follow the instructions in Esri Amazon Web Services CloudFormation templates to identify the ID.
  • An Amazon Virtual Private Cloud (VPC) and subnets.

    You can use one of the following CloudFormation templates to create a VPC: VPC with two public subnets or VPC with two public and private subnets with a NAT Gateway.

  • The arcgis-datastore-cluster.template.json CloudFormation template.

Required tasks

Complete the following tasks before running this template:

  • You must have an ArcGIS GIS Server site that will be used as a hosting server for an ArcGIS Enterprise portal. Be sure the site doesn't already have the same type of ArcGIS Data Store registered to it that you will create with this template.

    You can use the single machine ArcGIS Server site template or multiple machine ArcGIS Server site template to create the site.

  • Configure passwords in AWS Secrets Manager (optional but recommended).

    You can configure the passwords for accounts such as the site administrator username and the Windows arcgis user password in AWS Secrets Manager. This provides you with a secret Amazon Resource Name (ARN). Use the ARN in place of a password in the template parameters when you launch a stack. If you don't use AWS Secrets Manager for storing passwords, you must type passwords in plain text in the template parameter when launching the stack.

    Note:

    When creating a secret ARN in AWS Secrets Manager for a password to be used with Esri CloudFormation templates, you must use the Other types of secrets secret type and use the Plaintext option. For more information on creating an Amazon Resource Name for passwords, see AWS CloudFormation and ArcGIS.

Tip:

By default, CloudFormation deletes partially created resources if stack creation fails. This is helpful because it removes unusable deployments from your account, but it can make it difficult to troubleshoot. To retain the stack in its failed state, disable the Rollback on failure CloudFormation stack creation option before launching the stack. See Setting AWS CloudFormation options in the AWS help for more information.

Parameters

Refer to the following tables for descriptions of the parameters used in this CloudFormation template. Tables are grouped by parameter type.

Amazon EC2 Configuration

Parameter nameRequired or notParameter description

Platform Type

Required

Choose the operating system platform. Supported types are as follows:

  • Windows
  • Linux

For specific operating system versions, see Operating systems supported when using CloudFormation to ArcGIS deploy on AWS.

EC2 Instance AMI ID

Optional

You can leave this parameter value empty. If you do, CloudFormation templates will use the latest Amazon Machine Image (AMI) ID for Microsoft Windows Server 2019 or Ubuntu Server 22.04 LTS based on the type of platform you selected.

Note:

You cannot leave this parameter empty if you deploy in AWS GovCloud on a Linux platform. See Esri Amazon Web Services CloudFormation templates for instructions for finding the ID for the base Ubuntu AMI from Canonical. If you deploy on a supported Linux operating system other than Ubuntu, you can find the AMI ID using AWS Management Console.

To use your own custom AMI, type the AMI ID using one of the following:

  • Provide the AMI ID in the format 'ami-xxxxx'.
  • If you stored the AMI ID in the AWS SSM parameter store, provide the SSM parameter name in the format AWS'{{resolve:ssm:[a-zA-Z0-9_.-/]+:\d+}}'.

If you use a custom AMI, ensure that it meets the following requirements:

  • EC2 instance metadata service (with either version V2 or V1) must be activated.
  • The latest version of AWS Systems Manager (SSM) Agent must be installed.
  • If the custom AMI has a Windows operating system, AWS Tools for PowerShell must be installed.

EC2 Instance Keypair Name

Required

Choose an EC2 keypair name to allow remote access to EC2 instances.

Elastic IP Address Allocation ID

Optional

Provide the Allocation ID of an elastic IP address in this format, eipalloc-XXXXXXXX.

You need an elastic IP address to map the site domain name to the elastic IP address and access the EC2 instance via RDP/SSH from outside of the AWS environment.

Leave this parameter empty if you don't want to assign an elastic IP address to the EC2 instance.

Amazon VPC Configuration

Parameter nameRequired or notParameter description

VPC ID

Required

Choose a VPC ID.

Note:

This must be the same VPC as the ArcGIS GIS Server site to which you're adding this data store.

Subnet ID 1

Required

Choose a subnet ID. The subnet ID that you select must be within the VPC you have selected above.

If you used an Esri CloudFormation template to create the VPC, you can get the subnet ID from that template's output parameters.

Subnet ID 2

Required

Choose a second subnet ID. This must be a different subnet ID than you used for the Subnet ID 1 parameter. The subnet ID that you select must be within the VPC you specified for this deployment.

If you used an Esri CloudFormation template to create the VPC, you can get the subnet ID from that template's output parameters.

ArcGIS Server Configuration

Parameter nameRequired or notParameter description

Site Administrator User Name

Required

Provide a username for the ArcGIS Server primary site administrator of the ArcGIS GIS Server site that will be used as the portal's hosting server.

Site Administrator User Password

Required

Provide the password for the ArcGIS Server primary site administrator. You can either type a plain text password or the ARN of your secret ID from AWS Secrets Manager. For more information on creating an Amazon Resource Name for passwords, see AWS CloudFormation and ArcGIS.

The password or secret ID you provide must match that of the existing primary site administrator.

ArcGIS Server EC2 Instance ID

Required

Choose an EC2 instance ID where your ArcGIS Server site is configured.

Note:
If you are registering the database with a multimachine ArcGIS Server site, you can choose any EC2 instance ID from the ArcGIS Server site. Do not choose a file server EC2.

ArcGIS Data Store Configuration

Parameter nameRequired or notParameter description

Type

Required

Choose the ArcGIS Data Store type to deploy in cluster mode. Valid values are spatiotemporal or tileCache.

Tip:

To create a relational data store, use the single-machine data store or primary-standby data store template.

Number of EC2 instances

Required

Choose the number of EC2 instances to include in the data store cluster. The default is 3.

Valid values are 1, 3, or 5.

Note:

If you need a greater number of EC2 instances, use AWS Management Console to modify the AWS Auto Scaling group so it includes the necessary number of instances. You must have an odd number of instances in a spatiotemporal big data store, and it is recommended that you have an odd number of instances in a tile cache data store cluster as well.

EC2 Instance Type

Required

Choose the EC2 instance type to use for the ArcGIS Data Store instances. The default is m5.2xlarge.

Instances are configured in an AWS Auto Scaling group.

EC2 Instance Root Drive Disk Space

Required

Provide the size of the root drive in GB. The default is 200 GB. Minimum is 100 GB. Maximum is 4096 GB.

Windows arcgis user password

Conditional

Provide a password for the arcgis user. The arcgis user is a local Windows login used to run the ArcGIS software services; therefore, this password is only required if you deploy on Windows.

You can either enter a plain text password or the ARN of your secret ID from AWS Secrets Manager.

It's a best practice to manage your passwords in AWS Secrets Manager. For information on creating an Amazon Resource Name for passwords, see AWS CloudFormation and ArcGIS.

Backup Store Type

Required

Choose the ArcGIS Data Store backup store type. The default type is CloudStore.

  • FileSystemArcGIS Data Store backups are stored on an ArcGIS file server EC2 instance created by this stack.
  • CloudStoreArcGIS Data Store backups are stored in an Amazon S3 bucket created by this stack.
  • None—No automatic backups are created for this ArcGIS Data Store type in this deployment.

Backup Store EC2 Instance Type

Conditional

Choose an EC2 instance type for ArcGIS Data Store backup machine. The default instance type is m5.2xlarge.

Note:

This is required only if you specified FileSystem as the backup store type.

Backup Store EC2 Instance Root Drive Disk Space

Conditional

Provide the size of the root drive in GB. The default is 200 GB. Minimum is 100 GB. Maximum is 4096 GB.

Note:

This is required only if you specified FileSystem as the backup store type.

Outputs

When your stack is created successfully, you can see the following output parameters on the Outputs tab of the CloudFormation stack in AWS Management Console.

Output nameOutput description

DeploymentLogsURL

This is the URL for the Amazon CloudWatch logs where all deployment logs are stored. You can refer to these logs for troubleshooting purposes if your deployment fails.

StopStackFunctionName

This is the Stop Stack Lambda function URL. You can use this lambda function to stop all EC2 instances in the stack.

StartStackFunctionName

This is the Start Stack Lambda function URL. You can use this lambda function to start all EC2 instances in the stack that you previously stopped.

Considerations

The following are important points to consider after creating a CloudFormation stack containing ArcGIS deployments:

  • When you deploy ArcGIS Data Store on AWS using this template, EC2 instances are configured in an AWS Auto Scaling group. After the stack is created, you can add more machines (EC2 instances) to the ArcGIS Data Store cluster or remove machines by adjusting the size of the AWS Auto Scaling group. Use AWS Management Console to increase or decrease the capacity of the Auto Scaling group. See the AWS documentation for instructions on changing the capacity limits of an Auto Scaling group to add a machine to or remove a machine from an ArcGIS Data Store cluster.

    Caution:

    Only adjust the capacity by one machine at a time when you change the capacity of a spatiotemporal big data store cluster.

    To find the AWS Auto Scaling group created for a CloudFormation stack, review the Resources tab of the related CloudFormation stack. The logical ID of the resource is AutoScalingGroup.

    Although it is technically possible to change the AWS Auto Scaling group size to zero, keep at least one instance running in the Auto Scaling group; otherwise, the ArcGIS Data Store will not function.

  • Do not delete any AWS resource created by this CloudFormation template. If you want to know what AWS resources have been created by this template, refer to the Resources tab of this stack in the AWS Management Console. Each resource created by an Esri CloudFormation template also has metadata tags. However, some of the resources do not show tags in the AWS Management Console.
  • You can use the AWS Lambda functions that appear in your stack outputs (listed in the Outputs section of this page) to stop EC2 instances in this stack when they are not in use and start them again when required. These functions are useful to help you manage costs.

    When you run the Lambda function to stop the EC2 instances in the deployment, the function returns a message that the instances are stopped. However, if the instances participate in an Auto Scaling group, the function must also detach the instances from the group. This can take up to 10 minutes to complete. Therefore, wait at least 10 minutes before you run the Lambda start function to restart the deployment.

  • If you use AWS Secrets Manager for passwords, such as the site administrator user password or the Windows arcgis user password, and later (after you create the deployment) you change those passwords, be sure you update the appropriate AWS Secrets Manager ARN's with the updated passwords.

Troubleshooting

If you observe any failures when creating this CloudFormation stack, see Troubleshoot ArcGIS deployments on AWS.