The Esri arcgis-datastore-cluster.template.json Amazon Web Services (AWS) CloudFormation template creates an ArcGIS Data Store spatiotemporal big data store on multiple Amazon Elastic Compute Cloud (EC2) instances. Using a cluster of instances allows you to scale the data store to meet storage demands.
This template configures the ArcGIS Data Store instances in an AWS Auto Scaling group, which offers advantages such as capacity planning based on certain criteria, better fault tolerance, better availability, and better cost management.
When you create a spatiotemporal big data store cluster with this template, all machines are assigned the data and coordinator roles. To deploy a cluster that contains separate machines for the data and coordinator roles instead, see Deploy an ArcGIS Data Store spatiotemporal big data store cluster with separate roles using CloudFormation.
Tip:
Use this template as part of creating an ArcGIS Enterprise deployment as described in Deploy ArcGIS Enterprise components on separate machines using CloudFormation.You have the following backup storage options when using this template:
- Store the backup files in Amazon Simple Storage Service (S3).
- Store the backup files on a separate file server (EC2 instance).
The following architecture diagram provides an example of the resources that are created when you use this template to create a spatiotemporal big data store in AWS:
In the diagram above, backups are in an S3 bucket. Automatic backups can be created in an S3 bucket or an EC2 file server.
License:
Certain icons in the diagram are used with permission from Amazon Web Services.
Prerequisites
Prerequisites can be grouped by the items—such as files and accounts—that you must obtain and the tasks you must perform before running the CloudFormation template.
Required items
You need the following before running this template:
- An Amazon Web Services account.
The account must have access to basic AWS services such as CloudFormation, Amazon Elastic Compute Cloud (EC2), Amazon Virtual Private Cloud (VPC), Amazon Simple Storage Service (S3), Systems Manager, Amazon CloudWatch, Lambda, AWS Identity and Access Management (IAM), Amazon DynamoDB, Secrets Manager, AWS Certificate Manager, and Amazon Relational Database Service (RDS).
- If you create a deployment on Ubuntu EC2 instances in AWS GovCloud, you need an AMI ID. If you want to use the base canonical Ubuntu AMI, follow the instructions in Esri Amazon Web Services CloudFormation templates to identify the ID.
- An Amazon Virtual Private Cloud (VPC) and subnets.
You can use one of the following CloudFormation templates to create a VPC: VPC with two public subnets or VPC with two public and private subnets with a NAT Gateway.
- The arcgis-datastore-cluster.template.json CloudFormation template.
Required tasks
Complete the following tasks before running this template:
- You must have an ArcGIS GIS Server site that will be used as a hosting server for an ArcGIS Enterprise portal. Ensure that the site doesn't already have the same type of ArcGIS Data Store registered to it that you will create with this template.
You can use the single machine ArcGIS Server site template or multiple machine ArcGIS Server site template to create the site.
- Configure passwords in AWS Secrets Manager (optional but recommended).
You can configure the passwords for accounts such as the site administrator username and the Windows arcgis user password in AWS Secrets Manager. This provides you with a secret Amazon Resource Name (ARN). Use the ARN in place of a password in the template parameters when you launch a stack. If you don't use AWS Secrets Manager for storing passwords, you must type passwords in plain text in the template parameter when launching the stack.
Note:
When creating a secret ARN in AWS Secrets Manager for a password to be used with Esri CloudFormation templates, you must use the Other types of secrets secret type and use the Plaintext option. For more information about creating an Amazon Resource Name for passwords, see AWS CloudFormation and ArcGIS.
- If you intend to use a shared file server for multiple system directories across the deployment, run the template to create the EC2 instance before you launch this stack.
Tip:
By default, CloudFormation deletes partially created resources if stack creation fails. This is helpful because it removes unusable deployments from your account, but it can make it difficult to troubleshoot. To retain the stack in its failed state, disable the Rollback on failure CloudFormation stack creation option before launching the stack. See Setting AWS CloudFormation options in the AWS help for more information.
Parameters
Refer to the following tables for descriptions of the parameters used in this CloudFormation template. Tables are grouped by parameter type.
Amazon EC2 Configuration
| Parameter name | Required or not | Parameter description |
|---|---|---|
Platform Type | Required | Choose the operating system platform. Supported types are as follows:
For specific operating system versions, see Operating systems supported when using CloudFormation to ArcGIS deploy on AWS. |
EC2 Instance AMI ID | Optional | You can leave this parameter value empty. If you do, CloudFormation templates will use the latest Amazon Machine Image (AMI) ID for Microsoft Windows Server 2022 or Ubuntu Server 22.04 LTS based on the type of platform you selected. Note:You cannot leave this parameter empty if you deploy in AWS GovCloud on a Linux platform. See Esri Amazon Web Services CloudFormation templates for instructions for finding the ID for the base Ubuntu AMI from Canonical. If you deploy on a supported Linux operating system other than Ubuntu, you can find the AMI ID using AWS Management Console. To use your own custom AMI, type the AMI ID using one of the following:
If you use a custom AMI, ensure that it meets the following requirements:
|
EC2 Instance Keypair Name | Required | Choose an EC2 keypair name to allow remote access to EC2 instances. |
Elastic IP Address Allocation ID | Optional | Provide the Allocation ID of an elastic IP address in this format, eipalloc-XXXXXXXX. You need an elastic IP address to map the site domain name to the elastic IP address and access the EC2 instance via RDP/SSH from outside of the AWS environment. Leave this parameter empty if you don't want to assign an elastic IP address to the EC2 instance. |
Amazon VPC Configuration
| Parameter name | Required or not | Parameter description |
|---|---|---|
VPC ID | Required | Choose a VPC ID. Note:This must be the same VPC as the ArcGIS GIS Server site to which you're adding this data store. |
Subnet ID 1 | Required | Choose a subnet ID. The subnet ID that you select must be within the VPC you have selected above. If you used an Esri CloudFormation template to create the VPC, you can get the subnet ID from that template's output parameters. |
Subnet ID 2 | Required | Choose a second subnet ID. This must be a different subnet ID than you used for the Subnet ID 1 parameter. The subnet ID that you select must be within the VPC you specified for this deployment. If you used an Esri CloudFormation template to create the VPC, you can get the subnet ID from that template's output parameters. |
ArcGIS Server Configuration
| Parameter name | Required or not | Parameter description |
|---|---|---|
Site Administrator User Name | Required | Provide a username for the ArcGIS Server primary site administrator of the ArcGIS GIS Server site that will be used as the portal's hosting server. |
Site Administrator User Password | Required | Provide the password for the ArcGIS Server primary site administrator. You can either type a plain text password or the ARN of your secret ID from AWS Secrets Manager. For more information on creating an Amazon Resource Name for passwords, see AWS CloudFormation and ArcGIS. The password or secret ID you provide must match that of the existing primary site administrator. |
ArcGIS Server EC2 Instance ID | Required | Choose an EC2 instance ID where your ArcGIS Server site is configured. Note:If you are registering the database with a multimachine ArcGIS Server site, you can choose any EC2 instance ID from the ArcGIS Server site. Do not choose a file server EC2. |
ArcGIS Data Store Configuration
| Parameter name | Required or not | Parameter description |
|---|---|---|
Type | Required | This template creates an ArcGIS Data Store spatiotemporal big data store in cluster mode. A valid input value is spatiotemporal. Tip:To create a relational data store or graph store, use the single-machine data store or primary-standby data store template. |
Number of EC2 instances | Required | Choose the number of EC2 instances to include in the data store cluster. The default is 3. Valid values are 1, 3, or 5. Note:If you need a greater number of EC2 instances, use AWS Management Console to modify the AWS Auto Scaling group so it includes the necessary number of instances. You must have an odd number of instances in a spatiotemporal big data store. |
EC2 Instance Type | Required | Choose the EC2 instance type to use for the ArcGIS Data Store instances. The default is m5.2xlarge. Instances are configured in an AWS Auto Scaling group. |
EC2 Instance Root Drive Disk Space | Required | Provide the size of the root drive in GB. The default is 500 GB. Minimum is 200 GB. Maximum is 4096 GB. |
Windows arcgis user password | Conditional | Provide a password for the arcgis user. The arcgis user is a local Windows login used to run the ArcGIS software services; therefore, this password is only required if you deploy on Windows. All arcgis users in the same deployment—for example, all Windows machines (including file servers) in an ArcGIS Enterprise deployment or in a stand-alone ArcGIS Server site—must use the same password. You can either enter a plain text password or the ARN of your secret ID from AWS Secrets Manager. It's a best practice to manage your passwords in AWS Secrets Manager. For information on creating an Amazon Resource Name for passwords, see AWS CloudFormation and ArcGIS. |
Backup Store Type | Required | Choose the ArcGIS Data Store backup store type. The default type is CloudStore.
|
ArcGIS File Server Instance ID | Conditional | If you used the file share CloudFormation template to create a shared file server for this deployment, provide the ID of that EC2 instance. If an EC2 instance ID is provided, the ArcGIS Data Store backups will be stored on this instance. |
Backup Store Instance Type | Conditional | Choose an EC2 instance type for ArcGIS Data Store backup machine. The default instance type is m5.2xlarge. Note:This parameter is required if you choose FileSystem as the backup store type but do not provide an ID for the ArcGIS File Server Instance ID parameter. |
Backup Store Instance Root Drive Disk Space | Conditional | Provide the size of the root drive in GB. The default is 500 GB. Minimum is 200 GB. Maximum is 4096 GB. Note:This parameter is required if you choose FileSystem as the backup store type but do not provide an ID for the ArcGIS File Server Instance ID parameter. |
Outputs
When your stack is created successfully, you can see the following output parameters on the Outputs tab of the CloudFormation stack in AWS Management Console.
| Output name | Output description |
|---|---|
DeploymentLogsURL | This is the URL for the Amazon CloudWatch logs where all deployment logs are stored. You can refer to these logs for troubleshooting purposes if your deployment fails. |
StopStackFunctionName | This is the Stop Stack Lambda function URL. You can use this lambda function to stop all EC2 instances in the stack. |
StartStackFunctionName | This is the Start Stack Lambda function URL. You can use this lambda function to start all EC2 instances in the stack that you previously stopped. |
Considerations
Consider the following when creating a CloudFormation stack containing ArcGIS deployments:
- When you deploy ArcGIS Data Store on AWS using this template, EC2 instances are configured in an AWS Auto Scaling group. After the stack is created, you can add more machines (EC2 instances) to the ArcGIS Data Store cluster or remove machines by adjusting the size of the AWS Auto Scaling group. Use AWS Management Console to increase or decrease the capacity of the Auto Scaling group. See the AWS documentation for instructions on changing the capacity limits of an Auto Scaling group to add a machine to or remove a machine from an ArcGIS Data Store cluster.
Caution:
Only adjust the capacity by one machine at a time when you change the capacity of a spatiotemporal big data store cluster.
To find the AWS Auto Scaling group created for a CloudFormation stack, review the Resources tab of the related CloudFormation stack. The logical ID of the resource is AutoScalingGroup.
Although it is technically possible to change the AWS Auto Scaling group size to zero, keep at least one instance running in the Auto Scaling group; otherwise, ArcGIS Data Store will not function.
- Do not delete any AWS resource created by this CloudFormation template. To find out what AWS resources have been created by this template, refer to the Resources tab of the stack in the AWS Management Console. Each resource created by an Esri CloudFormation template also has metadata tags. However, some of the resources do not show tags in the AWS Management Console.
- You can use the AWS Lambda functions that appear in the stack outputs (listed in the Outputs section of this page) to stop EC2 instances in the stack when they are not in use and start them again when required. These functions are useful for managing costs.
When you run the Lambda function to stop the EC2 instances in the deployment, the function returns a message that the instances are stopped. However, if the instances participate in an Auto Scaling group, the function must also detach the instances from the group. This can take up to 10 minutes to complete, so wait at least 10 minutes before you run the Lambda start function to restart the deployment.
- If you use AWS Secrets Manager for passwords, such as the site administrator user password or the Windows arcgis user password, and later (after you create the deployment) you change those passwords, update the appropriate AWS Secrets Manager ARN's with the updated passwords.
Troubleshooting
If you observe any failures when creating this CloudFormation stack, see Troubleshoot ArcGIS deployments on AWS.