Manage ArcGIS Data Store backups

You need backups to recover data in the event of a disaster such as data corruption or data store failure. If you create backups of the data stores implemented through ArcGIS Data Store, and place the backup files in a safe location, you can set up a new ArcGIS Data Store, access the backup files, and restore the data if the data store crashes and cannot be restarted.

Read the ArcGIS Data Store backup considerations, then use the information in the remaining sections to configure and manage ArcGIS Data Store backups.

1. Change the default backup location for a relational data store, change the initial default location for a tile cache data store running in primary-standby mode, or define a default backup location for all other data store types.
2. Configure automatic backups.
3. Manually create backups as necessary.

ArcGIS Data Store backup considerations

Keep the following in mind when implementing a backup and recovery strategy for ArcGIS Data Store:

• Backups allow you to recover data if a disaster occurs, such as when your server fails or floodwaters destroy your server. If your backup is on the server that is destroyed in a flood, you cannot recover your data. Therefore, you need to store backup files on a different server than the data store and, if possible, in a different physical location than the data store.
• ArcGIS Data Store backup files only contain the relational, graph, object, tile cache, or spatiotemporal big data stores. Backup files do not maintain a backup of the GIS Server site, the ArcGIS Enterprise portal, or user-managed data stores you register with the GIS Server site. You must create backups of those components separately.

  If you use a relational or tile cache data store (or both), you could use the webgisdr tool installed with Portal for ArcGIS to create a backup. When you use the webgisdr tool, a backup is also created of the portal, hosting server, and federated servers. See ArcGIS Enterprise backups for information about using this tool. You'll still need to create separate backups of the user-managed data stores you register with the GIS Server site and the spatiotemporal big data store, object store, and graph store.

• ArcGIS Data Store backups help you recover data lost if the ArcGIS Data Store machine fails or data gets corrupted. They do not provide high availability. If you require that your hosted feature, spatiotemporal, video service, and scene layers and hosted knowledge graphs continue to be available even if a single ArcGIS Data Store machine fails, add a machine to each data store type to make them highly available. You can add multiple machines to tile cache (cluster) data stores, spatiotemporal big data stores, and object stores.
• Due to changes in the underlying storage mechanisms and in ArcGIS software, the data store backups you create with older ArcGIS Data Store versions cannot be used to restore data to newer ArcGIS Data Store versions. Therefore, always create a full backup of each of your data stores after you upgrade ArcGIS Data Store. The only exception to this is relational data stores; backup files from the previous release may work. To determine what backup files can be used to restore the data store, run the listbackups utility.
• When you deploy ArcGIS Enterprise on-premises but your data store backup files are in cloud storage, creating backups and restoring from those backups will take longer than when backup files are stored in local file shares.
• There is no automatic cleanup of graph store, object store, tile cache data store, or spatiotemporal big data store backup files. There is no automatic cleanup of any backups you create using the backupdatastore utility.
• When you create a backup of the object store, feature layer query caches are not included.

ArcGIS Data Store backup locations and behavior

All ArcGIS Data Store types require a default backup location. The backups automatically created by ArcGIS Data Store are always created in the default backup location.

You can define additional backup locations that you can specify for output when you run the backupdatastore utility to create full backups. This type of backup is also referred to as a manual backup. If you do not specify a location when you run the backupdatastore utility, the default backup location is used.

The following table summarizes the differences in backup locations for each type of data store. Read the information pertinent to the type of data store (or stores) you manage.

Data store	Default backup location	Locations for manual backups
Graph store	You must register a default backup location before you can create a backup of the graph store.	You can register additional file share backup locations for the graph store.
Object store	You must register a default backup location before you can create a backup of the object store.	You can register additional file share backup locations for the object store.
Relational data store	Default backups are written to a subfolder in the ArcGIS Data Store content directory until you change it. Change the default backup location to be a file share on a separate machine.	You can register additional backup locations for the relational data store, including other file shares, Amazon Simple Storage Service (S3) buckets, and Microsoft Azure Blob storage containers.
Tile cache data store in cluster mode	You must register a default backup location before you can create a backup of a tile cache data store running in cluster mode. The default backup location can be a file share location, an Amazon S3 bucket, or a Microsoft Azure Blob storage container.	You can register additional backup locations for the tile cache data store, including other file shares, Amazon S3 buckets, and Microsoft Azure Blob storage containers.
Tile cache data store in primary-standby mode	Default backups are written to a subfolder in the ArcGIS Data Store content directory until you change it. Change the initial default backup location to a file share on a separate machine or to an Amazon S3 bucket or a Microsoft Azure Blob storage container.	You can register additional backup locations for the tile cache data store, including other file shares, Amazon S3 buckets, and Microsoft Azure Blob storage containers.
Spatiotemporal big data store	You must register a default backup location before you can create spatiotemporal big data store backup files. The default backup location can be a file share location, an Amazon S3 bucket, or a Microsoft Azure Blob storage container.	You can register additional backup locations for the spatiotemporal big data store, including other file shares, Amazon S3 buckets, and Microsoft Azure Blob storage containers.

Change the default backup location for a relational data store

When you create a relational data store, a backup location is automatically configured on the same machine as the data store. By default, ArcGIS Data Store creates backups of relational data stores in /usr/arcgisdatastore/backup/relational. This is the default backup location.

However, it is not a good practice to keep backups on the same machine as the data store. If the primary data store machine fails, you can't access the backup files and, therefore, cannot restore your hosted layer data. Another reason you should not leave backup files and the data store on the same machine is that the backup files can rapidly fill the disk space on the machine. If you run low on disk space, the data store will be placed in read-only mode to avoid data corruption, and you cannot publish new hosted layers.

Therefore, define a different, secure location to store backups for the relational data store. To do this, use the change operation with the configurebackuplocation utility.

For relational data stores, you can designate a shared file directory on another machine to use for the default backup location.

Note:

• Ensure there is enough storage space to hold all the files that are included in a data store backup.
• The rate at which the default backup location fills up depends on your backup schedule and the number of days you retain backups. Monitor the size of the backup directory and adjust these settings and storage sizes as needed.
• All machines in the same data store must have access to the default backup location. For example, when you register a file share backup location for the relational data store, both the primary and standby machines in the relational data store must have write access to the file share location.

Follow these steps to change the default backup location for a relational data store:

1. Create a shared directory on another machine to store backup files.
2. Grant read and write access on the shared network directory to the account that installed ArcGIS Data Store.
3. Run the configurebackuplocation utility with the change option to specify the shared directory as the default output location for relational data store backups.

   If you run this utility for a relational data store to use a shared directory after users published hosted feature layers and an automatic backup already took place, the configurebackuplocation utility will move your existing relational data store backup files from the default backup location to the shared directory.

   In this example, the backup location for a relational data store is changed to a shared directory named ds_backups on a computer named sysshare. The backup location is named reldefbu.

   ./configurebackuplocation.sh --operation change --store relational 
   --location "type=fs;location=/net/sysshare/ds_backups;name=reldefbu"
   
   You are going to change the backup location of the data store. Existing backups will be copied to the new location and it could take a few moments.
   Please do not interrupt the process once it has started.
   
   Do you want to continue (Yes or No)? Yes

   For full syntax and additional examples, see ArcGIS Data Store command utility reference.

Change the default backup location for a tile cache data store (primary-standby mode)

When you create a tile cache data store deployed in primary-standby mode, a backup location is automatically configured on the same machine as the data store. The default location is /usr/arcgisdatastore/backup/tilecache. As with the relational data store, the default backup location should be changed to a remote location. Unlike the relational data store, though, you use the register and setdefault operations to establish a new default backup location for a tile cache data store run in primary-standby mode.

Follow these steps to change the default backup location for a tile cache data store that is running in primary-standby mode:

1. Create one of the following to store automatic data store backup files:
   • A shared directory on another machine
   • An Amazon S3 bucket under your Amazon Web Services account
   • A Microsoft Azure Blob storage container under your Azure Blob storage account
2. Grant read and write access to the location you configured in the previous step to the account that installed ArcGIS Data Store.
3. Run the configurebackuplocation utility with the register operation to add a shared directory or a cloud storage location for the tile cache data store.

   In this example, a backup location in an Azure Blob storage container named scenebackups is added to the tile cache data store. The backup location is named tc_defaultbu. Credentials to access the container are provided.

   ./configurebackuplocation.sh --operation register --store tileCache 
   --location "type=azure;location=scenebackups;name=tc_defaultbu;username=myazureaccountlogin;password=zpw4myazureaccount"

4. Now designate the newly registered backup location as the default backup location for the tile cache data store.

   Note:

   This utility will not move existing automatic backups for tile cache data stores that exist in the previous default backup location.

   In the following example, the backup location registered in the previous step (tc_defaultbu) is set as the default backup location.

   ./configurebackuplocation.sh --operation setdefault --store tileCache 
   --location "name=tc_defaultbu"

Tip:

To confirm the new location is now the default location, you can run the configurebackuplocation utility with the list operation.

For full syntax and additional examples, see ArcGIS Data Store command utility reference.

Register an initial default backup location

Upon creation, graph stores, object stores, tile cache data stores (cluster mode), and spatiotemporal big data stores do not have a default backup location. You must register at least one backup location before you can create backups of these types of data stores.

Register a secure, shared default backup location where ArcGIS Data Store can place files from scheduled (automatic) backups.

You can register a file share, Amazon S3 bucket, or Microsoft Azure Blob storage container as the default backup location.

Note:

• Ensure there is enough storage space to hold all the files that are included in a data store backup. Tile cache data stores and graph stores can be quite large and spatiotemporal big data stores tend to be even larger.
• The rate at which the default backup location fills up depends on your backup schedule and how many manual backups you write to the default backup location. Monitor the size of the backup directory and adjust the schedule and storage sizes as needed.
• All machines in the same data store must have access to the default backup location. For example, when you register a file share backup location for a spatiotemporal big data store, all machines in the spatiotemporal big data store must have write access to the file share location.

Follow these steps to configure a default location for data store backup files for a tile cache data store (cluster mode), spatiotemporal big data store, object store, or graph store:

1. Create one of the following to store data store backup files:
   • A shared directory on another machine
   • An Amazon S3 bucket under your Amazon Web Services account
   • A Microsoft Azure Blob storage container under your Azure Blob storage account
2. Grant read and write access to the location you configured in the previous step to the account that installed ArcGIS Data Store.
3. Run the configurebackuplocation utility with the register option to specify the default output location for data store backups.

   See the sections that follow these steps for examples for each type of data store.

   For full syntax and additional examples, see the ArcGIS Data Store utility reference.

Tip:

If your backup directory goes offline for more than a few minutes, perform a full manual backup of the data store as soon as a backup location is available.

Graph store example

In this example, a default backup location is registered for the graph store. The location is a file share directory named graphbu on sysshare. The backup location name is graph_defaultbu.

./configurebackuplocation.sh --operation register --store graph 
--location "type=fs;location=/net/sysshare/graphbu;name=graph_defaultbu"

Object store example

In this example, a default backup location is registered for the object store. The location is a file share directory named videobu on netshare. The backup location name is object_defaultbu.

./configurebackuplocation.sh --operation register --store object 
--location "type=fs;location=/net/netshare/videobu;name=object_defaultbu"

Spatiotemporal big data store example

In this example, a backup location is registered for a spatiotemporal big data store. The location is an Azure Blob storage location named dsbackups. The backup location is named sbds_defaultbu. Credentials to access the Blob storage location are provided.

./configurebackuplocation.sh --operation register --store spatiotemporal 
--location "type=azure;location=dsbackups;name=sbds_defaultbu;username=azureaccountlogin;password=zpw4azureaccount"

Tile cache data store (cluster mode) example

In this example, a default backup location is registered for a tile cache data store deployed in cluster mode. The location is an S3 bucket with a subfolder named scene. The bucket resides in the US East (Ohio) AWS region. The backup location name is tc_defaultbu. Credentials to access the bucket are provided.

./configurebackuplocation.sh --operation register --store graph 
--location "type=s3;location=backups/scene;name=tc_defaultbu;region=us-east-2;username=abc12345;password=dearliz@a0"

Register additional backup locations

You can register additional backup locations. When you run the backupdatastore utility to manually create backups, you can specify one of these predefined locations.

Follow these steps to add another location for backup files:

1. Create another location for backup files.
   • To register a shared directory (file share), create the directory on another machine. Ensure the storage space is large enough to hold all backup files. Also ensure that the login you use when you connect to the ArcGIS Data Store machine to run the backupdatastore utility has write access to this directory. If you create an additional shared directory for a tile cache (cluster) or spatiotemporal big data store, all machines in the same ArcGIS Data Store deployment must have access to this shared directory. If you create an additional shared directory for relational or tile cache (primary-standby) data store backups, the standby machine must have access to the shared directory.
   • To register an S3 bucket, create the bucket under your Amazon Web Services account. Choose a bucket size that can accommodate your backup files.
   • To register an Azure Blob storage container, create the container under your Azure Blob storage account.
2. Run the configurebackuplocation utility with the register operation to register this additional backup location.

   In this example, a second file share location is registered to store manual backups for the relational data store. The backup location is given the name rbu_manual.

   ./configurebackuplocation.sh --operation register --store relational
    --location "type=fs;location=/net/sysshar2e/ds_manual_backups;name=rbu_manual"

   In this example, an Azure Blob storage container is registered to store manual backups for the tile cache data store. The backup location is given the name tcbu_manual.

   ./configurebackuplocation.sh --operation register --store tileCache
    --location "type=azure;location=bucontainer;name=tcbu_manual;username=myazureaccountlogin;password=zpw4myazureaccount"

   The steps and examples for registering additional backup locations for tile cache (cluster) data store, graph store, object store, and spatiotemporal big data store are the same when adding backup locations as when registering an initial backup location.

Specify one of the backup locations as the default location

When you have multiple backup locations defined for a graph store, object store, tile cache data store, or spatiotemporal big data store, you can designate one of them as the default backup location. After you do that, backups created by ArcGIS Data Store (automatic backups) will write the backup files to that location.

Tip:

To change the default location for a relational data store, use the change operation.

Run the configurebackuplocation utility with the setdefault operation and specify the registered backup location. If you named the backup location when you registered it, you can use the name to designate the location.

Graph store example

In the following example, the graph store has multiple file share backup locations. The setdefault operation is used to designate one of them as the new default backup location.

./configurebackuplocation.sh --operation setdefault --store graph --location "location=/net/server/gsbackups24"

Object store example

In the following example, one of the backup locations configured with the object store—s3_backups—is set as the default backup location.

./configurebackuplocation.sh --operation setdefault --store object --location "name=s3_backups"

Spatiotemporal big data store example

In the following example, one of the spatiotemporal big data store's additional backup locations (one named bu_container) is designated as the new default backup location:

./configurebackuplocation.sh --operation setdefault --store spatiotemporal --location "name=bu_container"

Tile cache data store example

In the following example, one of the tile cache data store's additional backup locations in an S3 bucket in the Asia Pacific (Singapore) region is designated as the new default backup location:

./configurebackuplocation.sh --operation setdefault --store tileCache --location "type=s3;location=bu_bucket;
username=hijklmn1234567;password=z9y8x7w6v5u4t3s2r1q0;region=ap-northeast-1"

Determine all backup locations

It can become confusing to keep track of all the data store backup locations when you have multiple data store types and multiple locations for each. Or maybe you took over the job of ArcGIS Data Store administrator from someone else. In these cases, you can run the configurebackuplocation utility with the list operation to query ArcGIS Data Store for a list of all the backup locations for a specific data store type.

In the following example, all registered backup locations are returned for the spatiotemporal big data store in an ArcGIS Data Store deployment. To get a list for the other data store types, specify that type with the --store option, using relational, tileCache, graph, or object.

./configurebackuplocation.sh --operation list --store spatiotemporal

Manage automatic backups

By default, ArcGIS Data Store creates a full backup of relational data stores every four days, but you can change how often ArcGIS Data Store creates a full relational data store backup by running the updatebackupschedule utility. For all other data store types, there is no backup schedule until you set one using the updatebackupschedule utility.

Change backup frequency

If your organization members publish and edit large numbers of hosted layers, or you archive large volumes of streaming data, increase the frequency of backups.

Note:

By default, incremental backups are disabled for relational data stores. If you enable point-in-time recovery, incremental backups are created either when the log files are full or every five minutes, whichever comes first. The database controls incremental backup creation; you cannot control the frequency with which incremental backups are created.

Your backup location must have enough room to store all the backup files. Backup size varies depending on the amount of data you have, but if you use default backup settings, backups contain two full backups. The size of these files depends on the amount and size of your data. If you reenable point-in-time recovery for relational data stores, the backups also include seven days of incremental backup files by default.

If you decide to create backups manually and want to disable automatic backups, set the backup frequency to 0. If you disable automatic backups, you must create the backups yourself to guard against data loss in the event of machine failure or other data catastrophe.

Note:

When you disable automatic backups, you cannot use point-in-time recovery for the relational data store; all relational data store backups will be full backups.

The updatebackupschedule utility is installed in the <ArcGIS Data Store installation directory>/datastore/tools directory.

1. Open a command shell.
2. Run the updatebackupschedule utility to specify the backup frequency you require.

   The syntax to run the utility is as follows:

   updatebackupschedule [--store relational|tileCache|spatiotemporal|graph|object] 
   [--starttime <local server time>] --frequency <number of days>

   For example, type the following to schedule full relational data store backups at 3:00 a.m. (local server time) every day:

   ./updatebackupschedule.sh --store relational --starttime 03:00:00 --frequency 1

   In this example, a backup of the spatiotemporal big data store is scheduled for 11:30 p.m. (local server time) every three days:

   ./updatebackupschedule.sh --store spatiotemporal --starttime 23:30:00 --frequency 3

Change how long automatic relational data store backup files are retained

The backup directory retains relational data store backup files for seven days by default. This means if you keep the default backup frequency (every four days) and retention schedules (seven days), the backup directory contains two full backups. If you enable point-in-time recovery, the backup directory also contains seven days of incremental backup files. The size of these files depends on the amount and size of your data. The machine that stores your backups must have enough disk space for all these files. If you increase the backup frequency, consider decreasing the retention period for the backup files. In the previous section, the backup frequency was increased to every day. To prevent your backup directory from becoming too large, decrease the backup file retention period.

The syntax to run the updatebackupretaindays utility is as follows:

updatebackupretaindays <number of days>

In the following example, backup file retention time is changed to four days:

./updatebackupretaindays.sh 4

Manually create and delete backups

Even if you use automatic backups, there may be times when you want to create a backup for a specific purpose outside of the usual backup schedule, such as before upgrading the system or to create a secondary full backup in a different location.

If you disable automatic backups, it is recommended that you create manual backups on a regular basis.

You can use the deletebackup utility to delete backup files that you create for relational data stores and object stores.

Run a utility to create a data store backup

You can use the backupdatastore utility to make a full backup of the specified data store. You may want to manually create a full backup before making a large number of changes to the data store or before upgrading the data store. Or you may want to create a backup to preserve a copy of the data in a particular state; for example, at the end of the first phase of a project.

Running the backupdatastore utility for a relational data store always creates a full backup of the data store.

The first time you run the backupdatastore utility for a tile cache data store, backup copies are made of all existing tile cache data store databases. Similarly, the first time you run the backupdatastore utility for a spatiotemporal big data store, a full backup is created. Because both of these types of data stores can be very large, each time you run the backupdatastore utility after the first time, the utility only creates backup copies of data that was created since the last time you ran the utility.

The login you use to connect to the data store machine to run the backupdatastore utility must have read and write access to the data store backup location.

The syntax to run the backupdatastore utility is as follows:

backupdatastore [<backup file name>] [--store {relational | tileCache | spatiotemporal | graph | object}] [--location <backup_arguments>] [--prompt {yes | no}]

If you do not specify the --location option, backup files are written to the data store's default backup location.

Provide a meaningful backup name for the file so you can find it when you want to restore the data. If you do not specify a name, the utility assigns a default name to the file. The default name is in the format datastorename-timestamp. For example, if your data store is named corpds and you create the backup on July 10, 2014, at 14:25:49:554 UTC, the backup file name is corpds-20140710142549554.

You will be asked to confirm that you want to create a backup. Type yes or y to proceed with backup creation.

Tip:

If you want to script manual backups, include a flag to suppress the confirmation prompt as in the following example:

backupdatastore --store tileCache --prompt no

In this example, the data store generates the backup file name. This is necessary in a script to ensure a unique backup file name.

Delete manual data store backups

When you no longer need a relational data store, graph store, or object store backup file that you created using the backupdatastore utility, run the configurebackuplocaton utility with the --operation option set to list to get all the backup locations for a data store. Next, run the listbackups utility with each location to get the names of the backup files in each location. When you determine which backup files must be deleted, run the deletebackup utility for each location and file to remove the unneeded file. For example, after you upgrade your data store and confirm all layers are performing as expected, you can delete the data store backups you created before upgrading.

In this example, the relational data store backup file preupgrade1104_bu is deleted from backup location /net/systemserver/backups:

./configurebackuplocation.sh  --operation list --store relational 
Backups locations for relational data store:
=====================================================================================
Type  Location                                         isDefault 
=====================================================================================
fs    /net/ourserver/datastore/backups/rel                true
fs    /net/systemserver/backups                           false

./listbackups.sh --store relational --location "/net/systemserver/backups"

Backup_Name                      Status           Backup_Time         Mode
====================================================================================
phase1proj_bu                    BackupComplete   2023-03-08 14:12    manual
phase2proj_bu                    BackupComplete   2023-06-21 11:43    manual
preupgrade1104_bu                BackupComplete   2022-11-04 09:30    manual


./deletebackup.sh preupgrade1104_bu --store relational --location "/net/systemserver/backups"
You are attempting to delete backup 'preupgrade1104_bu'. This operation is irreversible.

Do you wish to continue (Yes or No)?yes

Operation completed successfully