Manage data store backups

You need backups to recover your data in the event of a disaster such as data corruption or data store failure. If you create backups of your data stores and place them in a safe location, you can set up a new ArcGIS Data Store, access your backup files, and restore the data if for any reason your data store crashes and cannot be restarted.

Read the ArcGIS Data Store backup considerations, then use the information in the remaining sections to configure and manage ArcGIS Data Store backups.

1. Define a backup location.
2. Configure automatic backups.
3. Manually create backups as necessary.

ArcGIS Data Store backup considerations

Keep the following in mind when implementing a backup and recovery strategy for your data stores:

• Backups allow you to recover your data if a disaster occurs, such as when your server fails or floodwaters destroy your server. If your backup is on the server that is destroyed in a flood, you cannot recover your data. Therefore, you need to store your backup files on a different server than your data store.
• ArcGIS Data Store backup files only contain the relational, tile cache, or spatiotemporal big data stores. Backup files do not maintain a backup of the GIS Server site, your portal, or data stores you register with the GIS Server site in ArcGIS Server Manager or an ArcGIS Desktop application. You must create backups of those components separately.

  If you use a relational or tile cache data store (or both), you could use the webgisdr tool installed with Portal for ArcGIS to create a backup. When you use the webgisdr tool, a backup is also created of your portal, hosting server, and federated servers. See ArcGIS Enterprise backups in the Portal for ArcGIS Administrator Guide for more information on using this tool. You'll still need to create separate backups of the data stores you register with the GIS Server site in ArcGIS Server Manager or an ArcGIS Desktop application.

• ArcGIS Data Store backups help you recover data lost if the ArcGIS Data Store machine fails or data gets corrupted. They do not provide high availability. If you require that your hosted feature, spatiotemporal, and scene layers continue to be available even if a single ArcGIS Data Store machine fails, add a standby machine to your relational and tile cache data stores to make them highly available. You can add multiple machines to your spatiotemporal big data store to make it highly available.
• Due to changes in the underlying storage mechanisms and in ArcGIS software, the data store backups you create with older ArcGIS Data Store versions cannot be used to restore data to newer ArcGIS Data Store versions. Therefore, you should create a full backup of each of your data stores after you upgrade ArcGIS Data Store.

Define a backup location

Backup locations and behavior differ among the data store types. Read the information pertinent to the type of data store (or stores) you manage.

• Relational data stores

  When you create a relational data store, a backup location is automatically configured on the same machine as the data store. By default, ArcGIS Data Store creates backups of relational data stores in /usr/arcgisdatastore/backups/relational. If the primary data store machine fails, you can't access the backup files and, therefore, cannot restore your hosted feature layer data.

  Another reason you should not leave backup files on the same machine as the data store is that these files can rapidly fill the disk space on the machine. If you run low on disk space, the relational data store will be placed in read-only mode to avoid data corruption.

  Therefore, define a file directory for automatic backups in a secure, shared file directory on another machine rather than in the default location. This file directory is considered the default backup location. The backups automatically created by ArcGIS Data Store always go to the default backup location. You can change the location for default backups using the change operation with the configurebackuplocation utility.

  You can register additional backup locations for your relational data store, including other file shares, Amazon Simple Storage Service (S3) buckets, and Microsoft Azure Blob storage containers. These additional locations can be used to store full backups you create using the backupdatastore utility.

• Tile cache data stores

  When you create a tile cache data store, a backup location is automatically configured, but it is on the same machine as the data store. By default, ArcGIS Data Store creates backups of tile cache data stores in /usr/arcgisdatastore/backups/tilecache. If the primary data store machine fails, you can't access the backup files and, therefore, cannot restore your scene layer data.

  Another reason you should not leave backup files on the same machine as the data store is that these files can rapidly fill the disk space on the machine. If you run low on disk space, the tile cache data store will shut down.

  Therefore, you should register a secure, shared file directory on another machine for your backups rather than the default location. Define a shared file directory for automatic backups before you create backups using the backupdatastore utility. This file directory is considered the default backup location. The tile cache data store backups created by ArcGIS Data Store always go to this location. You can change the location for default backups using the change operation with the configurebackuplocation utility.

• Spatiotemporal big data stores

  Upon creation, spatiotemporal big data stores do not have a default backup location. You must register at least one backup location before you can create spatiotemporal big data store backup files. You can register a file share, Amazon S3 bucket, or Microsoft Azure Blob storage container. You can also specify multiple backup locations for spatiotemporal big data stores and set one as the default location. The backups automatically created by ArcGIS Data Store always go to the default backup location. If this location runs low on disk space, the spatiotemporal big data store will shut down.

  You can write manual, full backups to additional backup locations you create using the backupdatastore utility. To do this, you must register another backup location.

Register a default backup location

Register a secure, shared default backup location where ArcGIS Data Store can place files from scheduled (automatic) backups.

Follow these steps to configure a shared directory to store data store backup files:

1. Create a shared directory on another machine to store backup files.

   Note:

   Ensure there is enough storage space to hold all the files that are included in a data store backup. The number of files and size of the files vary depending on your data and the type of data store you are using. Tile cache data stores can be quite large and spatiotemporal big data stores tend to be even larger. Data store backups are created automatically, and the rate at which your backup location fills up depends on your backup schedule and the number of days you retain backups. Be sure to monitor the size of the backup directory, and adjust these settings and storage sizes as needed.

   Also note that all spatiotemporal big data machines in the same ArcGIS Data Store deployment must have access to this shared directory.

2. Grant read and write access on the shared network directory to the account that installed ArcGIS Data Store.
3. Run the configurebackuplocation utility to specify your shared directory as the default output location for data store backups.

   Upon creation, spatiotemporal big data stores do not have a default backup location. You must register at least one backup location before you can create spatiotemporal big data store backup files. You can register a file share, Amazon S3 bucket, or Microsoft Azure Blob storage container.

   If users published hosted feature layers and an automatic backup took place before you configured the recommended shared directory for backups, the configurebackuplocation utility will move your existing relational data store backup files from the default backup location to the shared directory.

   Due to the potentially large size of tile cache data stores, configuring a new location for tile cache data store backups does not copy existing data. Therefore, be sure to specify a shared backup location before users publish any scene layers.

   In this example, the backup location for a relational data store is changed to a shared directory named ds_backups on a computer named sysshare.

   ./configurebackuplocation.sh --operation change --store relational 
   --location /net/sysshare/ds_backups
   
   You are going to change the backup location of the data store. Existing backups will be copied to the new location and it could take a few moments.
   Please do not interrupt the process once it has started.
   
   Do you want to continue (Yes or No)? Yes

   In this example, a backup location is registered for a spatiotemporal big data store. The location is a shared directory named bigdatabus on sysshare.

   ./configurebackuplocation.sh --operation register --store spatiotemporal 
   --location /net/sysshare/bigdatabus

   For full syntax and additional examples, including examples of configuring cloud storage backup locations, see ArcGIS Data Store utility reference.

Tip:

If your remote backup directory goes offline for more than a few minutes, perform a full manual backup of the data store as soon as the shared backup location is available.

Register additional backup locations

For spatiotemporal big data stores and relational data stores, you have the option to register additional backup locations. These can be used to store backups you manually create using the backupdatastore utility.

Note:

You need to clean up manual backup storage locations; ArcGIS Data Store does not delete files for you.

Follow these steps to add another location for backup files you create for a spatiotemporal big data store or relational data store.

1. Create another location for backup files.
   • To register a shared directory, create the directory on another machine. Be sure the storage space is large enough to hold all backup files and be sure the login you use when you connect to the ArcGIS Data Store machine to run the backupdatastore utility has write access to this directory. If you're creating a second shared directory for a spatiotemporal big data store, all spatiotemporal big data machines in the same ArcGIS Data Store deployment must have access to this shared directory.
   • To register an S3 bucket, create the bucket under your Amazon Web Services account. Choose a bucket size that can accommodate your backup files.
   • To register an Azure Blob storage container, create the container under your Azure Blob storage account.
2. Run the configurebackuplocation utility with the register operation to register this additional backup location.

Manage automatic backups

By default, ArcGIS Data Store creates a full backup of data stores every four days, but you can change how often the data store creates a full backup by running the updatebackupschedule utility.

Change backup frequency

If your portal members publish and edit large numbers of hosted layers, or you archive large volumes of streaming data, increase the frequency of backups.

Note:

By default, incremental backups are disabled for relational data stores. If you enable point-in-time recovery, incremental backups are created either when the log files are full or every five minutes, whichever comes first. The database controls incremental backup creation; you cannot control the frequency with which incremental backups are created.

Your backup location must have enough room to store all the backup files. Backup size varies depending on the amount of data you have, but if you use default backup settings, backups contain two full backups and seven days of incremental backup files. The size of these files depends on the amount and size of your data. If you reenable point-in-time recovery for relational data stores, the backups also include seven days of incremental backup files by default.

Be aware that there is no automatic cleanup of tile cache or spatiotemporal big data store backup files. If you increase the frequency of backup for these data stores, you will likely need to clean up the backup location more frequently, too.

If you decide to create backups manually and want to disable automatic backups, set the backup frequency to 0. If you disable automatic backups, you must create the backups yourself to guard against data loss in the event of machine failure or other data catastrophe.

The updatebackupschedule utility is installed in the <ArcGIS Data Store installation directory>/datastore/tools directory.

1. Open a command shell.
2. Run the updatebackupschedule utility to specify the backup frequency you require.

   The syntax to run the utility is as follows:

   updatebackupschedule [--store relational|tileCache|spatiotemporal] [--starttime <local server time>] --frequency <number of days>

   For example, type the following to schedule full relational data store backups at 3:00 a.m. (local server time) every day:

   ./updatebackupschedule.sh --store relational --starttime 03:00:00 --frequency 1

   In this example, a backup of the spatiotemporal big data store is scheduled for 11:30 p.m. (local server time) every three days:

   ./updatebackupschedule.sh --store spatiotemporal --starttime 23:30:00 --frequency 3

Change how long automatic relational data store backup files are retained

The backup directory retains relational data store backup files for seven days by default. This means if you keep the default backup frequency (every four days) and retention schedules (seven days), the backup directory contains two full backups. If you reenable point-in-time recovery, the backup directory also contains seven days of incremental backup files. The size of these files depends on the amount and size of your data. The machine that stores your backups must have enough disk space for all these files. If you increase the backup frequency, consider decreasing the retention period for the backup files. In the previous section, the backup frequency was increased to every day. To prevent your backup directory from becoming too large, decrease the backup file retention period.

The syntax to run the updatebackupretaindays utility is as follows:

updatebackupretaindays <number of days>

In the following example, backup file retention time is changed to four days:

./updatebackupretaindays.sh 4

Manually create and delete backups

Even if you use automatic backups, there may be times when you want to create a backup for a specific purpose outside of the usual backup schedule, such as before upgrading the system or to create a secondary full backup in a different location.

If you disable automatic backups, you should create manual backups periodically.

All backup files that you create manually, even for relational data stores, must be deleted manually.

Run a utility to create a data store backup

You can use the backupdatastore utility to make a full backup of the feature layer data in your data store. You may want to manually create a full backup before making a large number of changes to the data store or before upgrading the data store. Or you may want to create a backup to preserve a copy of the data in a particular state, for example, at the end of the first phase of a project.

The first time you run the backupdatastore utility for a tile cache data store, backup copies are made of all existing tile cache data store databases. Similarly, the first time you run the backupdatastore utility for a spatiotemporal data store, a full backup is created. Because both of these types of data stores can be very large, each time you run the backupdatastore utility after the first time, the utility only creates backup copies of data that was created since the last time you ran the utility.

The login you use to connect to the data store machine to run the backupdatastore utility must have read and write access to the data store backup location.

The syntax to run the backupdatastore utility is as follows:

backupdatastore [<backup name>] [--store {relational|tileCache|spatiotemporal}] [--location <backup_arguments>] [--prompt <yes | no>]

The --location parameter is supported for spatiotemporal big data stores and relational data stores. Arguments for this parameter are as follows and must be separated with semicolons (;):

• type=: Valid types are fs (file share), s3 (Amazon Simple Storage Service (S3) bucket), or azure (Microsoft Azure Blob storage container).
• name=: If you assigned names to the backup locations you configured for your spatiotemporal big data store, you can use the location name to specify where you want backup files created when running the backupdatastore utility.
• location=: If you do not specify a backup location name, you must specify the backup type and location. For file shares, provide the file path. For S3 buckets, provide the bucket name. For Azure Blob storage containers, provide the container name.

Provide a meaningful backup name for the file so you can find it when you want to restore the data. If you do not specify a name, the utility assigns a default name to the file. The default name is in the format datastorename-timestamp. For example, if your data store is named corpds and you create the backup on July 10, 2014, at 14:25:49:554 UTC, the backup file name is corpds-20140710142549554.

You will be asked to confirm that you want to create a backup. Type yes or y to proceed with backup creation.

Tip:

If you want to script manual backups, include a flag to suppress the confirmation prompt as in the following example:

backupdatastore --store tilecache --prompt no

In this example, the data store generates the backup file name. This is necessary in a script to ensure a unique backup file name.

Delete manual data store backups

If you no longer need to keep a data store backup file you created using the backupdatastore utility, you can run the listbackups utility to get the name of the specific backup file, and run the deletebackup utility to remove the unneeded file. For example, once you have upgraded your data store and confirmed all layers are performing as expected, you can delete the data store backup you created before upgrading.

In this example, the database preupgrade1104_bu is deleted:

./listbackups.sh --store relational

Backup_Name                      Status           Backup_Time         Mode
====================================================================================
phase1proj_bu                    BackupComplete   2014-03-08 14:12    manual
phase2proj_bu                    BackupComplete   2014-06-21 11:43    manual
preupgrade_bu                    BackupComplete   2014-10-04 09:30    manual
ds_gdt1oomh-20141103160748082    BackupComplete   2014-11-01 03:00    scheduled


/deletebackup preupgrade1104_bu
You are attempting to delete backup 'preupgrade1104_bu'. This operation is irreversible.

Do you wish to continue (Yes or No)?yes

Operation completed successfully