Manage data store backups

You need backups to recover your data in the event of a disaster such as data corruption or data store failure. If you create backups of your data stores and place them in a safe location, you can set up a new ArcGIS Data Store, access your backup files, and restore the data if for any reason your data store crashes and cannot be restarted.

Note:

Be aware that backup files only contain the data stores. Backup files do not maintain a backup of the GIS Server site or your portal. Rather, backups help you recover data lost if the data store machine fails or data gets corrupted. If you want your hosted feature and scene layers to work even if the machine the data is stored on fails, set up an additional data store machine to make the data store highly available.

If you use a relational or tile cache data store (or both), you could use the webgisdr tool installed with Portal for ArcGIS to create a backup instead. When you use the webgisdr tool, a backup is also created of your portal, hosting server, and federated servers. See ArcGIS Enterprise backups in the Portal for ArcGIS Administrator Guide for more information on using this tool.

Backups allow you to recover your data if a disaster occurs, such as when your server fails or floodwaters destroy your server. If your backup is on the server that is destroyed in a flood, you cannot recover your data. Therefore, you need to store your backup files on a different server than your data store.

Define a backup location

When you create a relational data store or tile cache data store, a backup location is automatically configured, but it is on the same machine as the data store. You need to configure a secure, shared file directory on another machine for each of these types of data stores. By default, ArcGIS Data Store creates backups of relational data stores in /usr/arcgisdatastore/backups. This means backup files are stored on the same machine as the relational or tile cache data store. If the data store or primary machine fails, you can't access the backup files and, therefore, cannot restore your hosted feature layer and scene layer data. For this reason, you should store your backups in a location other than the default.

Another reason you should not leave backup files on the same machine as the data store is that these files can rapidly fill the disk space on the machine. If you run low on disk space, the relational data store will be placed in read-only mode to avoid data corruption. On tile cache and spatiotemporal big data store machines, the data store will shut down when the machine runs low on disk space.

For relational and tile cache data stores, you must define a file directory for automatic backups. This file directory is considered the default backup location for these data stores. The backups automatically created by ArcGIS Data Store always go to the default backup location.

Starting with 10.6.1, you can define additional backup locations for your relational data store, including other file shares, Amazon Simple Storage Service (S3) buckets, and Microsoft Azure Blob storage containers. These additional locations can be used to store full backups you create using the backupdatastore utility.

Upon creation, spatiotemporal big data stores do not have a default backup location. You must register at least one backup location before you can create spatiotemporal big data store backup files. You can register a file share, Amazon S3 bucket, or Microsoft Azure Blob storage container. You can also specify multiple backup locations for spatiotemporal big data stores and set one as the default location. The backups automatically created by ArcGIS Data Store always go to the default backup location. You can write manual, full backups to additional backup locations you create using the backupdatastore utility.

You can manually create backups of relational, tile cache, and spatiotemporal big data stores. To create a backup of a spatiotemporal big data store, you must register a shared network location. You must also register a location for tile cache backups before backups can be created. Registration of a shared network location for tile cache data stores is optional but strongly recommended.

Register a default backup location

Register a secure, shared default backup location where ArcGIS Data Store can place files from scheduled (automatic) backups.

Follow these steps to configure a shared directory to store data store backup files:

1. Create a shared directory on another machine to store backup files.

   Note:

   Ensure there is enough storage space to hold all the files that are included in a data store backup. The number of files and size of the files vary depending on your data and the type of data store you are using. Tile cache data stores can be quite large and spatiotemporal big data stores tend to be even larger. Data store backups are created automatically, and the rate at which your backup location fills up depends on your backup schedule and the number of days you retain backups. Be sure to monitor the size of the backup directory, and adjust these settings and storage sizes as needed.

   Also note that all spatiotemporal big data machines in the same ArcGIS Data Store deployment must have access to this shared directory.

2. Grant read and write access on the shared network directory to the account that installed ArcGIS Data Store.
3. Run the configurebackuplocation utility to specify your shared directory as the default output location for data store backups.

   Upon creation, spatiotemporal big data stores do not have a default backup location. You must register at least one backup location before you can create spatiotemporal big data store backup files. You can register a file share, Amazon S3 bucket, or Microsoft Azure Blob storage container.

   If users published hosted feature layers and an automatic backup took place before you configured the recommended shared directory for backups, the configurebackuplocation utility will move your existing relational data store backup files from the default backup location to the shared directory.

   Due to the potentially large size of tile cache data stores, configuring a new location for tile cache data store backups does not copy existing data. Therefore, be sure to specify a shared backup location before users publish any scene layers.

   In this example, the backup location for a relational data store is changed to a shared directory named ds_backups on a computer named sysshare.

   ./configurebackuplocation.sh --operation change --store relational 
   --location /net/sysshare/ds_backups
   
   You are going to change the backup location of the data store. Existing backups will be copied to the new location and it could take a few moments.
   Please do not interrupt the process once it has started.
   
   Do you want to continue (Yes or No)? Yes

   In this example, a backup location is registered for a spatiotemporal big data store. The location is a shared directory named bigdatabus on sysshare.

   ./configurebackuplocation.sh --operation register --store spatiotemporal 
   --location /net/sysshare/bigdatabus

   For full syntax and additional examples, including examples of configuring cloud storage backup locations, see ArcGIS Data Store utility reference.

Tip:

If your remote backup directory goes offline for more than a few minutes, perform a full manual backup of the data store as soon as the shared backup location is available.

Register additional backup locations for manual backups

For spatiotemporal big data stores, you have the option to register additional backup locations. Beginning with 10.6.1, you also have the option to register additional backup locations for relational data stores. These can be used to store backups you manually create using the backupdatastore utility.

Note that you need to clean up manual backup storage locations; ArcGIS Data Store does not delete files for you.

Follow these steps to add another location for backup files you manually create for a spatiotemporal big data store or relational data store.

1. Create another location for backup files.
   • To register a shared directory, create the directory on another machine. Be sure the storage space is large enough to hold all backup files and be sure the login you use when you connect to the ArcGIS Data Store machine to run the backupdatastore utility has write access to this directory. If you're creating a second shared directory for a spatiotemporal big data store, all spatiotemporal big data machines in the same ArcGIS Data Store deployment must have access to this shared directory.
   • To register an S3 bucket, create the bucket under your AWS account. Choose a bucket size that can accommodate your backup files.
   • To register an Azure Blob storage container, create the container under your Azure Blob storage account.
2. Run the configurebackuplocation utility with the register operation to register this additional backup location.

Automatic backups

By default, ArcGIS Data Store creates a full backup of data stores every four days, but you can change how often the data store creates a full backup by running the updatebackupschedule utility.

Change backup frequency

If your users publish and edit large numbers of hosted feature layers, you should increase the frequency with which full relational data store backups are created. If your users publish large numbers of scene layers, increase how frequently tile cache data store backups are created. If the ArcGIS GeoEvent Server site archives large volumes of streaming data or many users are frequently running GeoAnalytics Tools, increase the frequency of spatiotemporal big data store backups.

Note:

By default, incremental backups are disabled for relational data stores. If you enable point-in-time recovery, incremental backups are created either when the log files are full or every five minutes, whichever comes first. The database controls incremental backup creation; you cannot control the frequency with which incremental backups are created.

Your backup location must have enough room to store all the backup files. Backup size varies depending on the amount of data you have, but if you use default backup settings, backups contain two full backups and seven days of incremental backup files. The size of these files depends on the amount and size of your data. If you reenable point-in-time recovery for relational data stores, the backups also include seven days of incremental backup files by default.

Be aware that there is no automatic cleanup of tile cache or spatiotemporal big data store backup files. If you increase the frequency of backup for these data stores, you will likely need to clean up the backup location more frequently, too.

If you decide to create backups manually and want to disable automatic backups, set the backup frequency to 0. If you disable automatic backups, you must create the backups yourself to guard against data loss in the event of machine failure or other data catastrophe.

The updatebackupschedule utility is installed in the <ArcGIS Data Store installation directory>/datastore/tools directory.

1. Open a command shell.
2. Run the updatebackupschedule utility to specify the backup frequency you require.

   The syntax to run the utility is as follows:

   updatebackupschedule [--store relational|tileCache|spatiotemporal] [--starttime <local server time>] --frequency <number of days>

   For example, type the following to schedule full relational data store backups at 3:00 a.m. (local server time) every day:

   ./updatebackupschedule.sh --store relational --starttime 03:00:00 --frequency 1

   In this example, a backup of the spatiotemporal big data store is scheduled for 11:30 p.m. (local server time) every three days:

   ./updatebackupschedule.sh --store spatiotemporal --starttime 23:30:00 --frequency 3

Change how long automatic relational data store backup files are retained

The backup directory retains relational data store backup files for seven days by default. This means if you keep the default backup frequency (every four days) and retention schedules (seven days), the backup directory contains two full backups. If you reenable point-in-time recovery, the backup directory also contains seven days of incremental backup files. The size of these files depends on the amount and size of your data. The machine that stores your backups must have enough disk space for all these files. If you increase the backup frequency, consider decreasing the retention period for the backup files. In the previous section, the backup frequency was increased to every day. To prevent your backup directory from becoming too large, decrease the backup file retention period.

The syntax to run the updatebackupretaindays utility is as follows:

updatebackupretaindays <number of days>

In the following example, backup file retention time is changed to four days:

./updatebackupretaindays.sh 4

Manual backups

Even if you use automatic backups, there may be times when you want to create a backup for a specific purpose outside of the usual backup schedule, such as before upgrading the system or to create a secondary full backup in a different location.

If you disable automatic backups, you should create manual backups periodically.

All backup files that you create manually, even for relational data stores, must be deleted manually.

Run a utility to create a data store backup

You can use the backupdatastore utility to make a full backup of the feature layer data in your data store. You may want to manually create a full backup before making a large number of changes to the data store or before upgrading the data store. Or you may want to create a backup to preserve a copy of the data in a particular state, for example, at the end of the first phase of a project.

The first time you run the backupdatastore utility for a tile cache data store, backup copies are made of all existing tile cache data store databases. Similarly, the first time you run the backupdatastore utility for a spatiotemporal data store, a full backup is created. Because both of these types of data stores can be very large, each time you run the backupdatastore utility after the first time, the utility only creates backup copies of data that was created since the last time you ran the utility.

The login you use to connect to the data store machine to run the backupdatastore utility must have read and write access to the data store backup location.

The syntax to run the backupdatastore utility is as follows:

backupdatastore [<backup name>] [--store {relational|tileCache|spatiotemporal}] [--location <backup_arguments>] [--prompt <yes | no>]

The --location parameter is supported for spatiotemporal big data stores and relational data stores. Arguments for this parameter are as follows and must be separated with semicolons (;):

• type=: Valid types are fs (file share), s3 (AWS S3 bucket), or azure (Azure Blob storage container).
• name=: If you assigned names to the backup locations you configured for your spatiotemporal big data store, you can use the location name to specify where you want backup files created when running the backupdatastore utility.
• location=: If you do not specify a backup location name, you must specify the backup type and location. For file shares, provide the file path. For S3 buckets, provide the bucket name. For Azure Blob storage containers, provide the container name.

Provide a meaningful backup name for the file so you can find it when you want to restore the data. If you do not specify a name, the utility assigns a default name to the file. The default name is in the format datastorename-timestamp. For example, if your data store is named corpds and you create the backup on July 10, 2014, at 14:25:49:554 UTC, the backup file name is corpds-20140710142549554.

You will be asked to confirm that you want to create a backup. Type yes or y to proceed with backup creation.

Tip:

If you want to script manual backups, include a flag to suppress the confirmation prompt as in the following example:

backupdatastore --store tilecache --prompt no

In this example, the data store generates the backup file name. This is necessary in a script to ensure a unique backup file name.

Delete manual data store backups

If you no longer need to keep a data store backup file you created using the backupdatastore utility, you can run the listbackups utility to get the name of the specific backup file, and run the deletebackup utility to remove the unneeded file. For example, once you have upgraded your data store and confirmed all layers are performing as expected, you can delete the data store backup you created before upgrading.

In this example, the database preupgrade1104_bu is deleted:

./listbackups.sh --store relational

Backup_Name                      Status           Backup_Time         Mode
====================================================================================
phase1proj_bu                    BackupComplete   2014-03-08 14:12    manual
phase2proj_bu                    BackupComplete   2014-06-21 11:43    manual
preupgrade_bu                    BackupComplete   2014-10-04 09:30    manual
ds_gdt1oomh-20141103160748082    BackupComplete   2014-11-01 03:00    scheduled


/deletebackup preupgrade1104_bu
You are attempting to delete backup 'preupgrade1104_bu'. This operation is irreversible.

Do you wish to continue (Yes or No)?yes

Operation completed successfully