Manage data store backups

You need backups to recover your data in the event of a disaster such as data corruption or data store failure. If you create backups of your data stores and place them in a safe location, you can set up a new ArcGIS Data Store, access your backup files, and restore the data if for any reason your data store crashes and cannot be restarted.

Backups let you recover your data if a disaster occurs, such as when your server fails or flood waters destroy your server. If your backup is on the server that is destroyed in a flood, you cannot recover your data. Therefore, you need to store your backup files on a different server than your data store.

Note:

Be aware that backup files only contain the data stores. Backup files do not maintain a backup of the GIS Server site or your portal. Rather, backups help you recover data lost if the data store machine fails or data gets corrupted. If you want your hosted feature and scene layers to work even if the machine the data is stored on fails, set up an additional data store machine to make the data store highly available.

If you use a relational or tile cache data store (or both), you could use the webgisdr tool installed with Portal for ArcGIS to create a backup instead. When you use the webgisdr tool, a backup is also created of your portal and hosting server. See ArcGIS Enterprise backups in the Portal for ArcGIS Administrator Guide for more information on using this tool.

Define a backup location

Register a secure, shared backup location where ArcGIS Data Store will place the backup files. Relational data stores are configured to automatically create backups of your data and need this location defined. By default, ArcGIS Data Store creates backups of relational data stores in /usr/arcgisdatastore/backups. That means backup files are stored on the same machine as the relational data store. If the data store or primary machine fails, you can't access the backup files and, therefore, cannot restore your hosted feature layer data. For that reason, you should store your backups in a location other than the default.

Also, be aware that leaving backup files on the same machine as the data store can rapidly fill the disk space on the machine. If you run low on disk space, the relational data store will be placed in read-only mode to avoid data corruption. On tile cache and spatiotemporal big data store machines, the data store will shut down when the machine runs low on disk space. You can use the changedbproperties utility to control the disk space threshold at which the relational data store is placed in read-only mode and the tile cache and spatiotemporal big data stores shut down.

You can manually create backups of relational, tile cache, and spatiotemporal big data stores. To create a backup of a spatiotemporal big data store, you must register a shared network location. You must also register a location for tile cache backups before backups can be created. Registration of a shared network location for tile cache data stores is optional but strongly recommended.

The following diagram shows a relational data store composed of one computer (the primary server) for storing your data, and a network drive to store backup files. Storing the backup files on a machine separate from the data store protects you from losing the backup files if the machine running the data store fails.

Follow these steps to configure a shared directory to store data store backup files:

1. Create a shared directory on another machine to store backup files.

   Note:

   Ensure there is enough storage space to hold all the files that are included in a data store backup. The number of files and size of the files will vary depending on your data and the type of data store you are using. Tile cache data stores can be quite large and spatiotemporal big data stores tend to be even larger. Relational data store backups are created automatically, and the rate at which your backup location fills up depends on your backup schedule and the number of days you retain backups. Be sure to monitor the size of the backup directory, and adjust these settings and storage sizes as needed.

   Also note that all spatiotemporal big data machines in the same ArcGIS Data Store deployment must have access to this shared directory.

2. Grant read and write access on the shared network directory to the account that installed ArcGIS Data Store.
3. Run the configurebackuplocation utility to specify your shared directory as the output location for data store backups.

   When you create a relational data store, a default backup location is created at the same time. If hosted feature layers were published before you configured this shared backup location, the configurebackuplocation utility will move your existing data store backup files from the default backup location to the shared directory.

   Tile cache data stores also have a default backup location when created. However, due to the potentially large size of tile cache data stores, configuring a new location for tile cache data store backups does not copy existing data. Therefore, be sure to specify a shared backup location before publishing any scene layers.

   Spatiotemporal big data stores do not have a default backup location. You must register a shared backup location before you can create spatiotemporal big data store backup files.

   In this example, the backup location for a relational data store is changed to a shared directory named ds_backups on a computer named sysshare.

   ./configurebackuplocation.sh --operation change --store relational 
   --location /net/sysshare/ds_backups
   
   You are going to change the backup location of the data store. Existing backups will be copied to the new location and it could take a few moments.
   Please do not interrupt the process once it has started.
   
   Do you want to continue (Yes or No)? Yes

   In this example, a backup location is registered for a spatiotemporal big data store. The location is a shared directory named bigdatabus on sysshare.

   ./configurebackuplocation.sh --operation register --store spatiotemporal 
   --location /net/sysshare/bigdatabus

Tip:

If your remote backup directory goes offline for more than a few minutes, perform a full manual backup of the data store as soon as the shared backup location is available.

Manually create a data store backup

You can use the backupdatastore utility to make a full backup of the feature layer data in your data store. You may want to manually create a full backup prior to making a large number of changes to the data store or prior to upgrading the data store. Or you may want to create a backup to preserve a copy of the data in a particular state, for example, at the end of the first phase of a project.

The first time you run the backupdatastore utility for a tile cache data store, backup copies are made of all existing tile cache data store databases. Similarly, the first time you run the backupdatastore utility for a spatiotemporal data store, a full backup is created. Because both of these types of data stores can be very large, each time you run the backupdatastore utility after the first time, the utility only creates backup copies of data that was created since the last time you ran the utility.

The login you use to connect to the data store machine to run the backupdatastore utility must have read and write access to the data store backup location.

The syntax to run the backupdatastore utility is as follows:

backupdatastore [<backup_name>] --store {relational | tilecache | spatiotemporal}

Provide a meaningful name for the file so you can find it when you want to restore the data. If you do not specify a name, the utility assigns a default name to the file. The default name is in the format datastorename-timestamp. For example, if your data store is named corpds and you create the backup on July 10, 2014, at 14:25:49:554 UTC, the backup file name is corpds-20140710142549554.

You will be asked to confirm that you want to create a backup. Type yes or y to proceed with backup creation.

Tip:

If you want to script manual backups, include a flag to suppress the confirmation prompt as in the following example:

backupdatastore --store tilecache --prompt no

In this example, the data store will generate the backup file name. This is necessary in a script to ensure a unique backup file name.

Change backup frequency for relational data stores

By default, ArcGIS Data Store creates a full backup of relational data stores every four days, but you can change how often the data store creates a full backup by running the updatebackupschedule utility. If your users will be publishing and editing large numbers of hosted feature layers, you should increase the frequency with which full data store backups are created.

Beginning with 10.5, incremental backups are disabled. If you re-enable point-in-time recovery, incremental backups will be created either when the log files are full or every five minutes, whichever comes first. The database controls incremental backup creation; you cannot control the frequency with which incremental backups are created.

Your backup location must have enough room to store all the backup files. Backup size varies depending on the amount of data you have, but if you use default backup settings, backups will contain two full backups and seven days of incremental backup files. The size of these files depends on the amount and size of your data. If you re-enable point-in-time recovery, the backups will also include seven days of incremental backup files by default.

If you decide to create backups manually and want to disable automatic backups, set the backup frequency to 0. Be aware that ArcGIS Data Store will not create any backups if you disable automatic backups. You must create the backups yourself.

The updatebackupschedule utility is installed in the <ArcGIS Data Store installation directory>/datastore/tools directory.

1. Open a command shell.
2. Run the updatebackupschedule utility to specify the backup frequency you require.

   The syntax to run the utility is as follows:

   updatebackupschedule --starttime <local server time> --frequency <number of days>

   For example, type the following to schedule full backups at 3:00 a.m. (local server time) every day:

   ./updatebackupschedule.sh --starttime 03:00:00 --frequency 1

Change how long relational data store backup files are retained

The backup directory retains relational data store backup files for seven days by default. That means if you keep the default backup frequency (every four days) and retention schedules (seven days), the backup directory will contain two full backups. If you re-enable point-in-time recovery, the backup directory will also contain seven days of incremental backup files. The size of these files depends on the amount and size of your data. The machine that stores your backups must have enough disk space for all these files. If you increase the backup frequency, consider decreasing the retention period for the backup files. In the previous section, the backup frequency was increased to every day. To prevent your backup directory from becoming too large, decrease the backup file retention period.

The syntax to run the updatebackupretaindays utility is as follows:

updatebackupretaindays <number of days>

In the following example, backup file retention time is changed to four days:

./updatebackupretaindays.sh 4

Delete manual data store backups

If you no longer need to keep a data store backup file you created using the backupdatastore utility, you can run the listbackups utility to get the name of the specific backup file, and run the deletebackup utility to remove the unneeded file. For example, once you have upgraded your data store and confirmed all layers are performing as expected, you can delete the data store backup you created prior to upgrading.

In this example, the database preupgrade1104_bu is deleted:

./listbackups.sh --store relational

Backup_Name                      Status           Backup_Time         Mode
====================================================================================
phase1proj_bu                    BackupComplete   2014-03-08 14:12    manual
phase2proj_bu                    BackupComplete   2014-06-21 11:43    manual
preupgrade_bu                    BackupComplete   2014-10-04 09:30    manual
ds_gdt1oomh-20141103160748082    BackupComplete   2014-11-01 03:00    scheduled


/deletebackup preupgrade1104_bu
You are attempting to delete backup 'preupgrade1104_bu'. This operation is irreversible.

Do you wish to continue (Yes or No)?yes

Operation completed successfully