Recover a data store

If you lose access to the data in a data store—for example, an ArcGIS Data Store machine crashes or an editor deletes the wrong data from a hosted feature layer—you can restore the data using backup files.

In the case where one or more of the machines in a data store crashes and cannot be recovered, install ArcGIS Data Store on a new machine and restore the latest available full backup to the new machine using the restoredatastore utility.

How you do that will vary depending on the type of data store and how many or which machines you're replacing. The following are examples of some of these differences:

• If you replace all the machines in a tile cache (cluster) or spatiotemporal big data store or object store, you'll run the restoredatastore utility twice to accommodate all the data.
• If the relational data store is configured for point-in-time recovery, you can recover hosted feature layer data to a specific point in time by restoring on top of the existing relational data store and specifying a date and time to which you want to restore.
• After you replace a machine in a tile cache data store cluster and restore data to the new machine, you redistribute scene cache data.

Important information about restoring data stores

Keep the following information in mind in regards to ArcGIS Data Store recovery:

• You must ensure your backup and restore procedures work by testing them on development or test deployments. You don't want to discover a flaw in your workflow when you need to restore a production deployment.
• The backup files you use for recovery are compressed. When you restore data, these files must be uncompressed to a staging location. That means you need to have a staging location that can accommodate this data. By default, the files are staged in the ArcGIS Data Store data directory. If you have a lot of data in the relational or tile cache data store, you can set up a separate staging location and specify that location to stage files for data store recovery. Specify the separate staging location using the changestaginglocation utility.
• Before you run the restoredatastore utility, ensure no one is publishing or editing layers. These new services and edits will be lost, as they are not included in the backup file you are restoring.
• Clients and services cannot access existing data in the data store while you are restoring it.
• When you deploy ArcGIS Enterprise on-premises but your data store backup files are in cloud storage, creating backups and restoring from those backups will take longer than when backup files are stored in local file shares.

Restore a relational data store, graph store, or tile cache data store (primary-standby mode) after a crash

If the hardware stops functioning or the data store is unavailable and cannot be brought back online for a single-machine relational data store, tile cache data store (running in primary-standby mode), or graph store, install ArcGIS Data Store on a new machine and apply your backup files to the new installation using the restoredatastore utility. This re-creates the data store on the new machine and keeps the data store registered with the existing GIS Server site.

Tip:

If the primary machine in a primary-standby deployment of a relational data store, graph store, or tile cache data store fails, the standby machine becomes the primary machine. Remove the failed machine from the deployment. Then you can install ArcGIS Data Store on a new machine and configure it as the standby machine. You do not need to restore from a backup file because the data will replicate from the primary machine to the standby machine. Similarly, if a standby machine fails and cannot be recovered, you do not have to restore data. Remove the standby machine from the deployment, install ArcGIS Data Store on a new machine, and configure it as the standby machine.

1. Install ArcGIS Data Store on a new machine.

   Do not proceed with configuration. Close the Data Store Configuration wizard when it opens at the end of the installation.

2. Open a command shell.
3. Run the restoredatastore utility from <ArcGIS Data Store installation directory>\datastore\tools to restore the most recent backup of the relational, graph, or tile cache data store.

   See the utility reference or type --help at the command line for syntax.

   This example restores the most recent backup of a relational data store from /net/fortknox/backups/datastore/relational to /usr/arcgisdatastore. Because the data store remains bound by default to the GIS Server site with which it was registered, you do not have to specify --bound true, but you do need to specify the GIS Server site URL and administrator credentials.

   ./restoredatastore --store relational --target most-recent --source-loc /net/fortknox/backups/datastore/relational --data-dir /usr/arcgisdatastore --server-url https://gisserver.domain.com:6443 --server-admin siteadmin --server-password MySApw

   If the directory you specify with the --data-dir option does not already exist, the directory will be created. If it does exist, you must have write access to it and it must be empty. If not, the restore operation will fail.

4. Confirm that you want to restore the data store by typing yes.

Restore an existing relational data store to a specific point in time

If the relational data store is configured for point-in-time recovery, you can use backup files to restore the relational data store to a point in time that you specify. For example, if an editor makes extensive edits to data in a hosted feature layer and discovers the next day that those edits were incorrect, you can restore the relational data store to the state it was prior to those edits.

Times specified must be in Coordinated Universal Time (UTC).

Note:

Any feature layers published to the portal after the point in time to which you restore the relational data store will no longer be valid, because the data used by those layers will no longer exist in the data store.

Also note that if you disabled automatic backups, you cannot restore to a specific point in time.

1. Open a command shell.
2. Run the restoredatastore utility from <ArcGIS Data Store installation directory>\datastore\tools to restore the data to a specific point in time.

   The syntax to restore the relational data store to a specific point in time is as follows:

   restoredatastore --store relational --target <date and time> --server-url <ArcGIS Server URL> --server-admin <ArcGIS Server admin user> --server-password <ArcGIS Server admin password>

   For example, type the following to restore the data store from files to the state it was in at 5:00 p.m. UTC on March 20, 2014:

   ./restoredatastore --store relational --target 2014-03-20-17:00:00 --server-url https://gisserver.domain.com:6443 --server-admin siteadmin --server-password MySApw
   
   You are going to restore the data store from a data store backup. This process could take a long time, 
   depending on the size of your data. Please do not interrupt the process once it has started.
   
   Do you want to continue (Yes or No)? Yes

3. Confirm that you want to restore the data store by typing yes.

   Tip:

   If you want to script the restoredatastore utility, include a flag to suppress the confirmation prompt, as in the following example:

   restoredatastore --store relational --target 2014-03-20-17:00:00 --prompt no

   Your hosted feature layer data is restored to the point in time you specified.

Restore a clustered tile cache data store to redistribute scene cache data after a crash

If a machine in a tile cache data store running in cluster mode crashes and you cannot bring it back online, replace the lost machine and restore the tile cache data store to the new machine. When you restore the data, ArcGIS Data Store redistributes copies of the scene cache data across all machines in the data store, including the new machine.

Follow these instructions to detect and replace a failed tile cache data store machine and restore to redistribute scene cache data for high availability:

1. If you suspect there is a problem with one of your tile cache data store machines, run the describedatastore utility to confirm that a machine is inaccessible.
2. If this indicates a problem with a machine, attempt to recover the machine.

   For example, if a network or power cable is unplugged, plug it in and start the machine.

3. If you can recover the machine, bring it back online and run the describedatastore utility again to confirm the machine is back online and participating in the tile cache data store.

   Do some tests of the scene layers to make sure they are accessible. If all is well, you're done. However, if you cannot recover the machine, proceed to the next step.

4. Run the removemachine utility with the --force option to remove the failed machine from the tile cache data store.

   In this example, the machine tilecache3 is inaccessible and cannot be recovered. Because ArcGIS Data Store cannot access the machine, you must specify --force true to remove the machine from the tile cache data store.

   ./removemachine tilecache3 --store tilecache --force true

5. Obtain a new machine with the same operating system as the old machine. Install ArcGIS Data Store on it, and add it to the existing tile cache data store by configuring it with the same hosting ArcGIS Server site.
6. Run the ArcGIS Server validate REST command to confirm that your existing scene cache data is highly available.

   If you receive a message that there is only one copy of one or more of your scene layer caches, proceed with the remaining steps.

7. Confirm you have configured a shared backup location for the tile cache data store. If you have not, configure one now.
8. Ensure no one publishes or updates any scene layers or is building any scene caches.

   To detect if a scene cache is being built or rebuilt, sign in to ArcGIS Server Manager for the hosting server, open Site > Jobs, choose System/SceneCachingControllers from the Services drop-down list, and click Query. If the server is building scene caches, the job statuses appear when you query. Do not proceed until these cache jobs complete. If no scene cache jobs are listed, you can proceed.

   To ensure no one publishes new layers or edits, you can place ArcGIS Enterprise in read-only mode. However, be aware that this affects everyone using the portal.

9. Create a backup of the tile cache data store.

   In this example, a backup file named myscenecachebu is created and stored in a backup location in an Amazon Simple Storage Service (S3) bucket you registered as a backup location for the tile cache data store.

   ./backupdatastore myscenecachebu --store tilecache --location type=s3;location=mys3bucket

10. Use the restoredatastore utility with the replicatedata option set to true to overwrite the contents of the tile cache data store and rebalance the content across all the machines.

    In the following example, the myscenecachebu file is used to restore the tile cache data store, and the replicatedata option is set to true to indicate to ArcGIS Data Store that two copies of each scene cache should be distributed across all machines in the tile cache data store.

    ./restoredatastore.sh --store tilecache --target myscenecachebu --serverurl https://gisserver.domain.com:6443 --server-admin myadmin --server-password myAdminPWd! --replicatedata true

11. If you placed ArcGIS Enterprise in read-only mode, put it back in read-write mode.
12. Run the validate command again to ensure there are two copies of all scene caches.

Restore a data store after loss of all machines

If you have to replace all the machines in a tile cache or spatiotemporal big data store, or object store—for example, a flood destroys your data center or you have to revert to an older version of ArcGIS Enterprise—follow these steps to restore the data store:

1. Install ArcGIS Data Store on a new machine.

   Do not proceed with configuration. Close the Data Store Configuration wizard when it opens at the end of the installation.

2. Open a command shell.
3. Run the restoredatastore utility from <ArcGIS Data Store installation directory>\datastore\tools and include the --loaddata operation set to false to prepare the first machine for restoring the spatiotemporal big data store.

   See the utility reference or type --help at the command line for syntax.

   This example prepares the new machine to restore the most recent backup of a spatiotemporal big data store from /net/myserver/backups/spatiotemporal to /usr/arcgisdatastore. The --loaddata operation is set to false so the data is not yet loaded.

   ./restoredatastore --store spatiotemporal --target most-recent --source-loc /net/myserver/backups/spatiotemporal --data-dir /usr/arcgisdatastore --loaddata false --server-url https://gisserver.domain.com:6443 --server-admin siteadmin --server-password MySApw

   If the directory you specify with the --data-dir option does not exist, the data store creates it. If the directory does exist, you must have write access to it and it must be empty. If not, the restore operation fails.

   To load data for a tile cache data store, you would specify --store tilecache and the tile cache backup location with the --source-loc operation in the previous example.

4. Install ArcGIS Data Store on additional machines and run the Data Store Configuration wizard or configuredatastore utility on each to add machines to the hosting server.
5. Run the restoredatastore utility again without the --loaddata option or with --loaddata set to true to restore all the data.

   Data will be distributed across all tile cache or spatiotemporal big data store machines.

   This example restores the data to the newly configured spatiotemporal big data store using the same backup file (on /net/myserver/backups/spatiotemporal). The --loaddata option is set to true by default, so you don't have to specify it when you load the data.

   ./restoredatastore --store spatiotemporal --target most-recent --source-loc /net/myserver/backups/spatiotemporal --data-dir /usr/arcgisdatastore --server-url https://gisserver.domain.com:6443 --server-admin siteadmin --server-password MySApw

Restore a spatiotemporal big data store or object store when the target machine can hold all the data in the backup file

If the first machine on which you create your replacement spatiotemporal big data store or object store (cluster mode) contains enough disk space and memory to hold all the data in your backup file or if you are restoring to your existing data store machines, you only need to run the restoredatastore utility once. Once restored, you can add the other new machines to your spatiotemporal big data store or object store, and ArcGIS Data Store will replicate the data across the other machines.

1. Install ArcGIS Data Store on a new machine.

   Do not proceed with configuration. Close the Data Store Configuration wizard when it opens at the end of the installation.

2. Open a command shell.
3. Run the restoredatastore utility from <ArcGIS Data Store installation directory>\datastore\tools and include the --loaddata operation set to true the spatiotemporal big data store.

   See the utility reference or type --help at the command line for syntax.

   This example prepares the new machine to restore the most recent backup of a spatiotemporal big data store from /net/myserver/backups/spatiotemporal to /usr/arcgisdatastore. The --loaddata operation is set to true because this new machine can hold all the data from the backup file.

   ./restoredatastore --store spatiotemporal --target most-recent --source-loc /net/myserver/backups/spatiotemporal --data-dir /usr/arcgisdatastore --loaddata true --server-url https://gisserver.domain.com:6443 --server-admin siteadmin --server-password MySApw

   If the directory you specify with the --data-dir option does not exist, the data store creates it. If the directory does exist, you must have write access to it and it must be empty. If not, the restore operation fails.

   Note that you could omit the --loaddata operation in this case, as data is loaded by default.

   If the machine cannot accommodate all the data from the backup file, the restore operation will fail.

4. Install ArcGIS Data Store on additional machines and run the Data Store Configuration wizard or configuredatastore utility on each to add spatiotemporal big data store machines to the hosting server.

   ArcGIS Data Store will distribute and replicate the data across the other machines.

Restore a relational or tile cache data store to a different OS

You can use your backup files to restore a relational or tile cache data store to a machine running a different operating system (OS).

For example, if the tile cache data store is running on a Microsoft Windows server, but your information technology department is switching to Linux servers, you can use the backup files created from an ArcGIS Data Store installation on the Windows server to restore to a tile cache data store on the Linux server.

1. Install ArcGIS Data Store on the new server.

   Do not proceed with configuration.

2. If you don't already have a full backup of the existing relational or tile cache data store, use the backupdatastore utility to create one.
3. Place the backup file in a shared location that can be accessed by the new ArcGIS Data Store installation.
4. Run the restoredatastore utility to restore to the new machine. Use the --server-url operation to specify the URL of the portal's existing hosting server (a GIS Server site) when you restore. That way, existing hosted feature and scene layers will continue to work.

   In this example, a tile cache data store is restored and registered with the GIS Server site https://mygisserver.domain.com:6443. The backup file is in the shared directory /net/backups/tilecache and the ArcGIS Data Store directory on the new machine is /arcgis/datastore.

   ./restoredatastore.sh --store tileCache --source-loc /net/backups/tilecache --data-dir /arcgis/datastore --server-url https://gisserver.domain.com:6443 --server-admin siteadmin --server-password MySApw