Data storage and management is a vital aspect of your ArcGIS Enterprise deployment. It shapes how your organization accesses, manages, contributes, and edits data, and it provides the foundation for how your data can be used. ArcGIS Enterprise allows you to store source data for your web services and layers in user-managed data storage locations or data storage that is managed by ArcGIS.
User-managed storage locations are the data sources that you or others in your organization manage, such as a database, folder, or cloud storage locations. Conversely, when you use data storage locations managed by ArcGIS—such as ArcGIS Data Store—you do not manage or access the underlying databases. In a single ArcGIS Enterprise deployment, you will likely use both types of data storage; you don't have to choose just one or the other.
Tip:
For more information on these terms and concepts, see the Data in ArcGIS: User Managed and ArcGIS Managed technical paper.
When you publish a web map, layer, or service to ArcGIS Enterprise, you determine how your data will be managed. The first step in this process is to decide if you'll copy data or register your data with one (or more) of the federated servers that comprise your ArcGIS Enterprise portal.
Register or copy the data
When you publish from ArcGIS Pro, you determine the location of the data used by your web layers. For most data sources, you can either register your data source—in which case the web layers access the data in the data source—or have ArcGIS copy the data to a location managed by ArcGIS, which can be the ArcGIS Data Store, a federated server, or the hosting server. If your data source is a cloud data warehouse, you always register the data source, but you can create a snapshot of the data when you publish. This makes a copy of the subset of data included in the query layer that accesses the cloud data warehouse and places it in the ArcGIS Data Store relational data store for the web service to access. You can refresh the contents of this snapshot from the web layer's item page in the portal. Doing so overwrites the data in the relational data store with data from the registered data source.
You can also add files to your organization and publish from the ArcGIS Enterprise portal. In this case, the data is always copied to one of the data stores provided through ArcGIS Data Store or the hosting server, depending on the type of layer you publish.
Note:
A hosting server is the federated ArcGIS Server site the organization administrator has designated to run the services that power most hosted web layers. This site can run other services, but its intended purpose is to run hosted services. There is only one hosting server per ArcGIS Enterprise organization.
Register data stores
When you add a user-managed data store and publish web layers, the web layers reference the data in the data source. If the data in the registered data source changes, you will see those changes in the web layer. The only exception to this is when you create a snapshot for data published from a cloud data warehouse.
The following are cases when registering data is recommended or required:
- You have multiple clients accessing and updating the source data.
If you have apps directly editing the source data, apps editing the source data through services, or conversion or ETL processes that load data from contractors to your source, publish map or feature layers that reference the data source. That way, the people who use the layers can see changes to the data as they are made in the source.
- You use versioned data from an enterprise geodatabase.
If you publish from a map that contains versioned enterprise geodatabase data and you copy the data, the copied data no longer participates in the version. Edits made through the published feature layer cannot take advantage of multiuser editing functionality.
- You use archive-enabled data from an enterprise geodatabase.
Data owners enable archiving so they can see changes in the data over time. If you copy data from the source when you publish a feature layer, it's no longer part of the archive and you cannot see the changes made to the data after it's copied.
- You have large feature classes or feature classes with complex geometries.
The greater the number of features and the more complex the shapes, the longer it takes to copy the data. Examples of complex shapes include polygons or lines with thousands of vertices, such as coast lines or meandering rivers.
- You're publishing imagery data.
Imagery data is generally large, and copying that data between machines can take a long time. Because of their potential size, image files referenced by mosaic datasets are never copied even when you choose to copy the data when you publish.
- To save resources on the hosting server, you have created raster tile, vector tile, or scene caches to store in folder or cloud data stores that you control, and you will publish one of the following to reference the appropriate cache: a tile layer, vector tile layer, or scene layer.
- You're working with data or file types that can only be published from a registered data source.
If you publish the following types of data from ArcGIS Pro, you must register your data source with an ArcGIS Server site and publish to that site:
- Utility networks
- Geoprocessing scripts or models
- Dynamic maps
- Geodatabases
- Streaming data (requires an ArcGIS GeoEvent Server site)
- KML
- You don't want ArcGIS to clean up the data when you delete the web layer.
Data that is copied to ArcGIS Enterprise is automatically deleted by the system when the service or portal item associated with the data is deleted. If users only interact with the data through the web layer (in other words, the web layer essentially is the data), you want the data and web layer to be deleted at the same time. However, if the service or portal item is only one way that people access the data, you need the data to remain in the data source. In that case, register the data source with a federated server and publish.
Tip:
If you copy the data when you publish a feature layer and later decide you need to keep the data, export the data from the hosted feature layer in the portal, move the data into a database or enterprise geodatabase you register with ArcGIS Server, and publish.
- Your database connection references a cloud data warehouse.
For information on registering data, see How to register your data with ArcGIS Server.
Copy data
Copying your data is like taking a snapshot of your source data at the time you publish. Unlike items created from registered data, items created from copied data do not receive dynamic updates from the data source as it changes. If you don't need your web layer to access the source data, copying the data when you publish is a suitable workflow.
The following are cases when you may prefer to copy data:
- You're loading a file to the portal and want to publish from it.
- Users outside your firewall need access to the data.
- You and other users will only access the data through the web layer.
- You're using an app or functionality that requires the data to be stored in ArcGIS Data Store.
- You're working with a type of data that requires you to copy the data when you publish.
When you publish 3D multipatch data or vector tiles from ArcGIS Pro, you must copy the cache data when publishing. However, while multipatch caches and vector tile caches are always copied, you have the option to leave the associated feature data in a registered data store or copy it.
- You're publishing from data in a cloud data warehouse, but your ArcGIS Enterprise portal is not in the cloud. In this case, making a snapshot of the data may improve performance when querying the web layer.
Copied data can be either user-managed or managed by ArcGIS.
Data managed by ArcGIS
Data managed by ArcGIS is the hosted data in your ArcGIS Enterprise portal. The services built from hosted data reside on the portal's hosting server*, and the data is always stored in ArcGIS Data Store or on the hosting server.
*Knowledge graph services that users create in ArcGIS Pro run on the portal's ArcGIS Knowledge Server site, not on the hosting server.
Many common workflows and their subsequent outputs in ArcGIS Enterprise depend on the ability to create hosted layers. Hosted layers are not only created as a direct action from uploading a dataset and explicitly choosing to publish it as a new layer. Hosted layers are also created as the output of many actions in ArcGIS Enterprise, such as running analysis tools and as part of distributed collaboration workflows where feature layers are copied.
Where to publish
Members of your organization can publish from files in the portal, from ArcGIS Pro, from data store items, or using ArcGIS API for Python.
Publish from files in the portal
You can publish hosted web layers from certain files you add to your organization.
When you publish from files in the portal, the services for the resultant layers always run on the hosting server.
The following table lists the files you can upload, the hosted web layers you publish from them, and where the layer's data resides:
| Files | Type of layer | Data storage location for the layer |
|---|---|---|
CSV file, Microsoft Excel file, GeoJSON file, zipped shapefile, zipped file geodatabase | ArcGIS Data Store (relational data store) | |
Tile package (.tpk or .tpkx), service definition (.sd) file, or vector tile package (.vtpk) | Files on the hosting server | |
Scene layer package (.slpk) | ArcGIS Data Store (tile cache data store) | |
Image collection (a .zip file containing image files) | Image files converted to a mosaicked image and placed in the raster data store that is registered with the selected federated ArcGIS Image Server site |
For information on publishing each type of layer, see Publish hosted feature layers, Publish hosted tile layers, Publish hosted vector tile layers, Publish hosted scene layers, and Publish hosted imagery layers.
Publish from ArcGIS Pro
When you add layers to maps and scenes in ArcGIS Pro, you can share the layers as web layers. Depending on what type of layer you create, the layer's data is copied to ArcGIS Data Store, a federated server, or the hosting server or it remains in your registered data source. When you choose to keep the data in your registered data source, you also choose the federated ArcGIS Server site on which the service will run.
Copy all data
When you choose an option under Copy all data when publishing from ArcGIS Pro, it means the data used by the resultant web layer will not be the same as the source data in your map or scene. There are certain web layers that require you to copy all data. They include the following:
- Vector tile layers
Vector tile layers are shared (published) from point, line, polygon, or multipoint feature layers in your map. Layer data is cached and the tiles are stored on the portal's hosting server in the same way as when you upload a vector tile package and publish in the portal. The service runs on the hosting server and a tile layer is created in your portal. When publishing from ArcGIS Pro 2.8 or later, you also have the option of creating a feature layer with the vector tile layer. The feature layer can be copied to the relational data store or stay in a registered data store.
See Author a map for vector tile creation in the ArcGIS Pro help for information on how to create a map that meets the requirements for publishing a vector tile layer.
- Tile layers
Tile layers are published from maps in ArcGIS Pro. Publishing a tile layer creates a cached map service on the portal's hosting server and a tile layer item in your organization. The tile caches are stored on the hosting server in the same way as when you publish a tile package or service definition file in the portal. See Author a web map in the ArcGIS Pro help for information on publishing maps and layers as tile layers.
- Scene layers
When you share a scene layer from ArcGIS Pro, both a hosted scene layer and a feature layer are created in your organization. The scene service is always on the hosting server and the scene cache is stored in the tile cache data store in the same way as when you publish a scene layer package in the portal. However, you decide whether the associated feature layer references registered data or the data is copied to the relational ArcGIS Data Store.
When you publish a feature layer in ArcGIS Pro, you have a choice of where to store the data. If you choose Copy all data when publishing, data is copied to the relational ArcGIS Data Store.
When you publish a mosaic dataset or raster dataset as an imagery layer from ArcGIS Pro, you should reference the source data in most cases. For information on publishing imagery layers, see Web imagery layer in the ArcGIS Pro help.
Copy data when you publish web tools
You can share a geoprocessing script or model from ArcGIS Pro to your organization. When you do this, you decide whether to copy the data used in the script or model to the server—which creates a static copy of the data the service uses—or create a reference that the service can access.
You can publish a web tool to the hosting server by copying data if the data size is small. When the data is large, copying data can take a long time, and it is not recommended.
If you publish a web tool with the geoprocessing service on a federated server, the data is stored on the federated server if you choose to copy the data. This creates a geoprocessing service on the selected federated server and a web tool item in your organization.
Publish a hosted map image layer from a hosted feature layer
Starting with ArcGIS Pro 2.5, you can publish a hosted map image layer from a hosted feature layer or hosted spatiotemporal feature layer that you own. The data stays in the relational data store or spatiotemporal big data store, and the hosted map image layer displays that data.
You might do this if you want to make the feature layer data available in a truly read-only format. You can keep your hosted feature layer editable so you can make updates to the data and share the hosted map image layer with a wider audience. Hosted map image layers may perform better when displaying large volumes of data, and they support a broader range of symbology than hosted feature layers. Therefore, you might publish a hosted map image layer from your hosted feature layer to use the map image layer as reference data in maps you share.
See the ArcGIS Pro help for more information on publishing hosted map image layers from hosted feature layers.
Reference registered data
If you want your web layers to reference your source data, you must register your data source with the ArcGIS Server site where you want the service to run. This ensures that the service can access the data. See Manage registered data stores in the ArcGIS Pro help for more information. Starting with ArcGIS Pro 2.5, a data store item is created in the active portal for the registered folder or database data source. Starting with ArcGIS Pro 3.0, you can register a cloud storage location.
When you publish feature layers from a registered database, the data remains in the source database or enterprise geodatabase, and a feature layer item and map image layer item are created in your organization. A map service with feature access enabled is also created on the federated server you selected when you published. If the database connection accesses a cloud data warehouse, the data remains in the source, but only a feature layer item is created in your organization.
For imagery layers, the data remains in the source geodatabase or folder, an image service is created on the federated ArcGIS Image Server site, and an imagery layer item is created in your organization.
To publish a map image layer from a map in ArcGIS Pro, you must register the data source (or sources) with the federated server to which you publish. All the data in the map stays in your registered data source, a map service is created on the federated server, and a map image layer is created in the organization.
For more information on publishing maps, features, and imagery to federated servers, see Layers published to your portal's federated servers.
Publish web tools using referenced data
When you share a geoprocessing script or model from ArcGIS Pro, you can choose to reference registered data rather than copy all data. If you want the tool to reference the data, the data must be in an enterprise geodatabase that you registered with the federated server.
Publish from data store items in the portal
When you or other members of your organization add data store items to the portal, the types of layers you publish depend on the type of data store.
- Database data store items—You, the data store item owner, can bulk publish feature layers and map image layers that reference the data accessed through the data store item if the data source is a database or enterprise geodatabase.
- Folder and cloud data store items—You and anyone you share the data store item with can create imagery layers from folder and cloud storage data store items. The imagery layers reference the image files accessible through the data store. You can also load precreated caches to the folders or cloud storage locations and publish a tile, vector tile, or scene layer that references the cache in those locations.
Publish using ArcGIS API for Python
You can use the Item class in the GIS module of ArcGIS API for Python to publish items to your ArcGIS Enterprise portal using Python scripts and notebooks. See the ArcGIS API for Python sample notebooks for content publishers for scenario-based examples.