Data storage and management is a vital aspect of your ArcGIS Enterprise deployment. It shapes how your organization accesses, manages, contributes, and edits data, and it provides the foundation for how your data can be used. At a high level, there are two ways to think of data management in ArcGIS Enterprise: either user-managed or managed by ArcGIS. In a single ArcGIS Enterprise deployment, you will likely have both types of data.
When you publish a web map, layer, or service to ArcGIS Enterprise, you determine how your data will be managed. The first step in this process is to decide if you'll copy data or register your data with one (or more) of the federated servers that comprise your ArcGIS Enterprise portal.
Register or copy the data
When you publish from ArcGIS Pro or ArcMap, you determine the location of the data used by your web layers. You can either register your data source—in which case the web layers access the data in the data source—or have ArcGIS copy the data to a location managed by ArcGIS, which can be the ArcGIS Data Store, a federated server, or the hosting server.
You can also add files to the portal and publish within the ArcGIS Enterprise portal website. In this case, the data is always copied to one of the data stores provided through ArcGIS Data Store or the hosting server, depending on the type of layer you publish.
Note:
A hosting server is the federated ArcGIS Server site the portal administrator has designated to run the services that power most hosted web layers. This site can run other services, but its intended purpose is to run hosted services. There is only one hosting server per portal.
Register data sources
When you register a data store and publish ArcGIS Server web layers, the web layers always reference the data in the registered data store. If the data in the registered data store changes, you will see those changes in the web layer.
Registered data stores are always user-managed.
The following are cases when registering data is recommended or required:
- You have multiple clients accessing and updating the source data.
If you have apps directly editing the source data, apps editing the source data through services, or conversion or ETL processes that load data from contractors to your source, publish map or feature layers that reference the data source. That way, the people who use the layers can see changes to the data as they are made in the source.
- You use versioned data from an enterprise geodatabase.
If you publish from a map that contains versioned enterprise geodatabase data and you copy the data, the copied data no longer participates in the version. Edits made through the published feature layer cannot take advantage of multiuser editing functionality.
- You use archive-enabled data from an enterprise geodatabase.
Data owners enable archiving so they can see changes in the data over time. If you copy data from the source when you publish a feature layer, it's no longer part of the archive and you cannot see the changes made to the data after it's copied.
- You have large feature classes or feature classes with complex geometries.
The greater the number of features and the more complex the shapes, the longer it takes to copy the data. Examples of complex shapes include polygons or lines with thousands of vertices, such as coast lines or meandering rivers.
- You're publishing imagery data.
Imagery data is generally large, and copying that data between machines can take a long time. Because of their potential size, image files referenced by mosaic datasets are never copied even when you choose to copy the data when you publish.
- You're working with data or file types that can only be published from a registered data source.
If you publish the following types of data from ArcGIS Pro or ArcMap, you must register your data source with an ArcGIS Server site and publish to that site:
- Utility networks (ArcGIS Pro only)
- Geoprocessing scripts or models
- Dynamic maps
- Geodatabase (ArcMap only)
- Streaming data (requires an ArcGIS GeoEvent Server site)
- Schematics (ArcMap only)
- KML
- You don't want ArcGIS to clean up the data when you delete the web layer.
Data that is copied to ArcGIS Enterprise is automatically deleted by the system when the service or portal item associated with the data is deleted. If users only interact with the data through the web layer (in other words, the web layer essentially is the data), you want the data and web layer to be deleted at the same time. However, if the service or portal item is only one way that people access the data, you need the data to remain in the data source. In that case, register the data source with a federated server and publish.
Tip:
If you copy the data when you publish a feature layer and later decide you need to keep the data, export the data from the portal item, move the data into a database or enterprise geodatabase you register with ArcGIS Server, and publish.
For information on registering data, see How to register your data with ArcGIS Server.
Copy data
Copying your data is like taking a snapshot of your source data at the time you publish. Unlike items created from registered data, items created from copied data do not receive dynamic updates from the data source as it changes. If you don't need your web layer to access the source data, copying the data when you publish is a suitable workflow.
The following are cases when you may prefer to copy data:
- You're loading a file to the portal and want to publish from it.
- Users outside your firewall need access to the data.
- You and other users will only access the data through the web layer.
- You're using an app or functionality that requires the data to be stored in ArcGIS Data Store.
- You're working with a type of data that requires you to copy the data when you publish.
When you publish 3D multipatch data or vector tiles from ArcGIS Pro, you must copy the data when publishing. However, while multipatch caches are always copied, you have the option to leave the associated feature data in a registered data store or copy it.
Copied data can be either user-managed or managed by ArcGIS.
Data managed by ArcGIS
Data managed by ArcGIS is the hosted data in your ArcGIS Enterprise portal. The services built from hosted data always reside on the portal's hosting server, and the data is always stored in ArcGIS Data Store or on the hosting server.
Many common workflows and their subsequent outputs in ArcGIS Enterprise depend on the ability to create hosted layers. Hosted layers are not only created as a direct action from uploading a dataset and explicitly choosing to publish it as a new layer. Hosted layers are also created as the output of many actions in the ArcGIS Enterprise portal, such as running analysis tools and as part of distributed collaboration workflows where feature layers are copied.
Where to publish
Members of your organization can publish from files in the portal, from ArcGIS Pro, from ArcMap, or using ArcGIS API for Python.
Publish from files in the portal
You can publish hosted web layers from certain files you add to your portal.
When you publish from files in the portal, the services for the resultant layers always run on the portal's hosting server.
The following table lists the files you can upload, the hosted web layers you publish from them, and where the layer's data resides:
| Files | Type of layer | Data storage location for the layer |
|---|---|---|
CSV file, Microsoft Excel file, GeoJSON file, zipped shapefile, zipped file geodatabase | ArcGIS Data Store (relational data store) | |
Tile package (.tpk), service definition (.sd) file, or vector tile package (.vtpk) over 2 GB | Files on the hosting server | |
Scene layer package (.slpk) | ArcGIS Data Store (tile cache data store) | |
Image collection (a .zip file containing image files) | Image files converted to a mosaicked image and placed in the raster data store that is registered with the selected federated ArcGIS Image Server site |
For information on publishing each type of layer, see Publish hosted feature layers, Publish hosted tile layers, Publish hosted vector tile layers, Publish hosted scene layers, and Publish hosted imagery layers.
Publish from ArcGIS Pro
When you add layers to maps and scenes in ArcGIS Pro, you can share the layers as web layers. Depending on what type of layer you create, the layer's data is copied to ArcGIS Data Store, a federated server, or the hosting server or it remains in your registered data source. When you choose to keep the data in your registered data source, you also choose the federated ArcGIS Server site on which the service will run.
Copy all data
When you choose Copy all data when publishing from ArcGIS Pro, it means the data used by the resultant web layer will not be the same as the source data in your map or scene. There are certain web layers that require you to copy all data. They include the following:
- Vector tile layers
Vector tile layers are shared (published) from point, line, polygon, or multipoint feature layers in your map. Layer data is cached and the tiles are stored on the portal's hosting server in the same way as when you upload a vector tile package and publish in the portal website. The service runs on the hosting server and a tile layer is created in your portal. See Author a map for vector tile creation in the ArcGIS Pro help for information on how to create a map that meets the requirements for publishing a vector tile layer.
- Tile layers
Tile layers are published from maps in ArcGIS Pro. Publishing a tile layer creates a cached map service on the portal's hosting server and a tile layer item in your portal. The tile caches are stored on the hosting server in the same way as when you publish a tile package or service definition file in the portal website. See Author a web map in the ArcGIS Pro help for information on publishing maps and layers as tile layers.
- Scene layers
When you share a scene layer from ArcGIS Pro, both a hosted scene layer and a feature layer are created in your portal. The scene service is always on the hosting server and the scene cache is stored in the tile cache data store in the same way as when you publish a scene layer package in the portal website. However, you decide whether the associated feature layer references registered data or the data is copied to the relational ArcGIS Data Store.
When you publish a feature layer in ArcGIS Pro, you have a choice of where to store the data. If you choose Copy all data when publishing, data is copied to the relational ArcGIS Data Store.
When you publish a mosaic dataset or raster dataset as an imagery layer from ArcGIS Pro, you should reference the source data in most cases. For information on publishing imagery layers, see Web imagery layer in the ArcGIS Pro help.
Tip:
If you are transitioning from ArcMap, the following table lists the publishing workflows in ArcMap that are equivalent to publishing and copying data in ArcGIS Pro:
| Layer published with Copy all data in ArcGIS Pro | Equivalent publishing workflow in ArcMap |
|---|---|
Publish a vector tile layer to an ArcGIS Enterprise portal | There is no equivalent ArcMap workflow. |
Publish a tile layer to an ArcGIS Enterprise portal | Publish a service with tiled mapping capabilities to My Hosted Services when signed in to an ArcGIS Enterprise portal. |
Publish a scene layer to an ArcGIS Enterprise portal | There is no equivalent ArcMap workflow. |
Publish a feature layer to an ArcGIS Enterprise portal | Publish a map service with feature access capabilities to My Hosted Services when signed in to an ArcGIS Enterprise portal. |
Copy data when you publish web tools
You can share a geoprocessing script or model from ArcGIS Pro to your portal. When you do this, you decide whether to copy the data used in the script or model to the server—which creates a static copy of the data the service uses—or create a reference that the service can access. If you choose to copy the data, the data is stored on the federated server.
Publishing a web tool creates a geoprocessing service on the selected federated server and a web tool item in the portal.
Reference registered data
If you want your web layers to reference your source data, you must register your data source with the ArcGIS Server site where you want the service to run. This ensures that the service can access the data. See Manage registered data stores in the ArcGIS Pro help for more information.
When you publish feature layers from a registered database, the data remains in the source database or enterprise geodatabase, and a feature layer item and map image layer item are created in the portal. A map service with feature access enabled is also created on the federated server you selected when you published.
For imagery layers, the data remains in the source geodatabase or folder, an image service is created on the federated ArcGIS Image Server site, and an imagery layer item is created in the portal.
If you want to publish a map image layer from a map in ArcGIS Pro, you must register the data source (or sources) with the federated server to which you publish. All the data in the map stays in your registered data source, a map service is created on the federated server, and a map image layer is created in the portal.
Tip:
If you are transitioning from ArcMap, the following table lists the publishing workflows in ArcMap that are equivalent to publishing referenced data in ArcGIS Pro:
| Layer published with Reference registered data in ArcGIS Pro | Equivalent publishing workflow in ArcMap |
|---|---|
Publish a feature layer to a federated ArcGIS GIS Server site. | Publish a map service with feature access capabilities to a GIS Server that is licensed as an ArcGIS GIS Server and is federated with a portal. |
Publish an imagery layer to a federated ArcGIS Image Server site. | Publish a mosaic or raster dataset to a GIS Server that is licensed for ArcGIS Image Server and is federated with a portal. |
Publish a map image layer to a federated ArcGIS GIS Server site. | Publish a map service to a GIS Server that is licensed as an ArcGIS GIS Server and is federated with a portal. |
For more information on publishing maps, features, and imagery to federated servers, see Layers published to your portal's federated servers.
Publish web tools using referenced data
When you share a geoprocessing script or model as a web tool from ArcGIS Pro, you can choose to reference registered data rather than copying all data. If you want the tool to reference the data, the data must be in an enterprise geodatabase that you registered with the federated server.
Publish from ArcMap
When you publish from ArcMap, you can publish to the portal or to an ArcGIS Server site. If the ArcGIS Server site is federated with the portal, a portal item is created when you publish.
Publish to the portal
To publish hosted feature layers or hosted tile layers to your ArcGIS Enterprise portal from ArcMap, configure ArcGIS Desktop to connect to your portal, sign in to the portal from your map, and publish a service with feature access capabilities or a service with tiled mapping capabilities to My Hosted Services.
When you publish a service with feature access capabilities to My Hosted Services, the data is copied to the relational ArcGIS Data Store, a hosted feature service is created on the portal's hosting server, and a hosted feature layer item is created in the portal. When you publish a service with tiled mapping capabilities to My Hosted Services, cached map tiles and a hosted map service are created on the hosting server and a hosted tile layer is created in the portal.
See Use ArcGIS Desktop with your portal for information on connecting and publishing to a portal from ArcMap.
Publish to a federated ArcGIS Server site
You can create a publisher GIS Server connection to a federated server from ArcMap and publish the services described in ArcGIS Server web services. If you publish feature data as a map service with feature access, WFS access, or WMTS access, your source database must be registered with the federated ArcGIS Server site. If it isn't, you receive a warning that the data will be copied when you publish. If you proceed to publish, the data is copied to the federated server. If you publish a map service with feature access (a feature service) and the data is copied to the federated server, you cannot edit the data in the feature service or the feature layer item in the portal.
Publish using ArcGIS API for Python
You can use the Item class in the GIS module of ArcGIS API for Python to publish items to your ArcGIS Enterprise portal using Python scripts and notebooks. See the ArcGIS API for Python sample notebooks for content publishers for scenario-based examples.