Output templates in a big data file share

Note:

At ArcGIS Enterprise 10.9.1 or higher it is recommended that you add or edit big data file shares through your portal contents page instead of ArcGIS Server Manager. Learn more about adding big data file shares in the portal.

Big data file shares are registered as a data store through your portal contents page. A big data file share outlines one or more input datasets using a big data file share manifest, as well as templates for writing results back to your big data file share location. When you write a result back to a big data file share, the big data file share automatically updates to include the new output dataset as an input dataset. This topic describes the formatting of the output templates and their possible values. For an overview of big data file shares, and how input datasets, output templates, and hints files work, see Get started with big data file shares.

Note:

This topic is about editing output templates. To learn about applying a hints file for delimited files, see Hints file.

The output templates are composed of one or more templates. The number of templates depends on the types of files and formatting you want to use to store output results. In the following example, there are three output templates:

"outputTemplates":[
  {.. template1 ..},
  {.. template2 ..},
  {.. template3 ..}
]

In each template, there are five top-level objects that may be applicable. Of these objects, name, title, and format are required.

{
 "name": "ExampleTemplate",
 "title": "My Example Template",
 "format": {},
 "geometry": {},
 "time": {}
}

Name

The name object is required and defines the name of the dataset. The name must be unique within the manifest and cannot include any special characters.

Title

The title object is required and defines the title that is shown when running a tool. The title must be unique within the manifest and can include spaces.

Format

The format object is required and defines the output file type and its format.

Syntax	Example
"format" : { "type" : "< delimited | shapefile | orc | parquet >", "extension" : "< csv | tsv | shp | orc | parquet >", "fieldDelimiter" : "< delimiter >", "recordTerminator: "< terminator >", "quoteChar": "< character for quotes>", "delimited.write.header" : < true | false >, "encoding" : "< encoding format >", "dateFields" : "< date format >" }	Example using a shapefile: "format" : { "type": "shapefile", "extension": "shp", "dateFields": "YYYY/MM/dd HH:mm:ss" } Example using a delimited file: "format" : { "type": "delimited", "extension": "csv", "fieldDelimiter": ",", "recordTerminator": "\n", "quoteChar" "\"", "hasHeaderRow": true, "encoding" : "UTF-8" } Example using a parquet file: "format" : { "type": "parquet" } Example using an ORC file: "format" : { "type": "orc" }
Description
type—A required property that defines the output data type. This can be delimited, shapefile, parquet, or orc. extension—A required property denoting the file extension for delimited files. fieldDelimiter—This is only used and is required when type is delimited. This field represents what separates fields in the delimited file. recordTerminator—This is always set to \n on Linux, and \r\n on Windows. If you modify this option, those defaults are still used. quoteChar—This is only required when type is delimited. The character denotes how quotes are specified in the delimited file. delimited.write.header—This is only required when type is delimited. This property specifies whether the first row in a delimited file is the field names. encoding—This is always set to UTF-8. If you modify this option, that default is still used. dateFields—How fields of type date will be formatted. These are the fields that are not representing the time of feature (instant or interval). See Time formats to learn about formatting date fields. Delimited file settings cannot be created though ArcGIS Server Manager. To modify delimited file settings, edit the templates by browsing to your Administrator Directory for your GeoAnalytics Server and clicking data > items > bigDataFileShares and your big data file share and clicking edit. Make your changes and click update.

Geometry

The geometry object is optional. If it's not supplied, the geometry field will be written in XYZ for points stored in delimited, ORC, and parquet files, and written in WKT for lines and polygons in delimited, ORC, and parquet files. For shapefiles, the fields will be written in a shape field.

Syntax	Example
"geometry" : { "spatialReference" : { "wkid": <wkidNum>, "latestwkid" : <latestWkidNum> }, "encodings":[ { "types" : ["< esriGeometryType >"], "fields" [ { "name": "<fieldName1>", "formats": ["<fieldFormat1>"] }, { "name": "<fieldName2>", "formats": ["<fieldFormat2>"] } }, {...} ] }	Example using a delimited file with x- and y-values: "geometry" : { "spatialReference" : { "wkid": 3857 }, "encodings": [{ "types": ["esriGeometryPoint"], "fields": [{ "name": "XValue", "formats": ["x"] }, { "name": "YValue", "formats": ["y"] } ] }, { "types": ["esriGeometryPolyline", "esriGeometryPolygon" ], "fields": [{ "name": "geometry", "formats": ["geojson"] }] } ] } Example using a delimited file with x-, y-, and z-values: "geometry" : { "spatialReference": { "wkid": 4326 }, "encodings": [{ "types": ["esriGeometryPoint"], "fields": [{ "name": "Longitude", "formats": ["x"] }, { "name": "Latitude", "formats": ["y"] }, { "name": "Height", "formats": ["z"] } ] }] } Example using a .tsv file: "geometry" : { "spatialReference": { "wkid": 3857 }, "encodings": [{ "types": ["esriGeometryPolygon", "esriGeometryPoint"], "fields": [{ "name": "Shapelocation", "formats": ["WKT"] }] }] }
Description
Note: The table above outlines the default options for the geometry object, if you do not specify it, based on geometry and file type. The following properties are listed as required or optional, assuming a geometry is used: spatialReference—A required property denoting the spatial reference for outputting datasets. wkid—A field denoting the spatial reference, where wkid or latestWkid is required for a dataset with a geometry. latestWkid—A field denoting the spatial reference at a given software release, where wkid or latestWkid is required for a dataset with geometry. encodings—This is required. This specifies how to encode the output geometry. The property is composed of two parts: the geometry type and the fields to which the geometry will be written. types—The geometry types specified in this encoding. Options include the following: esriGeometryPoint esriGeometryPolyline esriGeometryPolygon fields—This is required. This specifies the name of one or more fields used to store geometry results, as well as the formats of the fields in which the geometry is stored. name—A required property to name the field used for the geometry of the output dataset. This denotes the name of the field used to represent the geometry. There can be multiple instances of this. formats—A required property for delimited datasets with a spatial representation. This denotes the format of the field used to represent the geometry. There can be multiple formats. The geometry field of shapefiles cannot be specified. The following format types are supported: WKT—The default for lines and polygons stored as delimited, ORC, and parquet files. GeoJSON EsriJSON X, Y,Z—The default for points stored as delimited, ORC, and parquet files. EsriShape—This option is only available for ORC and parquet files. WKB—This option is only available for ORC and parquet files.

Time

The time object is optional. If it is not specified, and the results are time-enabled, the default will be used.

Syntax	Example
"time" : { "timeReference" : { "timeZone" : "<timeZone >" }, "encodings":[{ "types" : ["< instant | interval >",] "fields": [ { "name": "<fieldName1>", "formats": ["<fieldFormat1>"], "role": "< start | end >" } ] }] }	Example using an instant, with multiple formats in the time fields: "time": { "timeReference": { "timeZone": "UTC" }, "encodings": [{ "types": ["instant"], "fields": [{ "name": "iso_time", "formats": [ "yyyy-MM-dd HH:mm:ss", "MM/dd/yyyy HH:mm" ] }] }] } Example using an interval, with multiple fields used for startTime: "time": { "timeReference": { "timeZone": "-0900" }, "encodings": [{ "types": ["interval"], "fields": [{ "name": "time_start", "formats": ["HH:mm:ss"], "role": "start" }, { "name": "date_start", "formats": ["yyyy-MM-dd"], "role": "start" }, { "name": "datetime_ending", "formats": ["yyyy-MM-dd HH:mm:ss"], "role": "end" } ] },{ "types": ["instant"], "fields": [{ "name": "time", "formats": ["HH:mm:ss"], "role": "start" }, { "name": "date", "formats": ["yyyy-MM-dd"] } ] }] }
Description
Note: Since the time object is optional, the following properties are listed as required or optional, assuming that time is used: encodings—This is required if you are formatting time. This specifies how to encode the output time. The property is composed of two parts: the time type and the fields to which the time will be written. types—The geometry types specified in this encoding. Options include the following: Instant—For a single moment in time. Interval—For a time interval represented by a start and stop time. fields—This is required. This specifies the name of one or more fields used to store time results, as well as the formats of the fields in which the time is stored. name—A required property to name the field used for the time for the output dataset. This denotes the name of the field used to represent the time. There can be multiple instances of this. formats—A required field that denotes the format of the field used to represent the time. There may be multiple formats for a single field (as shown above). There can be multiple instances of this object. To learn how fields can be formatted, see Time formats. role—A required field when timeType is interval. It can represent either the startTime or endTime of a time interval. timeReference—A required field if the dataset is time-enabled, denoting the time zone (timeZone). timeZone—A optional field of timeReference that denotes the time zone format of the data. timeZone can be formatted as follows: Using the full name of the time zone: Pacific Standard Time Using the abbreviation of the time zone: PST Using the time zone offset in hours: -0900