The Watch a Folder for New JSON Files input connector can be used to read and adapt event data, formatted as generic JSON, from a system file. For examples illustrating how ArcGIS GeoEvent Server expects typical generic JSON data records to be formatted, refer to the Esri Community blog JSON Data Structures - Working with Hierarchy and Multicardinality.
When multiple event data records are specified in a generic JSON file, organize them in an array as illustrated below. The Generic JSON inbound adapter will manage each object in the array as a separate event record. The JSON illustrated below will be adapted to produce three individual event data records for a GeoEvent Service to process.
Usage notes
Keep the following in mind when working with the Watch a Folder for New JSON Files input connector c:
- Use this input connector to read event data, formatted as generic JSON, from a system file and adapt it to create event data records for processing.
- This input connector pairs the Generic JSON inbound adapter with the File inbound transport.
- The input connector watches the specified system folder and will read an entire file as soon as the file appears in the folder.
- The entire file’s content will be reread if changes are made to the file and are saved.
- All files in a watched folder will be reread, from the beginning of the files, in the following situations:
- The input connector’s parameters are updated and saved.
- The input connector is stopped and restarted (or the ArcGIS GeoEvent Server service is restarted).
- A generic JSON record does not need to contain data that represents a geometry.
- The adapter supports the ability to construct a point geometry from x, y, and z attribute values.
- The registered server folder, specified in the Input Folder Data Store parameter, can be specified using either an absolute path or UNC path. If a UNC path is used, the Windows service account running GeoEvent Server needs read/write permission to the folder.
- It is recommended that you use absolute paths—for example, C:\GeoEvent\input—for the Input Folder Data Store parameter.
- The Input Directory parameter allows you to specify a subfolder relative to the registered server folder.
- The Include Subfolders parameter allows you to specify whether folders beneath the folder specified in the Input Folder Data Store parameter should be searched recursively. Often, organizing data with different schemas into different folders, and changing Include Subfolders from its default to disable recursive search, allows a more direct and simpler configuration of this input connector.
- This input connector includes a Learning Mode parameter, which can be useful when you need to allow the input connector to modify a GeoEvent Definition it has constructed. The purpose of this parameter is to temporarily accept that event data received will have a variable schema or data structure. The input connector will use a sample of received data records to learn more about the variable data structure and append new, previously unobserved attribute fields to an existing GeoEvent Definition.
Allowing a GeoEvent Definition to be changed in real-time can negatively impact the design of real-time analytics in a GeoEvent Service. It is recommended that if schema variance is expected in your inbound event data, you use learning mode for as brief a period of time as possible to produce a GeoEvent Definition that supports all expected variants of your inbound data. You can then turn off the learning mode, and copy and tailor the autogenerated GeoEvent Definition for production deployment.
- Network latency can adversely impact the ability of GeoEvent Server to retrieve high volumes of event data.
Parameters
The following are the parameters for the Watch a Folder for New JSON Files input connector c:
| Parameter | Description |
|---|---|
Name | A descriptive name for the input connector used for reference in GeoEvent Manager. |
Input Folder Data Store | The registered system folder under which files reside. |
Input Directory | A subfolder directly under the registered system folder. Input Directory should be left blank if no subfolder under the registered system folder exists. |
Input File Filter | A regular expression pattern used to identify files appropriate for this input to ingest and adapt to create event data records for processing. The default is .*\.json, which matches any file name (.*) ending with the literal suffix (.json). While this parameter is not required and can be left blank, it is recommended that you specify a pattern that matches the file name of any file whose schema matches the GeoEvent Definition this input has been configured to use and exclude files (by name) that you do not want the input to ingest. |
Default Spatial Reference | The well-known ID (WKID) of a spatial reference to be used when a geometry is constructed from attribute field values whose coordinates are not latitude and longitude values for an assumed WGS84 geographic coordinate system, or when geometry strings are received that do not include a spatial reference. A well-known text (WKT) value or the name of an attribute field containing the WKID or WKT can also be specified. |
Include Subfolders | Specifies whether subfolders under Input Folder Data Store and Input Directory (optional) are used for files. The default is Yes; however, organizing data with different schemas into different folders and changing this parameter to No to disable recursive search allows a simpler configuration.
|
Delete Files After Processing | Specifies whether the files in the registered system folder are deleted after their content has been processed. Even if a file's content cannot be adapted, no event records are created, and no real-time event processing occurs, the inbound transport still deletes a file whose contents were successfully read. The default is No.
Files that are not deleted are reread from the beginning of the file if the input connector's properties are changed and saved or if the input is stopped and restarted, for example, if the ArcGIS GeoEvent Server service is restarted. |
Create GeoEvent Definition | Specifies whether a new or an existing GeoEvent Definition is used for the inbound event data. A GeoEvent Definition is required for GeoEvent Server to interpret the inbound event data attribute fields and data types.
|
GeoEvent Definition Name (New) (Conditional) | The name assigned to a new GeoEvent Definition. If a GeoEvent Definition with the specified name already exists, the existing GeoEvent Definition is used. The first data record received is used to determine the expected schema of subsequent data records, and a new GeoEvent Definition is created based on that first data record's schema. This parameter is shown when Create GeoEvent Definition is set to Yes and is hidden when it's set to No. |
GeoEvent Definition Name (Existing) (Conditional) | The name of an existing GeoEvent Definition to use when adapting received data to create event data for processing by a GeoEvent Service. This parameter is shown when Create GeoEvent Definition is set to No and is hidden when it's set to Yes. |
Expected Date Format | The pattern used to match expected string representations of date/time values and convert them to Java Date values. The pattern's format follows the Java SimpleDateFormat class convention. While the preferred pattern for date/time values in GeoEvent Server is the ISO 8601 standard, several string representations of date/time values commonly recognized as date values can be converted to Java Date values without specifying an Expected Date Format value. These include the following:
If the date/time values received use a convention other than one of those listed above, you must specify an expected date format pattern so GeoEvent Server can adapt the date/time values. |
Construct Geometry from Fields | Specifies whether the input connector will construct a point geometry using coordinate values received as attributes. The default is No.
|
X Geometry Field (Conditional) | The attribute field in the inbound event data containing the x-coordinate part (for example, horizontal or longitude) of a point location. This parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when it's set to No. |
Y Geometry Field (Conditional) | The attribute field in the inbound event data containing the y-coordinate part (for example, vertical or latitude) of a point location. This parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when it's set to No. |
Z Geometry Field (Conditional) | The attribute field in the inbound event data containing the z-coordinate part (for example, depth or altitude) of a point location. If no value is provided, the z-value is omitted, and a 2D point geometry is constructed. This parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when it's set to No. |
JSON Object Name | The name of a JSON element that can be used as the root node of a substructure within the received JSON data. When JSON Object Name is used to specify a JSON element by name, the adapter searches for substructures whose object name matches the specified element name. Only data within the identified substructure is considered. When left blank, which is the default, the uppermost JSON object is used as the root of the entire JSON structure. |
Learning Mode | Specifies whether Learning Mode is active or unavailable. When Learning Mode is set to Yes, the inbound adapter appends new fields to a GeoEvent Definition it has created and is maintaining.
Learning Mode can be useful when you need to allow the input connector to modify a GeoEvent Definition it has constructed. The purpose of this parameter is to temporarily accept that event data received has a variable schema or data structure. The input connector uses a sample of received data records to identify more about the variable data structure and append new, previously unobserved, attribute fields to an existing GeoEvent Definition. |
Read File as Text Lines | Specifies how the content of the file is read and parsed. The default is Yes.
When working with a text file with one or more JSON array objects, it is recommended that you read the entire file's content rather than read the file's individual lines of text. JSON documents are best read as a single document. |
Max Number of Lines per Batch (Conditional) | The maximum number of lines that are read from the file in each batch or interval. The default is 1000 lines. Reduce this value if each event record contains many attributes to limit the amount of data sent to the Text adapter as a batch. This parameter is shown when Read File as Text Lines is set to Yes and is hidden when it's set to No. |
Batch Flush Interval (milliseconds) (Conditional) | The number of milliseconds to wait before reading another batch of lines from the file. The default is 500. Reduce this value if file size is expected to be very large or if additional time is necessary to process each batch of lines retrieved from a file. This parameter is shown when Read File as Text Lines is set to Yes and is hidden when it's set to No. |
Number of Lines to Skip from Start of File (Conditional) | The number of lines that are skipped from the start of the file. The default is 0. Increase this value to skip a specific number of lines—for example, header lines specifying attribute field names or data types, because they do not contain actual data for processing. This parameter is shown when Read File as Text Lines is set to Yes and is hidden when it's set to No. |
As GeoJSON | Specifies whether the incoming geometry is parsed as a GeoJSON geometry object rather than as a feature JSON. For the default value, it is assumed that when a geometry is received as a string, the value is a feature JSON as described in Geometry objects.
Refer to the GeoJSON specification for valid and complete examples of GeoJSON geometry object formats. A single GeoJSON geometry object is assumed to have a member with the name coordinates whose value is an array. The structure of the array is determined by the type of geometry. |