The Poll an External Website for JSON input connector can be used to retrieve and adapt event data records, formatted as generic JSON, from an external website. For examples illustrating how ArcGIS GeoEvent Server expects typical generic JSON data records to be formatted, see the Esri Community blog JSON Data Structures - Working with Hierarchy and Multicardinality.
When multiple event data records are returned as a batch, they should be organized in an array as illustrated below. The Generic JSON inbound adapter will manage each object in the array as a separate event record.
Usage notes
Keep the following in mind when working with the Poll an External Website for JSON input connector:
- Use this input connector to query a web server or web service for data, formatted as generic JSON, and adapt it to create event data records for processing.
- This input connector pairs the Generic JSON inbound adapter with the HTTP inbound transport.
- The adapter interprets generic JSON as opposed to feature JSON or GeoJSON.
- A generic JSON record does not need to contain data that represents a geometry.
- Typical event data, organized in a JSON array, is illustrated above.
- The adapter will manage both single JSON records and JSON records organized in an array.
- The HTTP Method parameter specifies the type of HTTP request used to query a web service for data. The default is to use an HTTP/GET request.
- The adapter supports the ability to construct a point geometry from x, y, and z attribute values.
- Query parameters can be specified using the Parameters parameter to help select data of interest.
- Specifying query parameters as part of an HTTP/GET request can improve performance by narrowing the number of items returned.
- The name and values of any specified query parameter depend on what requests are supported by the external web service.
- This input connector assumes the external web server or web service will accept HTTP/GET requests without requiring authentication. For external websites requiring authentication, consider creating a custom input connector that pairs the Generic JSON inbound adapter with an HTTP inbound transport that supports basic, certificate, or OAuth authentication.
- The Receive New Data Only parameter is used to request that a feed return only previously unobserved or newly updated data. This capability depends on the data feed correctly incorporating Last-Modified into the header of its response to an HTTP request. If the response header does not indicate a date or time the content was last updated, GeoEvent Server will assume that data in the response represents new or recently updated data. You can read more about the use of Last-Modified in response headers in the RFC 7232 HTTP specification.
- This input connector includes a Learning Mode parameter, which can be useful when you need to allow the input connector to modify a GeoEvent Definition it has constructed. The purpose of this parameter is to temporarily accept that event data received will have a variable schema or data structure. The input connector will use a sample of received data records to learn more about the variable data structure and append new, previously unobserved attribute fields to an existing GeoEvent Definition.
Allowing a GeoEvent Definition to be changed in real-time can negatively impact the design of real-time analysis in a GeoEvent Service. It is recommended that if schema variance is expected in your inbound event data, you use learning mode for as brief a period of time as possible to produce a GeoEvent Definition that supports all expected variants of your inbound data. You can then turn off learning mode, and copy and tailor the autogenerated GeoEvent Definition for production deployment.
Parameters
The following are the parameters for the Poll an External Website for JSON input connector:
| Parameter | Description |
|---|---|
Name | A descriptive name for the input connector used for reference in GeoEvent Manager. |
URL | The base URL of the external website hosting the data as JSON. GeoEvent Server will issue a poll request to the URL to retrieve the JSON data. |
Create GeoEvent Definition | Specifies whether a new or an existing GeoEvent Definition is used for the inbound event data. A GeoEvent Definition is required for GeoEvent Server to interpret the inbound event data attribute fields and data types.
|
GeoEvent Definition Name (New) (Conditional) | The name assigned to a new GeoEvent Definition. If a GeoEvent Definition with the specified name already exists, the existing GeoEvent Definition is used. The first data record received is used to determine the expected schema of subsequent data records, and a new GeoEvent Definition is created based on that first data record's schema. This parameter is shown when Create GeoEvent Definition is set to Yes and is hidden when it's set to No. |
GeoEvent Definition Name (Existing) (Conditional) | The name of an existing GeoEvent Definition to use when adapting received data to create event data for processing by a GeoEvent Service. This parameter is shown when Create GeoEvent Definition is set to No and is hidden when it's set to Yes. |
HTTP Method | Specifies the HTTP request method to use when GeoEvent Server polls the external server providing the data feed. The default is to use an HTTP/GET request.
Technically, a polling input should always use an HTTP/GET request when querying an external web server or web service for data. Changing this parameter to specify either POST or PUT will reveal additional properties that are not relevant to this input connector. |
Post From Post Body Post body MIME Type Post Parameters (Conditional) | These parameters are hidden by default when HTTP Method is set to Get. For more information, refer to the Considerations and limitations section below. |
Default Spatial Reference | The well-known ID (WKID) of a spatial reference to be used when a geometry is constructed from attribute field values whose coordinates are not latitude and longitude values for an assumed WGS84 geographic coordinate system, or when geometry strings are received that do not include a spatial reference. A well-known text (WKT) value or the name of an attribute field containing the WKID or WKT can also be specified. |
Receive New Data Only | Specifies whether the adapter searches for a Last-Modified value in the data feed response’s header and attempts to use this value to avoid processing data that's already been received but not updated. The parameter’s name can be misleading. The transport always receives whatever data a feed includes in its response to the connector’s query. When a Receive New Data Only value is specified, however, the transport caches the date/time of its last request and only relays data it receives to the adapter if the response from the data feed includes a Last-Modified value that is greater than the transport’s cached value. Specifying a Last-Modified value is optional. In practice, not many data feeds include it in their responses. The default for this parameter is Yes, but if no Last-Modified value is found, the parameter is ignored as the date/time needed to determine whether data in the response is new or updated is not available.
For more information about using a Last-Modified value in response headers, refer to the RFC 7232 HTTP specification. |
Frequency (in seconds) | The interval (in seconds) an external web server or web service is polled for data. The default is 10 seconds (one poll request is made every 10 seconds). |
Header Parameter Name:Value List | The parameters to include in a request’s HTTP header when an external web server or web service is polled. The header parameters should be specified using Name:Value pairs (colon included) separated by commas. |
Construct Geometry from Fields | Specifies whether the input connector will construct a point geometry using coordinate values received as attributes. The default is No.
|
X Geometry Field (Conditional) | The attribute field in the inbound event data containing the x-coordinate part (for example, horizontal or longitude) of a point location. This parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when it's set to No. |
Y Geometry Field (Conditional) | The attribute field in the inbound event data containing the y-coordinate part (for example, vertical or latitude) of a point location. This parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when it's set to No. |
Z Geometry Field (Conditional) | The attribute field in the inbound event data containing the z-coordinate part (for example, depth or altitude) of a point location. If no value is provided, the z-value is omitted, and a 2D point geometry is constructed. This parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when it's set to No. |
JSON Object Name | The name of a JSON element that can be used as the root node of a substructure within the received JSON data. When JSON Object Name is used to specify a JSON element by name, the adapter searches for substructures whose object name matches the specified element name. Only data within the identified substructure is considered. When left blank, which is the default, the uppermost JSON object is used as the root of the entire JSON structure. |
Parameters | A query parameter (or multiple query parameters) that are appended to the base URL. Query parameters are specific to each web service and are not supported by all services. When supported, query parameters help specify data of particular interest. This parameter has no default value. Query parameters are typically incorporated into the URL parameter. |
Use URL Proxy | Specifies whether GeoEvent Server uses a URL proxy when making HTTP requests. The default is No.
|
URL Proxy (Conditional) | The URL proxy that is used when making an HTTP request. This parameter is shown when Use URL Proxy is set to Yes and is hidden when it's set to No. |
Expected Date Format | The pattern used to match expected string representations of date/time values and convert them to Java Date values. The pattern's format follows the Java SimpleDateFormat class convention. While the preferred pattern for date/time values in GeoEvent Server is the ISO 8601 standard, several string representations of date/time values commonly recognized as date values can be converted to Java Date values without specifying an Expected Date Format value. These include the following:
If the date/time values received use a convention other than one of those listed above, you must specify an expected date format pattern so GeoEvent Server can adapt the date/time values. |
Acceptable MIME Types (Client Mode) (Conditional) | Specifies the MIME types this connector accepts when receiving a response to its query to a data feed. The external web service’s response specifies the actual MIME types for any data sent to GeoEvent Server. This connector supports the following MIME types, which are the default for this parameter: text/json,application/json For more information about official MIME types, refer to Internet Assigned Numbers Authority (IANA). For more information about the MIME types not used by ArcGIS Server, refer to MIME types not used by ArcGIS Server. This parameter is shown when HTTP Method is set to Get and is hidden when it's set to either Post or Put. |
Acceptable MIME Types (Server Mode) (Conditional) | Specifies the MIME types the input connector accepts. Use commas to separate values, for example, text/plain,application/xml,text/xml. MIME types typically consist of a type and subtype structure and are used when a server must specify the internet media types of data it accepts from clients. The default is application/json, which is the official MIME type for the exchange of JSON between a client and a server. For more information about official MIME types, refer to Internet Assigned Numbers Authority (IANA). For more information about the MIME types not used by ArcGIS Server, refer to MIME types not used by ArcGIS Server. Note:This parameter is only available when the connector is configured in server mode. |
HTTP Timeout (in seconds) | The maximum time (in seconds) the input connector waits for a response from the external web service it has queried. The value must be an integer between 1 and 2147483. The default value is 30 seconds (the HTTP request times out after 30 seconds if no response is received). |
Learning Mode | Specifies whether Learning Mode is active or unavailable. When Learning Mode is set to Yes, the inbound adapter appends new fields to a GeoEvent Definition it has created and is maintaining.
Learning Mode can be useful when you need to allow the input connector to modify a GeoEvent Definition it has constructed. The purpose of this parameter is to temporarily accept that event data received has a variable schema or data structure. The input connector uses a sample of received data records to identify more about the variable data structure and append new, previously unobserved, attribute fields to an existing GeoEvent Definition. Allowing a GeoEvent Definition to be changed on the fly can adversely impact the design of real-time analytics in a GeoEvent Service. If schema variance is expected in the inbound event data, it is recommended that you use Learning Mode for as brief a period of time as possible to produce a GeoEvent Definition that supports all expected variants of the inbound data. Learning mode can then be turned off and the automatically generated GeoEvent Definition copied and tailored for production deployment. |
Considerations and limitations
Always use an HTTP/GET request, which does not include a Content-Body, for polling input, and use query parameters to specify data of interest. The parameters below are only applicable to output connectors that send data to an external web server or web service. Ensure that the HTTP Method parameter is set toHTTP/GET (the default) for the parameters below to remain hidden.
| Parameter | Description |
|---|---|
Post From (Conditional) | Specifies whether this connector uses an HTTP/POST or HTTP/PUT request.
This parameter is shown when HTTP Method is set to either Post or Put and is hidden when it's set to Get. |
Post Body (Conditional) | The data that is included in the Content-Body of an HTTP/POST or HTTP/PUT request. This parameter is shown when Post From is set to Content Body and is hidden when it's set to Parameters. |
Post body MIME Type (Conditional) | The MIME type of data that is included in the HTTP request’s Content-Body. MIME types typically consist of a type and subtype structure and are used to specify the internet media types used and accepted by an HTTP client and server. This parameter is shown when Post From is set to Content Body and is hidden when it's set to Parameters. |
Post Parameters (Conditional) | The parameters that are included as part of the HTTP request. Specify the Parameters value using Name:Value pairs (colon included) separated by commas. This parameter has no default value. This parameter is shown when Post From is set to Parameters and is hidden when it's set to Content Body. |