The Field Enricher (Feature Service) Processor is used to enrich (or join) event records with attribute data from a published feature service’s feature layer or nonspatial table. The attributes are appended as new fields in the processed event records.
Examples
The following are example uses of the Field Enricher (Feature Service) Processor:
- The Field Enricher (Feature Service) Processor can be used to give nonspatial data a spatial component. For example, attribute data from a sensor can be enriched with a geometry (location) coming from a related feature layer. This allows the ability to visualize the sensor on a map as it receives real-time attribute updates. The sensor can then be symbolized based on the changing attributes.
- Using the Field Enricher (Feature Service) Processor, real-time vehicle data from an automatic vehicle location (AVL) feed can be enriched with daily driver assignment information stored in a related feature layer. As drivers are assigned vehicles for the day, the real-time location and status of the vehicle can be enriched with the corresponding driver’s information from the feature layer. This is useful for monitoring incidents in real time.
Usage notes
Keep the following in mind when working with the Field Enricher (Feature Service) Processor:
- The Field Enricher (Feature Service) Processor requires a registered ArcGIS Server connection be specified, including the service folder, the feature service name, and the target layer in the feature service. The data types for each appended field will be carried over from the enrichment source—they are not specified when configuring the processor.
- Event record enrichment relies on a table join. You can specify a field name from the feature service's table and the name of the field on which a join can be performed. While the actual field name from the feature service's table must be provided, the field on which the join will be performed can be specified using either the name of the field or a tag applied to a field in the GeoEvent Definition associated with the event being processed.
- A comma-separated list of the fields to be included in the enrichment can be constructed by selecting fields or entering them manually. Optionally, specify the tags GeoEvent Server should apply to each new field it creates as a second list of comma-separated values.
- The processor works by caching polled feature records. This improves performance and reduces the total number of requests made to the feature service each time an event record is processed. As an event record is received, the processor polls the feature service for the corresponding feature record. Once polled, the feature record is cached and reused based on the duration of time configured by the Cache Refresh Time Interval (minutes) parameter. The total number of feature records that can be cached is controlled by the Maximum Number of Feature Records parameter.
- GeoEvent data enrichment alters the event record's schema, which requires GeoEvent Server to create a new GeoEvent Definition. The new GeoEvent Definition will be managed by GeoEvent Server and deleted if changes are made to the processor or the GeoEvent Service in which the processor is used.
- ArcGIS Enterprise, ArcGIS Online, and ArcGIS Server (stand-alone) map services or feature services can be used with the Field Enricher (Feature Service) Processor.
- When identifying an existing field to use in the GeoEvent Join Field parameter, it is not required to specify a GeoEvent Definition first. Choosing a GeoEvent Definition from the Definition menu only narrows the list of available fields to choose from in the Field menu.
Parameters
The following are the parameters for the Field Enricher (Feature Service) Processor:
| Parameter | Description |
|---|---|
|
Name |
A descriptive name for the processor used for reference in GeoEvent Manager. |
|
Processor |
Specifies the processor selected. |
| Registered Server Connection | The ArcGIS Server, ArcGIS Enterprise, or ArcGIS Online connection registered with GeoEvent Server as a data store. Registered server connections cache information about feature services, feature layers, and layer properties. Provide the registered data store connection containing the feature service used for event data enrichment. Nota:The processor allows you to register a new ArcGIS Server, ArcGIS Enterprise, or ArcGIS Online data store connection by clicking Register ArcGIS Server. Alternatively, a new data store connection can be registered in GeoEvent Manager by browsing to Site > Data Stores. |
Reference to Layer Type | Defines the layer type to reference.
|
| Folder | The ArcGIS Server services folder or ArcGIS Enterprise portal/ArcGIS Online content item folder where the map or feature service for event record enrichment is located. |
| Service | The name of the feature service from which feature records will be polled. |
| Layer | The name of the feature layer from which feature records will be polled. |
Service Layer URL (Conditional) | The URL to the feature service layer to use for enrichment. The parameter is shown when Reference to Layer Type is set to Service Layer URL and is hidden when set to Browse to Layer. |
| Feature Layer Join Field | The name of the field from the feature layer used to perform an attribute join with the processed event records. Joins are made on the corresponding data value in this field and the field specified in the GeoEvent Join Field parameter. |
| Target Fields | The target fields used for the enriched data obtained from the feature layer. The default is New Fields.
|
| Enrichment Fields | Specifies the fields from the feature layer to use to enrich (join to) processed event records. Use a comma-separated list, with no spaces, to choose more than one field for event record enrichment (for example, DriverName,Driver_ID,Route). Nota:Use Select Fields as an alternative method for choosing fields from the feature layer. |
| GeoEvent Join Field | The name of the field from the event record used to perform an attribute join with data from the feature layer. Joins are made on the corresponding data value in this field and the field selected in the Feature Layer Join Field parameter. Nota:Use the Definition menu to identify the GeoEvent Definition of the inbound event record. Choosing a GeoEvent Definition will limit the number of fields to choose from. Use the Field menu to identify the name of an existing field to be used for the join. |
| Field Tags (Conditional) | The tag, or a comma-separated list of tags, to apply to the new fields appended to the processed event record. The order of the tags must match the order of the specified enrichment fields (for example, GEOMETRY,TRACK_ID,TIME_START). Nota:The tags must already exist in GeoEvent Manager, the processor will not create new tags. The parameter is shown when Target Fields is set to New Fields and is hidden when set to Existing Fields. |
| New GeoEvent Definition Name (Conditional) | The name assigned to the new GeoEvent Definition. The new GeoEvent Definition will combine the schema of the inbound event record with the new fields used for storing the enrichment values from the feature layer. The parameter is shown when Target Fields is set to New Fields and is hidden when set to Existing Fields. |
| Cache Refresh Time Interval | Specifies the amount of time individual feature records will be stored in memory (cached) after being used to enrich an event record. Cached feature records are used to enrich additional corresponding event records. Once the cache refresh time interval for a feature record is exceeded, the processor will clear the feature record from its cache to make room for additional feature records. A new request will be made to the feature service for the feature record upon receiving another event record requiring a join. The default is 1. Nota:Cached feature records are stored in memory. |
Cache Refresh Time Units | Specifies the unit of time for the Cache Refresh Time Interval parameter. The default is minutes. |
| Maximum Number of Feature Records | Specifies the maximum number of feature records to be stored in memory (cached). When the number of feature records exceeds the value specified in this parameter, the oldest feature records in the cache are cleared to make room for the new feature records. This process occurs regardless of the value specified in the Cache Refresh Time Interval (minutes) parameter. The default is 1000. |
Considerations and limitations
There are several considerations to keep in mind when using the Field Enricher (Feature Service) Processor:
- Consider finding a balance when adjusting the cache settings for this processor. An increase to the time interval that feature records are cached will result in a reduction in the number of requests made to the feature service. This will improve the total throughput performance of the processor since feature records will be joined from memory instead of from repeated network requests. The trade-off with this approach is that the feature records used for joins will be inherently older. It is recommended that the value set in the Cache Refresh Time Interval (minutes) parameter be increased when the feature records are not updating often. For example, if the feature records are updated every 72 hours, consider adjusting the cache refresh time to be nearly the same, rather than every minute (the default). This will reduce the number of service requests made for feature data that is otherwise not changing and can be stored in memory.
- Increasing the value in the Maximum Number of Feature Records parameter will result in more feature records stored in memory (cached). Consider making this change when the value for the Cache Refresh Time Interval (minutes) parameter is also increased. Feature records that are cached for longer may accumulate in greater numbers depending on the rate of inbound event data. The trade-off with increasing the value in the Maximum Number of Feature Records parameter is that more system memory is used.
- Reducing the value in the Maximum Number of Feature Records parameter is often not recommended. Doing so can have an adverse impact on the performance of the processor. For example, setting the value of the Maximum Number of Feature Records parameter to 0 means that no feature records will be cached. If the number of unique event records coming through is 2,000 per second, the processor will effectively have to issue 2,000 requests to the feature service per second, every second, to obtain the feature records for enrichment. This may not only reduce the overall performance of GeoEvent Server, but also negatively impact the external server that must handle the high number of requests.
- When the maximum number of feature records size is exceeded, GeoEvent Server will remove the oldest cached feature record to accommodate the new feature record. For example, if the value for the Maximum Number of Feature Records is set to 100, and 101 unique event records are processed, the first cached feature record will be purged to make room for the 101st feature record. This happens regardless of whether the cache refresh time interval is exceeded.
- When this processor is used for the first time in a GeoEvent Service with a high-volume, high-velocity data feed, it is not uncommon for a backlog of inbound event records to accumulate. The reason being, GeoEvent Server must issue a series of requests to the feature service to build its cache of feature records to join on for the first time. Once the feature records begin to accumulate in the cache, the processor can join subsequent event records from memory, thereby improving processing performance.
- Network latency can have a significant impact on the performance of the processor. For example, if the time it takes to retrieve a feature record increases from 100 milliseconds to 200 milliseconds because of fluctuations in network latency, the number of event records that can be processed in that same unit of time will be reduced by half. The reason being, it takes twice as long to retrieve the feature record for a valid join to happen.