The Field Mapper Processor is used to translate (or map) data from one event record schema or structure to another event record schema or structure. A Field Mapper Processor has the following primary uses:
- To flatten hierarchical or multicardinal structures before routing the event record to an add or update feature output
- To handle necessary changes to attribute field names or capitalization—for example, mapping TrackID to track_id
- To selectively map fields of interest to new fields to simplify an event record.
- To add new attribute fields to an event record’s schema allowing other processors such as Field Enricher (Feature Service) / Field Enricher (File) to write to existing fields rather than new fields.
- To perform explicit data type conversions such as string to long integer or vice versa.
- To perform field calculations (similar to the Field Calculator Processor) when mapping the incoming event source field values to the target fields.
All event data imported into GeoEvent Server should have an associated GeoEvent Definition that defines the attribute field names, data types, and overall data structure. A Field Mapper Processor can be used to map one GeoEvent Definition to another.
Examples
The following are example uses of the Field Mapper Processor:
- The processor can be used to handle necessary changes to attribute field names or capitalization. For example, data ingested with attribute field names such as ReportedDateTime can be remapped to transfer data values into an attribute field named reported_dt. This is often necessary when the attribute field names in the received schema do not comply with back-end RDMS requirements that field names be either all uppercase or all lowercase.
- The processor can be used to selectively map fields of interest to new fields so the target event structure has fewer attribute fields than the source. For example, if you configure an input to construct a geometry from received coordinate values, a Field Mapper Processor can be used to remove the coordinate values from the event record’s structure before routing the data to an output. It can also be used to anonymize event data by removing a customer’s name and street address while keeping the name of a city, state, or other general information.
- The processor can be used to add more fields to an event structure. This is useful when using a Field Enricher (Feature Service) Processor, as it allows the processor to write to existing fields rather than create new fields. Using the processor before these other processors allows you to add fields to an event structure so that processors downstream can write data into the new fields.
- The individual source field text box entries accept field calculations like the Field Calculator Processor. This allows multiple field values to be used in conjunction with the built-in functions to create complex value mappings. For example, a string field speed value “12.1 kph” could be converted to miles per hour using the formula toInteger(replace(' kph',''))*0.6214.
- The processor can be used to perform explicit data type conversions such as a string to long (for example, “3201” to 3201) and long to string (for example, 3201 to “3201”). Since GeoEvent Server handles date values as long integers measured in epoch milliseconds, you can explicitly cast a date to a long integer, but you cannot use a Field Mapper Processor to convert a date to a string or a string to a date. Instead, use the toString( ) function in a Field Calculator Processor to convert a date to a string.
- The processor can be used to flatten a multicardinal or hierarchical event structure. This is typically done prior to routing event records to an add or update feature output; feature records in a geodatabase do not support hierarchical groups or multicardinal lists of data values. For example, illustrated below is a scenario in which three data values (pressure, temperature, and humidity) are grouped under a parent element named data. The processor can be used to flatten the event record’s hierarchy by mapping the attributes grouped under data to peers alongside the attributes id, status, and calibrated.
- When mapping values organized within a group, hierarchical structure, use dot (.) notation to access the nested fields. For example, to map a pressure field nested under a grouping data field, map data.pressure to a field named pressure.
- When mapping values organized within an array or list, or multicardinal structure, use index notation common to programming languages such as C# and Java. For example, given a list of values named observations, to map the third value from the list to a new attribute named third, you would map observations[2] to the field named third. Arrays in programming are almost always zero-based, meaning the index of the first element in the array is zero.
- For more information, see the Accessing data values section in the JSON Data Structures - Working with Hierarchy and Multi-cardinality blog on Esri Community for examples of hierarchical and multicardinal data.
Usage notes
Keep the following in mind when working with the Field Mapper Processor:
- When configuring the Field Mapper Processor, select a source GeoEvent Definition and a target GeoEvent Definition. Fields from the target GeoEvent Definition appear to the right, with drop-down lists available from the source GeoEvent Definition, allowing you to select the appropriate fields to map. The data type of each field is displayed to help identify the appropriate mappings for each field.
- A GeoEvent Definition for an event record at one stage of real-time processing may not align with the schema of a published feature service. The event record will need to be processed by the Field Mapper Processor to translate each event record's schema to match the schema of the target features before a GeoEvent Service can update features in a feature service. For example, many real-time data feeds (especially those providing generic JSON) frequently provide data structures with data organized as groups or as lists with multiple values. Such data structures do not align with feature service schemas whose data structures are flat and expect discrete data types (date, string, long, and so forth). The processor can be used to map data from hierarchical or multicardinal data structures to create a simpler, flat data structure such as a GeoEvent Definition imported from a feature service.
- The drop-down list available for the source GeoEvent Definition can be overwritten by typing in the text box. Formulas identical to those used in the Field Calculator Processor can now be used in each source field select text box. Referencing a source field in the formula is done by providing the case-sensitive name of the source field.
- If you are updating features in a published feature service and do not want one or more fields of existing features to be modified, leaving those fields unmapped in the processor will result in null values being written to the unmapped attributes of the feature records. Instead of leaving unmapped fields in the processor, eliminate the fields that should not be updated. For more information, see the Using a partial GeoEvent Definition to update feature records blog on Esri Community.
Parameters
The following are the parameters for the Field Mapper Processor:
| Parameter | Description |
|---|---|
| Name | A descriptive name for the processor used for reference in GeoEvent Manager. |
| Processor | The name of the selected processor. |
| Source GeoEvent Definition | The name of the GeoEvent Definition from which attribute values will be taken. Note:When mapping values organized within a group or hierarchical structure, use dot (.) notation to access the nested fields. Use array index notation common to programming languages such as C# and Java when mapping values organized within a list or multicardinal structure. When using formulas to calculate the value of a target field, use the same notation that is used for the Expression field in the Field Calculator Processor. |
| Target GeoEvent Definition | The name of the GeoEvent Definition whose attribute fields will receive values from the source data structure. |
Considerations and limitations
There are several considerations to keep in mind when using the Field Mapper Processor:
- If the Field Mapper Processor is unable to complete a data type conversion, a null value will be written to the mapped field. An example would be when mapping an attribute field whose data type is a string to an attribute field whose data type is a double. The processor cannot convert the literal string value twelve to a numeric value, so it writes a null value into the target field.
- The processor is particular about the event data it receives. For example, say the processor is configured to map event records associated with a Car GeoEvent Definition to a structure described by a Vehicle GeoEvent Definition. If the processor receives an event record with attribute data for a vessel, it will assume another processor downstream is expecting event records whose data structure matches the GeoEvent Definition of a Vessel and will allow the event record to pass through the processor unmodified.
- The processor will not drop or discard event records if the field mapping between the source and target GeoEvent Definitions is incorrect. Event records will continue to be processed using the source GeoEvent Definition schema and the target GeoEvent Definition will be ignored. For example, if the intent is to map a field from the source schema named TrackID to a field in the target schema named track_id and instead a typo is made so that the source field has been specified as TrackIDs, the processor will not map any of the fields. Instead, the processor will pass through the original event record using the source GeoEvent Definition. This is also the case if the dot or index notation for referencing hierarchical values is specified incorrectly.
Field name delimitation
Characters normally recognized as arithmetic functions or operators should not be used in an attribute field's name. The expression parser used by the processor cannot differentiate between an expression subtract accuracy from gps and gps-accuracy (in which the dash is part of the attribute field name) unless the field name itself is wrapped in a syntax specifically designed to separate a field name from the rest of the expression.
When data from an external source includes allowable metacharacters in an attribute field name that conflicts with characters normally recognized as arithmetic functions—for example, plus (+), minus (-), multiply (*), or divide (/)—you have to wrap the attribute name within ${} to instruct the expression parser to interpret the field name separately from the rest of the expression.
In the example below, given a sensor record represented as generic JSON, you would have to code references to the key gps-accuracy in the attribute substructure to work around the fact that a literal dash is included in the field name.
The ampersand (@) character in the keys @deviceID and @address is allowed, but is not recommended. For more information, see Field name restrictions. If the schema of the data cannot be updated or changed, a Field Mapper Processor should be used as early as possible in your event processing workflow to remove the nonalphanumeric character from the field name. The expressions illustrated on Line 1 and Line 2 accomplish this without the need to wrap the attribute name within ${}to delimit the field name.
The expression shown on Line 3 uses the required dot notation (.) to address the @noteskey in the attributes substructure. Again, while allowed, the inclusion of the ampersand (@) character in the field name is not recommended. The metacharacter does not cause the expression parser any particular ambiguity, so wrapping the attribute name within ${} is not required.
The inclusion of a dash (-) in the key gps-accuracy does produce ambiguity. So the expression shown on Line 4 uses the ${} syntax to specifically delimit the field name. The dot notation required to address the gps-accuracy key in the attributes substructure is incorporated into the ${}syntax.
Knowing that a Field Mapper Processor is necessary early in the GeoEvent Service to promote the field naming guidelines, the solution illustrated configured the input to not create a geometry from the received coordinates. Instead, the expression shown on Line 5 uses a function to produce a geometry on the fly and map it to a field named Geometry. Note the use of the dot notation in the function's parameters to address the Longitude and Latitude keys in the coordinates substructure.