Field Mapper Processor

Summary

The Field Mapper Processor can be used to translate (or map) data from one event record schema or structure to another event record schema or structure. A Field Mapper Processor has five primary uses:

• To flatten hierarchical or multi-cardinal structures before routing the event record to an add or update feature output.
• To handle necessary changes to attribute field names and/or capitalization. For example, mapping TrackID to track_id.
• To selectively map fields of interest to new fields in order to simplify an event record.
• To add new attribute fields to an event record’s schema allowing other processors such as Field Enricher (Feature Service) / Field Enricher (File) to write to existing fields rather than new fields.
• To perform explicit data type conversions such as string to long integer or vice versa.
• To perform field calculations (similar to the Field Calculator Processor) when mapping the incoming event source field values to the target fields.

All event data ingested into GeoEvent Server should have an associated GeoEvent Definition that defines the attribute field names, data types, and overall data structure. A Field Mapper Processor can be used to map one GeoEvent Definition to another.

Examples

• A Field Mapper Processor can be used to handle necessary changes to attribute field names and/or capitalization. For example, data ingested with attribute field names such as ReportedDateTime can be remapped to transfer data values into an attribute field named reported_dt. This is often necessary when the attribute field names in the received schema do not comply with a back-end RDMS requirements that field names be either all uppercase or all lowercase.
• The processor can be used to selectively map fields of interest to new fields so the target event structure has fewer attribute fields than the source. For example, if you configure an input to construct a geometry from received coordinate values, a Field Mapper Processor can be used to remove the coordinate values from the event record’s structure before routing the data to an output. It can also be used to anonymize event data by removing a customer’s name and street address while keeping the name of a city, state, or other general information.
• The processor can be used to add additional fields to an event structure. This is useful when using a Field Enricher (Feature Service) Processor as it allows the processor to write to existing fields rather than create new fields. Using the processor before these other processors allows you to add fields to an event structure so that processors downstream can write data into the new fields.
• The individual source field text box entries accept field calculations like the Field Calculator Processor. This allows multiple field values to be used in conjunction with the built in functions to create complex value mappings. For example, a string field speed value “12.1 kph” could be converted to miles per hour using the formula toInteger(replace(' kph',''))*0.6214.
• The processor can be used to perform explicit data type conversions such as a string to long (for example, “3201” to 3201) and long to string (for example, 3201 to “3201”). Since GeoEvent Server handles date values as long integers measured in epoch milliseconds, you can explicitly cast a date to a long, but you cannot use a Field Mapper Processor to convert a date to a string or a string to a date. Instead, use the toString( ) function in a Field Calculator Processor to convert a date to a string.
• The processor can be used to flatten a multi-cardinal or hierarchical event structure. This is typically done prior to routing event records to an add or update feature output; feature records in a geodatabase do not support hierarchical groups or multi-cardinal lists of data values. For example, illustrated below is a scenario in which three data values (pressure, temperature, and humidity) are grouped beneath a parent element named data. The processor can be used to flatten the event record’s hierarchy by mapping the attributes grouped beneath data to peers alongside the attributes id, status, and calibrated.

  

• When mapping values organized within a group, hierarchical structure, use dot (.) notation to access the nested fields. For example, to map a pressure field nested beneath a grouping data field, map data.pressure to a field named pressure.

  

• When mapping values organized within an array or list, or multi-cardinal structure, use index notation common to programming languages like C# and Java. For example, given a list of values named observations, to map the third value from the list to a new attribute named third you would map observations[2] to the field named third. Arrays in programming are almost always zero-based, meaning the index of the first element in the array is zero.
• For more information, see the Accessing data values section in JSON Data Structures - Working with Hierarchy and Multi-cardinality blog on GeoNet for examples of hierarchical and multi-cardinal data.

Usage notes

• When configuring the Field Mapper Processor, select a source GeoEvent Definition and a target GeoEvent Definition. Fields from the target GeoEvent Definition appear to the right, with drop-down lists available from the source GeoEvent Definition, allowing you to select the appropriate fields to map. The data type of each field is displayed to help identify the appropriate mappings for each field.
• A GeoEvent Definition for an event record at one stage of real-time processing might not align with the schema of a published feature service. The event record will need to be processed by the Field Mapper Processor to translate each event record's schema to match the schema of the target features before a GeoEvent Service can update features in a feature service. For example, many real-time data feeds (especially those providing generic JSON) frequently provide data structures with data organized as groups or as lists with multiple values. Such data structures do not align with feature service schemas whose data structures are flat and expect discrete data types (date, string, long, and so forth). The processor can be used to map data from hierarchical and/or multi-cardinal data structures to create a simpler, flat data structure such as a GeoEvent Definition imported from a feature service.
• The drop-down list available for the source GeoEvent Definition can be overwritten by typing in the text box. Formulas identical to those used in the Field Calculator Processor can now be used in each source field select text box. Referencing a source field in the formula is done by providing the case-sensitive name of the source field.
• If you are updating features in a published feature service and do not want one or more fields of existing features to be modified, leaving those fields unmapped in the processor will result in null values being written to the unmapped attributes of the feature records. Instead of leaving unmapped fields in the processor, simply eliminate the fields that should not be updated. For more information, see Using a partial GeoEvent Definition to update feature records blog on GeoNet.

Parameters

Parameter	Description
Name	A descriptive name for the processor used for reference in GeoEvent Manager.
Processor	The name of the selected processor.
Source GeoEvent Definition	The name of the GeoEvent Definition from which attribute values will be taken. Remarque : When mapping values organized within a group or hierarchical structure, use dot (.) notation to access the nested fields. Use array index notation common to programming languages like C# and Java when mapping values organized within a list or multi-cardinal structure. When using formulas to calculate the value of a target field, use the same notation that is used for the Expression field in the Field Calculator Processor.
Target GeoEvent Definition	The name of the GeoEvent Definition whose attribute fields will receive values from the source data structure.

Considerations and limitations

• If the Field Mapper Processor is unable to complete a data type conversion, a null value will be written to the mapped field. An example would be when mapping an attribute field whose data type is a string to an attribute field whose data type is a double. The processor cannot convert the literal string value twelve to a numeric value, so it writes a null value into the target field.
• The processor is particular about the event data it receives. For example, say the processor is configured to map event records associated with a Car GeoEvent Definition to a structure described by a Vehicle GeoEvent Definition. If the processor receives an event record with attribute data for a vessel, it will assume another processor downstream is expecting event records whose data structure matches the GeoEvent Definition of a Vessel and will allow the event record to pass through the processor unmodified.
• The processor will not drop or discard event records if the field mapping between the source and target GeoEvent Definitions is incorrect. Event records will continue to be processed using the source GeoEvent Definition schema and the target GeoEvent Definition will be ignored. For example, if the intent is to map a field from the source schema called TrackID to a field in the target schema called track_id and instead a typo is made so that the source field has been specified as TrackIDs, the processor will not map any of the fields. Instead, the processor will pass through the original event record using the source GeoEvent Definition. This is also the case if the dot or index notation for referencing hierarchical values is specified incorrectly.