Subscribe to a Kafka Topic for Text

The Subscribe to a Kafka Topic for Text input connector can be used to retrieve and adapt event data records, formatted as delimited text, from an Apache Kafka Topic. The text delimiter is usually a comma, so this type of text data is sometimes referred to as comma separated value data, but ArcGIS GeoEvent Server can use any ASCII printable character as a delimiter to separate data attribute values. For more information about getting started with Apache Kafka, see Apache Kafka Introduction.

Usage notes

Keep the following in mind when working with the Subscribe to a Kafka Topic for Text input connector:

• This input connector can be used to consume data as formatted or delimited text from a Kafka Topic. This input connector is a consumer of Kafka.
• This input connector pairs the Text inbound adapter with the Kafka inbound transport.
• Delimited text does not have to contain data that represents a geometry. Therefore, this input connector can be used to retrieve and process nonspatial data from Kafka.
• The adapter supports the ability to construct a point geometry from x, y, and z attribute fields.
• The Message Separator and Attribute Separator parameters are required to parse delimited text from Kafka. The message separator indicates the character that identifies the end of a data record; the default is newline (\n). The attribute separator specifies the character used to separate one attribute value from another in a single line of text; the default is a comma (,). Any ASCII printable character, specified as the character or Unicode value, can be used as a message or attribute separator.
• The Kafka inbound transport supports TLS 1.2 and SASL security protocols for authenticating with a Kafka cluster or broker.

Parameters

The following are the parameters for the Subscribe to a Kafka Topic for Text input connector:

Parameter	Description
Name	A descriptive name for the input connector used for reference in GeoEvent Manager.
Override with Custom Kafka Properties	Specifies whether the default GeoEvent Server Kafkaclient properties are overridden. The default is No. Yes—The default Kafka client properties exposed by the transport are overridden. A folder registered with GeoEvent Server that contains a Kafka.properties file with the correct formatting for a valid Kafkaconfiguration must be specified. Refer to Apache Kafka Configuration for a list of supported configurations and expected formatting for the specified .properties file. No—The default Kafka client properties exposed by the transport are not overridden. Kafka Bootstrap Servers and Consumer Group ID values must be specified.
Kafka Bootstrap Servers (Conditional)	A list of hostname:port pairs used to establish the initial connection to the Kafka cluster. Hostname:port pairs must be comma separated, for example, broker0.example.com:9092,broker1.example.com:9092,broker2.example.com:9092. This parameter is shown when Override with Custom Kafka Properties is set to No.
Topic Name(s)	The name of a Kafka topic, or list of Kafka topics, to consume data of interest from. Separate multiple topics with a semicolon. The following are examples: topic1 topic1;topic2;topic3;topic4 Note: Specifying multiple Kafka topics is supported in ArcGIS GeoEvent Server 10.8 and later. Note: The data order from a specific topic is not guaranteed.
Number of Consumers	The number of consumers for each consumer group. The default number of consumers is 1. Note: The number of consumers is limited by the number of partitions on the Kafka topic. Refer to Apache Kafka Introduction for more information about consumer instances.
Consumer Group ID (Conditional)	An optional string that uniquely identifies the consumer group for a set of consumers. This is also known as the consumer group name. If no Consumer Group ID is specified, GeoEvent Server assigns a static consumer group ID called geoevent-consumer. This static consumer group ID is shared across all instances of the Kafka connector where the Consumer Group ID is not specified. It is recommended that you specify a custom Consumer Group ID. Refer to Apache Kafka Introduction for more information about consumer groups. This parameter is shown when Override with Custom Kafka Properties is set to No.
Registered Folder for the Kafka Properties File (Conditional)	The folder registered with GeoEvent Server that contains the Kafka .properties file. The Kafka .properties file defines the custom Kafka properties when Override with Custom Kafka Properties is set to Yes. Ensure that the folder registered with GeoEvent Server is the full path to where the Kafka .properties file is located. This parameter is shown when Override with Custom Kafka Properties is set to Yes.
Kafka Properties File Name (Conditional)	The name of the Kafka .properties file that contains the custom Kafka properties for client configuration. Specify the name of the file without the .properties extension. If the name of the custom Kafka properties file is sample.properties, specify this parameter value as sample. This parameter is shown when Override with Custom Kafka Properties is set to Yes.
Start from Beginning	Specifies whether records are consumed from the topic starting at the beginning offset or from the last offset committed for the consumer. The default is Yes. Yes—Records are consumed from the topic at the beginning offset. No—Records are consumed from the last offset committed. Note: For more information about offsets, refer to Apache Kafka Configuration.
Event Separator	A single literal character that indicates the end of an event data record. Unicode values can be used to specify a character delimiter. Do not enclose the character in quotes. A newline (\n) is a common end-of-record delimiter.
Field Separator	A single literal character used to separate one attribute value from another in a message. Unicode values can be used to specify a character delimiter. Do not enclose the character in quotes. A comma is a common attribute delimiter.
Create GeoEvent Definition	Specifies whether a new or an existing GeoEvent Definition is used for the inbound event data. A GeoEvent Definition is required for GeoEvent Server to interpret the inbound event data attribute fields and data types. Yes—A new GeoEvent Definition is created based on the schema of the first event record received. No—No GeoEvent Definition is created. Choose an existing GeoEvent Definition that matches the schema of the inbound event data.
GeoEvent Definition Name (New) (Conditional)	The name assigned to a new GeoEvent Definition. If a GeoEvent Definition with the specified name already exists, the existing GeoEvent Definition is used. The first data record received is used to determine the expected schema of subsequent data records, and a new GeoEvent Definition is created based on that first data record's schema. This parameter is shown when Create GeoEvent Definition is set to Yes and is hidden when it's set to No.
GeoEvent Definition Name (Existing) (Conditional)	The name of an existing GeoEvent Definition to use when adapting received data to create event data for processing by a GeoEvent Service. This parameter is shown when Create GeoEvent Definition is set to No and is hidden when it's set to Yes.
Construct Geometry from Fields	Specifies whether the input connector will construct a point geometry using coordinate values received as attributes. The default is No. Yes—Values from specified event attribute fields are used to construct a point geometry. No—No point geometry is constructed. It is assumed that an attribute field contains a value that can be interpreted as a geometry or the event record is nonspatial (does not have a geometry).
X Geometry Field (Conditional)	The attribute field in the inbound event data containing the x-coordinate part (for example, horizontal or longitude) of a point location. This parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when it's set to No.
Y Geometry Field (Conditional)	The attribute field in the inbound event data containing the y-coordinate part (for example, vertical or latitude) of a point location. This parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when it's set to No.
Z Geometry Field (Conditional)	The attribute field in the inbound event data containing the z-coordinate part (for example, depth or altitude) of a point location. If no value is provided, the z-value is omitted, and a 2D point geometry is constructed. This parameter is shown when Construct Geometry from Fields is set to Yes and is hidden when it's set to No.
Default Spatial Reference	The well-known ID (WKID) of a spatial reference to be used when a geometry is constructed from attribute field values whose coordinates are not latitude and longitude values for an assumed WGS84 geographic coordinate system, or when geometry strings are received that do not include a spatial reference. A well-known text (WKT) value or the name of an attribute field containing the WKID or WKT can also be specified.
Expected Date Format	The pattern used to match expected string representations of date/time values and convert them to Java Date values. The pattern's format follows the Java SimpleDateFormat class convention. While the preferred pattern for date/time values in GeoEvent Server is the ISO 8601 standard, several string representations of date/time values commonly recognized as date values can be converted to Java Date values without specifying an Expected Date Format value. These include the following: "2019-12-31T23:59:59"—The ISO 8601 standard format 1577836799000—Java Date (epoch long integer; UTC) "Tue Dec 31 23:59:59 -0000 2019"—A common web services string format "12/31/2019 11:59:59 PM"—A common format used in the United States (12-hour clock) "12/31/2019 23:59:59"—A common format used in the United States (24-hour clock) If the date/time values received use a convention other than one of those listed above, you must specify an expected date format pattern so GeoEvent Server can adapt the date/time values.
Language for Number Formatting	The locale identifier (ID) used for locale-sensitive behavior when formatting numbers from data values. The default is the locale of the machine GeoEvent Server is installed on. For more information, refer to Java Supported Locales.
Authentication Required	Specifies whether the connection to the Kafka cluster, or Kafka broker, requires authentication. The default is No. Yes—Authentication is required to connect to the Kafka cluster or broker. No—Authentication is not required to connect to the Kafka cluster or broker.
Authenticate Using (Conditional)	Specifies the security protocol that is used to secure the Kafka cluster. The security protocol options are TLS 1.2 and SASL. TLS 1.2—The security protocol used by the Kafka cluster is TLS 1.2. Ensure that the Kafka cluster's PKI file (x509 certificate) is imported into the trust store of the ArcGIS Server machine with which ArcGIS GeoEvent Server is configured. For details, refer to the Import the certificate into ArcGIS Server section in Configure ArcGIS Server with an existing CA-signed certificate for specific instructions on importing certificates. SASL—The security protocol used by the Kafka cluster is SASL. Only SASL, SSL, and Kerberos are supported. Note: When using Kerberos, ensure that the operating system user account running ArcGIS GeoEvent Server has read access to the keytab file in the Kerberos setup and configuration. This parameter is shown when Authentication Required is set to Yes.
Registered Folder for Credential File (Conditional)	The folder registered with GeoEvent Server that contains the Kafka cluster's PKI file (x509 certificate). Ensure that the folder registered with GeoEvent Server is the full path to where the Kafka cluster's certificate is located. This parameter is shown when Authentication Required is set to Yes. It is applicable to TLS 1.2 only.
Credential Configuration File (Conditional)	The name of the Kafka cluster's PKI file (x509 certificate). The certificate and its associated private key must be stored in the PKCS#12 format, which is represented by a file with either the .p12 or .pfx extension. Provide the name of the file in addition to the extension. Note: Only the certificate file name and extension are supported for this parameter. Do not specify the relative paths to the certificate for this parameter. Register the full path to the certificate file using the Registered Folder for Credential File parameter. This parameter is shown when Authentication Required is set to Yes. It is applicable to TLS 1.2 only.
Keystore Password (Conditional)	The password for the Kafka cluster's PKI file (x509 certificate). This is also known as the certificate's private key. This parameter is shown when Authentication Required is set to Yes. The parameter is applicable to TLS 1.2 only.
SASL Authentication Type (Conditional)	Specifies the type of SASL authentication mechanism supported by the Kafka cluster. SASL authentication type options are SASL GSSAPI (Kerberos) and SASL PLAIN. SASL/GSSAPI (Kerberos)—The Kafka cluster uses SASL/GSSAPI Kerberos authentication. SASL/PLAIN—The Kafka cluster uses SASL/PLAIN authentication. This parameter is shown when Authentication Required is set to Yes. It is applicable to SASL only.
Kerberos Principal (Conditional)	The Kerberos principal for the specific user, for example, [e-mail]. This parameter is shown when Authentication Required is set to Yes. The parameter is applicable to SASL/GSSAPI (Kerberos) only.
Use Key Tab (Conditional)	Specifies whether the key tab in the Kerberos settings is used. The default is Yes. Yes—The key tab is used in the Kerberos settings. No—The key tab is not used in the Kerberos settings. This parameter is shown when Authentication Required is set to Yes. It is applicable to SASL/GSSAPI (Kerberos) only.
Store Key (Conditional)	Specifies whether the key is stored in the Kerberos settings. The default is Yes. Yes—The key is stored in the Kerberos settings. No—The key is not stored in the Kerberos settings. This parameter is shown when Authentication Required is set to Yes. It is applicable to SASL/GSSAPI (Kerberos) only.
Username (Conditional)	The username that is used to authenticate with the Kafka cluster. This is also known as a connection string with certain cloud providers. Refer to the documentation of the chosen cloud provider for the correct syntax. This parameter is shown when Authentication Required is set to Yes. It is applicable to SASL/PLAIN only.
Password (Conditional)	The password used to authenticate with the Kafka cluster. Refer to the documentation of the chosen cloud provider for the correct syntax. This parameter is shown when Authentication Required is set to Yes. This parameter is applicable to SASL/PLAIN only.

Considerations and limitations

There are several considerations to keep in mind when using the Subscribe to a Kafka Topic for Text input connector:

• Insufficiently managing and optimizing consumers will result in certain instances of the Subscribe to a Kafka Topic for Text input connector not retrieving any data. The number of consumers, in a consumer group, is limited by the number of partitions on a Kafka Topic. If the number of consumers in a consumer group exceeds the number of partitions available on a Kafka Topic, the excess consumers will not consume data. Consider optimizing the number of consumers to best align with the number of partitions on the Kafka Topic or implement different consumer groups for each connector to avoid this. For more information on consumers and consumer groups, see the Kafka documentation.
• The Subscribe to a Kafka Topic for Text input connector is a client consumer of Kafka. Apply the same considerations to this input connector as would be required for any other client consumer of Kafka. For example, if this input connector is not receiving data from a Kafka Topic but a separate client consumer of Kafka is, consider the factors that are involved with having two client consumers. This includes, but is not limited to, the configured consumer group ID, number of partitions available on the Topic, and the number of existing consumers. Alternatively, if the input connector is stopped and started in quick succession, consider the implications for Kafka from a consumer point-of-view. A rebalancing of the Kafka Topic's partitions may prevent the input connector from immediately rejoining as a consumer under the same consumer group.