Changing the location of Kafka and ZooKeeper data files

Location of the Kafka and ZooKeeper data files

The ArcGIS GeoEvent Gateway uses a local folder for Kafka on-disk topic queues and ZooKeeper configuration files.

• On Windows, the default directory is: C:\ProgramData\Esri\GeoEvent-Gateway
• On Linux, the default directory is: /home/arcgis/.esri/GeoEvent-Gateway

When installing ArcGIS GeoEvent Server to a drive or file path other than the default, the GeoEvent Gateway still uses the default folders above for the Kafka and ZooKeeper files. This can pose a problem if, for example, an organization’s standard practice is to limit the available disk on the C drive and instead install software on another drive. If this is the case, you may need to reconfigure the GeoEvent Gateway to write data files to a folder other than the default folders referenced above. This change must be done after installing GeoEvent Server.

Follow the steps below, for either Windows or Linux, to change the location of the Kafka on-disk topic queues and ZooKeeper configuration files.

Remarque :

The steps below will delete the existing GeoEvent Server configuration. It is recommended you perform these changes immediately after installing GeoEvent Server, before configuring GeoEvent Server or deploying custom components.

Remarque :

The system account used to run the ArcGIS GeoEvent Server and ArcGIS GeoEvent Gateway services (on Windows) or the equivalent daemons (on Linux) must have Full Control to read/write, create, and delete files to the file path you choose for the GeoEvent Gateway data files.

Remarque :

Windows UNC pathnames are not supported. When specifying a gateway data directory, use a local server directory such as C:\ProgramData\Esri or D:\Data\Esri on Windows or a path beneath the user account used to run GeoEvent Server such as /home/arcgis/.esri on Linux.

Before getting started

If inputs, outputs, or other elements such as GeoEvent Services or GeoEvent Definitions are configured in GeoEvent Server, follow the steps below to export the current configuration before proceeding with the steps in the next sections.

1. Open ArcGIS GeoEvent Manager and navigate to Site > GeoEvent > Configuration Store.
2. Click Export Configuration to create a backup of the existing configuration.

   If you have custom components (connectors and/or processors) deployed, it is recommended you perform the next step to remove them from GeoEvent Server before proceeding with the steps in the next section.

3. With the ArcGIS GeoEvent Server service running, move any existing *.jar files in the deploy folder to a temporary directory location. The deploy folder can be found at:

   <ArcGIS Server installation directory>\GeoEvent\deploy

Changing the location on Windows

1. Stop the ArcGIS GeoEvent Server and ArcGIS GeoEvent Gateway Windows services.
2. In File Explorer, navigate to the GeoEvent Gateway configuration files in the etc folder in the directory location below:

   <ArcGIS Server installation directory>\GeoEvent\gateway\etc

3. Open the kafka.properties file in a text editor and update the gateway.data.dir and log.dirs property values with your values.

   The default values for these properties are:

   • gateway.data.dir=C://ProgramData//Esri//GeoEvent-Gateway//
   • log.dirs= kafka/logs,kafka/logs1,kafka/logs2

   Remarque :

   The gateway data directory should specify an existing folder. The log directories specifies a comma separated list of subfolders the GeoEvent Gateway will create when the service is restarted.

   Remarque :

   ArcGIS GeoEvent Server 10.8.1 and earlier releases specified a single subfolder kafka/logs. Starting with 10.9, multiple partitions and topics are configured requiring three subfolders be specified (for example, kafka/logs,kafka/logs1,kafka/logs2).

4. Save and close the kafka.properties file.
5. In File Explorer, navigate to the GeoEvent Gateway configuration files in the etc folder in the directory location below:

   <ArcGIS Server installation directory>\GeoEvent\gateway\etc

6. Open the zookeeper.properties file in a text editor and update the gateway.data.dir and data.dir property values with your values.

   The default values for these properties are:

   • gateway.data.dir=C://ProgramData//Esri//GeoEvent-Gateway//
   • data.dir=zookeeper-data

   Remarque :

   The gateway data directory should specify an existing folder. The data directory specifies a configuration subfolder the GeoEvent Gateway will create when the service is restarted.

7. In File Explorer, navigate to and delete the files and folders in the data folder at:

   <ArcGIS Server installation directory>\GeoEvent\data

8. Restart the ArcGIS GeoEvent Gateway Windows service.
9. Wait several minutes for the ArcGIS GeoEvent Gateway to start running, then start the ArcGIS GeoEvent Server Windows service.
10. Open ArcGIS GeoEvent Manager and confirm you can sign in.
11. Optionally, if you had custom components deployed previously, move the *.jar files back into the deploy folder at:

    <ArcGIS Server installation directory>\GeoEvent\deploy

12. Optionally, if you exported the GeoEvent Server configuration previously, import it in ArcGIS GeoEvent Manager by navigating to Site > GeoEvent > Configuration Store and click Import Configuration to import the configuration.

Remarque :

If new files and folders are not created beneath the specified gateway data directory, after the GeoEvent Gateway is restarted, it is likely a permission issue with the system account used to run the ArcGIS GeoEvent Server and ArcGIS GeoEvent Gateway essentially not being able to read/write files to the specified folder.

Changing the location on Linux

1. Stop the ArcGIS GeoEvent Server and ArcGIS GeoEvent Gateway Linux daemons.
2. Navigate to the GeoEvent Gateway configuration files in the etc folder in the directory location below:

   <ArcGIS Server installation directory>\GeoEvent\gateway\etc

3. Open the kafka.properties file in a text editor and update the gateway.data.dir and log.dirs property values with your values.

   The default values for these properties are:

   • gateway.data.dir=/home/arcgis/.esri/GeoEvent-Gateway
   • log.dirs= kafka/logs,kafka/logs1,kafka/logs2

   Remarque :

   The gateway data directory should specify an existing folder. The data directory specifies a configuration subfolder the GeoEvent Gateway will create when the daemon is restarted.

   Remarque :

   ArcGIS GeoEvent Server 10.8.1 and earlier releases specified a single subfolder kafka/logs. Starting with 10.9, multiple partitions and topics are configured requiring three subfolders be specified (for example, kafka/logs,kafka/logs1,kafka/logs2).

4. Save and close the kafka.properties file.
5. Navigate to the GeoEvent Gateway configuration files in the etc folder in the directory location below:

   <ArcGIS Server installation directory>\GeoEvent\gateway\etc

6. Open the zookeeper.properties file in a text editor and update the gateway.data.dir and data.dir property values with your values.

   The default values for these properties are:

   • gateway.data.dir=/home/arcgis/.esri/GeoEvent-Gateway
   • data.dir=zookeeper-data

   Remarque :

   The gateway data directory should specify an existing folder. The data directory specifies a configuration subfolder the GeoEvent Gateway will create when the daemon is restarted.

7. Navigate to and delete the files and folders in the data folder at:

   /home/arcgis/server > rm -rf GeoEvent/data/*

8. Restart the ArcGIS GeoEvent Gateway daemon.
9. Wait several minutes for the ArcGIS GeoEvent Gateway to start running, then start the ArcGIS GeoEvent Server daemon.
10. Open ArcGIS GeoEvent Manager and confirm you can sign in.
11. Optionally, if you had custom components deployed previously, move the *.jar files back into the deploy folder at:

    <ArcGIS Server installation directory>\GeoEvent\deploy

12. Optionally, if you exported the GeoEvent Server configuration previously, import using ArcGIS GeoEvent Manager by navigating to Site > GeoEvent > Configuration Store and click Import Configuration to import the configuration.

Remarque :

If new files and folders are not created beneath the specified gateway data directory, after the GeoEvent Gateway is restarted, it is likely a result of a permission issue with the system account used to run the ArcGIS GeoEvent Server and ArcGIS GeoEvent Gateway essentially not being able to read/write files to the specified folder.

Additional considerations

The Kafka on-disk topic queues and ZooKeeper configuration files should be organized in the same data directory. For example, on Windows, the default directory for the Kafka on-disk topic queues is C:\ProgramData\Esri\GeoEvent-Gateway\kafka\logs and the ZooKeeper configuration files is C:\ProgramData\Esri\GeoEvent-Gateway\zookeeper-data. Kafka and ZooKeeper are tightly coupled, and their runtime files should be collocated.

The ArcGIS GeoEvent Server and ArcGIS GeoEvent Gateway services are constantly communicating with one another. Data from event records flows through Kafka topics as it is ingested, processed, and disseminated. The status of configured elements and information cached by GeoEvent Server is frequently updated in the ZooKeeper configuration store.

The GeoEvent Gateway runtime files must be stored locally on the server running the ArcGIS GeoEvent Gateway service (on Windows) or the equivalent daemon (on Linux). Storing the files on a network shared folder or a SAN / NAS should be avoided; doing so will impact performance, solution reliability, and stability.