GeoAnalytics health check

The health check operation indicates the status of compute resources and jobs within the GeoAnalytics Server compute platform and is the underlying framework that allows GeoAnalytics Server to distribute analysis across multiple GeoAnalytics Server cores and machines. The health check operation, which can be used for troubleshooting and monitoring, is available to the server or portal administrator and is accessed by logging in to your ArcGIS Server Administrator Directory using the URL format https://gis_geoanalytics_server.domain.com:6443/arcgis/admin. To navigate to the health check, start from your GeoAnalytics Server site and click System > Platform Services > Compute Platform > Health.

Use the health check operation to do the following:

• Verify and identify GeoAnalytics Server machines that are being used for analysis.
• Verify that the allotted memory and cores are being used.
• Check the amount of cores or memory available on the GeoAnalytics Server machines. This can be helpful when using the GeoAnalytics Server settings.

  Learn more about GeoAnalytics Server settings

• Check the number of GeoAnalytics Server jobs currently running or completed.
• Find the jobID of a GeoAnalytics Server job to view the REST endpoint.
• Troubleshoot and resolve error messages.

  Note:

  If you receive the error Unable to initialize distributed compute platform service, complete the health check operation and check the status of the worker nodes. Once you have checked the status of the nodes, complete the steps listed in Common GeoAnalytics Server problems and solutions.

Note:

The GeoAnalytics Server health check is also available through ArcGIS Monitor at 10.6.1.

Learn more about ArcGIS Monitor

The compute platform health check uses terminology from the underlying framework that distributes the analysis. Terms and concepts used throughout this documentation are outlined in the following table:

Term	Description
Node	A server machine.
Worker	A machine used for computation. The worker machine receives instructions from the master machine and executes them. There can be one or more worker machines to a master machine.
Master	The server that oversees the worker nodes. A server can be both a master node and worker node.
Job	A job refers to running a GeoAnalytics Server tool.

The health check is composed of keys that are either a single value or an array. When a GeoAnalytics Server job has been run and completed, the health check has the following keys for each GeoAnalytics Server machine in the site:

{
  "<geoanalytics server machine name>":{
  {.. url ..},
  {.. workers ..},
  {.. cores ..},
  {.. coresused ..},
  {.. memory ..},
  {.. memoryused ..},
  {.. activeapps ..},
  {.. completedapps ..},
  {.. activedrivers ..},
  {.. status ..}
}

When there is more than one machine in a site, there will be multiple geoanalytics service machine name keys. Only the master node machine will contain resource and job information. The worker node machines will contain only null and 0 values. Within each top-level server object, the following keys contain an array of values: workers, activeapps, completedapps, and activedrivers. The top-level keys' url, cores, coresused, memory, memoryused, and status keys have a single value.

Each key is explored in more detail using the following examples:

• Example 1: A three-machine GeoAnalytics Server site running a single job.
  • Each machine has 16 GB of RAM and 4 CPU cores.
  • The system properties use the default of a maximum of 80 percent of available RAM and cores.
  • GeoAnalytics Server is set to use 12 GB of RAM per machine.
  • GeoAnalytics Server is set to use 9 cores across all machines.
• Example 2: A three-machine GeoAnalytics Server site running two jobs simultaneously.
  • Each machine has 16 GB of RAM and 4 CPU cores.
  • The system properties use the default of a maximum of 80 percent of available RAM and cores.
  • GeoAnalytics Server is set to use 6 GB of RAM per machine.
  • GeoAnalytics Server is set to use 4 cores across all machines.

Site information and resources

The url, cores, coresused, memory, memoryused, and status keysare described in this section. If you have more than one machine in your GeoAnalytics Server site, there will be multiple url keys. One of the url keys will be populated with values as shown in the following examples. All other url keys will have values of 0, and no workers will be listed. The URL with populated values is the master node; the other URLs are machines that are represented as workers and are not the master node. The activedrivers key is always an empty array.

Example 1	Example 2
{ "machine1.example-esri.com": { "url": "spark://machine1.example-esri.com:7077", "workers": [ { ...worker1... }, { ...worker2... }, { ...worker3... } ], "cores": 9, "coresused": 9, "memory": 36864, "memoryused": 36864, "activeapps": [ ... ], "completedapps": [ ... ], "activedrivers": [], "status": "ALIVE" }, "machine2.example-esri.com": { "url": "spark://machine2.example-esri.com:7077", "workers": [], "cores": 0, "coresused": 0, "memory": 0, "memoryused": 0, "activeapps": [], "completedapps": [], "activedrivers": [], "status": "STANDBY" }, "machine3.example-esri.com": { "url": "spark://machine3.example-esri.com:7077", "workers": [], "cores": 0, "coresused": 0, "memory": 0, "memoryused": 0, "activeapps": [], "completedapps": [], "activedrivers": [], "status": "STANDBY" } }	{ "machine1.example-esri.com": { "url": "spark://machine1.example-esri.com:7077", "workers": [ { ...worker1... }, { ...worker2... }, { ...worker3... } ], "cores": 9, "coresused": 8, "memory": 36864, "memoryused": 36864, "activeapps": [ ... ], "completedapps": [ ... ], "activedrivers": [], "status": "ALIVE" }, "machine2.example-esri.com": { "url": "spark://machine2.example-esri.com:7077", "workers": [], "cores": 0, "coresused": 0, "memory": 0, "memoryused": 0, "activeapps": [], "completedapps": [], "activedrivers": [], "status": "STANDBY" }, "machine3.example-esri.com": { "url": "spark://machine3.example-esri.com:7077", "workers": [], "cores": 0, "coresused": 0, "memory": 0, "memoryused": 0, "activeapps": [], "completedapps": [], "activedrivers": [], "status": "STANDBY" } }
Description
url—The URL of the GeoAnalytics Server machine with the Spark prefix and Spark port. Only one URL can be listed per machine. cores—The total number of cores available to the GeoAnalytics Server across all GeoAnalytics Server machines in the site. This represents the total available cores multiplied by the percentageMaxAllowedComputeCores setting, which is 80 percent by default. For example, if you have a single machine with 16 cores, the percentageMaxAllowedComputeCores is set to 80 percent, and the number of cores to use is set to 2, the values will be floor(16 * .8) = 12.8. coresused—The total number of cores currently being used in GeoAnalytics Server analysis across all machines in the GeoAnalytics Server site. memory—The total amount of memory (MB) available to the GeoAnalytics Server across all GeoAnalytics Server machines in the site. This represents the amount of memory on each machine multiplied by the percentageMaxAllowedComputeMemoryand the number of machines. For example, if you have a single machine with 32 GB of RAM, the percentageMaxAllowedComputeCores is set to 50 percent, and the RAM to use is set to12 GB, the value will be floor(32 * .5) = 16. memoryused—The total amount of memory currently being used in GeoAnalytics Server analysis across all machines in the GeoAnalytics Server site. activedrivers—Always an empty array.

Workers

The workers key identifies the machines and information as part of the GeoAnalytics Server site.

Example 1	Example 2
{ "machine1.example-esri.com": { "url": "spark://machine1.example-esri.com:7077", "workers": [ { "id": "worker-20180302114515-10.33.9.162-56544", "host": "10.33.9.162", "port": 56544, "cores": 3, "coresused": 3, "coresfree": 0, "memory": 12288, "memoryused": 12288, "memoryfree": 0, "state": "ALIVE", "lastheartbeat": 1520469707880 }, { "id": "worker-20180302114111-10.33.10.149-56543", "host": "10.33.10.149", "port": 56543, "cores": 3, "coresused": 3, "coresfree": 0, "memory": 12288, "memoryused": 12288, "memoryfree": 0, "state": "ALIVE", "lastheartbeat": 1520469712961 }, { "id": "worker-20180302113155-10.33.9.165-56540", "host": "10.33.9.165", "port": 56540, "cores": 3, "coresused": 3, "coresfree": 0, "memory": 12288, "memoryused": 12288, "memoryfree": 0, "state": "ALIVE", "lastheartbeat": 1520469712874 } ], ... }, "machine2.example-esri.com": {...}, "machine3.example-esri.com": {...} } }	{ "machine1.example-esri.com": { "url": "machine1.example-esri.com:7077", "workers": [ { "id": "worker-20180302114515-10.33.9.162-56544", "host": "10.33.9.162", "port": 56544, "cores": 3, "coresused": 2, "coresfree": 1, "memory": 12288, "memoryused": 12288, "memoryfree": 0, "state": "ALIVE", "lastheartbeat": 1520469707880 }, { "id": "worker-20180302114111-10.33.10.149-56543", "host": "10.33.10.149", "port": 56543, "cores": 3, "coresused": 3, "coresfree": 0, "memory": 12288, "memoryused": 12288, "memoryfree": 0, "state": "ALIVE", "lastheartbeat": 1520469712961 }, { "id": "worker-20180302113155-10.33.9.165-56540", "host": "10.33.9.165", "port": 56540, "cores": 3, "coresused": 3, "coresfree": 0, "memory": 12288, "memoryused": 12288, "memoryfree": 0, "state": "ALIVE", "lastheartbeat": 1520469712874 } ], ... }, "machine2.example-esri.com": {...}, "machine3.example-esri.com": {...} } }
Description
id—A unique ID for the worker. host—The host address of the machine port—The port on the machine used for GeoAnalytics Server analysis. Each worker is assigned its own port. It is required that these ports are opened for communication. cores—The number of cores allocated to GeoAnalytics Server analysis on the machine. This is set using GeoAnalytics Server settings and represents the percentageMaxAllowedComputeCores setting. coresused—The number of cores currently used for GeoAnalytics Server analysis. If there are any GeoAnalytics Tools running, this value is expected to be greater than 0. coresfree—The number of cores that are currently free and can be immediately used for GeoAnalytics Server analysis on this machine. If this number is equal to cores on all machines, no GeoAnalytics Tools are currently running. memory—The amount of memory (MB) allocated to GeoAnalytics Server analysis on the machine. This is set using GeoAnalytics Server settings and represents the percentageMaxAllowedComputeMemory setting. memoryused—The amount of memory (MB) currently used for GeoAnalytics Server analysis by the specified machine. memoryfree—The amount of memory (MB) that is currently free and can be used immediately for GeoAnalytics Server analysis on this machine. The amount of memoryfree must be greater or equal to the amount set in the server setting ( Maximum allowed memory per job per machine (GB)) on each machine for GeoAnalytics Tools to run. state—The state of the current machine. ALIVE is on, ready to run a job or running one. STANDBY switches to ALIVE when a job is started. DEAD may have an error, and the GeoAnalytics Server should be restarted. UNKNOWN may have an error, and the GeoAnalytics Server should be restarted. If you cannot get a machine to switch to ALIVE or STANDBY from DEAD or UNKNOWN, contact Esri Support. lastheartbeat—The last time the current machine contacted the master node in milliseconds from epoch.

Completed apps

The completedapps key identifies the completed GeoAnalytics Server tools and information. This includes tools that successfully completed and tools that resulted in a failure. The completed apps information is cleared when GeoAnalytics Server is restarted.

Example 1	Example 2
{ "machine1.example-esri.com": { "url": "spark://machine1.example-esri.com:7077", "workers": [...], ..., "activeapps": [...], "completedapps": [ { "starttime": 1520469691617, "id": "app-20180307164131-0008", "name": "ArcGIS GeoAnalytics (jcb8bda25a3f14dc7835dca0b3267b42c)", "cores": 9, "user": "agstest", "memoryperslave": 12288, "submitdate": "Wed Mar 07 16:41:31 PST 2018", "state": "FINISHED", "duration": 21947 } ], "activedrivers": [], "status": "ALIVE" }, "machine2.example-esri.com": {...}, "machine3.example-esri.com": {...} }	{ "machine1.example-esri.com": { "url": "spark://machine1.example-esri.com:7077", "workers": [...], ..., "activeapps": [...], "completedapps": [ { "starttime": 1520469691617, "id": "app-20180307164131-0008", "name": "ArcGIS GeoAnalytics (jcb8bda25a3f14dc7835dca0b3267b42c)", "cores": 4, "user": "agstest", "memoryperslave": 6144, "submitdate": "Wed Mar 07 16:41:31 PST 2018", "state": "FINISHED", "duration": 21947 } ], "activedrivers": [], "status": "ALIVE" }, "machine2.example-esri.com": {...}, "machine3.example-esri.com": {...} }
Description
starttime—The time the GeoAnalytics Tools job started running in milliseconds from epoch. id—A unique ID for the job. name—The name of the job. This is always in the format ArcGIS GeoAnalytics (<jobID>). The jobID is the same jobID used at the REST endpoint when running a job. cores—The total number of cores used for analysis across all machines. user—The user who ran the job. This is the user who set up ArcGIS Server and will always be the same on your site. It is not the portal member that ran the job. memoryslavepernode—The amount of memory used on each machine. submitdate—The time the GeoAnalytics Tools job started running in a readable text format. state—The state of the current job. FINISHED means that the job has been completed. The job may have completed successfully or resulted in an error. duration—The total duration of the job in milliseconds.

Active apps

The activeapps key identifies the GeoAnalytics Tools that are currently running and their information. Once a job has been completed, the information is moved to the completedapps object.

Example 1	Example
{ "machine1.example-esri.com": { "url": "spark://machine1.example-esri.com:7077", "workers": [...], ..., "activeapps": [ { "starttime": 1520464610794, "id": "app-20180307151650-0007", "name": "ArcGIS GeoAnalytics (j4b986eb5c603426fae7abe8e9d5d7afa)", "cores": 9, "user": "agstest", "memoryperslave": 12288, "submitdate": "Wed Mar 07 15:16:50 PST 2018", "state": "RUNNING", "duration": 5105025 } ], "completedapps": [...] }, "machine2.example-esri.com": {...}, "machine3.example-esri.com": {...} }	{ "machine1.example-esri.com": { "url": "spark://machine1.example-esri.com:7077", "workers": [...], ..., "activeapps": [ { "starttime": 1520464610794, "id": "app-20180307151650-0007", "name": "ArcGIS GeoAnalytics (j4b986eb5c603426fae7abe8e9d5d7afa)", "cores": 4, "user": "agstest", "memoryperslave": 6144, "submitdate": "Wed Mar 07 15:16:50 PST 2018", "state": "RUNNING", "duration": 5105025 }, { "starttime": 1520469713951, "id": "app-20180307164153-0009", "name": "ArcGIS GeoAnalytics (j739627e916b34533be918fe98b0fcb90)", "cores": 4, "user": "agstest", "memoryperslave": 6144, "submitdate": "Wed Mar 07 16:41:53 PST 2018", "state": "RUNNING", "duration": 1868 } ], "completedapps": [...] }, "machine2.example-esri.com": {...}, "machine3.example-esri.com": {...} }
Description
starttime—The time the GeoAnalytics Tools started running in milliseconds from epoch. id—A unique ID for the job. name—The name of the job. This is always in the format ArcGIS GeoAnalytics (<jobID>). The jobID is the same job ID used at the REST endpoint when running a job. cores—The total number of cores used for analysis across all machines. user—The user who ran the job. This is the user who set up ArcGIS Server and will always be the same on your site. It is not the portal member who ran the job. memoryslavepernode—The amount of memory used on each machine. submitdate—The time the GeoAnalytics Tools started running in a readable text format. state—The state of the current job. RUNNING means the job has started. WAITING means the job will wait for one minute to determine if there are enough resources. If there aren't enough resources within a minute, the job will cancel. If there are resources, it will start RUNNING. duration—The current duration of the job in milliseconds.