Stream services

As the number of data sources providing real-time data streams increases, it is becoming more important that your applications can consume and immediately display this event data. Latency introduced by first storing this event data in a feature layer in an enterprise geodatabase so that clients can periodically poll a feature service to retrieve the data for display is proving to be problematic—especially when working with high-volume data streams.

The existing paradigm must change. Data persistence must be addressed as an archival task undertaken in parallel as event data is pushed in real-time to clients. It is recommended that you use a stream service—a type of ArcGIS Server service.

Introduction to stream services

A stream service emphasizes low latency, real-time data dissemination for client-server data flows. Clients connecting to a stream service begin receiving data immediately upon subscribing to the service. Clients can specify and reconfigure both spatial and attribute constraints without first unsubscribing and then reconnecting to the service.

GeoEvent Server must be licensed and installed in your enterprise GIS in order to leverage stream services. Stream service content can be incorporated into ArcGIS Online and Portal for ArcGIS web maps and ArcGIS Pro, as well as exposed through clients developed using ArcGIS API for JavaScript.

Stream services leverage WebSocket technology, which supports full-duplex, bidirectional communication. This allows clients to specify the data they want to receive without having to unsubscribe and reestablish their connection to the service. Clients can filter stream service data by specifying either spatial or attribute constraints.

Connecting to a stream service to receive a real-time data feed allows you to separate the basic need of immediate event visualization from the need to persist data in a database. By configuring the GeoEvent Server output to broadcast event data via a stream service, you can store the latest event data in an enterprise relational data store or spatiotemporal big data store, but doing so is not a requirement for data visualization.

The illustration below compares how feature data has traditionally been received, processed, and consumed with how real-time data is received and broadcast using stream services.

Notice in the upper portion of the illustration how, prior to stream services, real-time data had to first be in a feature class, which required an enterprise geodatabase. To display the data, client applications had to periodically poll the feature service to obtain new and updated features.

The lower portion of the illustration shows how stream services allow real-time data to be received and immediately pushed to clients via a WebSocket.

Refer to the Stream Services Tutorial available from GeoEvent Server tutorials for more information on working with stream services in GeoEvent Server.

Stream service publishing

Stream services are created and published in GeoEvent Manager as part of the workflow to configure a Send Features to a Stream Service Output Connector.

When configuring a Send Features to a Stream Service Output Connector, it is recommended that you use the ArcGIS Server or Portal for ArcGIS connection registered as the default in GeoEvent Server.

A Send Features to a Stream Service Output Connector configured and running as part of a GeoEvent Server instance must be incorporated into a GeoEvent Service configured and running on that same GeoEvent Server.

Stream services in the ArcGIS REST Services Directory

Stream services are listed in the ArcGIS REST Services Directory like any other ArcGIS Server service. Review a stream service's properties as well as use controls to broadcast event data and subscribe to receive event data from a stream service.

• Clicking Broadcast will open a web page from which you can provide an Esri Feature JSON representation of one or more features and send the features to clients connected to a stream service.
• Clicking Subscribe will open a web page from which you can connect to a stream service to see any features being streamed. If you have a high-volume stream of data, the form on this page can quickly become overloaded; use this page for short periods of time only to confirm that clients subscribing to a stream service are receiving data.

See Stream Services for more information on stream services in ArcGIS REST API.

Consume a stream service

From the stream service's properties page in the ArcGIS REST Services Directory, click ArcGIS JavaScript to generate an HTML page on-the-fly from which you can visualize data being broadcast by a stream service.

Right-click ArcGIS JavaScript and click View Source to review the JavaScript code. Developers can use and customize this code to create web applications to consume stream services.

Stream services can also be consumed by incorporating them into a web map and in ArcGIS Pro.

See Stream layers for more information on consuming stream services in ArcGIS Pro.

Filter a stream service

A stream service allows filtering on a per-client basis. Each client can request a filter be applied to the data before it is sent from the stream service. This filter will not affect the data flowing to any of the other clients. The filter can be specified at the time the connection is made or after the connection is established.

These settings and others can be applied at the time the connection is established by opening a WebSocket connection.

Set a spatial reference for the data stream

A stream service has a default spatial reference that can be found in the service description page in ArcGIS Server Manager. If the client wanted to receive the data in a different spatial reference other than the default, it must be set at connection time. Once the spatial reference is set, it cannot be changed for the life of the connection. A new WebSocket connection must be created to change the spatial reference. To override the default spatial reference, use the keyword outSR and know the well-known ID (WKID) of the desired spatial reference and add outSR=<WKID> to the URL. For example, to set the default spatial reference to WGS 1984 Web Mercator (Auxiliary Sphere), whose WKID is 3857, add outSR=3857 to the URL, such as ws://HOSTNAME:6180/arcgis/services/Vehicles/StreamService/0/subscribe?outSR=3857

Configure the filter

A stream service filter can include a spatial component, an SQL-like query component, and an outFields component. Each can be set and cleared independently so any, all, or none of them can be in effect at any point in time. To set one part of the filter, specify that part and leave the rest out. The other parts of the filter will remain unchanged.

Geometry filter

The geometry part of the filter is the same as the structure of the JSON Geometry Objects returned by ArcGIS REST API. To preserve the performance of the stream service, only envelopes are accepted as a geometry type.

The filter can also include a spatial relationship identified by the keyword spatialRel. This represents the spatial relationship applied on the input geometry while performing the query. The only supported spatial relationship is intersects (esriSpatialRelIntersects), which is the default.

The geometry filter is assumed to be in the same spatial reference as the connection, which can only be overridden at connection time. If a different spatial reference is specified as part of the geometry, the envelope is projected into the connection's spatial reference for use when filtering features from the data stream.

Where filter

The where filter is an SQL-like expression that filters features from the data stream by checking to see if their attributes match the specified where clause. The expression must be a quoted JSON string value. The supported operations include AND, OR, NOT, =, !=, <, <=, >, >=, IS NULL, IS NOT NULL, IN, and LIKE. Comparisons can be made between a field and a literal value such as ( 'field1 > 1' ) or between two like-typed fields such as ( 'field1 > field2' ). Parenthesis can be added to make precedence explicitly enforced.

• Numerical comparison—Example comparing a field (Altitude) to a numeric value: "Altitude < 1000".
• Field comparison—Example comparing two fields: "speed > maxSpeed".
• String Comparison—Strings can also be compared. Comparisons are case-sensitive, and literal values must be surrounded with single quotes, for example: "Departure_Airport='KZSE'".
• LIKE statements—Strings can be compared using the LIKE statement to make use of wildcards. The percent sign is any number of characters and the underscore can be any single character. The following example will accept names that have the letters am as the second and third characters, such as Samantha and James: "name LIKE '_am%'".
• IN statements—The IN statement allows lists of literal values to be specified. If the field matches any value in the list, the feature is passed through the stream.
  • The following is an example of a list of strings: "name IN ('Bob','Jane','Henry')".
  • The following is an example of the IN statement using a list of numeric values: "alertCode IN (404,500,505)".
  • You can also use a Boolean field in place of a Boolean expression. For example, if your features contain a field named active that is a Boolean type, you can write a where clause that would pass any features through that have the active field set to true, for example: "active".

Out fields filter

The fields within the data stream can be individually filtered using the out fields filter identified with the keyword outFields. The desired fields are defined using a comma-separated list of field names.