As the number of data sources providing real-time data streams increases, it is becoming more important that your applications are able to consume and immediately display this event data. Traditionally, latency introduced by first storing this event data in a feature class in an enterprise geodatabase so that clients can periodically poll a feature service to retrieve the data for display is proving to be problematic—especially when working with high-volume data streams.
The existing paradigm needs to change. Data persistence needs to be addressed as an archival task undertaken in parallel as event data is pushed in real-time to clients. Esri's answer is a new type of ArcGIS Server service—a stream service.
What is a stream service?
A stream service is a type of ArcGIS Server service that emphasizes low latency, real-time data dissemination, for client-server data flows. Clients connecting to a stream service begin receiving data immediately upon subscribing to the service. Clients can specify and reconfigure both spatial and attribute constraints without first unsubscribing and then reconnecting to the service.
GeoEvent Server must be licensed and installed in your enterprise GIS in order to leverage stream services. In the initial release, stream service content can be incorporated into ArcGIS Online and Portal for ArcGIS web maps as well as exposed through clients developed using the ArcGIS API for JavaScript. Future releases will support a wider variety of client subscriptions.
Stream services leverage WebSocket technology, which supports full-duplex, bidirectional communication. This enables clients to specify the data they want to receive without having to unsubscribe and reestablish their connection to the service. Clients are able to filter stream service data by specifying either spatial or attribute constraints.
Connecting to a stream service to receive a real-time data feed allows you to separate the basic need of immediate event visualization from the need to persist data in a database. By configuring the GeoEvent Server output to broadcast event data via a stream service, you have the option of storing the latest event data in an enterprise relational data store or spatiotemporal big data store, but doing so is not a requirement for data visualization.
The illustration below compares how feature data has traditionally been received, processed, and consumed with how real-time data is received and broadcast using stream services.
Notice in the illustration how, prior to stream services, real-time data had to first be in a feature class, which required an enterprise geodatabase. Client applications wanting to display the data had to periodically poll the feature service to obtain new and updated features.
The lower portion of the illustration shows how stream services allow real-time data to be received and immediately pushed to clients via a WebSocket.
Refer to the stream services tutorial available from GeoEvent Server tutorials for more information on working with stream services in GeoEvent Server.
Publishing a stream service
Stream services are created and published in GeoEvent Manager as part of the workflow to configure a Send Features to a Stream Service Output Connector.
When configuring a Send Features to a Stream Service Output Connector, it is recommended you use the ArcGIS Server or Portal for ArcGIS connection registered as the Default in GeoEvent Server.
Alternatively, if you have GeoEvent Server installed and licensed on a second server machine, you can register a data store connection to that second machine in GeoEvent Manager and specify the stream service be published to this second machine. This allows you to configure an output running on one machine to broadcast its feature records via a stream service running on a different machine.
A Send Features to a Stream Service Output Connector configured and running as part of a GeoEvent Server instance must be incorporated into a GeoEvent Service configured and running on that same GeoEvent Server.
Stream services in the ArcGIS REST Services Directory
Stream services are listed in the ArcGIS REST Services Directory like any other ArcGIS Server service. Users can click to review a stream service's properties. At the bottom of the stream service page, in the ArcGIS REST Services Directory, are links to broadcast event data and subscribe to receive event data from a stream service.
Click Broadcast to open a web page from which you can enter an Esri Feature JSON representation of one or more features and send the features to clients connected to a stream service.
Click Subscribe to open a web page from which you can connect to a stream service to see any features being streamed. If you have a high-volume stream of data, the form on this page can quickly become overloaded; use this page for short periods of time only to confirm that clients subscribing to a stream service should be receiving data.
See Stream services for more information on stream services in the ArcGIS REST Services Directory.
Consuming a stream service
In the illustration above of the stream service's REST page, notice the link at the top to view a stream service's content in an ArcGIS JavaScript map, an option commonly available for map services.
Click ArcGIS JavaScript to generate an HTML page on-the-fly from which you can visualize data being broadcast by a stream service.
If you right-click ArcGIS JavaScript and click View Source, you can review the JavaScript code. Developers can use and customize this code to create web applications to consume stream services.
Stream services can also be consumed by incorporating them into a web map and in ArcGIS Pro.
See Stream layers for more information on consuming stream services in ArcGIS Pro.
Filtering a stream service
The stream service allows filtering on a per-client basis. Each client can request a filter be applied to the data before it is sent from the stream service. This filter will not affect the data flowing to any of the other clients. The filter can be specified at the time the connection is made or after the connection has been established.
These settings and others can be applied at the time the connection is established by opening a WebSocket connection.
Setting a spatial reference for the data stream
A stream service has a default spatial reference that can be found in the service description page in ArcGIS Server Manager. If the client wanted to receive the data in a different spatial reference other than the default, it has to be set at connection time. Once the spatial reference is set, it cannot be changed for the life of the connection. A new WebSocket connection would need to be created in order to change the spatial reference. To override a default spatial reference you would need to use the keyword outSR and know the well-known ID (WKID) of the desired spatial reference and would add outSR=<WKID> to the URL. For example, to set the default to WGS 1984 Web Mercator (Auxiliary Sphere), whose WKID is 3857, add outSR=3857 to the URL such as: ws://HOSTNAME:6180/arcgis/services/Vehicles/StreamService/0/subscribe?outSR=3857
Configuring the filter
A stream service filter can include a spatial component, a SQL-like query component, and an outFields component. Each can be set and cleared independently so any, all, or none of them can be in effect at any point in time. To set one part of the filter, simply specify that part and leave the rest out. The other parts of the filter will remain unchanged.
The geometry filter
The geometry part of the filter is the same as the structure of the JSON Geometry Objects returned by the ArcGIS REST API. To preserve the performance of the stream service, only envelopes are accepted as a geometry type.
The filter can also include a spatial relationship identified by the keyword spatialRel. This represents the spatial relationship applied on the input geometry while performing the query. The only supported spatial relationship is intersects, which is the default (esriSpatialRelIntersects).
The geometry filter is assumed to be in the same spatial reference as the connection, which can only be overridden at connection time. If a different spatial reference is specified as part of the geometry, the envelope is projected into the connection's spatial reference for use when filtering features from the data stream.
The where filter
The where filter is a SQL-like expression that filters features from the data stream by checking to see if their attributes match the specified where clause. The expression should be a quoted JSON string value. The supported operations include AND, OR, NOT, =, !=, <, <=, >, >=, IS NULL, IS NOT NULL, IN, and LIKE. Comparisons can be made between a field and a literal value such as ( 'field1 > 1' ) or between two like-typed fields such as ( 'field1 > field2' ). Parenthesis can be added to make precedence explicitly enforced.
- Numerical comparison—Example comparing a field (Altitude) to a numeric value: "Altitude < 1000".
- Field comparison—Example comparing two fields: "speed > maxSpeed".
- String Comparison—Strings can also be compared. Comparisons are always case-sensitive, and literal values should be surrounded with single quotes. For example: "Departure_Airport='KZSE'".
- LIKE statements—Strings can be compared using the LIKE statement to make use of wildcards. The percent sign (%) is any number of characters and the underscore (_) can be any single character. The following example will accept people whose name has the second and third character equal to a name such as Samantha and James: "name LIKE '_am%'".
- IN statements—The IN statement allows lists of literal values to be specified. If the field matches any value in the list, then the feature is passed through the stream.
- The following is an example of a list of strings: "name IN ('Bob','Jane','Henry')".
- The following is an example of the IN statement using a list of numeric values: "alertCode IN (404,500,505)".
- It is also possible to use a Boolean field in place of a Boolean expression. For example, if your features contain a field named active that is a Boolean type, you could write a where clause that would pass any features through that have the active field set to true, for example: "active".
The out fields filter
The fields within the data stream can be individually filtered using the out fields filter identified with the keyword outFields. The desired fields would be defined using a comma-separated list of field names.