Skip to Content
SAP HANA streaming analytics

Streaming Web Service Configuration and Testing (for SAP HANA Smart Data Streaming SP10)

Tags:

Part A: Configuring the Streaming Web Service for SAP HANA Smart Data Streaming

This post will provide a hands on example of configuring, running and testing the Streaming Web Service (“SWS”) for SAP HANA Smart Data Streaming SP10. It will explain how to edit the configuration file, streamingweb.xml, and how to run and test the Streaming Web Service.

Detailed documentation for the Streaming Web Service can be found at: Help Portal, Streaming Web Service

Prerequisites

  • Access to a HANA system with the Smart Data Streaming SP10 option installed and configured.
  • HANA user ID with permissions to create, compile, deploy and run a Smart Data Streaming project. This can be the HANA SYSTEM user which is granted full permissions for streaming as part of the install process.
  • Ability to connect to the Linux host running the Smart Data Streaming as the <sid>adm user in order to start the Streaming Web Service.

What is the Streaming Web Service

The Streaming Web Service is a new addition to the SAP HANA Smart Data Streaming (“SDS”) adapter family. SWS is a scalable gateway which provides HTTP-based access to SAP HANA SDS. New to SP10, it provides greater scalability and performance than the older Web Service Provider (“WSP”). Like the WSP, the SWS can receive requests from multiple web services and publish data to multiple projects.  This contrasts with other adapters which only send and receive data from one source and publish to one project.

The SWS has 2 significant functional differences from the WSP:

  1. The REST and Websocket request and message formats supported by the SWS are different from the message formats previously required by the WSP, which means that upgrading applications to work with the new SWS will require changes to the JSON message body.
  2. SWS uses a token based authentication in contrast to the WSP user ID & password authentication. In order to publish to a stream or window using SWS, a client must first obtain an authorization token by sending a REST request with its credentials and requested permissions. The Streaming Web Service responds with a token which must be included in the Authorization header of subsequent requests. The token expires after a specified duration which can be set in the configuration file. When this occurs, the client must send a new authorization request.

Configuration

The configuration details for the SWS are stored in a file called streamingweb.xml, located in the directory $STREAMING_HOME/etc.

To edit this file you need to connect to the SDS node in your HANA system as the <sid>adm Linux user.

First, we will edit the config file, streamingweb.xml. Navigate to the directory $STREAMING_HOME/etc. Next, open the streamingweb.xml file with a text editor such as vi. You should see this:

The file is split into 4 sections: <server>, <workers>, < security>, <cluster>. There are several properties for each section that can be changed. If left blank, their default value will be used. Full documentation can be found here: Help Portal, Configuring the Streaming Web Service.

Under the server section, leave the <host> blank. This means it will be able to bind to any interface server. Also note down the <port> number, which should be 9093 by default. We will be using 9093 for this tutorial. Next, under the security section, ensure that <ssl-enabled> is set as its default value of “true”. Finally, under <token-validity-seconds> change the value to 3000.

If you would like to turn on tracing, there are options to do so under <workers>. This is done by changing <trace> and/or <trace-http> to “true”.

Leave the default values for the other entries. Once you are done, save and close the file.

Running the Service

On the SDS node, navigate to the $STREAMING_HOME/bin directory. To start the SWS run the following command:

./streamingwebservice

This runs the streamingwebservice executable, which starts the SWS. If your tracing is set to false, you should see something like this:

The Streaming Web Service is now up and running!

While running the SWS as a foreground process is useful, when configuring and testing the connectivity it is not ideal for production usage, as foreground processes will be shut down whenever the SSH session is closed. When running the SWS in production, it is generally recommended to start the process in the background and have it continue running even you disconnect from the SSH session. This can be done by running this command instead, when starting SWS:

nohup streamingwebservice&

Where:

  • nohup instructs the process to ignore shutdown signals when the SSH session is closed
  • The trailing “&” causes the streamingwebservice process to run as a background process

Note: all output will be redirected to the $STREAMING_HOME/bin/nohup.out file

Part B: Testing Streaming Web Service connectivity using the Chrome “Advanced Rest Client”

This section will walk through the basic message formats of Streaming Web Service, using the Google Chrome App “Advanced Rest Client”.

You can download the Advanced Rest Client for the Chrome browser here: https://chrome.google.com/webstore/detail/advanced-rest-client/hgmloofddffdnphfgcellkdfbfbjeloo

Prerequisite

For the purpose of this section of the tutorial, please go back to the streamingweb.xml configuration file, and set the value of <ssl-enabled> to “false”. You may change it back to “true” after you have finished testing SWS with the Advanced Rest Client.

Sample Streaming Project

Provided below is a sample streaming project consisting of a single input stream. We will use the MACHINEDATA input stream to receive JSON messages via a REST post.

CREATE INPUT STREAM MACHINEDATA SCHEMA (

        MACHINEID string ,

        EVENT_TIME msdate ,

        EVENT_NAME string ,

        EVENT_DESCRIPTION string ,

        EVENT_VALUE string ) ;

       

Note: This input stream is taken from the Freezer Monitoring tutorial. If you have already completed the Freezer Monitoring tutorial, you can use the existing Freezer Monitoring project for this tutorial.


Configuring the Streaming Project to Enable Web Services

Start by navigating to the “SAP HANA Streaming Development” perspective and opening the project configuration (.ccr) file for your streaming project. Under the “Advanced” tab in the Project Configurations window that appears, click on “Project Deployment (Non-HA)”. Enable Web Services by checking the corresponding box in the Project Deployment Details and setting the value in the drop down menu to “true”.

Note: If the check box to the left of the “Web Service Enabled” option is not checked, then the option value of true or false is ignored.

Start the Sample Project

Once you have configured the sample project to enable web services, compile and run the project on the SDS server. In the SAP HANA Streaming Run-Test perspective, open the stream view for the MACHINEDATA stream. This is where we will see the event records appear after we post them from the REST client.

Advanced Rest Client

When first opened, the Advanced Rest Client app will look like this:

Click the drop down arrow (circled in blue), and delete the auto generated “Key | Value” query parameter line(s). This is done by clicking the x. We need to do this or else the URL will be wrong.

There are a few relevant fields on this interface which we will be using, and they are labeled above as HOST, PATH, HTTP Methods, Header, and Payload.

Sending and Receiving Data

First, let’s test if SWS is running, by obtaining the version string. Type/change the following in the corresponding fields, and leave all other fields blank:

HOST:

http://< my.streaming.host.com>:9093

PATH:

/1/version

HTTP Method:
GET

The format for HOST is http://< my.streaming.host.com>:<port>. The <my.streaming.host.com> is the fully qualified domain name (“FQDN”) or TCP/IP address of the SDS node of the HANA system. Make sure the port number matches the one in the streamingwebservice.xml file which should be 9093 if you kept the default port number as recommended above.

Click send. Scroll to the bottom of the screen, and you should be able to see the returned string inside the Response field. The format is:

Obtaining the Authentication Token

Before you can publish messages, SWS requires that you request for a token for the specific resource you intend to access. The returned token then needs to be included in the headers of subsequent POSTs.

The Payload of the JSON message can be used to request access to multiple resources at once, and the format is the following:

[

{ "privilege":"view|write", "resourceType":"project|stream", "resource":"<workspace>/ <project>/<stream>"},

{ "privilege":"view|write", "resourceType":"project|stream", "resource":"<workspace>/ <project>/<stream>"},

……………….

]

Note: The token is only valid for a set number of seconds, which is defined in streamingwebservice.xml. The default value is 300, but we changed it to 3000 for the sake of this tutorial. After this time period, a new token will need to be requested.


For this test, we need to request a token that will allow us to publish to the MACHINEDATA stream of the freezer_monitoring project running under the default workspace. Type/change the following in the corresponding fields, and leave all other fields blank:

HOST:

http://<my.streaming.host.com>:9093

PATH:

/1/authorization

HTTP Method:

POST

Header:

Authorization: Basic <auth-data>

Payload:

[

{ "privilege":"write", "resourceType":"stream", "resource":"default/freezer_monitoring/MACHINEDATA"}

]

Fill <auth-data> with the base64-encoded authorization data for the SYSTEM user. This can be generated automatically by the Chrome App, just toggle from the Raw tab to the Form tab, put “Authorization” in the key. Then click on the value field, and click “construct”.

Enter the username SYSTEM and password in the subsequent screen, and it will fill in the value field for you.

This is what it looks like:

Click send, and the response should contain the token. The format is:

Response: [ { "sws-token":"<token>" } ]

Copy the token onto your clipboard for the next step.

Publishing to a Stream

The data that will be written to the MACHINEDATA input stream in the SDS project needs to be defined as a JSON payload. The JSON message body can be sent in either of 2 formats:

  1. A “map” format
  2. An “array” format

The structure of the “map” format is:

[

{ "ESP_OPS":"<opcode>", "<field1>":<value1>, "<field2>":<value2>, ... },

{ "ESP_OPS":"<opcode>", "<field1>":<value1>, "<field2>":<value2>, ... }

...

]

Each line sends values to the corresponding fields, based on field name. Any unspecified fields will be sent the value of NULL.

The structure of the “array” format is:

[

[ "<opcode>", <value1>, <value2>, <value3>, ...],

[ "<opcode>", <value1>, <value2>, <value3>, ...]

...

]

With this method, every field must be sent a value, and the order they are typed must correspond to the schema.

For this tutorial we will use the map format. Enter the following into the corresponding fields of the Advanced Rest Client, and leave all other fields blank:

HOST:

http://dewdflhana2370.emea.global.corp.sap:9093

PATH:

/1/workspaces/default/projects/freezer_monitoring/streams/MACHINEDATA

HTTP Method:

POST

Header:

Authorization: SWS-Token "sws-token"="<token>"

Payload:

[

{"ESP_OPS":"i", "MACHINEID":"Adv Rest Cli", "EVENT_TIME": "2012-01-23T04:45:36.123", "EVENT_NAME":"DOOR", "EVENT_DESCRIPTION":"sws", "EVENT_VALUE":"DoorOpen"}

]

Paste the token you obtained previously in the place of <token>. Your screen should look like this:

Click send. If successful, you will see this:

You can now go to HANA Studio to check for the output:

Expired Tokens

If your token expired during this process, you will receive the error “419 Authentication Timeout”.

If this occurs, you will need to go back to the step Obtaining the Authentication Token, obtain a new token, and retry the step Publishing to a Stream.

Former Member