Skip to Content
SAP HANA streaming analytics

Publishing Event Data to Streaming Projects using REST

Tags:

Prerequisites

  • Access to a HANA system with the Smart Data Streaming option installed and configured.
  • HANA user ID with permissions to create, compile, deploy and run a Smart Data Streaming project. This can be the HANA SYSTEM user which is granted full permissions for streaming as part of the install process.
  • Ability to connect to the Linux host running the Smart Data Streaming as the <sid>adm user in order to start the Web Service Provider (if not already started).
  • This tutorial assumes that you have already configured and started the Web Service Provider as per the instructions in the Configuring the Web Service Provider (WSP) for use with SDS Projects tutorial.

Introduction

Using web services to post event data to SAP HANA Smart Data Streaming (“SDS”) projects is a popular choice. The Web Service Provider (“WSP”) component of Smart Data Streaming supports the posting of events through both REST and Websocket connections. When individual ‘ad hoc’ events may be posted with only one or a few posts originating from a given source, then using a REST post is the recommended method. An example where REST posts may be preferred is collecting click data from web pages, where any given browser session may send as few as 1 individual page click.

The benefit of a REST post in this case relative to using a Websocket connection is that a REST post is self-contained and includes the authentication credentials as part of the post. In contrast, Websocket connections require separate requests to open and authenticate the connection, then send the event data. When a continuous or at least an extended series of events is being posted from a single source, for example an automated sensor, then a Websocket is the preferred web service interface as it authenticates the connection once for the session rather than for each individual request. Again, when posting a single event, using a REST post is more efficient than using a WebSocket and incurring the overhead of opening and closing the WebSocket only to send a single post.

The purpose of this tutorial is provide an example of constructing and sending event data to a stream in an SDS project using a REST web service post. This tutorial is intended to supplement the existing SDS REST Provider documentation (REST Documentation) by pairing step by step instructions inspired by the documentation with a working example. Specifically, the following steps will be addressed:

  1. Configuring a streaming project to allow the use of REST posts
  2. Using an HTML page to initialize an xmlhttp Javascript object
  3. Creating JSON posts to be sent via the xmlhttp object to the streaming project
  4. Sending the JSON posts to the streaming project

Sample Streaming Project

Provided below is a sample streaming project consisting of a single input stream. When the project is configured as defined in the separate “Configuring the Web Service Provider (WSP) for use with SDS Projects ” tutorial, the MACHINEDATA input stream can be used to receive JSON messages via a REST post.

CREATE INPUT STREAM MACHINEDATA SCHEMA (

       MACHINEID string ,

       EVENT_TIME msdate ,

       EVENT_NAME string ,

       EVENT_DESCRIPTION string ,

       EVENT_VALUE string ) ;

          

Note: This input stream is taken from the Freezer Monitoring tutorial. If you have already completed the Freezer Monitoring tutorial, you can use the existing Freezer Monitoring project for this tutorial.

Configuring the Streaming Project to Enable Web Services

Start by navigating to the “SAP HANA Streaming Development” perspective and opening the project configuration (.ccr) file for your streaming project. Under the “Advanced” tab in the Project Configurations window that appears, click on “Project Deployment (Non-HA)”. Enable Web Services by checking the corresponding box in the Project Deployment Details and setting the value in the drop down menu to “true”.


Note: If the check box to the left of the “Web Service Enabled” option is not checked, then the option value of true or false is ignored.

Publishing to a Streaming Project from an HTML Page

Using Javascript, it is possible to send REST JSON posts to a running Streaming project from a web page. To create an HTML/Javascript page containing REST post functionality, the following steps are required:

  1. Create and open the XMLHttpRequest() object. The XMLHttpRequest()object will be used to construct and send a structured HTTP request containing the required JSON message to post the event to the Smart Data Streaming project via the Web Service Provider.
  2. Construct the JSON message within the XMLHttpRequest() object, containing
    1. Credentials to authenticate and post to the target Smart Data Streaming workspace, project and stream.
    2. Data for each expected column of the target input stream. The order must match the column order of the target input stream.
  3. Send the JSON message using the send() method of the XMLHttpRequest()object.

Create and Open the XMLHttpRequest()Object

  1. To create the XMLHttpRequest()object, we need to declare a variable and instantiate it as a new XMLHttpRequest()instance.

For example the javascript to create a new XMLHttpRequest object  can look like this:

var xmlhttp = new XMLHttpRequest();// new HttpRequest instance

Note: In the sample HTML page at the end of this tutorial, the variable is declared in the //global variable declarationssection and then instantiated in the setPostMethod() function. 

  1. The open()method of the XMLHttpRequest()object is used to set the request type and specify the URL that the request will be sent to. Since this is a REST request, the request type will be set to “POST”.

The URL to be used for the connection needs to follow the format below:

http://host:port/espws/restservice/stream/stream?action=insert&workspace=workspace&project=project


  • The host is the fully qualified domain name (“FQDN”) or TCP/IP address of the SDS node of the HANA system that is running the WSP.
  • The port is the REST port specified in the wsp.xml file (Default is 9091, for instructions on how to specify a different port check the “Configuring the Web Service Provider (WSP) for use with SDS Projects" tutorial)
  • stream is the name of the input stream we will be publishing our JSON posts to
  • workspace is the workspace that your’re streaming project is running within on the SDS server. This is most likely to be “default”.
  • project is the name of the streaming project you would like to publish to

For example the javascript to open the XMLHttpRequest()object can look like this:

xmlhttp.open("POST", http://my.streaming.host.com:9091/espws/restservice/stream/MACHINEDATA?action=insert&workspace=default&project=dmm165_sds, true);


Note: In the sample HTML page at the end of this tutorial, the xmlhttp.open() call is made in the buildAndSendJSON() function. 

Construct the JSON Message

Now that we have the XMLHttpRequest()object created and opened, we need to build out the JSON message and assign it to the body of the XMLHttpRequest()object. The JSON message needs to specify both the credentials to be used for authentication and the contents of the event record to be posted to the input stream in the streaming project. Structurally both the credentials and the content are objects in their own right which in turn are part of the full JSON object.

The JSON contents should look as follows:

{

"defaultCluster" : { "credentials":"<username>:<password>"},

"content" : {

"<column-name>":"<value>",

"<column-name>":"<value>",

"<column-name>":"<value>",

"<column-name>":"<value>"

}

}


Where,

  • <username> is a HANA user name with permissions to write to the desired input stream. This could be the HANA SYSTEM user, or you can grant a separate HANA user the required permissions. (See the APPENDIX – CREATING A HANA USER AND GRANTING SMART DATA STREAMING PERMISSIONS in the Freezer Monitoring Tutorial for instructions)
  • <password> is the HANA password for the username being specified
  • The "<column-name>":"<value>" pairs make up an array of the expected columns and the value for each column of the target input stream that we will be publishing our JSON posts to. Note that the column values must be provided in the order that the columns were declared when creating the input stream. For the MACHINEDATA sample that we are using that means you need to provide the "<column-name>":"<value>" pairs in the order of MACHINEID, EVENT_TIME, EVENT_NAME, …

To construct the JSON message we will first construct the credentials object and the content object then combine them into the full JSON object.

To create the credentials object, we use the following javascript to declare a new object with the name credentialsObject and then create and assign the "credentials" attribute.

var credentialsObject = {}; credentialsObject["credentials"]="sap:password";

Note: In the sample HTML page at the end of this tutorial, the credentialsObject is declared in the //global variable declarations section and populated in the init() function.

To create the contents object, we use the following javascript to declare a new object with the name contentObject and then create the array of "<column-name>":"<value>" pairs.

var contentObject = {};

contentObject["MACHINEID"]=document.getElementById("MachineID").value;

contentObject["EVENT_TIME"]=document.getElementById("EventTime").value;

contentObject["EVENT_NAME"]=document.getElementById("EventName").value;

contentObject["EVENT_DESCRIPTION"]=document.getElementById("EventDescription").value;

contentObject["EVENT_VALUE"]=document.getElementById("TemperatureValue").value;

Note: In the sample HTML page at the end of this tutorial, the contentObject is declared in the //global variable declarations section and populated in the gatherUserInputs()  function.

Now we need to combine the credentials and content objects to form the complete JSON object. To this we first declare a new object with the name jsonObject and then assign the credentialsObject object to the “defaultCluster” attribute and the contentObject object to the “content” attribute.

var jsonObject = {};

jsonObject["defaultCluster"]=credentialsObject;

jsonObject["content"]=contentObject;

Note: In the sample HTML page at the end of this tutorial, the jsonObject is declared in the //global variable declarations section and populated in the gatherUserInputs() function.

Send the JSON Message

At this point we have completed the construction of the JSON message body that will be posted to the Web Service Provider using the send() method of the XMLHttpRequest()object that we instantiated and opened earlier in the tutorial. The final step in sending the message is to convert the jsonObject to a string and call the send() method with the ‘stringified’ jsonObject as a parameter.

This can be done in a single javascript statement such as:

     xmlhttp.send(JSON.stringify(jsonObject));

Note: In the sample HTML page at the end of this tutorial, the xmlhttp.send() call is made in the buildAndSendJSON()  function.

*Don’t forget to start the WSP server!

 

REST_Sample.html Page

Provided below is a sample html page complete with the Javascript code to build and send REST JSON posts. In order to use this sample page to post REST events to your own Smart Data Streaming server, there are few edits you will need to make to the default credentials and project URL.

  1. The <HANA USER ID> and <HANA Password> assigned to the credentialsObject need to be set to the HANA user ID and password that you are using to connect to your Smart Data Streaming server.
  2. The connection URL (<host>) and project name (<project>) values will need to be set to match your Smart Data Streaming system as described in step 2 of the Create and Open the XMLHttpRequest()Object section above.

Finally, it is important to note that this web page must loaded from a web server, as opposed to loading it directly from your local file system. The use of the XMLHttpRequest()object requires that the web page have an “origin” set. This is done automatically when the browser loads the web page from a web server, however it is not set when loading an HTML document from the local file system.

<!DOCTYPE html>

<html>

   <head>

      <title>Machine Event Simulator - SAP HANA Smart Data Streaming</title>

      <style>

.indent{

                 margin-left:2%;

}

      </style>

      <script>

//global variable declarations

var xmlhttp;

var jsonObject = {};

var credentialsObject = {};

var contentObject = {};

  

//function is called on page load (called from body onLoad)

function init(){

        setPostMethod();

        setProjectServerURL();

        customServer();//decide whether to show input field for custom server address

        document.getElementById("EventTime").value = getCurrentDateTime();//set event time field to current time

        setEventValueInput();

        credentialsObject["credentials"]="<HANA USER ID>:<HANA Password> ";

}

  

function setProjectServerURL(){

        if (document.getElementById("RESTProjectURL").value != "customOption"){

              document.getElementById("projectServerURL").innerHTML = document.getElementById("RESTProjectURL").value;

        }else if (document.getElementById("RESTProjectURL").value == "customOption"){//clear SDS Project Server URL innerHTML

                    document.getElementById("projectServerURL").innerHTML = "";

                    document.getElementById("customServer").value="http://";

        }

  }

  

  //will be called upon user click of "Set Server" button

  function setCustomServerURL(){

        document.getElementById("projectServerURL").innerHTML = document.getElementById("customServer").value;

  }

  

  //function gets appropriate project URL based on user selected post method

  function getProjectURL(){

        return document.getElementById("RESTProjectURL").value;

  }

  

  //function displays appropriate project server drop down menu based on user selected post method

  function setPostMethod(){

        customServer();//decide whether to show custom server field

        xmlhttp = new XMLHttpRequest();// new HttpRequest instance

        setProjectServerURL();

  }

  

  //function returning a string representation of the current machine date and time

  function getCurrentDateTime(){

        var currentDateTime;

        var currentDate;

        var currentTime;

        var d = new Date();

        var currentMonth = d.getMonth()+1;

        currentDate = d.getFullYear()+"-"+currentMonth+"-"+d.getDate();

        currentTime = d.getHours()+":"+d.getMinutes()+":"+d.getSeconds()+"."+d.getMilliseconds();

        currentDateTime = currentDate+" "+currentTime;

        return currentDateTime; 

}

  

function formatDateTime(time){

        var newTime = "";

        var i = 0;

        while (time.charAt(i) != ' '){//get date

              newTime += time.charAt(i);

              i ++;

        }

        newTime += 'T';

        i ++;

        while (i < time.length){//get time

              newTime += time.charAt(i);

              i ++;

        }

        newTime += 'Z';

        console.log("new time: " + newTime);

        return newTime;

}

  

function gatherUserInputs(){

        contentObject["MACHINEID"]=document.getElementById("MachineID").value;

        contentObject["EVENT_TIME"]=document.getElementById("EventTime").value;

        EventTime = getCurrentDateTime();//reset event time field to current time

        contentObject["EVENT_NAME"]=document.getElementById("EventName").value;

        contentObject["EVENT_DESCRIPTION"]=document.getElementById("EventDescription").value;

        if (document.getElementById("EventName").value=="TEMP"){   

              contentObject["EVENT_VALUE"]=document.getElementById("TemperatureValue").value;

        } else if (document.getElementById("EventName").value=="DOOR"){

              contentObject["EVENT_VALUE"]=document.getElementById("DoorStatus").value;

        } else if (document.getElementById("EventName").value=="POWER"){

              contentObject["EVENT_VALUE"]=document.getElementById("PowerStatus").value;

        }

        

        jsonObject["defaultCluster"]=credentialsObject;

        jsonObject["content"]=contentObject;

        //console.log(JSON.stringify(jsonObj));

  }

  

  function buildAndSendJSON(){

        gatherUserInputs();

        xmlhttp.open("POST", getProjectURL(), true);

        xmlhttp.send(JSON.stringify(jsonObject));

  }

  

  //function to show input field when SDS Project to post to is "Specify Server"

  function customServer(){

        if(document.getElementById("RESTProjectURL").value == "customOption"){//show custom server field

              document.getElementById("customServerOption").style.display = "block";

        }else{//hide custom server field

              document.getElementById("customServerOption").style.display = "none";

        }

  }

  

  //function to display appropriate inputs

  function setEventValueInput(){

        if (document.getElementById("EventName").value=="TEMP"){   

                    document.getElementById("TemperatureValue").style.display = "inline";

                    document.getElementById("DoorStatus").style.display = "none";

                    document.getElementById("PowerStatus").style.display = "none";

        } else if (document.getElementById("EventName").value=="DOOR"){

                    document.getElementById("TemperatureValue").style.display = "none";

                    document.getElementById("DoorStatus").style.display = "inline";

                    document.getElementById("PowerStatus").style.display = "none";

        } else if (document.getElementById("EventName").value=="POWER"){

                    document.getElementById("TemperatureValue").style.display = "none";

                    document.getElementById("DoorStatus").style.display = "none";

                    document.getElementById("PowerStatus").style.display = "inline";

              }

  }

      </script>

   </head>

   <body onLoad="init()">

      <img src="https://www.sapstore.com/_ui/desktop/theme-sap-sapphire/images/vendors/logo-sap-small.png" height="15%" width="15%" style="display:inline;"/>

      <h1 style="display:inline;">Machine Event Simulator</h1>

      <div id="main" style="margin-left:5%; margin-top:5%;">

            <div style="float:right; margin-right:15%; width:40%;">

                  <h3>SDS Project Server URL:</h3>

                  <div class="indent">

                        <p id="projectServerURL"></p>

                  </div>

            </div>

      

            <div>

                  <h3>Select the SDS Project Server to post to:</h3>

                  <div id="RESTProjectServer">

                        <div class="indent">

                              <select id="RESTProjectURL" onChange="setPostMethod()">

                                    <option value="http://<host>:9091/espws/restservice/stream/MACHINEDATA?action=insert&workspace=default&project=<project>">A Server</option>

                               

                                                                              <option value="customOption">Specify Server</option>

                              </select>

                        </div>

                  </div>

            

                  <div id="customServerOption" class="indent">

                        <h4>Specify a server:</h4>

                        <form>

                              <input type="input" id="customServer" value=""/>

                              <input type="button" value="Set Server" onclick="setCustomServerURL()"/>

                        </form>

                  </div>

            </div>

      

            <div style="float:right; margin-right:15%;" id="responseField">

                  <!-- Will be filled with SDS Project Server URL by JS during runtime -->

            </div>

      

            <div id="singleDiv">

                  <h3>Single post event parameters:</h3>

                  <div class="indent">

                        <form>

                              Machine ID: <input type="text" name="machineid" value="2DDDBW3TP" id="MachineID"><br>

                              Event Time: <input type="text" name="eventtime" value="" id="EventTime"><br>

                              Event Name:

                              <select id="EventName" onChange="setEventValueInput()">

                                    <option value="DOOR">DOOR</option>

                                    <option value="POWER">POWER</option>

                                    <option value="TEMP">TEMP</option>

                              </select> <br>

                              Event Description: <input type="text" name="eventdescription" value="" id="EventDescription"><br>

                              Event Value: <input type="text" name="eventvalue" size="8" value="50" id="TemperatureValue">

                              <select id="DoorStatus">

                                    <option value="Door open">Door Open</option>

                                    <option value="Door close">Door Close</option>

                              </select>

                              <select id="PowerStatus">

                                    <option value="Power on">Power On</option>

                                    <option value="Power off">Power Off</option>

                              </select><br><br>

                        </form>

                  </div>

                  <button type="button" onclick="buildAndSendJSON()">Build & Send JSON Post</button>

            </div>

      </div>

   </body>

</html>