Skip to Content

FTP Adapters Smart Data Streaming

FTP Adapter Smart Data Streaming

Prerequisites

·         This tutorial assumes you have already completed the TowerMon tutorial and have the finished product open in your HANA Studio

·         You will need to have access to an FTP server – this tutorial was written using an IIS FTP server running on the same machine

·         It is recommended to make a copy of your TowerMon tutorial project before starting as both the FTP InputAdapter section and FTP Output Adapter sections begin where the TowerMon tutorial left off

FTP Input Adapter

1. The current project should look as follows (from the TowerMon tutorial) when the .cclnotationfile is open in SAP HANA Streaming Development perspective in HANA Studio.

                             

    • Add FTP CSV Input adapter

·         Open the Input Adapters drawer on the Palette by clicking on the drawer

·         Click on FTP CSV Input to select it. Click on the diagram to add the output adapter

2. Connect output adapter to iwWeatherStationReference

·         Choose the Connectortool in the Palette

·         Click the header of FTP_CSV_Input1 making it the source

·         Click the header of iwWeatherStationReference to make it the destination

 

3.      3. Remove the File_Hadoop_CSV_Input1 input adapter (used in the previous tutorial)

·         Right click the adapter and select Delete Element

4.       4. Set up the FTP_CSV_Input1input adapter

·         Hover over the FTP_CSV_Input1 adapter and click Edit Properties in the popup shape toolbar

·         First, set the Stream name, opcode expected value to FALSE so that the system doesn’t expect more columns than are filled in our data file

·         Second, we will need to set the name or IP address of the FTP server you wish to connect to. For this tutorial, I am using an FTP server running on my own machine so I will be entering the IP address.

·         Since I have set my FTP server to allow anonymous logins, I have set the Login Type value to anonymous, however if your FTP server requires a username and password, keep the Login Type as normal and enter the username and password values in the following 2 fields.

·         Next we will set the name of the file we will be using for input: weather-sensor-inventory.csv

·         Finally, we will set the absolute file path to the weather-sensor-inventory.csv file on our FTP server. On my FTP server, the file is in the test folder sitting in the root directory so I have entered /test/

Note: at the time of writing this tutorial, the file must not be in the root directory. I.e. You may not enter “/” as the File Path

·         Under the Advanced tab double click the Value field beside the Has Header cell and select TRUE

  5. Clean up and check your work

  • Use the layout tools to clean up your diagram
  • Click the Compile tool to   compile your project
  • Check the Console to see if it   compiled without errors.


  • If there are errors, check the Problems  view to see the errors and correct them

6.      6. Run the project and view results

·         Click the dropdown arrow beside the green Runbutton and select Run Streaming Project in Workspace esps://<streaming server>:<port>/<workspace>

·         The perspective will be switched to SAP HANA Streaming Run-Test and you will see a list of adapters and streams under the ftp_input_adapter project in the Server View

·         Double click the iwWeatherStationReference input stream and you should be able to see the data that has been streamed from the weather-sensor-inventory.csv file on your FTP server

FTP Output Adapter

  • ·       The current project should look as follows (from the TowerMon tutorial) when the .cclnotationfile is open in SAP HANA Streaming Development perspective in HANA Studio.

  • ·         Add FTP CSV Output adapter

·         Open the Output Adapters drawer on the Palette by clicking on the drawer

·         Click on FTP CSV Output to select it. Click on the diagram to add the output adapter

3.       Connect the SensorAlert filter to the new FTP_CSV_Output1 adapter

·         Choose the Connectortool in the Palette

·         Click the header of SensorAlert making it the source

·         Click the header of FTP_CSV_Output1 to make it the destination

Note: the arrow points from FTP_CSV_Output1 to SensorAlert even though we have connected them the opposite way, this is okay!

4.       Set up the FTP CSV Output adapter – slightly different than setting up the input adapter

·         Hover over FTP_CSV_Output1and click Edit Properties in the popup shape toolbar

·         First, we will need to set the name or IP address of the FTP server you wish to connect to. For this tutorial, I am using an FTP server running on my own machine so I will be entering the IP address.

·         Since I have set my FTP server to allow anonymous logins, I have set the Login Type value to anonymous, however if your FTP server requires a username and password, keep the Login Type as normal and enter the username and password values in the following 2 fields.

·         Next we will set the name of the file that the FTP Output adapter will be writing to: weather.csv

·         Finally, we will set the absolute file path to the weather-sensor-test.csv file will be written to on our FTP server. On my FTP server, the file is in the test folder sitting in the root directory so I have entered /test/

Note: at the time of writing this tutorial, the file must not be in the root directory. I.e. you may not enter “/” as the File Path or the input adapter will not be able to access the file

Note: By default, a file in the File Path directory having the chosen File Name will be overwritten. If the file does not already exist, it will be created. If you would like to append data to an existing file instead of overwrite it, set the Over Write value to FALSE

·         Under the Advancedtab, double click the Value field beside the Has Header cell and select TRUE so that the header values will be added to the file as well

5.       Subtle adjustment to the SensorAlerts filter

·         In order for our output adapter to receive any data, we are going to have to make an adjustment to the WHERE clause of our SensorAlerts filter

·         Expand the filter by pressing the ‘+’ in the popup shape toolbar

·         Now expand the Filter section, right click the Edit button and select Edit Where Clause

·         Change one of the ‘>’ signs to a ‘<’. For this tutorial I chose to change the second one

·         Press OK

           6. Clean up and check your work

      • Use the layout tools to clean up your diagram
      • Click the Compile tool to   compile your project
      • Check the Console to see if it   compiled without errors.
      • If there are errors, check the Problems  view to see the errors and correct them

            7. Run the project

      • Click the dropdown arrow beside the green Runbutton and select Run Streaming Project in Workspace esps://<streaming server>:<port>/<workspace>


      • The perspective will be switched to SAP HANA Streaming Run-Test and you will see a list of adapters and streams under the ftp_output_adapter project in the Server View


8.       Use the Playback tool to stream in some sample data from a file

·         Go to the Playback view in the lower left of the Studio

·         Click on the Select Project tool in the upper right corner of the Playback View

Note: this connects the playback tool to a project. If you only have one project running, one click will connect it. If you have multiple projects running it will prompt you to choose.

    • Click the Select Playback File tool and select the input-streams.csv file in the replay_data folder of the towermon3 project on your local Windows machine (C:\streamingdemos\towermon3\windows_files\replay_data\input_streams.csv)
    • Change the file type in the window to *.csv so you can see your file
    • (Optional) Change the Playback Mode to rec/ms and set a rate (eg: 0.1) to slow down the playback and get a better view of the alerts being generated over time.
    • Click the Start Playback button

    

  • Open SensorAlerts  in your ftp_output_adapter  project in Server View to view   data as it comes in

  

  • You will notice that a file has either been created or appended to in your FTP server

  

Note: The RecentWeatherData element is a join condition that joins   the isWeatherStream to the iwWeatherStationReference  element. In order for the join to return a record, there has to be a record   coming from the isWeatherStream element that   satisfies the join condition. Until the playback file is run, there is no   data in the isWeatherStream and thus no results   from the join condition in the RecentWeatherData  element

Appendix

If you encounter leading new line characters, ‘\n’, when using the FTP Input adapter, implement the following.

Open $STREAMING_HOME/adapters/framework/instances/ftp_csv_input/adapter_config.xmlin a text editor and change the “IncludeDelimiter” property to “false” as shown below.

     <Module type="formatter">

      <InstanceName>MyStreamingInputFormatter</InstanceName>

      <Name>StreamToStringFormatter</Name>

      <Next>CsvStringToEspFormatter</Next>

      <Parameters>

        <StreamToStringFormatterParameters>

          <Delimiter><![CDATA[\n]]></Delimiter>

                                  <IncludeDelimiter>false</IncludeDelimiter>

                                  <AppendString><![CDATA[\n]]></AppendString>

                                  <AppendPosition>front</AppendPosition>

                                  <IgnoreSpace>true</IgnoreSpace>                       

                   </StreamToStringFormatterParameters>

      </Parameters>

    </Module>

Tags:
Former Member

No comments