Update HADOOP as updates occur via SLT

[patrickbachmann]

Hi folks,

My boss posed an interested question to me today.  He wants to know if any of the tables that we are loading into HANA from SAP via SLT could somehow also be pulled into HADOOP on a delta basis.  So as inserts/updates/deletes are occurring to a particular table like MSEG for example, could HADOOP be somehow polling periodically and just process changes as they come into HANA?

Thanks,

-Patrick

[justin_molenaur2 · Accepted solution]

Two ways that are easy

- Enhance the SLT table via IUUC_REPL_MON, add logic for timestamp to be added to record when it's processed by SLT. Then have the Sqoop job use that timestamp for delta tracking, pulling only data >= timestamp

- Have HANA maintain the timestamp via calculated column; GENERATED ALWAYS AS X and have the current timestamp. Sqoop processes the same way from here.

Probably more transparent with the first option.

As far as I know there is no support for writing directly from SLT to Hadoop, but it was mentioned that you could possibly invoke a webservice in ABAP layer when SLT is processing the record - but it was very tricky. Might get more developed as time goes on, but the above should work no problem.

Regards,

Justin

[justin_molenaur2]

Ah, and another way I have seen it happen is

1) Create a new SLT configuration for the same source from which you are already replicating

2) Perform the initial loads for the desired tables

3) Pause master job

4) Perform Sqoop load in full

5) Restart Master job to load data to tables

6) Pause master job via FM IUUC_REPL_SUSPEND_REPLICATION

7) Load all delta data in schema with Sqoop to Hadoop

😎 Delete all data in schema

9) Start replication again via FM IUUC_REPL_RESUME_REPLICATION

The only tricky part is capturing Deletes, so you have to also do something in SLT to ignore the deletes and instead pass them to HANA with a "Deletion indicator" so Sqoop knows to delete.

This is clearly more data and processing, but you can use the rock solid trigger method for SLT.

Happy HANA,

Justin

[patrickbachmann]

Justin thanks for all the great feedback/ideas.  Let me just digest this for a bit now, see if I have any more questions and will close this thread shortly.

-Patrick

[patrickbachmann]

Ok my next question is if we were to move to ECC on HANA and SLT goes away;  Can we use Smart data streaming or event based replication or are we stuck using an ETL tool?

Thanks!

-Patrick

[justin_molenaur2]

Hmm, ok - slightly different scenario there then. Now you are just looking at how to get data out of ECC really. There may be some outbound tools like you mention, but I haven't really thought about that option to be honest.

Are you saying you won't have an external appliance to serve as your data warehouse?

Regards,

Justin

[patrickbachmann]

To be honest I'm not even sure myself.  I think we are just trying to understand all our options.  Thanks for the great suggestions.

-Patrick