cancel
Showing results for 
Search instead for 
Did you mean: 

Errors with HANA Hadoop Integration

ziyi_jiang
Explorer
0 Kudos

Hi,


I'm trying HANA Hadoop Integration using SAP HANA SPS09, Hadoop 2.7.1 and HIVE 1.2.1.

I have followed the videos on the youtube (SAP HANA Academy - SDA : Hadoop Enhancements - 1. Creating User [SPS09] - YouTube) and also read the HANA documents (Administration Guide) on this topic.


All I want to do is to follow the video to run WordCount using HANA Hadoop Integration.

So I have assigned all the necessary privileges to the user and created MR Job Archives, Remote Source and Virtual Functions as shown below:


I have created the mrjobs.

Then I have run the two SQLs successfully.

create remote source hadoop_source

adapter "hadoop"

configuration 'webhdfs_url=http://myserver:50070;webhcat_url=http://myserver:50111'

with credential type 'PASSWORD'

using 'user=hduser;password=hdpass';


create virtual function HADOOP_WORD_COUNT()

RETURNS TABLE ("word" NVARCHAR(60), "count" integer)

package DEV."dev.HanaShared::WordCount"

CONFIGURATION 'enable_remote_cache=true;mapred_jobchain=[{"mapred_input":"/data/mockingbird/",

"mapred_mapper":"sap.WordCountMapper",

"mapred_reducer":"sap.WordCountReducer"}]'

AT HADOOP_SOURCE;



Currently I'm experiencing an issue when executing the last step.

So I wish to check the results:


select * from HADOOP_WORD_COUNT();

But then the console gives me these errors:


Could not execute 'select * from HADOOP_WORD_COUNT()' in 74 ms 384 µs .

SAP DBTech JDBC: [2048]: column store error: search table error:  [2620] executor: plan operation failed;

I went to check the log file of my HANA and found these:


pop = executorPy.ceCustomCppPop() # pop1

pop.setNodeName('$$_SYS_SS2_RETURN_VAR_$$HADOOP_WORD_COUNT')

pop.setUseInternalTable()

pop.addViewAttribute('word', datatype=83, intDigits=60, sqlType=37, sqlLength=60)

pop.addViewAttribute('count', datatype=73, sqlType=3)

pop.setLocale('BINARY')

pop.setUserSchema('DEV')

pop.addPlanDebugOpDataInfo(, scenarioName = 'DEV:_SYS_SS_CE_166142_139899986562304_3_TMP')

pop.addKeyValuePair( 'CONFIGURATION','enable_remote_cache=true;mapred_jobchain=[{"mapred_input":"/data/mockingbird","mapred_mapper":"sap.WordCountMapper","mapred_reducer":"sap.WordCountReducer"}]')

pop.addKeyValuePair( 'PACKAGE_NAME','dev.HanaShared09::WC')

pop.addKeyValuePair( 'PACKAGE_SCHEMA','DEV')

pop.addKeyValuePair( 'REMOTE','HADOOP_SOURCE')

pop.addKeyValuePair( 'RETURN_TYPE_INFO','[{"ftcType":37,"index":0,"length":60,"name":"word","scale":0},{"ftcType":3,"index":1,"length":0,"name":"count","scale":0}]')

pop.addKeyValuePair( 'VUDF_NAME','DEV_HADOOP_WORD_COUNT')

pop.setPopId(2251)

pop.addViewAttr('count')

pop.addViewAttr('word')

pop.setExecuteUser('DEV')

[16093]{328130}[29/-1] 2015-10-21 14:24:36.867483 e cePlanExec       cePlanExecutor.cpp(07145) : Error during Plan execution of model DEV:_SYS_SS_CE_166142_139899986562304_3_RET (-1), reason: executor: plan operation failed;

[16060]{-1}[-1/-1] 2015-10-21 14:26:17.056104 w Logger           SavepointImpl.cpp(02149) : NOTE: BACKUP DATA needed to ensure recoverability of the database

[18169]{328227}[22/-1] 2015-10-21 14:26:35.373304 i TraceContext     TraceContext.cpp(00823) : UserName=SYSTEM, ApplicationUserName=xxxxxxxx, ApplicationName=HDBStudio, ApplicationSource=csns.admin.commands.AdministrationHandler$1$1.call(AdministrationHandler.java:338);csns.admin.commands.AdministrationHandler$1$1.call(AdministrationHandler.java:1);java.util.concurrent.FutureTask.run(FutureTask.java:266);java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142);java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617);java.lang.Thread.run(Thread.java:812);

[18169]{328227}[22/-1] 2015-10-21 14:26:35.373288 e PlanViz          PlanVizAction.cc(00045) : PlanVizContext is NULL!!

[18169]{328227}[22/-1] 2015-10-21 14:26:35.373324 e PlanViz          PlanVizAction.cc(00046) : Current session context: systemWatermark=30125,slaveInitCount=-1,version=5,contextId=104427,globalSessionId=328227,anchorGlobalSessionId=328227,version=0,user=SYSTEM,schema=SYSTEM,locale=en_US,collate=BINARY,client=,curr_id_val=-1,app=HDBStudio,app_user=xxxxxxx,dateformat=,reserveprefix=true,ddlautocommit=false,checkPasswordChangeNeeded=false,abapVarcharMode=false,largeNumberOfParametersSupport=false,isFederationCallbackSession=false,associatedConnectionId=0,totalRowCount=0,enableDeferredLobOperation=0,hasStatefulCtxBitmap=4,tmpTableCount=0,transactionIsolationLevel=1

[18169]{328227}[22/-1] 2015-10-21 14:26:35.373356 e PlanViz          PlanVizAction.cc(00047) : Stack trace:

1511995[thr=18169]: SqlExecutor at

1: 0x00007f4867900052 in Execution::ContextFunctions::dumpInfo(Execution::Context&, ltt::basic_ostream<char, ltt::char_traits<char> >&, bool, bool, bool, bool, bool)+0x390 at ContextFunctions.cpp:657 (libhdbbasis.so)

2: 0x00007f485bfd91aa in ptime::PlanVizActionParam::init(ptime::Env const&)+0x506 at PlanVizAction.cc:48 (libhdbrskernel.so)

3: 0x00007f485bf81ce2 in ptime::BuiltinProcedure_PLANVIZ_ACTION::execute(ptime::Env&) const+0x190 at PlanVizAction.h:14 (libhdbrskernel.so)

4: 0x00007f485b53f4f6 in ptime::Proc_call::execute(ptime::Env&) const+0x3a2 at qe_proc_call.cc:268 (libhdbrskernel.so)

5: 0x00007f485b54006c in ptime::Proc_call::operator()(ptime::Env&) const+0x728 at qe_proc_call.cc:141 (libhdbrskernel.so)

6: 0x00007f485be292ae in ptime::Query::_execute(ptime::Transaction&, char const*, ptime::Query::Plan*, ptime::Query::param_t*, ptime::Query::result_t*, bool)+0x5fa at query.cc:5249 (libhdbrskernel.so)

7: 0x00007f485be2f9db in ptime::Query::execute(ptime::Transaction&, char const*, ptime::Query::param_t*, ptime::Query::Plan*, ptime::Query::result_t*, ptime::Statement*, bool)+0x647 at query.cc:701 (libhdbrskernel.so)

8: 0x00007f485ab51b79 in ptime::Statement::execute_(Execution::Context&, bool, bool, bool, bool)+0x355 at Statement.cc:2054 (libhdbrskernel.so)

9: 0x00007f485ab7b03c in ptime::CallableStatement::execute(Execution::Context&, bool, bool, bool, bool, ptime::Statement::BatchProcessingState, bool, bool, bool)+0x588 at CallableStatement.cc:503 (libhdbrskernel.so)

10: 0x00007f4862863c8f in ptime::Session::executeQuery(Execution::Context&, ptime::Action&)+0xdb at sm_session.cc:1357 (libhdbsqlsession.so)

11: 0x00007f4862858ff0 in ptime::SessionHandler::handleEvent(Execution::Context&, ptime::AppEvent*)+0x4f0 at sm_handler.cc:846 (libhdbsqlsession.so)

12: 0x00007f486285a401 in ptime::SessionHandler::receiveMessage(Execution::Context&, ptime::CommEvent*)+0x960 at sm_handler.cc:647 (libhdbsqlsession.so)

13: 0x00007f486287907c in ptime::TcpReceiver::doWork(Execution::Context&, ptime::CommMgr*)+0xd78 at tcp_receiver.cc:505 (libhdbsqlsession.so)

14: 0x00007f4862879b0a in ptime::TcpReceiver::run(void*)+0x1d6 at tcp_receiver.cc:604 (libhdbsqlsession.so)

15: 0x00007f4875feb4d4 in TrexThreads::PoolThread::run()+0x810 at PoolThread.cpp:256 (libhdbbasement.so)

16: 0x00007f4875fecfb0 in TrexThreads::PoolThread::run(void*&)+0x10 at PoolThread.cpp:124 (libhdbbasement.so)

17: 0x00007f4867958439 in Execution::Thread::staticMainImp(void**)+0x875 at Thread.cpp:488 (libhdbbasis.so)

18: 0x00007f4867958ffd in Execution::Thread::staticMain(void*)+0x39 at ThreadMain.cpp:26 (libhdbbasis.so)


Judging from the logs it seems the plan executor fails before the job could be able to be started.


BTW

I can successfully see the controller.jar in hadoop

Also I have uploaded the jars in the lib folder following the administration guide.


I tried different HANAs including SPS09 and SPS10 to perform the integration, all give me the same error.


So my question is has anyone happened to face the same issue?

Or if there is anything wrong with my previous steps?

Thanks in advance.

Accepted Solutions (0)

Answers (1)

Answers (1)

Former Member
0 Kudos

Hi Jiang,

I am also facing some error while connecting to Hadoop using SAP HANA SPS 10.

I am getting  "SAP DBTech JDBC: [403]: internal error: Cannot get remote source objects: Credential not found" error while creating remote source.

I followed SAP Note 2177933. In this note, it is mentioned to provide any user name and password. so I have used "hanaes" user name and password.


CREATE REMOTE SOURCE "spark_demo" ADAPTER "sparksql" 

CONFIGURATION 'port=7860;ssl_mode=disabled;server=<actual server>' 

WITH CREDENTIAL TYPE 'PASSWORD' USING 'user=hanaes;password=hanaes'

Can you please let me know what credentials should I use in remote connection.

Thank you.

Best regards,

Ram

ziyi_jiang
Explorer
0 Kudos

Hi Ram,

You should use the credentials of the Linux user who have access to Hadoop, e,g, if you have created a user called hduser in Linux and configured the Hadoop with this user, you should enter the credentials of hduser. Also root user works.

Former Member
0 Kudos

Hi Jiang,

Thank you for the reply.

We created a new Linux user "hanauser" which has access to Hadoop system other than ec2 user and tried to create the remote source. But still we are facing same issue.

Do we need to have "aws.resolver-0.0.3.jar" as well or any other .jar file required for the connection.

As we are on HANA SPS 10, we need not to have JDBC connector installed right Jiang?

Please let me know if we need to establish any other connections.

Thank you.

Best regards,

Ram

ziyi_jiang
Explorer
0 Kudos

Hi Ram,

    We only need to pass a delivery unit called HANA_HADOOP_CTLR into HANA. You can refer to this post

    Also there are series of video tutorials in SAP HANA Academy - SDA : Hadoop Enhancements - 1. Creating User [SPS09] - YouTube . Hope this helps.

Best regards,

Ziyi Jiang

Former Member
0 Kudos

Hi Jiang,

Very thanks for the reply.

Presently we are working on SAP HANA SPS 10. We are now able to connect to Hadoop system successfully. But after creating virtual table, we are unable to see the content. we are getting the below error.

Could not execute 'SELECT TOP 1000 * FROM "HDPUSER"."spark_demo_products"' SAP DBTech JDBC: [403]: internal error: Error opening the cursor for the remote database for query "SELECT "spark_demo_products"."productid",

Did you face any error of this sort Jiang. Could you please help us if you have any solution for it?

Thank you.

Best regards,

Ram

ziyi_jiang
Explorer
0 Kudos

Hi Ram, I haven't met the errors before, maybe you should check the configuration.

Those posts may help:

Former Member
0 Kudos

Hello Ziyi Jiang,

Thanks a lot for the help.

As we are working on spark controller, I think this is little different with HIVE. But definitely I will check these posts.

Thank you.

Best regards,

Ram