on 10-04-2016 2:50 AM
Dear all,
Recently I read a post about how to migrate hadoop to IQ with xp_cmdshell -- http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/109181ef-1f84-3010-feac-9c99f0d66... --, in my memory -- this is a OS command call inside the DBMS so that we can do many OS operations from DBMS.
So I made some test exciting -- call xp_cmdshell('touch /tmp/t.tmp'); -- it works perfectly.
But then with a further test, here's a test shell to connect to db and print the current timestamp out --
asiq160@bigDataIQ:~/rs/reliance/monitor/zr3> cat t.sh
#!/bin/bash
source /asiq160/IQ.sh
dbisql -c "uid=dba;pwd=sql" -host 10.128.244.161 -port 2641 -nogui t.sql
asiq160@bigDataIQ:~/rs/reliance/monitor/zr3> cat t.sql
select now() ># /tmp/t.tmp;
And the t.sh can work fine when I call it under linux command prompt. But when I call it with ---
call xp_cmdshell('/asiq160/rs/reliance/monitor/zr3/t.sh');
--- in dbisql
it will not generate any output and without any error warning...
I'm confused. So does it means xp_cmdshell can't call a script or it means xp_cmdshell can't call the dbisql to connect back to the IQ server itself? Please kind help me out of this. Thanks in advance.
Regards
Eisen
Hello Eisen, I have just ran the same test as yourself on a IQ 16 Sp08 install on RHEL Linux and it worked fine.
I very much suspect you may be hitting the old gotcha that the OS fork call to spawn the child process that runs the unix command outside of IQ does an initial memory footprint check to ensure there is enough memory to spawn the process. The call checks that there is at least potentially the same amount of memory available (free RAM + Swap) as what the parent process is currently using (i.e. the IQ server process). If not the call fails with a ENOMEM error. In reality it does not need, nor uses anywhere near this amount.
To get it to work you need to ensure that once IQ has started that the amount of memory that IQ is using is < Free Ram + SWAP. You can test if this is the issue by starting up a demo db server or test server of your own that is configured with a very small amount of cache.
Please refer to 1969865 - xp_cmshell fails with return status 2 - SAP IQ for further information.
Regards Danie
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.
Dear Danie,
Thanks for your quick response... And this is an interesting answer. But after some check, I think maybe it's not my issue --
Here's my test server -- ASIQ16.10 on SUSE 11 . And the configuration is --
bigDataIQ:/iq_data/mpx # cat mpxtest.cfg
-x tcpip{port=2640}
-n mpxtest
-c 48m
-gc 20
-gd all
-gl all
-gm 10
-gp 4096
-ti 4400
-gn 25
-iqmc 1000
-iqtc 1000
So we can see -- the memory consuming is only slightly more than 2G. And the free memory on the test server is --
bigDataIQ:/iq_data/mpx # free
total used free shared buffers cached
Mem: 65979812 42524460 23455352 0 410648 32799692
-/+ buffers/cache: 9314120 56665692
Swap: 41943036 31192 41911844
So I think the free memory is enough to fork the child process... And there's no error or exception in iqmsg file. So I think it should be another issue. Thanks for your help and appreciate for any other ideas.
Regards
Eisen
can you try strace this in a system where only you are connected running this test , then see if the xp_cmdshell call is made and if you are getting past the CLONE call. You may be able to then see how far its getting and if it hits any other type of error?
Another thought, when you are trying this are there other user connections? I noticed your -gm setting is low at 10 (10 concurrent user connections to the database server), perhaps the dbisql connection in from the script being called is hitting this limit perhaps? You could try also turning on -z logging and request level logging (-zr ALL) and check the connection back in is being made and the "select now() ># /tmp/t.tmp;" SQL is being executed ?
Hi Eisen,
To check further, as suggested by Danie, whether memory is actually involved, you can trace the IQ process like this:
strace –f –tt –o strace_iq.out -p <IQ_PID>
where <IQ_PID> is the unix process ID of iqsrv16.
Test again xp_cmdshell
Then stop strace
Look for "ENOMEM" or "Cannot allocate memory" in strace_iq.out.
Also take a look in stderr file and OS messages.
Regards
Tayeb.
Dear all,
Thanks a lot. Yeah... in the strace log file, I can see --
7382 08:36:15.951829 close(3) | = 0 |
7382 08:36:15.952020 munmap(0x7f44b0c85000, 4096) = 0
7382 08:36:15.952213 mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|0x40000, -1, 0) = -1 ENOMEM (Cannot allocate memory)
7382 08:36:15.952478 open("/proc/self/coredump_filter", O_RDWR) = 3
7382 08:36:15.952686 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
7382 08:36:15.952904 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f44b0c85000
7382 08:36:15.953100 read(3, "00000033\n", 1024) = 9
7382 08:36:15.953297 lseek(3, 0, SEEK_SET) = 0
7382 08:36:15.953493 write(3, "0x73", 4) = 4
7382 08:36:15.953691 close(3) | = 0 |
But ... it makes me more confused -- why it can't allocate another 2G from the 23G free memory to fork child process?
Regards
Eisen
Hi, this is happening in a mmap call rather than the child process clone call. You may need to raise an incident with SAP Product Support to progress this further. Please include the full strace and also please check what you values are in /proc/sys/vm/overcommit_memory and overcommit_ratio.
User | Count |
---|---|
101 | |
13 | |
13 | |
11 | |
11 | |
7 | |
6 | |
5 | |
4 | |
4 |
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.