Error in Webdispatcher Startup

[sandeep_kumar60]

HI ,

Can you people help me or guide me in analyzing and finding a solution for the below error .

My O.S is AIX

When i try to start the Webdispatcher in the Portal System i come across the following error,this error occurred all of a sudden , which we used to perform every week  sucessfully.

There are no changes made to id or any parameters,

cd /usr/sap/SID/globalwebdisp/portal_webdisp

./sapwebdisp pf=/usr/sap/SID/globalwebdisp/portal_webdisp/sapwebdisp.pfl&

[1] 1781814

sapportdb:gppadm 17> *** ERROR => CTrcOpen: fopen dev_webdisp

---------------------------------------------------

trc file: "dev_webdisp", trc level: 1, release: "700"

---------------------------------------------------

sysno      20

sid

systemid   324 (IBM RS/6000 with AIX)

relno      7000

patchlevel 0

patchno    261

intno      20050900

make:      multithreaded, ASCII, 64 bit, optimized

pid        1781814

[Thr 01] Sun Oct 21 12:23:10 2012

[Thr 01] Profile configuration error detected, use temporary corrected setup

[Thr 01] Shared Pool 10: ipc/shm_psize_10 = 19880000 (too small)

[Thr 01] Shared Pool 10: (smaller than min requirement 130168712)

[Thr 01] *** WARNING => maximum number of sockets supported on this

host (1997) less than parameter icm/max_sockets (8192) [icxxrout_mt. 2791]

[Thr 01] started security log to file dev_icm_sec

[Thr 01] SAP Web Dispatcher running on: sapportdb.geometric.local

[Thr 01] MtxInit: 30001 0 2

[Thr 01] ***LOG Q0I=> NiIBindSocket: could not delete file '/tmp/.sapstream65000' for hdl 0: unlink (1: Not owner) [nixxi.cpp 3207]

[Thr 01] *** ERROR => NiIBindSocket: could not delete file '/tmp/.sapstream65000' (hdl 0; errno=1) [nixxi.cpp    3214]

[Thr 01] ***LOG Q0I=> NiIBindSocket: could not delete file '/tmp/.sapstream64999' for hdl 0: unlink (1: Not owner) [nixxi.cpp 3207]

[Thr 01] *** ERROR => NiIBindSocket: could not delete file '/tmp/.sapstream64999' (hdl 0; errno=1) [nixxi.cpp    3214]

[Thr 01] ***LOG Q0I=> NiIBindSocket: could not delete file '/tmp/.sapstream64998' for hdl 0: unlink (1: Not owner) [nixxi.cpp 3207]

[Thr 01] *** ERROR => NiIBindSocket: could not delete file '/tmp/.sapstream64998' (hdl 0; errno=1) [nixxi.cpp    3214]

[Thr 01] IcmInit: listening to admin port: 64997

[Thr 01] *** ERROR => shmat(5242989,0x0,SHM_RND) (13: Permission denied) [shmux_mt.c   1924]

[Thr 01] *** ERROR => shmctl(5242989,IPC_RMID,0) 0xffffffffffff278) (1: Not owner) [shmux_mt.c   1935]

[Thr 01] *** ERROR => ShmCleanup: ShmCreate(63,0,SHM_ATTACH,...) failed. rtc=1 [shmux_mt.c   4269]

[Thr 01] *** ERROR => ShmCleanup: Inv. Key=63  [shmux_mt.c   4310]

[Thr 01] *** ERROR => shmat(5242989,0x0,SHM_RND) (13: Permission denied) [shmux_mt.c   1924]

[Thr 01] *** ERROR => shmctl(5242989,IPC_RMID,0) 0xffffffffffff278) (1: Not owner) [shmux_mt.c   1935]

[Thr 01] *** ERROR => ShmCleanup: ShmCreate(63,0,SHM_ATTACH,...) failed. rtc=1 [shmux_mt.c   4269]

[Thr 01] *** ERROR => ShmCleanup: Inv. Key=63  [shmux_mt.c   4310]

[Thr 01] *** ERROR => shmat(5242989,0x0,SHM_RND) (13: Permission denied) [shmux_mt.c   1924]

[Thr 01] *** ERROR => shmctl(5242989,IPC_RMID,0) 0xffffffffffff278) (1: Not owner) [shmux_mt.c   1935]

[Thr 01] *** ERROR => ShmCleanup: ShmCreate(63,0,SHM_ATTACH,...) failed. rtc=1 [shmux_mt.c   4269]

[Thr 01] *** ERROR => ShmCleanup: Inv. Key=63  [shmux_mt.c   4310]

[Thr 01] *** ERROR => IcmInit: IcmShmCreate failed (rc=-1) [icxxrout_mt. 1798]

[Thr 01] *** ERROR => shmat(5242989,0x0,SHM_RND) (13: Permission denied) [shmux_mt.c   1924]

[Thr 01] *** ERROR => shmctl(5242989,IPC_RMID,0) 0xffffffffffff608) (1: Not owner) [shmux_mt.c   1935]

[Thr 01] *** ERROR => ShmCleanup: ShmCreate(63,0,SHM_ATTACH,...) failed. rtc=1 [shmux_mt.c   4269]

[Thr 01] *** ERROR => ShmCleanup: Inv. Key=63  [shmux_mt.c   4310]

[Thr 01] *** ERROR => shmat(5242987,0x0,SHM_RND) (13: Permission denied) [shmux_mt.c   1924]

[Thr 01] *** ERROR => shmctl(5242987,IPC_RMID,0) 0xffffffffffff608) (1: Not owner) [shmux_mt.c   1935]

[Thr 01] *** ERROR => ShmCleanup: ShmCreate(82,0,SHM_ATTACH,...) failed. rtc=1 [shmux_mt.c   4269]

[Thr 01] *** ERROR => ShmCleanup: Inv. Key=82  [shmux_mt.c   4310]

[Thr 01] *** ERROR => ShmCleanup(62) failed 3 [mpixx_mt.c   3244]

[Thr 01] SAP Web Dispatcher: WebDisp shared memory cleanup done

shared memory cleanup done

[Thr 01] *** ERROR => ThrMtxDelete() failed: 12 [icrxx_mt.c   3196]

[Thr 01] *** SAP Web Dispatcher shutdown completed (pid: 1781814) ***

*** SAP Web Dispatcher shutdown completed (pid: 1781814) ***

[1]    Exit 255             ./sapwebdisp pf=/usr/sap/SID/globalwebdisp/portal_webdisp/sapwebdisp.pfl

[Former Member · Accepted solution]

Hi S Kumar,

The log here shows two errors -

1.

[Thr 01] Profile configuration error detected, use temporary corrected setup

[Thr 01] Shared Pool 10: ipc/shm_psize_10 = 19880000 (too small)

[Thr 01] Shared Pool 10: (smaller than min requirement 130168712)

To resolve this, set up your ipc shm size as recommended here. You can check that by running

sappfpar check pf=<webdispatcher profile>

and do a clean restart of the system. Clean restart means by running "cleanipc" command with this syntax -

cleanipc <Instance number of webdisp> remove

Also run it for SCS instance number and restart.

2. [Thr 01] *** ERROR => NiIBindSocket: could not delete file '/tmp/.sapstream65000' (hdl 0; errno=1) [nixxi.cpp    3214]

Look for two things here -

1. space in /tmp filesystem by using " df -k /tmp" and see it is not full.

2. Permission of /tmp. It should be 755.

Please go through these steps. These will help you in getting back your system up and running.

Regards,

Ashutosh Shukla

[sandeep_kumar60 · Accepted solution]

HI,

This error had been came for the first time from the past 5-6 years which we are performing the activity regularly ,

we had not even done a single change in the profile parameters or any thing , but we cant understand the cause for this issue either we had changed the permissions of the users or the files, and we had followd the process as we are doing earlier.

The strange thing i need to understand is y this issue automatically all of sudden with out any changes.

what caused to rise this issue?

could any one help me in that view

Thanks

[Former Member · Accepted solution]

Have you cleaned up the shared memory as suggested by Ashutosh?

Have a clean restart of your system.

stopsap --> cleanipc --> ipcrm --> startsap

Regards,

Vishal

[sandeep_kumar60 · Accepted solution]

HI Vishal ,

I think cleanipc may work , but why should i do , what is making me to perform that extra command .

I need to know why this issue came

Reason is most important to me than the solution.

Pls assist me in understanding the issue and the cause.

[Former Member · Accepted solution]

Hello Kumar,

Shared memory is allocated when you start the SAP instance.But in your case somehow it is not able to clean the shared memory segments.

So in order to cleanup those sahres memory segment, I've asked you to do cleanipc.

But For web dispatcher, you can start it with cleanup command:

sapwebdisp
pf=<profile name> [-f <trace file> -t <trace level>  -cleanup -shm_attach_mode <mode>
-auto_restart -version]

<mode> = cleanup

For more info, Please gothrough

http://help.sap.com/saphelp_nw04s/helpdata/en/ae/4bed847ca94816ba46b182cc1c3a24/frameset.htm

Cheers,

Vishal

[sandeep_kumar60]

HI every one

As suggest by you people , i had a restart of the System from OS level , and started the sap as we do everytime and staretd the webdisp,

The webdispacther is now working and up .

Thanks for the Suggestions provided by you all.

but now iam experinecing one more issue for the same.

Previously when i start the webdisp i use to get 3 lines of Output

as webdispatcher is up and runnihjng,

icmbnd: handle for "xyz.gabc.com:80" (only on this adapter) successfully sent to server

icmbnd: handle for "xyz.ggabc.com:443" (only on this adapter) successfully sent to server

*** SAP Web Dispatcher up and operational (pid: 1319112) ***

But Now when i run the same commands , i am getting a lot of linesof output in console ,

can u pls suggest me why iam reciving these lines of output in console as i had not done any changes in any file any where.

[Thr 01] Sun Nov  4 11:04:49 2012

[Thr 01] Profile configuration error detected, use temporary corrected setup

[Thr 01] Shared Pool 10: ipc/shm_psize_10 = 19880000 (too small)

[Thr 01] Shared Pool 10: (smaller than min requirement 130168712)

[Thr 01] *** WARNING => maximum number of sockets supported on this

host (1997) less than parameter icm/max_sockets (8192) [icxxrout_mt. 2791]

[Thr 01] started security log to file dev_icm_sec

[Thr 01] SAP Web Dispatcher running on: sapportdb.gabc.local

[Thr 01] MtxInit: 30001 0 2

[Thr 01] IcmInit: listening to admin port: 65000

[Thr 01] WLM Tag 'ICM' successfully set for this process

[Thr 01] SAP Web Dispatcher: WebDisp shared memory cleanup done

[Thr 01] IcrCoreInitSessionTable: Session table initialized

[Thr 515] *** WARNING => HttpPlugInInit: Parameter icm/HTTPS/trust_client_with_issuer or icm/HTTPS/trust_client_with_subject not set => do not trust any intermediary

X.509 cert data will be removed from header [http_plgrt_m 743]

[Thr 515] *** WARNING => HttpAdmHandlerInit: archive /usr/sap//SYS/exe/run/wdispadmin.SAR does not exist [http_adm_mt. 291]

[Thr 515] HttpExtractArchive: files from archive ./wdispadmin.SAR in directory . are up to date

[Thr 515] HttpSubHandlerAdd: Added handler HttpAdminHandler(slot=0, flags=4101) for /sap/wdisp/admin:0

[Thr 515] CsiInit(): Initializing the Content Scan Interface

[Thr 515]            IBM RS/6000 with AIX (mt,ascii,SAP_CHAR/size_t/void* = 8/64/64)

[Thr 515] CsiInit(): CSA_LIB = "/usr/sap//SYS/exe/run/libsapcsa.o"

[Thr 515] *** ERROR => DlLoadLib()==DLENOACCESS - dlopen("/usr/sap//SYS/exe/run/libsapcsa.o") FAILED

  "     0509-022 Cannot load module /usr/sap//SYS/exe/run/libsapcsa.o.

        0509-026 System error: A file or directory in the path name does not exist."  (errno=2,No such file or directory) [dlux_mt.c    445]

[Thr 515] *** ERROR => HttpAuthHandlerInit: url: / -> failed -> content filter deactivated [http_auth_mt 319]

[Thr 515] HttpSubHandlerAdd: Added handler HttpAuthHandler(slot=1, flags=12293) for /:0

[Thr 515] HttpSubHandlerAdd: Added handler HttpWebDispHandler(slot=2, flags=28677) for /:0

[Thr 515] Started service 8600 for protocol HTTP on host "xyz.gabc.com"(only on this adapter) (processing timeout=600, keep_alive_timeout=60)

[Thr 515] =================================================

[Thr 515] = SSL Initialization    platform tag=(rs6000_5.2_64)

[Thr 515] =   (700_REL,Nov  6 2010,mt,ascii,SAP_UC/size_t/void* = 8/64/64)

[Thr 515]   profile param "ssl/ssl_lib" = "/usr/sap/SID/globalwebdisp/secudir/libsapcrypto.so"

           resulting Filename = "/usr/sap/SID/globalwebdisp/secudir/libsapcrypto.so"

[Thr 515] Sun Nov  4 11:04:50 2012

[Thr 515] =   found SAPCRYPTOLIB  5.5.5C pl17  (Oct 11 2005) MT-safe

[Thr 515] =   current UserID: "sidadm",  env-var USER="sidadm"

[Thr 515] =   found SECUDIR environment variable

[Thr 515] =   using SECUDIR=/usr/sap/SID/globalwebdisp/secudir

[Thr 515]   profile param "ssl/server_pse" = "/usr/sap/SID/globalwebdisp/secudir/SAPSSL.pse"

           resulting Filename = "/usr/sap/SID/globalwebdisp/secudir/SAPSSL.pse"

[Thr 515] =  secudessl_Create_SSL_CTX():  PSE "/usr/sap/SID/globalwebdisp/secudir/sec/SAPSSLC.pse" not found,

[Thr 515] =      using PSE "/usr/sap/SID/globalwebdisp/secudir/SAPSSL.pse" as fallback

[Thr 515] =  secudessl_Create_SSL_CTX():  PSE "/usr/sap/SID/globalwebdisp/secudir/sec/SAPSSLA.pse" not found,

[Thr 515] =      using PSE "/usr/sap/SID/globalwebdisp/secudir/SAPSSL.pse" as fallback

[Thr 515] ******** Warning ********

[Thr 515] *** No SSL-client PSE "SAPSSLC.pse" available

[Thr 515] *** -- this will probably limit SSL-client side connectivity

[Thr 515] ********

[Thr 515] = Success -- SapCryptoLib SSL ready!

[Thr 515] =================================================

icmbnd: handle for "XYZ.gabc.com:443" (only on this adapter) successfully sent to server

[Thr 01] Started service 443 for protocol HTTPS on host: "xyz.gabc.com"(only on this adapter) (processing timeout=600, keep_alive_timeout=60)

[Thr 258] **** SigHandler: signal 20 received

[Thr 01] IcmCreateWorkerThreads: created worker thread 0

[Thr 01] IcmCreateWorkerThreads: created worker thread 1

[Thr 01] IcmCreateWorkerThreads: created worker thread 2

[Thr 01] IcmCreateWorkerThreads: created worker thread 3

[Thr 01] IcmCreateWorkerThreads: created worker thread 4

[Thr 01] IcmCreateWorkerThreads: created worker thread 5

[Thr 01] IcmCreateWorkerThreads: created worker thread 6

[Thr 01] IcmCreateWorkerThreads: created worker thread 7

[Thr 01] IcmCreateWorkerThreads: created worker thread 8

[Thr 01] IcmCreateWorkerThreads: created worker thread 9

[Thr 3343] IcmWatchDogThread: watchdog started

*** SAP Web Dispatcher up and operational (pid: 1302646) ***

[Former Member]

[Thr 01] *** WARNING => maximum number of sockets supported on this
host (1997) less than parameter icm/max_sockets (8192) [icxxrout_mt. 2791]

You can use the "ulimit -n" command to obtain the current value for the maximum number of open sockets and 'chuser nofiles=<new value> <user>' to change this value.

[Thr 515] *** WARNING => HttpPlugInInit: Parameter icm/HTTPS/trust_client_with_issuer or icm/HTTPS/trust_client_with_subject not set => do not trust any intermediary

For more info about configuration please read X.509-Based Logon to NetWeaver AS from SAP Web Dispatcher.

[Thr 515] *** WARNING => HttpAdmHandlerInit: archive /usr/sap//SYS/exe/run/wdispadmin.SAR does not exist [http_adm_mt. 291]
[Thr 515] HttpExtractArchive: files from archive ./wdispadmin.SAR in directory . are up to date
[Thr 515] HttpSubHandlerAdd: Added handler HttpAdminHandler(slot=0, flags=4101) for /sap/wdisp/admin:0
[Thr 515] CsiInit(): Initializing the Content Scan Interface
[Thr 515]            IBM RS/6000 with AIX (mt,ascii,SAP_CHAR/size_t/void* = 8/64/64)
[Thr 515] CsiInit(): CSA_LIB = "/usr/sap//SYS/exe/run/libsapcsa.o"
[Thr 515] *** ERROR => DlLoadLib()==DLENOACCESS - dlopen("/usr/sap//SYS/exe/run/libsapcsa.o") FAILED
  "     0509-022 Cannot load module /usr/sap//SYS/exe/run/libsapcsa.o.
        0509-026 System error: A file or directory in the path name does not exist."  (errno=2,No such file or directory) [dlux_mt.c    445]
[Thr 515] *** ERROR => HttpAuthHandlerInit: url: / -> failed -> content filter deactivated [http_auth_mt 319]

Please check INSTANCE_NAME, DIR_INSTANCE profile parameters from RZ11 tcode or in profile files.

[Thr 515] =================================================
[Thr 515] = SSL Initialization    platform tag=(rs6000_5.2_64)
[Thr 515] =   (700_REL,Nov  6 2010,mt,ascii,SAP_UC/size_t/void* = 8/64/64)
[Thr 515]   profile param "ssl/ssl_lib" = "/usr/sap/SID/globalwebdisp/secudir/libsapcrypto.so"
           resulting Filename = "/usr/sap/SID/globalwebdisp/secudir/libsapcrypto.so"
[Thr 515] Sun Nov  4 11:04:50 2012
[Thr 515] =   found SAPCRYPTOLIB  5.5.5C pl17  (Oct 11 2005) MT-safe
[Thr 515] =   current UserID: "sidadm",  env-var USER="sidadm"
[Thr 515] =   found SECUDIR environment variable
[Thr 515] =   using SECUDIR=/usr/sap/SID/globalwebdisp/secudir
[Thr 515]   profile param "ssl/server_pse" = "/usr/sap/SID/globalwebdisp/secudir/SAPSSL.pse"
           resulting Filename = "/usr/sap/SID/globalwebdisp/secudir/SAPSSL.pse"
[Thr 515] =  secudessl_Create_SSL_CTX():  PSE "/usr/sap/SID/globalwebdisp/secudir/sec/SAPSSLC.pse" not found,
[Thr 515] =      using PSE "/usr/sap/SID/globalwebdisp/secudir/SAPSSL.pse" as fallback
[Thr 515] =  secudessl_Create_SSL_CTX():  PSE "/usr/sap/SID/globalwebdisp/secudir/sec/SAPSSLA.pse" not found,
[Thr 515] =      using PSE "/usr/sap/SID/globalwebdisp/secudir/SAPSSL.pse" as fallback
[Thr 515] ******** Warning ********
[Thr 515] *** No SSL-client PSE "SAPSSLC.pse" available
[Thr 515] *** -- this will probably limit SSL-client side connectivity
[Thr 515] ********
[Thr 515] = Success -- SapCryptoLib SSL ready!
[Thr 515] =================================================

For SSL configuration please read Configuring the SAP Web Dispatcher to Support SSL.

Regards,

Roman

[sandeep_kumar60]

Hi ROman ,

Thanks for the reply ,,

previously till last week we had started the webdisp sucess fully ,

the only ouput i used to get on the consloe is

icmbnd: handle for "xyz.gabc.com:80" (only on this adapter) successfully sent to server

icmbnd: handle for "xyz.ggabc.com:443" (only on this adapter) successfully sent to server

*** SAP Web Dispatcher up and operational (pid: 1319112) ***

After taht later from the nextweek , there were no changed made in any Profiles or the env variables ,

Suddenly i am getting the out put with some errors and warnings above.

I think all the configurations will be god enough as it is used to work earlier ,

The INSTANCE_NAME, DIR_INSTANCE are set correctly

*** WARNING => HttpAdmHandlerInit: archive /usr/sap//SYS/exe/run/wdispadmin.SAR does not exist [http_adm_mt. 291]

for the above line icant get a solution why the path is taken as usr/sap//sys---- here the sid is missing and inserted with double quotes

iam not able to find the exact reason why iam getting this huge erorr output on console.

As suggested by you , i had checked the ipc/shm_psize_10 , the value set for this variable is around 145 MB , but iam seeing as 10 MB in the log file, which i cant know from where it is picking the value.

and the other thing is i dont have sec directory mentioned above, this is not there from the past long time , and no erorr was poped up till now.

i need to find why this issues had been raised suddenly .

[Former Member]

Check environment variables for <sid>adm user.

Regards,
Roman

[sandeep_kumar60]

HI

Which variable i should i checkin that ?

i think the below variables are set perfectly

LD_LIBRARY_PATH=/secudir

DIR_LIBRARY=/usr/sap/SID/SYS/exe/run

SECUDIR=/usr/sap/SID/globalwebdisp/secudir

Any suggestions please

[Former Member]

library are searched beforehand in LB_LIBRARY_PATH

so adding

/usr/sap/SID/SYS/exe/run to LB_LIBRARY_PATH

ex:

LD_LIBRARY_PATH=/secudir:/usr/sap/SID/SYS/exe/run

also does any os level patch applied on your system

[Former Member]

Please attach instance profile for web dispatcher.

Regards,

Roman

[sandeep_kumar60]

H INikhil

No os level patching is done to the system ,

is it LD_Library _path or LB_Library_path ?

Roman 😆 i could not understand your point,c an u pls elaborate

[Former Member]

typo its only one thing LD_LIBRARY_PATH

[sandeep_kumar60]

Hi Roman

i did not understand your point exactly

could you please elaborate

[Former Member]

Hi Kumar,

When did the last time this system restarted? If its been a long time, say more than 3-4 months, then that itself explains the cause of such behaviour.

Java systems, with time, accumulate garbage in terms of memory leaks, error dumps and things like this. Though the garbage collector process runs for this very purpose and is of two types, but still as per SAP best practices, it needs to be recycled/restarted every 40 days which is what followed in my project as we have onsite SAP consultants.

Because of this reason, i suggested clean system restart after clearing up shared memory segments.

Hope this answers your query.

Regards,

Ashutosh Shukla

[Former Member]

hi

The problem seems to be with your /tmp file system. please check your /tmp permission for <sid>adm or the problem could be with the file system itself.

Thanks & regards

bala

[anja_engelhardt2]

moved to SAP NetWeaver Application Server by moderator