Oracle 10.2.0.2 upgrade causes causes sporadic signal 4/ORA-

[Former Member]

Hello

We have recently upgraded the Oracle database of this ECC 5.0 SAP

system from Oracle 9207 to Oracle 10.2.0.2 with 40 one-up patches. The

SAP kernel is 640 64-bit Unicode at patch level 175. We are receiving

intermittent signal 4/ORA-24909 errors after completing this upgrade.

As part of the upgrade, we followed the recommendation to install the

latest Oracle client software which we did. We followed note 1017418

to install the latest 64-bit 9208 Oracle client available on

sapnet. In our case, since it is the only SAP system on the server and

the Oracle DB is 10.2.0.2, we implemented the client per the step B.5

in note 1017418. Implementing the client per these instructions

creates the following directories:

qasadm> ls -la /oracle/client/92x_64

total 0

drwxr-sr-x 8 qasadm sapsys 256 Jan 15 2007 .

drwxrwsr-x 5 qasadm dba 256 Jan 28 14:19 ..

drwxr-sr-x 5 qasadm sapsys 256 Jan 15 2007 jdbc

drwxr-sr-x 2 qasadm sapsys 256 Jan 15 2007 lib

drwxr-sr-x 3 qasadm sapsys 256 Jan 15 2007 network

drwxr-sr-x 3 qasadm sapsys 256 Jan 15 2007 ocommon

drwxr-sr-x 4 qasadm sapsys 256 Jan 15 2007 oracore

drwxr-sr-x 3 qasadm sapsys 256 Jan 15 2007 rdbms

I have read all notes I can find trying to understand what may be

causing this issue, but everything I find seems to point to an Oracle

client issue. After the upgrade, the only change made to <sid>adm

(QASADM in our case) was the setting of ORACLE_HOME. I have attached

the envronment variables of <sid>adm to this message. <sid>adm is able

to read the new $ORACLE_HOME and the data pointed to by ORA_NLS33.

Note 952388 provides a direct hit for the issue we are having. Since

our database is now 10.2.x, we validated the following two points:

1) made sure note 1017418 was implemented (as mentioned above).

2) made sure one-up patch 5092134 was applied per note 1005758; I've

attached the OPATCH lsinventory output to confirm this criteria is met.

Please let us know what needs to be done to resolve this issue.

Thanks & Regards

[stefan_koehler]

Hello,

please post the following output:

> shell> ls -la /oracle/client/92x_64/ocommon/nls/admin/data

Is the ocommon/nsl/admin/data directory pointing to the $ORACLE_HOME/ocommon/nls/admin/data or have you copied the NLS files?

Regards

Stefan

[Former Member]

This is a Soft link..

ocommon/nsl/admin/data directory pointing to the $ORACLE_HOME/ocommon/nls/admin/data .

Thnaks

[stefan_koehler]

Hello,

>> ocommon/nsl/admin/data directory pointing to the $ORACLE_HOME/ocommon/nls/admin/data .

Ok, that is correct.

Please involve us a little bit more in your environment.

The SAP support has mentioned that you have more application servers.

1) Does the error occur on any application server?

2) Have you upgraded the client and NLS files on every application server?

3) I think you are using AIX (as you mentioned the AIX libraries) ... which version (with TL and SP)?

4) Please post the Stack Dump of the client... there is a known bug ... (Metalinknote #342791.1)

Regards

Stefan

[fidel_vales]

Hello Michael,

That error is is typical issue with the oracle client.

you show that you have the oracle client installed:

> qasadm> ls -la /oracle/client/92x_64

but, are you using it?

That would be the next check.

change to the kernel directory and perform the following:

> ldd dboraslib.<ext>

and paste here the output.

I've seen similar issues when the oracle client library has been "manualy" copied to a different directory (for example, the kernel directory) and the kernel is using this copied version (old one) and not the new one.

[Former Member]

Hello

oops No link established ocommon/nsl/admin/data directory pointing to the $ORACLE_HOME/ocommon/nls/admin/data .

I was looking into the wrong server

1. Yes the error occurs on 2 apps server.

2. Yes upgraded the client and NLS files on every application server.

3. AIX Version 5.3

Thanks

[Former Member]

Here is the output

find ./ -name dboraslib* -print

1. cd ./sapmnt/QAS/exe/

1. ldd dboraslib.o

dboraslib.o needs:

Cannot find /oracle/client/92x_64/lib/libclntsh.a(shr.o)

/oracle/client/92x_64/lib/libwtc9.so

/usr/lib/threads/libc.a(shr_64.o)

/usr/lib/libpthreads.a(shr_xpg5_64.o)

/usr/lib/libdl.a(shr_64.o)

/usr/lib/libodm.a(shr_64.o)

/usr/lib/threads/libc.a(aio_64.o)

/unix

/usr/lib/libcrypt.a(shr_64.o)

1. Then I did ls command to find these files. All the above files exists.

Edited by: Michael Brandes on Feb 6, 2008 8:37 PM

[Former Member]

Hello

Here is my reply:

1) No. It is the same 9208 client that exists on the application server, implemented as documented in note 1017418

2) Yes; there is no issue on the application server, but the client on the application server is installed differently than a DBCI.

3) The Metalink item you mentioned is not relevant for Oracle 10.2.0.2 and Oracle 9.2.0.8 combination.

As I mentioned earlier, the solution I am proposing eliminates the client issue and it will be required in the long term for the other landscapes running kernels linked with Oracle 9i.

[fidel_vales]

Hi,

here is something interesting:

> dboraslib.o needs:

> Cannot find /oracle/client/92x_64/lib/libclntsh.a(shr.o)

You mention that the file exists.

- as what user have you tried the ldd

- What are the authorizations of /oracle/client/92x_64 and subdirectories?

Regarding the other option, kerne 640_EXT, that should also solve the issue. As mentioned before, the error is "known" and caused by an issue with the 9i library.

What is a little complicated is to find the "root cause" ( where is the "wrong" library )

Edited by: Fidel Vales on Feb 6, 2008 10:15 PM

[Former Member]

Hello

Yes files are exists in the directory. Authorzations are correct on directories and subdirectories.

Can you please advise as the solution to the root cause problem?

Thanks

Edited by: Michael Brandes on Feb 6, 2008 10:27 PM

[fidel_vales]

Hello Michael,

I repeat the question, as which user have you executed the "ldd" command.

At this point, what I see is that the dboraslib cannot read the oracle 9.2.0.8 library. In that case you should not be able to run SAP. I wonder what user you have used for the check. Try the following on the AS where you get the error:

>

>- connect as <sid>adm

>- check the environment:

> env | grep -i PATH

>- check the path:

> ls -lRT /oracle/client

>- change to the kernel directory and

> ldd dboraslib.o

> ls -l libclntsh*

When it was the last time you got the error?

[Former Member]

Was this ever resolved?

[Former Member]

Hello

I’m not getting far with my response from SAP concerning this issue.

My initial response to this was that this was the typical SAP response “go upgrade the kernel and tell me if it still fails”. I know for a fact that a later version of the current 640 kernel will make no difference. This is an Oracle client issue and I also know that re-installing it will make no difference.

Instead of wasting further time with this message, I would like to pursue a long-term permanent fix that gets around the client issue. We will need to eventually apply this solution to all SAP systems still running a kernel linked with Oracle 9i (CRM, CRM-BW, SRM, ERP). If this solution resolves the sporadic problem we are experiencing in the ERP landscape after upgrading to Oracle 10.2.0.2, then we will add it as an additional step in our Oracle upgrade outage. I’m proposing that we implement the recently released 640_EX2 kernel which has been linked with Oracle 10 and current AIX compiler libraries.

I would like to try this in our Sanbox environment first, let it run for a few days, then make the same change in DEV/QAS.

Let me know if you have any other suggestions.

Thanks & regards

[Former Member]

Hello

Here is SAP’s latest response message from SAP OSS:

Dear Customer,

Thank you for your reply.

As you are in Upgrade Prepare, please maintain the environment setting

for DBS_LIBPATH_EXT until final tasks AFTER the SAP upgrade.

The ORA-3114 and ORA-24909 occurred ONLY on 01.02.08 at server az18u169.Not occurred after 01.02.08 and not happened to your application server.

ST22:

=====

Runtime Error SYSTEM_CORE_DUMPED

Date and Time 02/01/2008 08:13:50

ShrtText

Process terminated by signal 4.

SIGILL (4) means Illegal instruction and can be caused by any kernel

component. By looking at the stack trace:

=> Stack Trace:

Unknown at 0x0

nioqwa() at 0x900000002a5d980

upirtrc() at 0x9000000026cac8c

kpurcsc() at 0x9000000027d8d2c

kpuexecv8() at 0x9000000027f64ac

kpuexec() at 0x9000000027f83d8

OCIStmtExecute() at 0x90000000281e214

oci_execute_stmt() at 0x9000000021a19a8

stmt_fetch() at 0x9000000021ad4d8

This type of segmentation fault should be due to oracle client.

Please reinstall the oracle client 9208 as per note 1017418 and update

the SAP kernel release 640 to the latest version. The current SAP kernel release 640 patch level is very low, only 175. the latest one now is 218. Please refer note 19466 for more details.

Kindly please monitor the system if the ORA-3114 happen again in your

system, please check note note 534729 Oracle errors due to network

problem to make sure this is not cause by network problem.

Hope the above information helps.

Looking forward for your feedback.

Feel free to inform me if the issue still exist.

Thank you.

Best regards,

Ivy

Support Consultant

SAP Active Global Support (Technology)

-

---

Hi Ivy:

Thanks for responding. I have reviewed note 602843 and we are in

complete compliance with the requirements documented in this note. You

can verify this by checking the attached file (sidadm_env.log).

We are also in full compliance with note 836517 in that we have

upgraded the database of this system to 10g as a pre-requisite to

performing an SAP upgrade. We are beginning the PREPARE phase within

the next few days and hope to complete it two weeks prior to the actual

upgrade, as recommended in many SAP notes and Best Practices.

To the best of my knowledge, parameter DBS_LIBPATH_EXT is only used by

the upgrade tools to allow them access to the Oracle 10 instantclient.

It should not be used by the 640 kernel in the current non-upgraded SAP

system.

If you feel environment variable DBS_LIBPATH_EXT is the cause of the

problem we are experiencing, then please explain how we are to perform

the upgrade over a 3-4 week period?

I will not remove parameter DBS_LIBPATH_EXT until you provide me a

suitable work-around that resolves my existing problem and also allows

me to proceed with the upgrade.

Please respond with a solution. If you are not able to resolve this

problem, please escalate to SAP Oracle support.

Thanks.

-

---

Dear Customer,

Thank you for your connection.

I have logon to your system and analyzed your issue. Below was my

finding.

From the system log (SM21), I found many error as following only

occurred in server az18u169.

08:06:35 az18u169_QAS_00 DIA 003 510 E41 WE02 BY0 >

ORA-24909: call in progress. Current operation cancelled

08:06:35 az18u169_QAS_00 DIA 003 510 E41 WE02 AB2 >

Include ??? line 0000.

08:06:35 az18u169_QAS_00 DIA 003 510 E41 WE02 AB0 Run-time error "DBIF_REPO_SQL_ERROR" occurred

08:06:35 az18u169_QAS_00 DIA 003 510 E41 WE02 BY2 Database error 3114 at FET

08:06:35 az18u169_QAS_00 DIA 003 510 E41 WE02 BY0 >

ORA-03114: not connected to ORACLE

Please make sure that the environment setting is set as per note

602843. In server az18u169, the following environment setting is set.

DBS_LIBPATH_EXT=/oracle/client/10x_64/instantclient <<<

Please remove it and monitor if the issue still exist if you already

finished the SAP upgrade. This environment setting is only need when

you preparing to upgrade your SAP system as per note 836517.

Hope the above information helps.

Looking forward for your feedback.

Feel free to inform me if the issue still exist.

Thank you.

Best regards,

Ivy

Support Consultant

SAP Active Global Support (Technology)

-

---

Hello:

As I mentioned in the initial text of this message, the problem occurs

sporadically and I am able to receate this problem myself. If you look

in ST22, you will see the dump SYSTEM_CORE_DUMPED each time one of

these failures occur. It happens to both dialog and BTC processes,

performing various SAP functions (it is not limited to a specific SAP

functionality). If you look in SM21, at the time of the failure, you

will see the pattern (signal 4, ORA-24909 on REPOLOAD, ORA-03114). If

you look in SM37, BASIS jobs running as DDIC (RDDMNTAB, RDDVERSL,

RDDDICL3, RDDEXECL, etc.) fail some of the time but not all of the time.These failures did not occur before taking the steps necessary to

upgrade Oracle to 10.2.0.2.

I have opened the service connection and provided logon information in

the secure area.

Please log into the system and determine what needs to be done to

correct this problem.

Thanks.

-

---

Dear Customer,

Thank you for contacting with SAP support!

I would like to logon to your system to analyse this issue.

1. Please detail the exact, specific steps to duplicate the issue,

transaction codes, menu options, field data, etc.

2. Please open a remote connection and provide logon

details for a user within validity date.

Please enter Logon-information for your system into this message.

This is how (make sure to be in 'edit' mode):

In OSS: Via menu option "Goto" choose "Customer Logon Data".

In SAP Market Place: Click the Access Data link.

Client, Server, System & System type

One User-ID, Password (& Router-pw, if required).

Press

IMPORTANT: Remember to save the message to save the logon-data!

Please refer to note 508140 for details of logon security.

Looking forward for your feedback.

Feel free to inform me if the issue still exist.

Thank you.

Best regards,

Ivy

Support Consultant

SAP Active Global Support (Technology)

[Former Member]

Here is sidadm_env Log:

-

---

AUTHSTATE=files

CGI_DIRECTORY=/var/docsearch/cgi-bin

DBSID=QAS

DBS_LIBPATH_EXT=/oracle/client/10x_64/instantclient

DB_SID=QAS

DIR_LIBRARY=/usr/sap/QAS/SYS/exe/run

DOCUMENT_DIRECTORY=/usr/docsearch/html

DOCUMENT_SERVER_MACHINE_NAME=localhost

DOCUMENT_SERVER_PORT=49213

EDITOR=vi

ENV=.kshrc

ERRNO=0

FCEDIT=/usr/bin/ed

HOME=/home/QASadm

IBM_JAVA_OPTIONS='-Xmx256M -Xj9'

IFS='

'

INSTANCEDIR_CI=/usr/sap/QAS/DVEBM

INSTANCEDIR_DI=/usr/sap/QAS/D

INSTANCEDIR_GW=/usr/sap/QAS/G

INSTANCEDIR_J=/usr/sap/QAS/J

INSTANCEDIR_JC=/usr/sap/QAS/JC

INSTANCEDIR_LIST=/usr/sap/QAS/DVEBMGS00

INSTANCEDIR_S=/usr/sap/QAS/SCS

JAVA_HOME=/usr/java14_64

LANG=en_US

LC__FASTMSG=true

LIBPATH=/usr/lib:/lib:/usr/sap/QAS/SYS/exe/run:/usr/sap/QAS/SYS/exe/runU

LINENO=1

LOCPATH=/usr/lib/nls/loc

LOGIN=QASadm

LOGNAME=QASadm

MAIL=/usr/spool/mail/QASadm

MAILCHECK=600

MAILMSG='[YOU HAVE NEW MAIL]'

NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat

NLS_LANG=AMERICAN_AMERICA.UTF8

ODMDIR=/etc/objrepos

OPTIND=1

ORACLE_BASE=/oracle

ORACLE_HOME=/oracle/QAS/102_64

ORACLE_PSRV=QAS

ORACLE_SID=QAS

ORA_NLS33=/oracle/client/92x_64/ocommon/nls/admin/data

PATH=/oracle/QAS/102_64/bin:/usr/java14_64/bin:/home/QASadm:/usr/sap/QAS/SYS/exe/runU:/usr/sap/QAS/SYS/exe/run:/usr/bin:/etc:/usr/sbin:/usr/ucb:/home/QASadm/bin:/usr/bin/X11:/sbin:.

PPID=5677240

PS1=' ];QASadm@az18u169:$PWD

QASadm> '

PS2='> '

PS3='#? '

PS4='+ '

PWD=/home/QASadm

RANDOM=29314

SAPDATA_HOME=/oracle/QAS

SAPSYSTEMNAME=QAS

SECONDS=270

SECUDIR=/usr/sap/QAS/DVEBMGS00/sec

SHELL=/usr/bin/sh

TERM=xterm

TERM_DEFAULT=lft

THREAD=NOPS

TMOUT=0

TZ=MST7

USER=QASadm

_=set

d=/usr/sap/QAS/SYS/exe/run

dbms_type=ORA

dbs_ora_schema=SAPERP

dbs_ora_tnsname=QAS

i=1

p=/usr/sap/QAS/SYS/exe/run

rc1=0

rc2=0

rc3=1

rc4=1

rc5=1

rc6=1

Here is orinv Log:

-

---

Invoking OPatch 10.2.0.2.4

Oracle interim Patch Installer version 10.2.0.2.4

Copyright (c) 2007, Oracle Corporation. All rights reserved..

Oracle Home : /oracle/QAS/102_64

Central Inventory : /oracle/oraInventory

from : /etc/oraInst.loc

OPatch version : 10.2.0.2.4

OUI version : 10.2.0.2.0

OUI location : /oracle/QAS/102_64/oui

Log file location : /oracle/QAS/102_64/cfgtoollogs/opatch/opatch2008-01-31_12-30-10PM.log

Lsinventory Output file location : /oracle/QAS/102_64/cfgtoollogs/opatch/lsinv/lsinventory2008-01-31_12-30-10PM.txt

-

---

Installed Top-level Products (2):

Oracle Database 10g 10.2.0.1.0

Oracle Database 10g Release 2 Patch Set 1 10.2.0.2.0

There are 2 products installed in this Oracle Home.

Interim patches (40) :

Patch 6488148 : applied on Fri Jan 25 16:19:33 MST 2008

Created on 11 Dec 2007, 16:34:48 hrs US/Pacific

Bugs fixed:

6151963, 5245494, 6321245, 6488148, 6077514, 6007259, 4904838, 4708389

5763245, 5984705, 5380055, 5882954, 6350462, 4698023, 6370247, 6167221

4722645, 5705257, 6147372, 5838613, 5762598, 5043097, 5397482, 6082745

5949981, 5033476, 5690241, 5547058, 5126551, 5089444, 6011182, 5966822

6134565, 5997892, 5884780, 6051211, 5944076, 4712638, 6070954, 4483240

5449488, 5718007, 5483301, 5381446, 5057695, 5680702, 5548510, 5741121

4273361, 5766310, 4372359, 3901785, 5705630, 5131645, 5724540, 5399282

5113934, 6087237, 5990716, 5649765, 4648181, 5129407, 5172444, 5084239

5976822, 5694984

Patch 6455795 : applied on Fri Jan 25 16:19:04 MST 2008

Created on 21 Nov 2007, 23:50:04 hrs PST8PDT

Bugs fixed:

4430244, 5752105, 6140309, 5558627, 5376783, 5212539, 6455795, 5913430

5386204, 5093837, 5667736, 6033289, 5245129, 5959612, 3279497, 6086497

4771672, 5728380, 5657620

Patch 6447320 : applied on Fri Jan 25 16:18:42 MST 2008

Created on 30 Nov 2007, 13:51:21 hrs PST8PDT

Bugs fixed:

6447320

Patch 6435823 : applied on Fri Jan 25 16:18:23 MST 2008

Created on 18 Oct 2007, 02:54:54 hrs PST8PDT

Bugs fixed:

6084512, 5258887, 5029334, 6435823

Patch 6406234 : applied on Fri Jan 25 16:18:09 MST 2008

Created on 15 Sep 2007, 03:55:01 hrs PST8PDT

Bugs fixed:

4883174, 6398635, 4522561, 5391568, 6406234, 5573544, 4670363, 5181800, 5399901

Patch 6340979 : applied on Fri Jan 25 16:17:48 MST 2008

Created on 3 Sep 2007, 06:22:47 hrs PST8PDT

Bugs fixed:

4599763, 6053541, 5842686, 4772145, 5915774, 6340979, 5284303, 5709414

Patch 6153847 : applied on Fri Jan 25 16:17:27 MST 2008

Created on 28 Jun 2007, 03:25:28 hrs PST8PDT

Bugs fixed:

6153847

Patch 6046043 : applied on Fri Jan 25 16:17:07 MST 2008

Created on 11 May 2007, 07:13:44 hrs PST8PDT

Bugs fixed:

5579457, 6046043, 4460775

Patch 6005996 : applied on Fri Jan 25 16:16:53 MST 2008

Created on 18 Sep 2007, 22:48:13 hrs PST8PDT

Bugs fixed:

6005996

Patch 5941030 : applied on Fri Jan 25 16:16:36 MST 2008

Created on 20 Aug 2007, 11:09:46 hrs PST8PDT

Bugs fixed:

5941030

Patch 5762514 : applied on Fri Jan 25 16:16:17 MST 2008

Created on 25 Mar 2007, 01:07:27 hrs PST8PDT

Bugs fixed:

5762514

Patch 5737613 : applied on Fri Jan 25 16:15:58 MST 2008

Created on 11 Nov 2007, 21:37:01 hrs PST8PDT

Bugs fixed:

5458753, 5377099, 5737613

Patch 5636728 : applied on Fri Jan 25 16:15:19 MST 2008

Created on 15 Feb 2007, 18:54:07 hrs PST8PDT

Bugs fixed:

5636728

Patch 5635254 : applied on Fri Jan 25 16:15:01 MST 2008

Created on 22 Nov 2006, 22:54:11 hrs US/Pacific

Bugs fixed:

5635254

Patch 5618049 : applied on Fri Jan 25 16:14:13 MST 2008

Created on 22 Mar 2007, 23:23:00 hrs US/Pacific

Bugs fixed:

5618049

Patch 5581518 : applied on Fri Jan 25 16:13:54 MST 2008

Created on 3 May 2007, 00:01:41 hrs PST8PDT

Bugs fixed:

5581518, 5567141, 5508574

Patch 5571916 : applied on Fri Jan 25 16:13:35 MST 2008

Created on 18 Oct 2006, 03:23:28 hrs PST8PDT

Bugs fixed:

5571916

Patch 5530958 : applied on Fri Jan 25 16:13:16 MST 2008

Created on 29 Mar 2007, 12:33:29 hrs PST8PDT

Bugs fixed:

5530958

Patch 5496862 : applied on Fri Jan 25 16:12:57 MST 2008

Created on 15 Sep 2006, 11:00:42 hrs PST8PDT

Bugs fixed:

5496862

Patch 5442919 : applied on Fri Jan 25 16:12:35 MST 2008

Created on 2 Aug 2007, 16:14:36 hrs PST8PDT

Bugs fixed:

5442919

Patch 5369855 : applied on Fri Jan 25 16:12:16 MST 2008

Created on 25 Aug 2006, 09:46:25 hrs US/Pacific

Bugs fixed:

5369855

Patch 5363584 : applied on Fri Jan 25 16:11:57 MST 2008

Created on 15 Sep 2006, 06:25:34 hrs PST8PDT

Bugs fixed:

5363584

Patch 5345999 : applied on Fri Jan 25 16:11:38 MST 2008

Created on 30 Apr 2007, 05:45:55 hrs PST8PDT

Bugs fixed:

5345999

Patch 5308692 : applied on Fri Jan 25 16:11:19 MST 2008

Created on 4 Sep 2007, 03:30:37 hrs PST8PDT

Bugs fixed:

5308692

Patch 5188321 : applied on Fri Jan 25 16:11:01 MST 2008

Created on 6 Dec 2007, 21:47:38 hrs PST8PDT

Bugs fixed:

5188321

Patch 5103126 : applied on Fri Jan 25 16:10:40 MST 2008

Created on 3 Jul 2006, 04:46:19 hrs US/Pacific

Bugs fixed:

5103126

Patch 5099990 : applied on Fri Jan 25 16:10:21 MST 2008

Created on 16 Mar 2006, 10:36:22 hrs US/Pacific

Bugs fixed:

5092134

Patch 5063279 : applied on Fri Jan 25 16:10:01 MST 2008

Created on 8 Jun 2006, 10:48:49 hrs US/Pacific

Bugs fixed:

5063279

Patch 4952782 : applied on Fri Jan 25 16:09:43 MST 2008

Created on 27 Oct 2006, 00:19:33 hrs US/Pacific

Bugs fixed:

4952782

Patch 4899479 : applied on Fri Jan 25 16:09:24 MST 2008

Created on 16 Nov 2006, 03:21:30 hrs PST8PDT

Bugs fixed:

4899479

Patch 4883635 : applied on Fri Jan 25 16:09:06 MST 2008

Created on 6 Mar 2007, 20:04:22 hrs PST8PDT

Bugs fixed:

4883635

Patch 4864648 : applied on Fri Jan 25 16:08:48 MST 2008

Created on 31 Mar 2006, 03:36:35 hrs US/Pacific

Bugs fixed:

4864648

Patch 4773931 : applied on Fri Jan 25 16:08:35 MST 2008

Created on 7 Jun 2006, 02:47:20 hrs US/Pacific

Bugs fixed:

4773931

Patch 4742607 : applied on Fri Jan 25 16:08:18 MST 2008

Created on 22 Sep 2006, 03:28:30 hrs PST8PDT

Bugs fixed:

4742607

Patch 4707226 : applied on Fri Jan 25 16:08:00 MST 2008

Created on 12 May 2006, 02:24:40 hrs US/Pacific

Bugs fixed:

4707226

Patch 4668719 : applied on Fri Jan 25 16:07:48 MST 2008

Created on 13 Apr 2007, 00:28:08 hrs PST8PDT

Bugs fixed:

4668719

Patch 4638550 : applied on Fri Jan 25 16:07:28 MST 2008

Created on 26 Apr 2006, 22:06:12 hrs US/Pacific

Bugs fixed:

4638550

Patch 5090357 : applied on Fri Jan 25 16:07:16 MST 2008

Created on 16 Mar 2006, 14:33:45 hrs US/Pacific

Bugs fixed:

4604970

Patch 4518443 : applied on Fri Jan 25 16:06:57 MST 2008

Created on 28 Feb 2007, 00:48:29 hrs PST8PDT

Bugs fixed:

4518443

Patch 3748430 : applied on Fri Jan 25 16:06:44 MST 2008

Created on 16 May 2006, 04:37:08 hrs US/Pacific

Bugs fixed:

3748430

-

---

OPatch succeeded.

Thanks