LiveCache crash situation..

[Former Member]

Hi,

This is regarding SCM 4.0 and LiveCache 7.4.3 on AIX. SCM DB is Oracle and LiveCache DB is SAP-DB by default.

Our LiveCache was not available few days ago. I went to LC10 and found out that it was green but It would not let me to do anything and gave sql errors.

After few tries I had to restart to resolve the situation.

The situation was resolved after restart.

I went to LC10 - Prob Analysis - Db Analyser - Bottlenecks to analyse what happened. This is what I found -

| * W1 |Data cache hitrate (SQL Pages) 98.19%, 178668 of 9879774 accesses failed

| |CON: DC_Hit < 99 && ( PReads ) > MAX_IDLE_IO_ALL_DEVS

| |VAL: 98.19 < 99 && ( 178668 ) > 6772

	ACT: In addition to enlarging the data cache (note the paging risk of the operating system), search for the cause of the high read activity. Frequently, individual SQL statements cause a high percentage of the total logical and physical read activit
	DES: For a running database application the data cache hitrate should not be less than 99%, otherwise too much data has to be read physically. Data cache hitrates less than 99% for intervals of 15 minutes or more must be avoided.

| * W3 |OMS cache hitrate 97.39%, 178668 of 6849819 accesses failed

| |CON: OMS_Hit < 99 && ( OMS_Fails > MAX_IDLE_IO_ALL_DEVS )

| |VAL: 97.39 < 99 && ( 178668 > 6772 )

| * W3 |Log queue overflows: 2271

| |CON: LQ_Overflows > 0

| |VAL: 2271 > 0

| * I |Number of physical reads: 178668

| |CON: PReads > ( 4 * MAX_IDLE_IO_ALL_DEVS )

| |VAL: 178668 > ( 4 * 6772 )

| * I |Number of physical writes: 108089

| |CON: PWrites > ( 4 * MAX_IDLE_IO_ALL_DEVS )

| |VAL: 108089 > ( 4 * 6772 )

| * I |Garbage collector tasks activated 26 times, currently active: 0

| |CON: GC_Activities > 0 || ActivGarbColls > 0

| |VAL: 26 > 0 || 0 > 0

| * I |Garbage collection deleted 12 objects and 1600888 history objects, amount of history pages now: 176769

| |CON: GC_ObjDel > 0 || GC_HistReleased > 0

| |VAL: 12 > 0 || 1600888 > 0

| * I |backup activity: number of pages read 0, written 162176

| |CON: BackUpReadPg > 0 || BackUpWrittenPg > 0

| |VAL: 0 > 0 || 162176 > 0

| * W1 |LOGQUE1: collision rate 10.04%, 157938 collisions, 783 waits (0.05%), 1573152 accesses on region 24

| |CON: REGION_COLLISION_PERCENT[""] > MAXCPU * 2

| |VAL: 10.04 > 3 * 2

I have a feeling that it ran out of LiveCache and could not find enough room to write new information. That is why it was green but not available. The above logs also gives me impression that we have to increase the size of the LiveCache.

Please let me know if I am not thinking in right direction if I have missed anything here.

Your input will be greately appreciated.

Thank you.

sume

[former_member229109]

Hello Sume,

Please review the SAP note 354700 for the liveCache analysis

of the crash & what information we need to analyze the

liveCache crash issue.

If the liveCache will be crashed, it will be offline & you

could see the liveCache crash errors in the knldiag.err file.

I recommend you to create the OSS message to the component

'BC-DB-LVC', if the liveCache will be crashed on your system.

But according your information, the liveCache was 'online',

you saw the 'green' light status of the liveCache in LC10

==> liveCache was not crashed.

< Please check errors in knldiag.err file >

You had the SQL connection problems from APO to liveCache.

-> What errors did you get?

-> Please check system log in /nsm21 around the time,

when you had problems & corresponding dev log file.

-> It will be helpful next time to check the activities in liveCache

using x_cons tool, for example, run 'x_cons <LC-name> show active' &

review the results.

< run 'x_cons <LC-name> help' or see the documentation about x_cons

tool. >

Please also review the SAP notes::

990602 FAQ: CCMS for MaxDB and liveCache

767598 Available documentation

822239 FAQ: MaxDB Interfaces

869267 FAQ: MaxDB LOG area

Thank you & best regards, Natalia Khlopina

[Former Member]

Thank you for the reply.

I did check all SM21, ST22, WP trace etc. I ran x_cons also using dbmcli command to check if it is runing out of sessions or anyother issue like long running sessions. I could not find any good information.

The knldiag.err and DB Analyser information was giving simmilar information.

Because there was no error, it did not log anything anywhere except some entries in DB Analyser.

The hit ration was going down based on the DB Analyser alerts so I am thinking it was not finding enough space in LC.

I am aware of the notes suggested.

Please let me know if anybody has any suggestion.

Thanks.

Sume

[former_member229109]

1.

"

Our LiveCache was not available few days ago. I went to LC10

and found out that it was green but It would not let me to do

anything and gave sql errors.

"

=> What SQL error did you see at that time?

-> Did you run test connection in /ndb59 for LCA connection?

-> Check dbm.prt & lcinit.his, if the liveCache was restarted in

LC10 at that time?

2.

"

The hit ration was going down based on the DB Analyser alerts so

I am thinking it was not finding enough space in LC.

"

Could you please explain what do you mean -'it was not finding enough

space in LC'?

http://dev.mysql.com/doc/maxdb -> MaxDB Library

-> Tools -> Database Analyzer

==>

A) Check <messages>:

'OMS Cache Hit Rate' ( you had W3 )

< Review 'User Response' recommendations. >

The hit rate when accessing the OMS data in the data cache is too low.

In a running liveCache application, the hit rate should not fall below

100%, otherwise the data must be read physically.

The hit rate may fall for short periods of time, for example, when objects

are accessed for the first time.

-> Did you have data cache usage 100%?

-> Did you have long running OMS version?

-> What is the value of the liveCache parameter CACHE_SIZE &

check the 'Data Area Usage' in the liveCache?

'Log queue overflows' ( you had W3 )

The log queue is full before the log pages are written to the log area.

-> What is the value of the liveCache parameter LOG_IO_QUEUE to check

the size of the log queue?

-> Check the performance of the hard disk on which the log volumes are

located. See the SAP note 748225.

B)

The results of the analysis are noted in the Database Analyzer in log files.

< Click on 'log files' >:

DBAN_CACHES_OCCUPANCY.csv

DBAN_FILLING.csv

3.

If you need further analysis of this problem, please create the OSS ticket, open the connection to your system via OSS & let us logon & check the issue.

Thank you & best regards, Natalia Khlopina