dbm.knl corrupt and backup fails

[former_member233435]

Hi,

Recently, backups on several servers started to fail. I schedule backups from dbacockpit. There program log shows:

*** DBA Action starts:
Timestamp: 20131117200023   Function: sda   Object: DATA
*** SAVE/RESTORE request:
SAVE DATA QUICK TO  '/mnt/backup/data/2_completeDataBackup.fcs' FILE PACKED BLOCKSIZE 8 NO CHECKPOINT MEDIANAME '2_completeDataB
*** SAVE/RESTORE request accepted:
OK
*** SAVE/RESTORE result:
ERR
-24988,ERR_SQL: SQL error
-903,Host file I/O error
3,Data backup failed
1,Backupmedium #1 (/mnt/backup/data/2_completeDataBackup.fcs)                                         
Backup error occured, Error code 3700 "hostfile_error"
*** SAVE/RESTORE request:
SAVE DATA CANCEL
*** SAVE/RESTORE request accepted:
OK
*** SAVE/RESTORE result:
ERR
-24988,ERR_SQL: SQL error
-104,DBM command impossible at this time
*** DBA Action ends:
Timestamp: 20131117232752   State:    Failure

While in Database Studio the following message popups:

The backup history file /sapdb/FCS/data/wrk/FCS/dbm.knl is corrupted in line 132
-24763,ERR_BHIST_INVALID_ENTRY: The line "52776432026B|DAT_000000048|SAVE WARM|2013-11-04 10:09:06|2013-11-04 10:09:06|2013-11-04 10:09:07|2013-11-04 11:58:18|  17380677|          |NO |2_completeDataBackup                                            |  17904640|         1|      -903| " is an invalid line for the backup history
-24757,ERR_FORMAT_SEPARATORNOP: The separator '|' cannot be found 15 times within string "52776432026B|DAT_000000048|SAVE WARM|2013-11-04 10:09:06|2013-11-04 10:09:06|2013-11-04 10:09:07|2013-11-04 11:58:18|  17380677|          |NO |2_completeDataBackup                                            |  17904640|         1|      -903| "
-24762,ERR_FORMAT_SEPARATORMISSING: String " " does not contain the expected separator '|'
The backup history file /sapdb/FCS/data/wrk/FCS/dbm.knl is corrupted in line 132
-24763,ERR_BHIST_INVALID_ENTRY: The line "52776432026B|DAT_000000048|SAVE WARM|2013-11-04 10:09:06|2013-11-04 10:09:06|2013-11-04 10:09:07|2013-11-04 11:58:18|  17380677|          |NO |2_completeDataBackup                                            |  17904640|         1|      -903| " is an invalid line for the backup history
-24757,ERR_FORMAT_SEPARATORNOP: The separator '|' cannot be found 15 times within string "52776432026B|DAT_000000048|SAVE WARM|2013-11-04 10:09:06|2013-11-04 10:09:06|2013-11-04 10:09:07|2013-11-04 11:58:18|  17380677|          |NO |2_completeDataBackup                                            |  17904640|         1|      -903| "
-24762,ERR_FORMAT_SEPARATORMISSING: String " " does not contain the expected separator '|'

MaxDB version on this system is 7.9.08.12.

When I open dbm.knl on OS level I can see that some lines mentioned in DatabaseStudio and some other lines contain 'garbage' characthers like:

52891247009D|DAT_000000066|SAVE WARM|2013-11-17 20:00:23|2013-11-17 20:00:23|2013-11-17 20:00:29|2013-11-17 23:11:46|  19487304|          |NO |2_completeDataBackup                                            |  17917200|         1|      -903|^A^@^@^@^@^@^@^@<A5>^<8E>^@^@^@^@^@pΚ<DF>^]^?^@^@wΚ<DF>^]^?^@^@<80>Κ<DF>^]^?^@^@|

After repeating same DB backup, I could notice that backup size is about the same size as previous successful backups (almost exactly the same size) and it seems backup fails when it tries to record activity in dbm.knl. I tried to remove grabaged lines from dbm.knl and first immediate backup works but later garbaged lines are again introduced to dbm.knl and database backups start to fail again.

This happens on several systems with 7.9 maxdb.

Any idea how to resolve this issue?

[Former Member · Accepted solution]

Hi Folks, editing the dbm.knl is not the recommended way to solve this issue. You should never edit the Backup history (dbm.knl). The backup history is used by the database tools to support you during recovery. The problem with corrupted backup history files is known and has been fixed as of MaxDB 7.9.08.18 and 7.8.02.39. Regards, Christiane

[former_member233435 · Accepted solution]

Hi again,

what is the status of the patch, it the release date known. Where can we check this information ourselves?

BR

Sasha

[Former Member · Accepted solution]

Hi Folks, 7.9.08.18 and 7.8.02.39 are already released for 4 weeks. Regards, Christiane

[Private_Member_739866 · Accepted solution]

Hi,

We have edited the dbm.knl file and it gives no error yet even we have restored and recovered the database after that

Regards,

Paramvirsinh Jetawat

[Former Member]

Hi

We also get corrupt backup history files from time to time.

To repair them I had to learn that it's not sufficient to "repair" the dbm.knl file. you also have to cope with dbm.mdf.

The correct way to do it is described in note http://service.sap.com/sap/support/notes/1089402

It seems that dbm.knl files are "re-created" from dbm.mdf is you don't resolve errors there also.

For your "real" problem: i can't help you, sorry. BUT we also have ~20 MaxDBs on 7.9.08.12 on SLES11.3 VMware and also use compressed backups with no problems during backup.

BR

Michael

[Private_Member_739866]

Hi,

Use below procedure

1. Go to /sapdb/SRD/data/wrk

2. cp -rp dbm_knl dbm_knl_OLD

3. vi +<Line in which the corruption occured> dbm.knl

• see there must be a corruption in the given line and it will specified with a green color
• delete that part and make the format as per the above line

4. save the file

5. execute the procedure again from Database studio or Database Manager

(Note: Make sure that you have taken the backup first of dbm.knl file Because any extra entry may cause another corruption)

(FYI : This file is only log file of successful backups and returns the return-code)

Regards,

Paramvirsinh Jetawat

SAP BASIS Consultant