Complete data backup job ends with error (db instance can't be stopped)

[Former Member]

Hello all you SAP gurus!

I have a problem with the database backup jobs. The backup itself seems to finish well, however the fields in DBA Planning Calender turns always red. The reading in the joblog of the complete data backup isn't showing anything in particular, however the joblog of the check complete data backup shows this:

*** SAVE/RESTORE result:
ERR
-24895,ERR_SHUTDOWN: shutdown of database occured
-24994,ERR_RTE: Runtime environment error
6,connection broken server state 6

And really, the database instance couldn't be stopped via linux console either. stopsap -t db shows this:

Trying to stop database ...
There are still active users connected to the database.
On order to shutdown the database with stopdb you first have to stop all
database applications. This includes MCOD instances and database utilities.
Trying: Permission denied.

However when the application server is restarted, the first shutdown of the db instance is without any problems. The next ones are with the mentioned error.

I tried to apply the note 1148232, but with no effort.

I can turn the database offline with dbmcli, however that doesn't solve my problem with backups.

I'm planning a kernel update of the system, so maybe that will help, but until then... any hints?

Thanks in advance.

Cheers!

- Stefan

[Former Member]

Got it! 😄

Remote database administration also popped up in my head, but I wasn't able to find it in out system landscape.

But today I've found an old system copy still running, which had remote database set for our mentioned system.

Thank you everybody for your hints and advices, I'm going to backup this thread, since lot of good ideas appeared here.

Many thanks!

Cheers!

- Stefan

[Former Member]

OK, a progress in sight.

Kernel update went fine, but the problem persists.

However I've found the reason of the error - the complete data backup was overlapping with the complete data backup check, hence the problem with database stop. But now I have other problem - it seems I can't delete the job with the complete data backup check through DB13. Even if I clear the whole calendar and then set the jobs anew, every other night at 0:00 complete data backup check starts and I can't do anything about it. It's not listed in the job list in SM37 and when opened via DB13, there's a line This action has not been scheduled via DB13.

So now what? Where else could this job be scheduled? The system is running on openSUSE and I read somewhere something about a cronjob (crontab) on the OS level, however I'm not sure anyone from our company set something like that. Moreover I'm not that familiar with cronjob, so any help in this field will be appreciated.

Take two: any hints?

[Former Member]

Hi Stefan,

Maybe SAP Note 564576 could be of help.

Check table SDBAP for orphan entries.

Regards,

Srikishan

[Former Member]

Hi Srikishan,

thanks for the note - looks promising.

As for the tables SDBAP and SDBAR - I can see there entries that were scheduled only by me, not a single one with a time 0:00... but I'm going to try to erase them (along with the DB13 erase) anyway and we'll see tomorrow.

[Former Member]

Failed again...

Even though the tables SDBAP and SDBAR are empty and the calendar in DB13 cleared, the backup jobs are started anyway with the same message This action has not been scheduled via DB13.

So where else schedule (or disable) the backup jobs?

[Former Member]

Hi Stefan,

As you mentioned the jobs are starting from the SAP system side - please check for the job entry in tcode SM37.

The jobs scheduled by DB13 should be of the format DBA: *

Also search for backup jobs incase they are not in the above format. You can also check with the start time condition which was mentioned as 00:00:00 every day.

Once the job is shown, select the job and from the menu 'Job' -> 'Change' remove the scheduling.

Sometimes the online backup is started from the OS level via shell scripts - like in our company. So checking on the OS level via the command crontab -l would also be a good idea.

Regards,

Srikishan

[Former Member]

Hi Srikishan,

thanks for your reply.

No jobs with string DBA in the name were found in SM37... though the backups were done (as seen in DB13).

No jobs with the program RSDBAJOB were found.

There are simply no jobs relevant to the backups shown in DB13.

When checked the shell scripts with the command crontab -l, I'll get this reading:

# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.XXXXkuLrrd installed on Thu Jan 20 11:31:25 2011)
# (Cron version V5.0 -- $Id: crontab.c,v 1.12 2004/01/23 18:56:42 vixie Exp $)
01 0,6,12,18 * * * /usr/bin/kinit -k SAPService/sapvmt15.xxxxx.xxxxx.sitexxxxxxx.SITE

Doesn't look a backup schedule job to me, but as I wrote above, I have no experience with crontab.

Looks like a ghost is scheduling my backups.

[Former Member]

Hi Stefan,

Ok now this looks like automation at its best

So the backup is neither fired from SAP system nor the operating system. Remaining source would be the Database side.

Could you check via the DBMGui if the automatic backup is scheduled. Also check the backup history logs to know the source of backup command.

It could be that the backup is scheduled via the dbmcli scheduler:

http://maxdb.sap.com/doc/7_7/45/1587ca16f85d78e10000000a11466f/content.htm

Regards,

Srikishan

[Former Member]

Hi Srikishan,

the result from the command scheduler_list_jobs is

T15>OK

the result from the command scheduler_state is

T15>OK
OFF

It appears the scheduler is switched off.

The backup history log from the backup_history_list -Inverted command shows this:

T15>OK
CONTINUE
4E4032CB002D|DAT_000000418|SAVE WARM|2011-08-08 21:02:36|2011-08-08 21:02:36|2011-08-08 21:02:39|2011-08-09 00:32:39|  23668508|          |NO |COM1                                                            |  11326752|         1|         0|
4E3FCA2D002B|LOG_000000426|SAVE WARM|2011-08-07 15:30:03|2011-08-08 13:36:09|2011-08-08 13:36:13|2011-08-08 13:36:17|  23612994|  23655659|   |LOG                                                             |     42680|         0|         0|
4E3E935E002A|LOG_000000425|SAVE WARM|2011-08-06 15:52:59|2011-08-07 15:30:03|2011-08-07 15:30:06|2011-08-07 15:30:10|  23570328|  23612993|   |LOG                                                             |     42680|         0|         0|
4E3D8FBD0029|DAT_000000417|SAVE WARM|2011-08-06 21:02:22|2011-08-06 21:02:22|2011-08-06 21:02:35|2011-08-07 00:30:17|  23578853|          |NO |COM3                                                            |  11326432|         1|         0|
4E3D473C0027|LOG_000000424|SAVE WARM|2011-08-05 16:52:56|2011-08-06 15:52:59|2011-08-06 15:53:00|2011-08-06 15:53:07|  23527662|  23570327|   |LOG

I'm tellin' ya, ghosts are runnin' our backups. 8-|

[Former Member]

Hi Stefan,

Is it livecache or MaxDB? If it is, did you checked RSLVCBACKUP program has been scheduled?

Best regards,

Orkun Gedik

[Former Member]

Hi Orkun,

It's liveCache. And nope, no RSLVCBACKUP is scheduled... at least according to SM37. (also searched 7 days back just to be sure - and really no RSLVCBACKUP report has been launched.)

[Former Member]

Hi Stefan,

Do you have a external backup tool in your landscape? It might be started, by remote. Or a DBACOCKPIT connected to this system. This is only an idea.

Best regards,

Orkun Gedik