HANA full backup not happening for PRD environment suddenly.

[Former Member]

Hello All,

we have an urgent issue in our hana db where complete data backup is not happening. We are on the followning version

SAP HANA Studio

Version: 2.0.16

When we trigger full backup, it throws the below error which was working fine until 4 days back.

447: backup could not be completed: [110026] The state 'ManagerCommandRunning' of the BackupManager does not allow the requested operation SQLSTATE: HY000

RETURN_STATUS not equal to 0, no backup of *.ini files

We have not changed any configuration.

1) We tried to kill the session using the below, but didnt work

Kill sesson using below command

ALTER SYSTEM CANCEL [WORK IN] SESSION <connection_id>

ALTER SYSTEM CANCEL SESSION <connection_id>

2) Followed OSS note 2310262, to cancel the runnning threads, but no

luck. Please note we cannot take a full system restart as per the note

now. We have to find an alternative to solve this issue.

3) tried to kill the backup using the command "backup cancel backupid", but no luck.

Can you please help.

Regards,

Vishwanath B

[Former Member]

Do you use any third party tool for backups ?

At times, it wont get killed from Studio level. Try killing the session form OS level.

[Former Member]

Hi Viswanath,

Please increase the below parameter( by default 600 Sec) and try to execute back again.

global.ini -> [backup] -> backint_response_timeout

Let me know which Hana version and Whih third backup tool you are using for backint backup.

"Backint did not respond for 600 seconds" bug is fixed in Hana version 102.06 as per the note 2290067.

Regrads

Maruthi

[Former Member]

sorry guys for the late reply...really caught up with the issue of backup. Still no luck.

We tried every possible thing like updating tsm drivers, redifining libraries, executing via hdbsql, via hana console.....even backup to file failed at the last moment, with the error that backup of backup catalog failed.

Currently we have asked backup team to reinstall tdp hana backint agent.

Lets see.

[davidebruno]

but why is not possible to schedule a restart of the HANA db?

I think that in 15 minutes you will solve the issue..

[Former Member]

Can you paste the backup.log file here

[Former Member]

we can go ahead with the server reboot to kill the defunct hdbbackint process.The client has agreed for this now. But before that we want to know the below queries:

What is the gurantee that this wont happen again. Why does this happen in the first place ?

The backup was normally running till june 16th.

[Former Member]

Hello,

You say even your backup to file is failing so that does lend itself to it being a backint problem.

As has been previously asked.

1. What revision are you on ?

2. Upload the backup.log file so people can see what is going on.

3. Have you opened a message with SAP ?

KR,

Amerjit

[Former Member]

Hello

The complete and log backup to filesystemm is running successfully now.

But the issue is here with backint.

Regards,

Vishwanath B

[Former Member]

we have done a server reboot...the backup job is scheduled via crontab at 20:00 CET. Lets see if it works now.

[Former Member]

Hello All,

I have set the trace for backup for all services to DEBUG. Now when i try to run the fcomplete backup, it fails with the below error:

BackupMgr_Manager.cpp(01069) : SAVE DATA finished with error: [447] backup could not be completed, [110512] Backint reported 'BACKINT backup job into /usr/sap/SID/SYS/global/hdb/backint/COMPLETE_DATA_BACKUP_databackup_0_1 failed with wrong size / excepted: 147456 reported: 147454' in file '/var/tmp/hdbbackint_SID.vQ5h24

under /var/tmp, there is no such file called hdbbackint_SID.vQ5h24. Any hints or help here ?

Any clue ??

Regards,

VIshwanath B

[former_member183326]

Have you contact your backint provider? Seems they are in the best position to help with this.

[former_member708493]

hi,

please use correct/updated backint.

[Former Member]

hi guys,

the issue is still there.

It always hangs at this stage.

BackupExecuteTopologyAndSSFSBackupInProgress

SAP is also trying but in vain. Please help me.

Best Regards,

Vishwanath B

[Former Member]

This message was moderated.

[Former Member]

Hello,

For you to get more directed answers you do need to provide at least the following info:

1. Your HDB revision ?

2. Which third party tool (name and version) are you using ?

Have you at least tried to restart the backint agent of the third party tool you are using ?

KR,

Amerjit

[davidebruno]

The backup is done with integration with third party or to file system?

Because if you can modify the destination of the backup, you can do a full backup of the system to the filesystem and not to the third party tool, and when you can schedule a restart you will solve the issue

It's a work-around to be in a safe situation with a full backup

[Former Member]

Hello,

Its backup using third part tool BACKINT and as i said these errors are there in the logs

83436]{-1}[-1/-1] 2016-06-17 10:27:31.015593 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

[83436]{-1}[-1/-1] 2016-06-17 10:37:33.719805 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

[83436]{-1}[-1/-1] 2016-06-17 10:47:36.424135 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

[83436]{-1}[-1/-1] 2016-06-17 10:57:39.128339 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

[83436]{-1}[-1/-1] 2016-06-17 11:07:41.832513 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

[83436]{-1}[-1/-1] 2016-06-17 11:17:44.536783 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

Regards,

Vishwanath B

[davidebruno]

ok but, can you try to do a full backup on the file system, and not using the third party tool?

[Former Member]

We do not have enough space at the FS level to back it up. Its close to 1.5 TB.

It has been designed to take backup via backint...:(

[davidebruno]

you can mount a NFS share, we backup in this way, directly to EMC datadomain

with third party HP dataprotector we had a lot of issue not solved by the vendor..

[anandtigadikar]

For the time being, go with backup of NFS file system...

with help of your netapp engineer, get /hana/backup FS created  of around 2TB, which is shared across the nodes, if it's cluster environment and try to get manual backup over there, either thru HANA Studio or HDBSQL commandline.

If you don't know, how to change this, i can guide you.

But till the time, you resolve your BACKINT issue, this can be workaround...

[anandtigadikar]

To help more, change destination type in hana studio from backint to filesystem

[Former Member]

hello ,

I just switched on the trace for nameserver for backup and triggered the backup again. I can see something like this below. Can this be the cause. this kind of trace is not seen in trace file of any other servers were backups are running successfully.

RootKeyStore.cpp(00386) : Empty SSFS cache: reading from SSFS

Regards,

Vishwanath B

[former_member183326]

This has nothing to do with the issue.

Have you contacted you backint provider like previously asked below?

This seems to me to be an issue for your backint provider.

[davidebruno]

I think that you have to follow the OSS 2310262, and restart the system, or open a OSS message..

[Former Member]

Hello ,

Thanks for ther reply. that would be last option.

What i see since the error cropped up was as follows which i saw just now

83436]{-1}[-1/-1] 2016-06-17 10:27:31.015593 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

[83436]{-1}[-1/-1] 2016-06-17 10:37:33.719805 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

[83436]{-1}[-1/-1] 2016-06-17 10:47:36.424135 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

[83436]{-1}[-1/-1] 2016-06-17 10:57:39.128339 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

[83436]{-1}[-1/-1] 2016-06-17 11:07:41.832513 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

[83436]{-1}[-1/-1] 2016-06-17 11:17:44.536783 w Backup           Backup_Progress.cpp(00315) : Backint did not respond for 600 seconds

Can you help here. Thank you. How to check if backint is working fine.

Regards,

Vishwanath B

[former_member197667]

Hi Vishwanath,

Are you using third party utility for backup ?

If it is a started procedure and your facing problem, please check the follow: -

- check your OS version and HANA DB kernel level. if possible update the kernel.

- check your OS level permissions for kernel files and the backup destination.

- check your configuration.

Backing Up Customer-Specific Configuration Settings - SAP HANA Administration Guide - SAP Library

Regards,

Raghav