dbmcli does not support external commands anymore

[Former Member]

My backup script doesn't work with dbmcli version 7.9.08.10.

If you store your MaxDB data sets on NetApp-Filer volumes you can use NetApp-Snapshots to backup them. Since MaxDB version 7.8 this is supported by MaxDB with device type EXTERNAL.

After creating an new backup template (e.g. NETAPP) with device type EXTERNAL you can use the dbmcli commands

"backup_start NETAPP DATA"

and

"backup_finish NETAPP ExternalBackupID $SNAPSHOT"

The first command "backup_start NETAPP DATA" tells maxdb to do a savepoint and take a temporay MaxDB snapshot. Now you can backup the database with operating system tools e.g. draw a NetApp-Filer snapshot. The second dbmcli command registers the backup in the backup history list and deletes the temporay MaxDB snapshot.

Unfortunatly you have to submit the two dbmcli commands within the same database session. This is no problem if you do the backup of your MaxDBs in dialog mode. But if you want to script the backup you need to sumbit the NetApp snapshot command from within one dbmcli session between backup_start and backup_finish. I have done it until now with a HERE script:

dbmli -Uc -d <SID> >/dev/null 2>&1 <-HERE

db_connect

backup_start NETAPP DATA

!ssh sapfiler snap create dbvol $SNAPSHOT

backup_finish NETAPP ExternalBackupID $SNAPSHOT

db_release

quit

HERE

Earlier versions of MaxDB knew the statement "exec_command". This is long gone. But the undocumented feature "!" to start os commands survived in dbmcli until  7.9.08.05. Now they declared this bitter needed feature to a security risk. Since 7.9.08.06 you get an error (unkown command).

Searching for a solution I tried the "background_server" commands. But they only usable in dialog mode.

Any suggestions?

Regards

Günter Hoeth

[Former Member]

A workaround is to copy an older version of the dbmcli executable from another installation. A 7.9.07.10 file works fine with a 7.9.08.15 installation for example. The executable is located in /sapdb/clients/<SID>/bin (linux).

[Former Member]

Yes, that's what I did. I use this older version only in my backup script.

[Former Member]

Hi Günter,

you are right - MaxDB does not support executation of external commands via DBMCLI ! anymore because of security reasons.

Regards, Christiane

[Former Member]

Hi Christiane,

I understand the security risk - but SAP cannot give up important functionality for backup without supplying an alternative. Since 7.8 MaxDB supports I/P consitent EXTERNAL snapshots: 

db_connect

backup_start <template_name>

   <create the IO consistent Snapshot clone>

backup_finish <template_name> ExternalBackupID <external_backup_ID>

db_release

Now these snapshots are not longer possible. We need to start an external script from within dbmcli.

You have already a concept for that: Events are allowed to start scripts which are listed in dbm_whitelist.cfg.

Regards Günter

[Former Member]

Hello Günter,

use the pyton script  belo with the sapdb.dbm interface and call it with

x_python -d <SID> -u CONTROL,CONTROL

Kind regards,

Burkhard

#!/usr/bin/env python
import sys
import string
import optlib
import sapdb.dbm
import os

def _options ():
    return [
    ('n', 'host'            , ':', 'localhost'         , 'specifies servernode'),
    ('d', 'dbname'           , ':', 'SDB'              , 'specifies the database name'),
    ('u', 'userpwd'          , ':', 'DBM,DBM'          , 'specifies user and password'),
]

def main (options, args):
    session = sapdb.dbm.DBM (options.host, options.dbname, '', options.userpwd)
    result = session.cmd('backup_start NETAPP DATA')
    os.system('ssh sapfiler snap create dbvol $SNAPSHOT')
    result = session.cmd('backup_finish NETAPP ExternalBackupID $SNAPSHOT')

if __name__ == "__main__":
    optlib.optMain2 (main, _options ())

[Former Member]

Hello Burkhard,

thanks for your script. Never work with python before but I tried to start it. Had to call x_python it with full path even after setting the environement with  .dbenv.sh. Now I get an   ImportError: No module named sapdb.dbm.

I searched the whole volume /sapdb for such a module but could not find one.

Regards

Günter

[Former Member]

Hello Günter,

my fault. The module name has been changed from sapdb to sdb.

Change in the script the strings sapdb.dbm to sdb.dbm. It occures twice.

If you like to know from where python loads the modules you can use

x_python

>>> import sdb.dbm

>>> sdb.dbm

<module 'sdb.dbm' from 'D:\sapdb\clients\D73\lib\python2.3\sdb\dbm.pyd'

Kind regards,

Burkhard

[Former Member]

Hello Burkhard,

now I get following error:

/sapdb/J06/db/bin/x_python -d J06 -u CONTROL,CONTROL

Traceback (most recent call last):

  File "J06", line 5, in ?

    import sdb.dbm

ImportError: /sapdb/J06/db/lib/python2.3/sdb/dbmmodule.so: undefined symbol:

_ZN20RTEIPC_ObjectManager8InstanceEv

By the way:  There is no subdirectory "python2.3" in the MaxDB clients path at our site:

find /sapdb/* -name "*python*"

/sapdb/J06/db/bin/x_python

/sapdb/J06/db/pgm/python

/sapdb/J06/db/lib/python2.3

/sapdb/J06/db/lib/python23.zip

We are working with SLES11SP2 (64-bit). The python version is python-2.6.8-0.15.1.

sdbregview -l

Installation: Global    /sapdb/programs

Global Listener                7.9.08.03    valid    64 bit

Installation Compatibility     7.9.08.03    valid    64 bit

Installer                      7.9.08.03    valid

SAP Utilities Compatibility    7.9.08.03    valid    64 bit

Installation: J06    /sapdb/J06/db

Base                7.9.08.03      valid    64 bit

DB Analyzer         7.9.08.03      valid    64 bit

Database Kernel     7.9.08.03      valid    64 bit

Fastload API        7.9.08.03      valid    64 bit

JDBC                7.6.06.09      valid

Loader              7.9.08.03      valid    64 bit

Messages            MSG 0.10302    valid

ODBC                7.9.08.03      valid    64 bit

Redist Python       7.9.08.03      valid    64 bit

SAP Utilities       7.9.08.03      valid    64 bit

SQLDBC              7.9.08.03      valid    64 bit

SQLDBC 76           7.6.06.22      valid    64 bit

SQLDBC 77           7.9.08.03      valid    64 bit

Server Utilities    7.9.08.03      valid    64 bit

Installation: CL_J06    /sapdb/clients/J06

Base             7.9.08.03      valid    64 bit

Fastload API     7.9.08.03      valid    64 bit

JDBC             7.6.06.09      valid

Messages         MSG 0.10302    valid

ODBC             7.9.08.03      valid    64 bit

SAP Utilities    7.9.08.03      valid    64 bit

SQLDBC           7.9.08.03      valid    64 bit

SQLDBC 76        7.6.06.22      valid    64 bit

SQLDBC 77        7.9.08.03      valid    64 bit

Kind regards

Günter

[Former Member]

Hello Günter,

I have checked this and find out that the symbol ZN20RTEIPC_ObjectManager8InstanceEv is missing in 7.9.08.03 dbmmodule.so.

With 7.9.08.08 and 7.8.02.35 it works, You should install a newer version

The python package is not installed as default for a client with SAPINST. But you can install it separate by download the server or client and calling

SDBINST -package "Redist Python"

Kind regards,

Burkhard

[Former Member]

Hello Burkhard,

finaly, after updating the database of a test system to 7.9.08.11, I got the python script  running. But afterwards I went back to the NetApp snapshot I took before. Reasons:

1. Python is not officially supported by SAP (see statement of Daniel Dittmar in

http://scn.sap.com/thread/3156375). And after doing some researchs in the forum and in OSS I found many hints that Daniel is right, e.g.
http://scn.sap.com/thread/1356737 "What is the relationship between MaxDB and Python?"
http://scn.sap.com/thread/1348638 "Support for Python 2.5 / 2.6"
http://scn.sap.com/thread/3265891"7.8 feature state: WebDAV, Python?"

And that dbmmodule.so was missing in build 7.9.08.03 fits in this picture. Even the documentation in the python manuals get smaller and smaller from version to version. The last time that the module sdb.dbm was  mentioned was in the 7.7 manual (http://maxdb.sap.com/doc/7_7/default.htm)

2. Python does not support xuser. You have to deposit user names and passwords in every script.

Resume: I am looking for a reliable and officially supported solution. We have to backup daily 22 MaxDB databases - all stored on NetApp Filer NFS Volumes. Backup with EXTERNAL SNAPSHOTS and transfering the snapshots with NetApp snapvault to a remote secondary filer is a good functioning, simple and reliable backup concept. Therefore we will insist for SAP support using mor official support channels as a public forum like SCN can be.

So thanks for your help and

kind regards

Günter

[Former Member]

Please try NetApp SnapCreator Framework 4.0p1 for backing up MaxDB

You need a NetApp support account to download the software.

https://communities.netapp.com/community/products_and_solutions/databases_and_enterprise_apps/snapcr...

[brian_griffin3]

Hello Burkhard,

What is the proposed solution now that the python libraries are no longer supported on MaxDB 7.9?  The ability to execute a files ystem snapshot from a script is a key request from clients.

[thorsten_zielke]

Hello Brian,

let me recommend to start a 'dbmsrv' in background, that is give it a name of your choice and then let it run in background via special dbmcli command. You can then send commands to that named dbmsrv process whenever you wish. This should - at least in theory - cover all your reqirements.

Here is the syntax (to be executed via dbmcli):

background_server_execute <bg_server_name> [-no_reply] <command>

background_server_exit <bg_server_name>
background_server_get_reply <bg_server_name> -wait [<skip_bytes>]
background_server_reset <bg_server_name>
background_server_show_status <bg_server_name>
background_server_start <bg_server_name>
background_server_takeover <bg_server_name>

Unfortunately, that feature is a bit complicated to work with, so let me try to explain:

You should always keep in mind that you need to start a 'dbmcli session' first.
1. background_server start myname -> this sets a name for the background server
2. background_server_execute myserver db_connect -> the background server connects to the database using the credentials from the dbmcli session
3. background_server_get_reply myserver -wait -> this is certainly odd, but you must fetch the reply of each command, otherwise the next command will throw an error.
4. background_server_execute myserver backup_start <backup_template> data -> starts a backup
5. Exit the dbmsrv session

Now, to connect to that background session again, you need to explicitly take it over - you cannot simply start a session with the same name:
6. start a new dbmcli session
7. background_server_takeover myserver
8. background_server_execute myserver backup_finish <backup_template> ExternalBackupID <external_backup_ID>
9. background_server_get_reply myserver -wait
10. background_server_exit myserver

Hope this helps,
Thorsten

Edit: '-wait' option added

[brian_griffin3]

Thank you for the reply Thorsten!

To make sure I'm issuing multiple commands dbmcli commands in the same session, would you suggest I utilize the Script Mode or Session Mode?

It's my understanding I would do this via script mode, so I could pass all of the parameters consecutively.

I'm not aware of the format to use for passing multiple dbmcli commands via session mode (in the same session).  Perhaps you are suggesting I can run each of those background_server commands separately...each in their own session?

Something like this:

/sapdb/programs/bin/dbmcli -u <user>,<pw> -d LPD -uUTL -c background_server start backupsession

/sapdb/programs/bin/dbmcli -u <user>,<pw> -d LPD -uUTL -c background_server_execute backupsession db_connect

/sapdb/programs/bin/dbmcli -u <user>,<pw> -d LPD -uUTL -c background_server_get_reply backupsession

/sapdb/programs/bin/dbmcli -u <user>,<pw> -d LPD -uUTL -c background_server_execute backupsession backup_start COMP_DATA_SNAP data

Thoughts?

Best,

Brian

[brian_griffin3]

Hi Thorsten,

It appears that is is required to use the -i option (script mode), as the commands need to be executed consecutively.

I've created and tested the commands in the foreground, and they work as expected.  The issue remains, however, in passing the commands via an automated script. 

When I put the commands in a script, the dbmcli issues the first two commands successfully, but then bombs out.  It looks like the "background_server_get_reply backupsession" command executes before the previous command (background_server_execute backupsession db_connect) is able to completely finish the task.

This results in the message

"

ERR

-24647,ERR_BGS_REPLY: No reply of background DBM server backupsession available

-24652,ERR_BGS_STATE: Background DBM server backupsession is in state "command available", but state "ready" is required"

Is there a known way to work around this issue?

-Brian

[brian_griffin3]

Although it is less than ideal, I was able to break the commands into multiple script files due to the errors.  By having four separate text files with the dbm commands I need, I created a wrapper script (bash) to put 10 second pauses in between the areas where it would consistently fail.

This solved the issue, and is what we'll be using going forward.

Thanks for the tip on background_server feature.

Best,

Brian

[thorsten_zielke]

Hi Brian,

indeed, when I read your first reply I noticed that I should have mentioned the 'dbmcli -i' option to execute dbmcli commands from an input file in the same session, but by now you already know this 🙂

I am glad you got your script to work, but still if there is any bug on our side, we should fix it. Let me test it a bit and I will get back to you later...

And one more hint about dbmcli commands in general: you can get detailed help for every dbmcli command by using an 'explain' within a dbmcli session, e.g. 'explain background_server_show_status'.

Thorsten

[thorsten_zielke]

Hi Brian,

ok, it is a bug, but I have to admit that fixing it is not high on our priority list... I tried to find a workaround, but other than using separate input files as you already did, nothing worked for me.

Thorsten

Edit: Bug has already been fixed in newer versions of MaxDB 7.9 - problem was that the 'wait' option is mandatory when using multiple commands in a dbmcli -i input script, but the '-wait' option does not work in older releases including all of 7.8

[brian_griffin3]

Thanks for the update Thorsten.

For those out there who want to see what a final solution looks like, remember to issue the "background_server_takeover" command at the start of each script. 

Here here are my input scripts, which I call from a wrapper bash script:

snapstart1.txt

background_server_start backupsession

background_server_execute backupsession db_connect

exit

snapstart2.txt

background_server_takeover backupsession

background_server_get_reply backupsession

background_server_execute backupsession backup_start COMP_DATA_SNAP data

exit

snapstop1.txt

background_server_takeover backupsession

background_server_get_reply backupsession

background_server_execute backupsession backup_finish COMP_DATA_SNAP ExternalBackupID 60

snapstop2.txt

background_server_takeover backupsession

background_server_get_reply backupsession

background_server_exit backupsession

exit

I have a sed command in the wrapper script which increments the number of the ExternalBackupID each time, which is important so different numbers are stored in the backup history.

[thorsten_zielke]

Brian,

thank you for posting your solution - I think this approach should be made more visible to our customers. May I use these 'snap...' scripts in a new SAP note describing how to start external snapshots with the 'background server'?

Thorsten

Later comment: Although this solution does work, the script can be made simpler for MaxDB versions fully supporting the '-wait' option

[thorsten_zielke]

Brian,

one more thing - can you let me know which MaxDB version you are using? I know that in MaxDB 7.8 the background server still has some errors, which indeed would require you to use several input files (and unfortunately I had tested with 7.8 during our conversation). Now I am using 7.9 and things look much better...

Still, I am going to write a SAP note about this.

Thorsten

[thorsten_zielke]

ok, to be more precise:

If you use the 'dbmcli -i' input file option in combination with the background server, you must set the '-wait' option for the 'background_server_get_reply' command.

Due to a bug, the '-wait' option does not work as expected in MaxDB 7.8, so you indeed have to split the commands in several input files or upgrade to MaxDB 7.9.

Here is an example:

Input script 1 and MaxDB 7.9

background_server_start myserver

background_server_execute myserver db_state

background_server_get_reply myserver

background_server_execute myserver db_state

background_server_get_reply myserver

background_server_exit myserver

-> -24652,ERR_BGS_STATE: Background DBM server myserver is in state "command available", but state "ready" is required

Input script 2 and MaxDB 7.9

background_server_start myserver

background_server_execute myserver db_state

background_server_get_reply myserver -wait

background_server_execute myserver db_state

background_server_get_reply myserver -wait

background_server_exit myserver

-> works as expected

Thorsten

[brian_griffin3]

Hi Thorsten-

You may certainly use these commands in an SAP note.  I know others will find them valuable if they are using MaxDB and are instituting file level snapshots.  In our use case, we're using the snapshot capability of the EBS drives on Amazon Web Services.

I am using MaxDB 7.9.08.23 at the moment, so I'll test the -wait option and let you know if it works for me.

Best,

Brian

[thorsten_zielke]

... I have summarized this in the new SAP note 2282054 (Background-DBM-Server).