Content server migration - loadercli and I'm running nuts...

[markus_doehr2]

We have an (old) content server database of 150 GB on 7.3.0.52 and we're trying to migrate that database to Linux x86_64 bit. I tried the following before:

- tried to install/run 7.3 on SuSE SLES 11 SP1 - failing (new kernel threading)

- tried to restore the 7.3 backup on 7.5 - failing (host pagaes too old)

- tried to use 'loadercli' of 7.5 (Note 962019) - failing (ASCII --> Unicode)

- now trying to use 7.6.06.20 and stuck

I want to use pipes as transport and use loadercli on the target system (7.6).

I created two files (according to note 962019)

EXPORT USER
catalog outstream pipe '/home/sqdcos/trans/COS.CAT'
data outstream pipe '/home/sqdcos/trans/COS.DATA' PAGES
package outstream file '/home/sqdcos/trans/COS.export'

and

IMPORT USER
catalog instream pipe '/home/sqdcos/trans/COS.CAT'
data instream pipe '/home/sqdcos/trans/COS.DATA' PAGES
package outstream '/home/sqdcos/trans/COS.import'

The pipes are not existing.

I start the export which seems to work, the COS.CAT pipe is created.

As soon as I start the import, I get the following error message:

IMPORT USER
catalog instream pipe '/home/sqdcos/trans/COS.CAT'
data instream pipe '/home/sqdcos/trans/COS.DATA' PAGES
package outstream '/home/sqdcos/trans/COS.import'
// *
// M    Execute   PACKAGE  to transform  CATALOG
/ *
// M    Import    PACKAGE x'01000000A296EB4EB45800009B16031EC842BF0100000000'
// *
// M    Number of TABLES   transformed : 3
// *
// M    Processed command is a PAGES format based copy from database 
// with ASCII catalog to database with UNICODE catalog
// *
// M    Execute   PACKAGE  to transform  DATA
// *
// M    Number of TABLES   to transform: 0
// *
// E -25329:    The given data file '/home/sqdcos/trans/COS.DATA' was not 
// generated using EXPORT in PAGE Format (missing table description).

plus I get an additional pipe created "COS.DATA0000"

What am I missing here? I'm fiddling with this since hours and I can't figure what I'm doing wrong.

Markus

[steffen_schildberg]

Hi Markus,

I'll have a look at it. Am I right that it has to be solved by yesterday

Best,

Steffen

[markus_doehr2]

> I'll have a look at it. Am I right that it has to be solved by yesterday

Well - we want to do this migration "between the days" (x-mas and new year)... this is just a first test run.

I fiddled a bit around. It is working if I use RECORDS instead of PAGES, though this migration takes then roughly 40 hours. And the other interesting part is: the souce database 7.3 is 131 GB, loadercli says "finished" when the target database is 61 GB filled (so 50 %). Is this normal?

Markus

[steffen_schildberg]

Hi Markus,

what is the source platform (just to be able to test here)?

50% less data volume in the target sounds strange. What does the loader.log say - source and target - anything suspicious?

Best,

Steffen

[markus_doehr2]

what is the source platform (just to be able to test here)?

Source platform is SLES 9 32bit

Target is SLES 11 SP1 64bit

> 50% less data volume in the target sounds strange. 

it is, data is missing.

> What does the loader.log say - source and target - anything suspicious?

Not really, it looks "good":

loadercli -d COS -n xx.xx.xx.xx -u SAPR3,SAP -b COS_EXPORT.sql
Loader protocol: '/home/sqdcos/sdb/connd266/loader/log/loader.log'
Loader packages: '/home/sqdcos/sdb/connd266/loader/packages'

User SAPR3 connected to database COS schema SAPR3 on 191.1.1.29.
EXPORT USER
catalog outstream pipe '/home/sqdcos/trans/COS.CAT'
data outstream pipe '/home/sqdcos/trans/COS.DATA' RECORDS
package outstream file '/home/sqdcos/trans/COS.export'
Successfully executed:
---
Total number of tables (definition) exported: 3
Total number of tables (data)       exported: 3 (excluded: 0, failed: 0)

loadercli -d COS -u SAPR3,SAP -b COS_IMPORT.sql
Loader protocol: '/home/sqdcos/sdb/connd266/loader/log/loader_2011121600202813.log'
Loader packages: '/home/sqdcos/sdb/connd266/loader/packages'

User SAPR3 connected to database COS schema SAPR3 on local host.
IMPORT USER
catalog instream pipe '/home/sqdcos/trans/COS.CAT'
data instream pipe '/home/sqdcos/trans/COS.DATA' RECORDS
package outstream '/home/sqdcos/trans/COS.import'
Successfully executed:
---
Total number of tables (definition) imported: 3
Total number of tables (data)       imported: 3 (excluded: 0, failed: 0)

Could/should we use a higher version on the target system?

Markus

[steffen_schildberg]

Hi Markus,

no, this really looks good! I'd say: be happy ;-). Did you do any checks to see if the data is there? I've seen other systems where the Loader logged the execution as finished successfully but only for x of y tables where x < y. Would you mind to send me the COS.export and COS.import files? There should be logged (additionally) how many records were exported and imported respectively. If this matches everything is ok. Unfortunately does the Loader not check for such things at the end of the import.

Best,

Steffen

[markus_doehr2]

> no, this really looks good! I'd say: be happy ;-). Did you do any checks to see if the data is there?

Yes, it is missing. 6 to 7 out of 20 drawings are not existing whereas they exist on the source system.

> Would you mind to send me the COS.export and COS.import files?

Those were pipes because we don't have the space to export on file.

> There should be logged (additionally) how many records were exported and imported respectively. If this matches everything is ok. Unfortunately does the Loader not check for such things at the end of the import.

I don't see any numbers. loader.log contains the package ID as last entry, after that no more is visible.

Markus

[steffen_schildberg]

Hi Markus,

please send me the files named COS.export and COS.import. In contrary to your last post those should be files as you can see from this snippet taken from your command posted here:

package outstream file '/home/sqdcos/trans/COS.export'

These files are used to store the content of the Loader system table for a certain command after its execution - they shouldn't be really big.

Thx,

Steffen

[steffen_schildberg]

Hi Markus,

sorry for this piece wise requests: may I get the loader.log, too.

Thx,

Steffen

[markus_doehr2]

> please send me the files named COS.export and COS.import. In contrary to your last post those should be files as you can see from this snippet taken from your command posted here:

>

package outstream file '/home/sqdcos/trans/COS.export'

> These files are used to store the content of the Loader system table for a certain command after its execution - they shouldn't be really big.

Those files were never created (neither for source nor target).

Markus

[steffen_schildberg]

Well, if this is the case the export/import is simply crashed. Those files are written if the commands finish successfully. Is it an option to do the migration table wise?

Best regards,

Steffen

[markus_doehr2]

> Well, if this is the case the export/import is simply crashed.

and a crashed export/import says "successfull" in the logfiles? Seriously?

> Those files are written if the commands finish successfully. Is it an option to do the migration table wise?

There are only three tables and one of them contains all the LOBs (which is basically 149 GB of the 150 GB).

We tried to do the migration today - during runtime, apparently 7.3 creates a table lock when exported with loadercli (with records as well as with pages) and this disables people to work with it.

We're now trying to set up an environment where 7.3 and 7.6 can run, we will do a backup/restore over remote pipes. Unfortunately those two systems are on a different location so we can't just use one of our local systems to do the migration. And copying 150 GB over a WAN connection is also not an option.

Basically all this problems are caused by the "sapkprotp" programm, which doesn't work on Linux (it simpyl stops without any traces and error messages), we are fiddling with this issue since over 6 months now again. Interestingly enough, the whole procedure works if a sapkprotop on a Windows machine is used (OSS 664270/2009).

sigh

Markus

[steffen_schildberg]

> and a crashed export/import says "successfull" in the logfiles? Seriously?

>

Hmm, may be there is still a problem with the pipes and then the files do not get written. I've to check this, too. The check list gets longer

> We tried to do the migration today - during runtime, apparently 7.3 creates a table lock when exported with loadercli (with

> records as well as with pages) and this disables people to work with it.

>

You can omit the locks if you set the isolation level to 0 or 1 and switch the locks off during export using the commands

set isolation level 0

and

lock off

. Place them in your script before connecting to be sure.

Sorry,

Steffen

[markus_doehr2]

Hi Steffen,

I tried to find in the documentation how and where to set them, information is spread but I can't find anything specific what the proper syntax of that is.

siauqas:q01adm 61> cat export.control 
set isolation level 0 
lock off
EXPORT USER
catalog outstream file '/edvinst/CONTENT/COS.CAT'
data outstream file '/edvinst/CONTENT/COS.DATA' RECORDS
package outstream file '/edvinst/CONTENT/COS.export'

I get

Loader protocol: '/home/q01adm/sdb/siauqas/loader/log/loader.log'
Loader packages: '/home/q01adm/sdb/siauqas/loader/packages'

User SAPR3 connected to database COS schema SAPR3 on xx.xx.xx.xx
set isolation level 0 
lock off
EXPORT USER
catalog outstream file '/edvinst/CONTENT/COS.CAT'
data outstream file '/edvinst/CONTENT/COS.DATA' RECORDS
package outstream file '/edvinst/CONTENT/COS.export'
Error during execution
-->-25281
Syntax error at position 24 (Unexpected keyword: lock); check for typing errors

Defined maximum number of errors (1) reached


1 error(s) while processing command file 'export.control'.
---> See Loader log file for more information.

I found http://help.sap.com/saphelp_nw73/helpdata/en/ee/adaca262c34551b3ff176878e658f0/frameset.htm but I must admit, I have no clue where and how to set those parameters.

Markus

[markus_doehr2]

Hi Steffen,

after having fiddled today another some hours trying to find in the (with full respect) messy documentation of different loader versions and command line options (there was an option -I 0 with 7.4, it's gone) I give up on this.I guess I will have to be a developer to understand the difference between "command", "transport" and "session" mode and what command has to be put at what point in what sequence in what file. NO accusation to you, you were really helpful and trying to help a lot what I appreciate but this is just not going to work like this.

We will now try to get a backup over a slow WAN line and export the content of the database using sapkprotop to eventually get all the data on filesystem omitting the usage of a database (what was the original intent).

Thank you for your help!

Markus