Replication Down on LTI error

[Former Member]

Dear All,

I have an issue with replication - the replication was up and running until we got an error:

Replication Server refused the following invalid LTL command; LTI thread is now stopping. - the command is provided - basically it is rs_update.

The next error is Line 1, character 1117: Invalid token length encountered.

So this is $.SmallIntValue=NULL - but this is standard...

The details below.

E.

2015/07/13 16:43:05.182

ERROR

com.sybase.ra.lti.rs.LTLSession

_ds 4 ~,A{0x}000000004e92000111c700260000040000004e92000111c700040001049e000c,1 ~*620150707 20:48:01:440,6 ~,J{0x}5052445f53414d31325f52412e5052445f53616d706c656d616e6167656d656e7431322d79f76f0000 _ap  owner=~"$dbo ~"+Properties.rs_update _yd _bf ~G,LocalIDItem=~L!(7201977,~G-PropertyName=~X)Pedigree,~$,BigIntValue=NULL,~$)BitValue=NULL,~$.DateTimeValue=NULL,~$+FloatValue=NULL,~$)IntValue=NULL,~$+MoneyValue=NULL,~$-NumericValue=NULL,~$*RealValue=NULL,~$.SmallIntValue=NULL,~$-TinyIntValue=NULL,~$6UniqueIdentifierValue=NULL,~$-VarCharValue=NULL,~$*ValueType=~X.PedigreeValue,~$5XmlValue_As_nVarChar=~c"5{0x}003c0050006500640069006700720065006500560061006c00750065003e003c00500065006400690067007200650065002f003e003c002f0050006500640069006700720065006500560061006c00750065003e,~$7Attributes_As_nVarChar=NULL,~$<VarCharMaxValue_As_nVarChar=NULL,~$*Timestamp=~1){0x}000000003b37c843 _af ~G,LocalIDItem=~L!(7201977,~G-PropertyName=~X)Pedigree,~$,BigIntValue=NULL,~$)BitValue=NULL,~$.DateTimeValue=NULL,~$+FloatValue=NULL,~$)IntValue=NULL,~$+MoneyValue=NULL,~$-NumericValue=NULL,~$*RealValue=NULL,~$.SmallIntValue=NULL,~$-TinyIntValue=NULL,~$6UniqueIdentifierValue=NULL,~$-VarCharValue=NULL,~$*ValueType=~X.PedigreeValue,~$*Timestamp=~1){0x}000000003b37c890,~$7Attributes_As_nVarChar=NULL,~$5XmlValue_As_nVarChar=~d"';{0x}3c0050006500640069006700720065006500560061006c00750065003e003c00500065006400690067007200650065003e003c0041007500740068006f0072006900740079002000490044003d002200370065003300380066003000370061002d0061006500360031002d0034003400320063002d0061003300630034002d0037006200310062003700350061003100390030003100350022003e003c0050006100740068003e003c004900740065006d002000490044003d002200370065003300380066003000370061002d0061006500360031002d0034003400320063002d0061003300630034002d0037006200310062003700350061003100390030003100350022002f003e003c004900740065006d002000490044003d002200650037003900660033003900310030002d0066006500650061002d0034003600380037002d0062006200300035002d0031003700330064003100390065006500340064006400640022002f003e003c004900740065006d002000490044003d002200630037003700360032003000640032002d0062003000640036002d0034003100660037002d0038003100370066002d0030006300320035006200340066006500320061006400320022002f003e003c004900740065006d002000490044003d002200660061003300620061006100320031002d0065003700320065002d0034003300350035002d0061006100640030002d0064003600660038003600650066003900650066003400310022002f003e003c004900740065006d002000490044003d002200350033006300390033003600340065002d0066006200300030002d0034003100320039002d0039003700640036002d0034003000620031006400630066006300310036006600360022002f003e003c004900740065006d002000490044003d002200300063003100390038003900620064002d0035003500640038002d0034006100360032002d0038006300310039002d0061003900370031003200660039003200620062006500330022002f003e003c004900740065006d002000490044003d002200620034006400370035003400340066002d0032003400660063002d0034006200340064002d0062003600630063002d0063003900620066003700660030003800380061006200300022002f003e003c002f0050006100740068003e003c0050006100740068003e003c004900740065006d002000490044003d002200370065003300380066003000370061002d0061006500360031002d0034003400320063002d0061003300630034002d0037006200310062003700350061003100390030003100350022002f003e003c004900740065006d002000490044003d002200650037003900660033003900310030002d0066006500650061002d0034003600380037002d0062006200300035002d0031003700330064003100390065006500340064006400640022002f003e003c004900740065006d002000490044003d002200630037003700360032003000640032002d0062003000640036002d0034003100660037002d0038003100370066002d0030006300320035006200340066006500320061006400320022002f003e003c004900740065006d002000490044003d002200660061003300620061006100320031002d0065003700320065002d0034003300350035002d0061006100640030002d0064003600660038003600650066003900650066003400310022002f003e003c004900740065006d002000490044003d002200350033006300390033003600340065002d0066006200300030002d0034003100320039002d0039003700640036002d0034003000620031006400630066006300310036006600360022002f003e003c004900740065006d002000490044003d002200300063003100390038003900620064002d0035003500640038002d0034006100360032002d0038006300310039002d0061003900370031003200660039003200620062006500330022002f003e003c004900740065006d002000490044003d002200660039003200330036006200350038002d0037003400640034002d0034006500630038002d0038003700640036002d0063006300660032003100360031003800650039003000630022002f003e003c002f0050006100740068003e003c0050006100740068003e003c004900740065006d002000490044003d002200370065003300380066003000370061002d0061006500360031002d0034003400320063002d0061003300630034002d0037006200310062003700350061003100390030003100350022002f003e003c004900740065006d002000490044003d002200650037003900660033003900310030002d0066006500650061002d0034003600380037002d0062006200300035002d0031003700330064003100390065006500340064006400640022002f003e003c004900740065006d002000490044003d002200630037003700360032003000640032002d0062003000640036002d0034003100660037002d0038003100370066002d0030006300320035006200340066006500320061006400320022002f003e003c004900740065006d002000490044003d002200660061003300620061006100320031002d0065003700320065002d0034003300350035002d0061006100640030002d0064003600660038003600650066003900650066003400310022002f003e003c004900740065006d002000490044003d002200350033006300390033003600340065002d0066006200300030002d0034003100320039002d0039003700640036002d0034003000620031006400630066006300310036006600360022002f003e003c004900740065006d002000490044003d002200300063003100390038003900620064002d0035003500640038002d0034006100360032002d0038006300310039002d0061003900370031003200660039003200620062006500330022002f003e003c004900740065006d002000490044003d002200630061006400620031006600610063002d0038003900370033002d0034006100300031002d0039006100360064002d0033003300630035006200390031003500360039003000380022002f003e003c002f0050006100740068003e003c0050006100740068003e003c004900740065006d002000490044003d002200370065003300380066003000370061002d0061006500360031002d0034003400320063002d0061003300630034002d0037006200310062003700350061003100390030003100350022002f003e003c004900740065006d002000490044003d002200650037003900660033003900310030002d0066006500650061002d0034003600380037002d0062006200300035002d0031003700330064003100390065006500340064006400640022002f003e003c004900740065006d002000490044003d002200630037003700360032003000640032002d0062003000640036002d0034003100660037002d0038003100370066002d0030006300320035006200340066006500320061006400320022002f003e003c004900740065006d002000490044003d002200660061003300620061006100320031002d0065003700320065002d0034003300350035002d0061006100640030002d0064003600660038003600650066003900650066003400310022002f003e003c004900740065006d002000490044003d002200350033006300390033003600340065002d0066006200300030002d0034003100320039002d0039003700640036002d0034003000620031006400630066006300310036006600360022002f003e003c004900740065006d002000490044003d002200300063003100390038003900620064002d0035003500640038002d0034006100360032002d0038006300310039002d0061003900370031003200660039003200620062006500330022002f003e003c004900740065006d002000490044003d002200310062003900320035003100390031002d0066003700380031002d0034003000300039002d0038003400330066002d0064003900340064003400610037003400640036006200640022002f003e003c002f0050006100740068003e003c0050006100740068003e003c004900740065006d002000490044003d002200370065003300380066003000370061002d0061006500360031002d0034003400320063002d0061003300630034002d0037006200310062003700350061003100390030003100350022002f003e003c004900740065006d002000490044003d002200650037003900660033003900310030002d0066006500650061002d0034003600380037002d0062006200300035002d0031003700330064003100390065006500340064006400640022002f003e003c004900740065006d002000490044003d002200630037003700360032003000640032002d0062003000640036002d0034003100660037002d0038003100370066002d0030006300320035006200340066006500320061006400320022002f003e003c004900740065006d002000490044003d002200660061003300620061006100320031002d0065003700320065002d0034003300350035002d0061006100640030002d0064003600660038003600650066003900650066003400310022002f003e003c004900740065006d002000490044003d002200350033006300390033003600340065002d0066006200300030002d0034003100320039002d0039003700640036002d0034003000620031006400630066006300310036006600360022002f003e003c004900740065006d002000490044003d002200300063003100390038003900620064002d0035003500640038002d0034006100360032002d0038006300310039002d0061003900370031003200660039003200620062006500330022002f003e003c004900740065006d002000490044003d002200310061003300660030003000350036002d0032006600340032002d0034003200640066002d0062003400310032002d0030003500300062006600390035006300350062003700640022002f003e003c002f0050006100740068003e003c0050006100740068003e003c004900740065006d002000490044003d002200370065003300380066003000370061002d0061006500360031002d0034003400320063002d0061003300630034002d0037006200310062003700350061003100390030003100350022002f003e003c004900740065006d002000490044003d002200650037003900660033003900310030002d0066006500650061002d0034003600380037002d0062006200300035002d0031003700330064003100390065006500340064006400640022002f003e003c004900740065006d002000490044003d002200630037003700360032003000640032002d0062003000640036002d0034003100660037002d0038003100370066002d0030006300320035006200340066006500320061006400320022002f003e003c004900740065006d002000490044003d002200660061003300620061006100320031002d0065003700320065002d0034003300350035002d0061006100640030002d0064003600660038003600650066003900650066003400310022002f003e003c004900740065006d002000490044003d002200350033006300390033003600340065002d0066006200300030002d0034003100320039002d0039003700640036002d0034003000620031006400630066006300310036006600360022002f003e003c004900740065006d002000490044003d002200300063003100390038003900620064002d0035003500640038002d0034006100360032002d0038006300310039002d0061003900370031003200660039003200620062006500330022002f003e003c004900740065006d002000490044003d002200620032003800300030003800370063002d0035003200390031002d0034003700640038002d0039006600610032002d0030003800300061003700340038006200660036006500610022002f003e003c002f0050006100740068003e003c002f0041007500740068006f0072006900740079003e003c002f00500065006400690067007200650065003e003c002f0050006500640069006700720065006500560061006c00750065003e00,~$<VarCharMaxValue_As_nVarChar=NULL

Each time the replication is resumed this error pops up and the replication is down - any ideas how to troubleshoot that?

Thanks in advance,

Arek

[marcin_najs]

Hi.

Have you tried to capture the entire transaction that is causing the problem (from exceptions log)?

sysamin log_first_tran, dataserver, database;

connect rssd;

rs_helpexception <tran_id>, v;

Can you post larger errorlog fragment from the moment of RS thread(s) failure?

Best Regards

--
Marcin

[Former Member]

Marcin,

Thanks for reply - trying to capture the transaction to exception log is not working unfortunately. No error is being written to the exception log. I think this is because the error is being created when the SQL database transaction log is being red. SAP suggested to check SAP Note 2100592. LTITRACELTL log produced ~65k characters between get truncation statements so it may be the case... Some more entries from the error logs.

W.	2015/07/13 16:43:05.167	WARNING	com.sybase.ra.lti.rs.LTLSession	Replication Server refused the following invalid LTL command found in batch. Resending commands individually.
W.	2015/07/13 16:43:05.167	WARNING	com.sybase.ra.lti.rs.LTLSession	null
I.	2015/07/13 16:43:05.167	INFORMATION	com.sybase.ra.lti.rs.LTLSession	Changing batch mode to <FALSE>.
E.	2015/07/13 16:43:05.182	ERROR	com.sybase.ra.lti.rs.LTLSession	Replication Server refused the following invalid LTL command; LTI thread is now stopping.

Already posted error

I.	2015/07/13 16:43:05.229	INFORMATION	com.sybase.ra.lti.LTITaskBasedThreadManager	LTI Thread Manager received notification that thread LTI <RS Distribution> completed with <com.sybase.ra.lti.LTIException> exception.
E.	2015/07/13 16:43:05.229	ERROR	com.sybase.ra.lti.LTITaskBasedThreadManager	Rep Server refused an invalid LTL command.

E.

2015/07/13 16:43:05.229

ERROR

com.sybase.ra.lr.mssql.RAMSenderThread

Caught an unexpected fatal exception: Task java.util.concurrent.FutureTask@57423671 rejected from java.util.concurrent.ThreadPoolExecutor@3c046178[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 126]

I.	2015/07/13 16:43:05.245	INFORMATION	com.sybase.ra.lr.mssql.RAMLogReader	Transaction LogReader is Quiescing operations.
E.	2015/07/13 16:43:05.245	ERROR	com.sybase.ra.lr.mssql.RAMTranReader	The LogReader Operation Queue has been invalidated by another thread.
E.	2015/07/13 16:43:05.245	ERROR	com.sybase.ra.lr.mssql.RAMTranReader	com.sybase.ra.lr.LRException: The LogReader Operation Queue has been invalidated by another thread.
E.	2015/07/13 16:43:05.245	ERROR	com.sybase.ra.lr.mssql.RAMTranReader	com.sybase.ra.lr.OperationQueue.put(Unknown Source)
E.	2015/07/13 16:43:05.245	ERROR	com.sybase.ra.lr.mssql.RAMTranReader	com.sybase.ra.lr.mssql.RAMTranReader.processScan(Unknown Source)
E.	2015/07/13 16:43:05.245	ERROR	com.sybase.ra.lr.mssql.RAMTranReader	com.sybase.ra.lr.mssql.RAMTranReader.run(Unknown Source)
E.	2015/07/13 16:43:05.245	ERROR	com.sybase.ra.lr.mssql.RAMTranReader	com.sybase.ra.util.thread.RunnableTask.run(Unknown Source)
E.	2015/07/13 16:43:05.245	ERROR	com.sybase.ra.lr.mssql.RAMTranReader	com.sybase.ra.util.thread.TPWorker.run(Unknown Source)
E.	2015/07/13 16:43:05.245	ERROR	com.sybase.ra.lr.mssql.RAMTranReader	java.lang.Thread.run(Thread.java:791)
I.	2015/07/13 16:43:05.245	INFORMATION	com.sybase.ra.lr.mssql.RAMSenderThread	Transaction LogReader Internal Thread <T_LRSENDER> Exited.

[marcin_najs]

I'm sorry, my suggestion (about sysadmin log_first_tran) was pointless because problem occurs at LTL level (between RepAgent and RepServer) and transaction is not in DSI queue yet...

I think that description of the case (Note: #2100592 ) fits your problem:

"Reproducing the Issue:

* Compile a stored procedure with strings values with more than 32768 of length.

Cause:

* This is a Replication Server Limitation.

The limitation is based in to replicate no more than 32768 length of any String values.

Workaround:

* Split the stored procedures involved – Limit the string value length to 32768."

Maybe you should locate the procedure and try to apply this workaround... If it does not work - open new case in Tech Support.

Best Regards

--

Marcin