Error 14023 rep agent

[jmtorres]

Hello,  I'm

Looking for the cause of the following error on a WS with RS 15.7.1 on linux:

1. E. 2015/11/04 05:10:43. ERROR #14023 REP AGENT(dba.machi) - neric/exec/execint.c(7206)

SQM had an error writing to the inbound-queue.

1. I. 2015/11/04 05:50:07. ...... connected to server 'huasco' as user 'fin700v60_maint'.
2. I. 2015/11/04 06:00:07. Connection to server 'huasco' as user 'fin700v60_maint' has been faded out (closed).
3. I. 2015/11/04 06:02:08. ...... connected to server 'huasco' as user 'fin700v60_maint'.
4. E. 2015/11/04 06:08:17. ERROR #6120 dAIO( ) - d64/sqm/rsaiolinux.c(1015)

SQM detected a failing status from an outstanding AIO (Status not set.).

1. E. 2015/11/04 06:08:17. ERROR #6026 SQM(118:1 WS_dba.fin700v60) - /generic/sqm/sqmio.c(1362)

Block write failed for queue '118:1', segment 0, block 0. OS dependent error is 'Status not set.'

1. I. 2015/11/04 06:08:17. SQM stopping due to an exception: 118:1 WS_dba.fin700v60
2. E. 2015/11/04 06:08:17. ERROR #30024 DIST(118 WS_dba.fin700v60) - /generic/exec/dist.c(6367)

The distributor for 'WS_dba.fin700v60' failed while reading a transaction from it's stable queue.

1. E. 2015/11/04 06:08:17. ERROR #14023 REP AGENT(dba.fin700v60) - neric/exec/execint.c(532)

SQM had an error writing to the inbound-queue.

Thank you

Regards

Jose

[jmtorres]

Terry ,

Thanks a lot . Some final questions. I know I have to run :

alter connection to huasco.machi set dsi_sqt_max_cache_size to '<new_value>'

go

How  van i get the current  <run value> for this parameter for this connection ?

If i run just admin config, it'll show the value , but this is a global value?

Regards

Jose

[jmtorres]

Terry, i executed

admin config , "connection", huasco, machi , dsi_sqt_max_cache_size

go

and got this value : 9663676416  

sqt_max_cahe_size is set to 12021705523

Thanks

JM

[terry_penna]

Jose

Can you check your disks that your stable devices are on, the RS is reporting an error from the OS?  Are these disks available, do they have any errors?

This error is telling you that due to something in the OS concerning your hard disks is preventing the RS from reading the stable device.

Regards.

Terry

[jmtorres]

Hey guys,

After 2 reboots the intial error apparently went away. We're checking SDs anyway. But  I think w're on some memory issues. I've attached  part of Rep server errorlog .

memory-limit is set to 57 Gb

sqt_max_cache_size is set to 11Gb

Machine RAM is 60gb dedicated only to RS /15.7.1/EBF 24235 SP207 rs1571sp207/Linux AMD64

On thing I sholud point is one DB "machi" has a heavy transactional usage during day and at night SQL batch jobs are run at the PDB .

Thank you

Regards

Jose

RS.txt.zip

[terry_penna]

Hi Jose

From looking at your RS log I see the following after your final restart of the RS:

W. 2015/11/05 09:41:16. WARNING #24057 DSI(152 huasco.machi) - generic/sqt/sqtint.c(8029)

  _sqt_remove_largest_tran(146:1 WS_dba.machi): No candidate found for removal. Memory limit will be exceeded by SQM/TI thread.

W. 2015/11/05 09:41:16. WARNING #24057 DSI(152 huasco.machi) - generic/sqt/sqtint.c(8029)

  _sqt_remove_largest_tran(146:1 WS_dba.machi): No candidate found for removal. Memory limit will be exceeded by SQM/TI thread.

W. 2015/11/05 09:41:16. WARNING #24057 DSI(152 huasco.machi) - generic/sqt/sqtint.c(8029)

  _sqt_remove_largest_tran(146:1 WS_dba.machi): No candidate found for removal. Memory limit will be exceeded by SQM/TI thread.

W. 2015/11/05 09:41:16. WARNING #24057 DSI(152 huasco.machi) - generic/sqt/sqtint.c(8029)

  _sqt_remove_largest_tran(146:1 WS_dba.machi): No candidate found for removal. Memory limit will be exceeded by SQM/TI thread.

W. 2015/11/05 09:41:16. WARNING #24057 DSI(152 huasco.machi) - generic/sqt/sqtint.c(8029)

  _sqt_remove_largest_tran(146:1 WS_dba.machi): No candidate found for removal. Memory limit will be exceeded by SQM/TI thread.

W. 2015/11/05 09:41:16. WARNING #24057 DSI(152 huasco.machi) - generic/sqt/sqtint.c(8029)

This indicates that this DSI may still need more sqt cache?  However instead of increasing the global sqt_max_cache_size I would recommend altering the connection huasco.machi and set the dsi_sqt_max_cache_size higher for this connection.  Then you could lower the global sqt_max_cache_size if your other connections are not as busy. 

Lowering the global sqt_max_cache_size will give more memory to RS and can help prevent these warnings which I saw in the RS log previously, before your last restart. When these warnings occur replication will stop until more memory is freed up.  All of this memory is taken from the memory_limit parameter.

W. 2015/11/05 01:45:08. WARNING #14141 REP AGENT(dba.FIN700_GLB) - neric/exec/execint.c(368)

  WARNING: Replication Agent for dba.FIN700_GLB is sleeping due to memory controls (EXEC threshold '90 percent') being triggered.

You may also want to go to the Replication Server Wiki page and download the Monitors and Counters Analysis package to do some performance and tuning in your RS environment.

Monitors and Counters Analysis - SAP Replication Server - SCN Wiki

Regards

Terry

[kimon_moschandreou]

Hello,

Could you post the whole Rep server log? something is wrong with your stable device

Regards,

Kimon

[jmtorres]

HI Kimon,

I'll get that and post it

Thanks

Jose