Backlog in R/3 Outbound Queues to APO

[Former Member]

We are suddenly encountering a situation where LUW entries are accumulating in a much faster in the R/3 outbound queues than what CIF seemingly can handle.  We believe we have enough APO processing power since there is no backlog in the APO inbound queue.  Business-wise, nothing has changed and we don't think we have added any new process that would have suddenly changed the throughput of the system.  I wonder if anyone has experienced something similar.  I am looking for things that could have caused the backlog and I would appreciate any insights.

[Former Member]

Hi Sam,

I assume you are using inbound queues in both ECC and APO.  You have the ability to use parallelization for CIF in both ECC and APO.  You can set these parameters in CFM3 (RIMODAC2) for each of your integration models.

Best Regards,

Mike

[Former Member]

Thanks Mike.  I just verified that we don't have them set.  It would be worthwhile to consider.

When the R/3 outbound queue backlog occurs, the most immediate impact is dropping deliveries and posting goods issue.  When it gets very severe, we would deactivate the ATP check integration models (separate models) while leaving all other transaction models active.  In other words, we have switched to perform only a local R/3 availability check.  It is interesting to see the queues would begin to flush.  Once the queues are back to normal, we would then clean up the TQA's in APO, re-activate the ATP check integration models, and run the Correction Report for sales order and delivery requirements.  We are now back to perform the APO GATP availability check.

When switching between performing a local R/3 availability check and performing the APO GATP availability check, since we continue to transfer the transactions even with the local R/3 availability check, the difference is really the creation of TQA's in APO (with APO controlliing the transfer of requirements).  I may have simplified things too much, but functionally it really boils down haviing to process the additional instrution to process the TQA entries in the queues.  Mike, do you have any thoughts?

Thanks,

Sam

[Former Member]

Hi Sam,

I have to defer to the ATP experts on your issue.  I suggest you try the parallelization for ECC in CIF.

Best Regards,

Mike

[Former Member]

Sam,

  I just verified that we don't have them set.  It would be worthwhile to consider.

Are you saying you don't have inbound queues, or you don't have parallelization?

If you have activated inbound queues in APO, and you have no APO IB queue backlog, then the problem is not related to ATP or anything else in the APO application, it is strictly the CIF.

In any case, QRFC resources sound like a possible culprit in your system to me.  There may be inadequate sessions available for the scheduler to do its work.  SMQS in ERP.  Select Goto > QRFC Resources.  You should have green here.  If not, you and your basis team have to make some decisions.  There is a note that describes how resources should be allocated and managed.   https://service.sap.com/sap/support/notes/527481

Best Regards,

DB49

[Former Member]

We have both inbound and outbound queues.  We can't communicate without them.  We did not have parallelization selected in our integation models.

I am not suggesting that ATP is the issue but it is somehow contributing to the issue in some shape or form.  ATP is a symptom.  By deactivating the ATP check integration models, CIF ultimately returns to normal.  Obviously we have other things that contribute to the CIF traffic.  We have a lot more analysis to do.  But I thought it was interesting that after we turn off the global ATP check in APO, CIF traffic would return to normal.  There is probably more to it but I just can't ignore the major difference that stands between ATP being on and off, i.e., TQA and the transfer of requirements.

Our Basis team has been monitoring the system resources.  And they have not seen anything suspicious.  We will continue to do so.  I will confirm with the Basis team.

Thanks for the inputs and reference link.

[Former Member]

Sam,

I hope your Basis colleagues are knowledgeable in this matter.  Few of them fully understand the critical items in an APO-interfaced system that must be monitored.  Fewer still know how to 'troubleshoot' core interface issues.

I will assume that you have performed the transaction I suggested, and the QRFC resources did not show red. 

Since you are using inbound queues in R/3 > APO, and your R/3 outbound queues are still "accumulating in a much faster in the R/3 outbound queues than what CIF seemingly can handle", the thing to focus on would be problems on the R/3 side.  The main purpose of activating inbound queues in APO is to offload the processing from R/3.  Outbound queues just need to receive an acknowledgement from the APO IB queue to pass an LUW, there is no requirement for the R/3 OB queue to 'talk' to the APO application databases at all.  Make sure your inbound queue scheduler is registered and activated.  SMQR

One thing I have noticed that does occasionally cause inexplicable problems from time to time is if some type of logging is 'on', and the logs, for whatever reason, cannot be processed in a timely manner, or they are filled.  I would suggest that you review whether CIF logging is turned on for the CIF (it usually doesn't have to be); whether logs are being regularly reorganized, and whether 'someone' has recently turned on logging of additional data items.  In R/3, Logging can be activated/deactivated in CFC2.  The types of items that are logged are set in R/3 in CFC6.  Logs can be reorganized in CFGD.

Another opportunity is to keep your IMs fully consistent and 'lean'.

Periodically delete unnecessary Update counters RCFUPDCR

Periodically repair errors in CIFORDMAP table RCFORDCH

Periodically rebuild (generate) runtime versions of the active models.  RCIFIMAX  

Periodically delete unnecessary filter objects from the CIF_IM* tables RCIFIMDL

If all else fails, raise a message with SAP and have them tell you what is wrong.  You don't have to put up with this system behavior.  Your SAP support fees are paid just for such an occasion.  Deactivating ATP queues is merely a stopgap until you (or SAP) find the root cause.

Best regards,

DB49

[Former Member]

DB49,

Thanks for all the very useful information.  Everything appeared to be in working order.  We only deactivated the ATP check integration models to relief the traffic.  As soon as the outbound queues were unclogged, we re-activated the IM's.

We have OSS message opened with SAP.  They wanted us to apply a note, which we did although we did not expect it to do anything based on the description of the note.  If and when the queues begin to get clogged up, SAP should be more active in providing their support.

Thanks again,

Sam

[Former Member]

Sam,

My experience with SAP is that if you make the OSS message a high priority, they will respond very quickly.

Best Regards,

DB49

[Former Member]

Hi

Good point... This is SAP Best practice to deactivare IM for ATP check.. this will reduce the tracffic either side.. So once IM activated and next step is to de-activate the same and keep in same status.

Regards

PK

[Former Member]

De-activating the ATP check models was the last resort.  We have developed a procedure for dealing with situation when APO is down or liveCache should become unavailable.  We modified the procedure in order to handle CIF queue backlog.  If backlog persists, it means that there are underlying issues that must be addressed.  While de-activating the ATP check models works to relieve the backlog, you do lose the advanced ATP methodology such as rules-based ATP check.

We currently still have a very high OSS message opened.  After numerous exchanges with different processors, we still have not been able to identify the culprit.  SAP Development even came into our system and they made an observation that we need to reduce and control the number of queue names.  The ideal scenario for the queue scheduler would be "less" queue names with each queue name containing "more" queue entries.  Having made that recommendation, he indicated that he wasn't the right person to help in that regard.  Subsequenet processors came in and made no attempt to follow up with his recommendation.

I wonder if anyone knows how to influence and change the # of queue names generated by the system.

Regards,

Sam