"WSAEADDRINUSE" from custom app - how to debug? (blog post)

[JimSpath]

We are getting intermittent errors in a custom .NET app, with messages to the effect:

LOCATION CPIC (TCP/IP) on local host

ERROR partner not reached (host 1.2.3.174, service 3301)

TIME Tue Dec 08 06:48:56 2009

RELEASE 610

COMPONENT NI (network interface)

VERSION 35

RC -10

MODULE ninti.c

LINE 976

DETAIL NiPConnect2

SYSTEM CALL SO_ERROR

ERRNO 10048

ERRNO TEXT WSAEADDRINUSE: Address already in use

I wrote a semi-fictional blog post explaining the background and debugging so far:

/people/jim.spath/blog/2009/12/08/missing-process--another-abap-detective-pulp-true-story

Any other tips on how to debug would be appreciated. Reply here or on the blog (but not both :-).

Thanks.

Jim

[Former Member · Accepted solution]

A search for WSAEADDRINUSE gives several hits - including this forum article and 2 OSS-notes (734095 and 804139).

734095 is a prime suspect if you are seeing this symptom on a Windows platform.

[JimSpath · Accepted solution]

Peter - Thanks; I should have mentioned we looked at note 734095, as well as Microsoft technical articles on socket connection maximum ports allowed, as well as retry waits. The developer believes the code is not attempting multiple connects, despite what the error message tells us. The symptoms are sporadic (of course). Note 804139 does not seem to apply since no Java involved.

I will have them document these registry settings:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\TcpTimedWaitDelay

I suspect the code is doing something other than what the developer thinks. Proving this is challenging.

We're only seeing this on one client, so if the server were blocking ports more RFC connections would be affected. That leads us back to the Windows machine, and our custom code.

Jim

Edited by: Jim Spath on Dec 9, 2009 9:09 AM : "TcpTimedWaitDelay" omitted

[Former Member · Accepted solution]

Hi Jim,

I am having the same exact problem and was planning to open a new SDN question when I found your thread. I will just post my issue here and follow this thread if you don't mind. Otherwise, let me know. Thanks!!

I am seeing the below error which is almost if not exactly your error. This is on a Windows 2003 VM server so the parameters mentioned in note 734095 do not apply or I am not seeing the two parameters.

Connect to SAP gateway failed

Connect_PM GWHOST=167.161.16.91, GWSERV=3309, ASHOST=167.161.16.91, SYSNR=09

LOCATION CPIC (TCP/IP) on local host

ERROR partner not reached (host ?, service ?)

TIME Wed Dec 09 15:04:04 2009

RELEASE 640

COMPONENT NI (network interface)

VERSION 37

RC -10

COUNTER 468

The error is coming from an in house application that has worked for years and we just started seeing these errors. It is using the SAP Connector to make the connections.

I hope someone is able to help shed some light on this issue.

Best Regards,

Rick

[Former Member · Accepted solution]

Sorry, I posted the wrong error message:

LOCATION CPIC (TCP/IP) on local host

ERROR partner not reached (host 167.161.16.93, service 3309)

TIME Wed Dec 09 13:48:46 2009

RELEASE 620

COMPONENT NI (network interface)

VERSION 36

RC -10

MODULE ninti.c

LINE 976

DETAIL NiPConnect2

SYSTEM CALL SO_ERROR

ERRNO 10048

ERRNO TEXT WSAEADDRINUSE: Address already in use

COUNTER 4145

Thanks.

Rick

[JimSpath · Accepted solution]

Rick:

> worked for years and we just started seeing these errors

rings a bell, as this is one of the first things the development team said when they reported our problems. I believe it has been getting steadily worse since around April 2009. One of their first questions was to ask for a list of changes that have gone into the systems since then. That's quite a tall order, as you can imagine.

Let me gather more data and post an update in a day or so, possibly on a blog rather than here as I can use more formatting tools there.

Jim

[Former Member · Accepted solution]

We may have seen some intermittent issues before, but it really became more prevalent after applying the HR EOY patches in ERP at the end of November. This particular application will login and touch some HR data. However, I don't see why this would change the login behaviour.

I hope someone has some ideas. I don't see a lot of activity on this forum though.

Thanks,

Rick

[Former Member · Accepted solution]

Hi Jim,

We actually had two things change right before we noticed this intermittent issue. We applied some SAP ERP patches and also switched antivirus software.

Since we are on Windows 2003 servers, I didn't believe SAP note 734095 would apply to us. However, I found the below link and manually created the registry entry MaxUserPort and now our problem seems to be resolved.

http://technet.microsoft.com/en-us/library/cc758002(WS.10).aspx

Give this a try and let me know...

Thanks,

Rick

[JimSpath · Accepted solution]

Rick:

I asked the development team lead and he said:

> We upped the MaxUserPort to 65000 and rebooted, and there was no impact

But it certainly worth my time to verify this value is still set to what they think. Thanks for the update!

Jim

[dana_schreiber · Accepted solution]

Have you tried upping your gateway parameter for maximum entries

gw_max_conn

You might be able to correlate if the error you are getting is the result of this by going to smgw in the target system and reviewing the gateway trace log. Perhaps you will see some type of "max connections reached" in this log. Default value is normally 500. Systems that see high RFC communication traffic have to be increased. Ours is set to 2000.

[JimSpath · Accepted solution]

Yes, maximum connections is set to 2,000. We're going to alter client settings next to see what shakes loose.

See:

/people/jim.spath/blog/2009/12/22/abap-detective-missing-process-chapter-3-shadow-of-the-thin-client

/people/jim.spath/blog/2009/12/14/missing-process--abap-detective-story--chapter-2

/people/jim.spath/blog/2009/12/08/missing-process--another-abap-detective-pulp-true-story

Jim

[JimSpath · Accepted solution]

I am closing this thread. I wish I had a definitive answer to post, but all I know is that we moved the application to new hardware (presumably faster) and we get fewer errors. Thus the issue is unresolved, but apparently acceptable to users. I'll post again if I learn more.

Jim