WSAConnect failing with WSAENOBUFS

[EDN Admin]

Hello, we have written a telco grade AAA server, which is running in a production environment, on Windows Server 2003.
We increased the number of outbound connections by a small amount yesterday, and got a major outage, because WSAConnect started failing with WSAENOBUFS.
There are approximately 1000 connections on each server, of which about 500 are inbound, and the rest outbound. Of the outbound connections, 64 are permanent, and the rest are inbound connections which are created, used, and dropped; and outbound connections
to a database, which are connected, used, and after an inactive period, dropped. The count of 1000 connections includes connections in CLOSE_WAIT etc. after the inbound connection has been correctly closed.<br/>

On our test machine, we are able to open a few thousand connections without encountering this issue.
I have used Internet search, which comes up with two possible causes - the /3GB switch in boot.ini, which is not present, and the source port hitting the 5000 limit. As we never have more than a thousand connections on this particular server, I cannot easily
explain it as a 5000 limit thing, and we have not managed to reproduce the issue on our test servers, with the same software, and more connections, which means (to me) that there is something configured differently.

What do we need to look at? Other than the MaxUserPort in the TCP/IP registry area, and the /3GB in boot.ini?
Thank you very much!
This is the detail from one of the Production machines - and two have so far exhibited the behaviour:
<p style=" <span lang="EN-US" style="color:#1F497D [boot loader]
<p style=" <span lang="EN-US" style="color:#1F497D timeout=5
<p style=" <span lang="EN-US" style="color:#1F497D default=multi(0)disk(0)rdisk(0)partition(1)WINDOWS
<p style=" <span lang="EN-US" style="color:#1F497D
<p style=" <span lang="EN-US" style="color:#1F497D [operating systems]
<p style=" <span lang="EN-US" style="color:#1F497D multi(0)disk(0)rdisk(0)partition(1)WINDOWS="Windows Server 2003 Standard x64 Edition" /noexecute=optin /fastdetect
<p style=" <span style="text-decoration:underline <span lang="EN-US" style="color:#1F497D <span style="text-decoration:none
<p style=" <span lang="EN-US" style="color:#1F497D ------------------
<p style=" <span lang="EN-US" style="color:#1F497D System Information
<p style=" <span lang="EN-US" style="color:#1F497D ------------------
<p style=" <span lang="EN-US" style="color:#1F497D Time of this report: 9/11/2012, 12:14:11
<p style=" <span lang="EN-US" style="color:#1F497D Machine name: XXXX01HW
<p style=" <span lang="EN-US" style="color:#1F497D Operating System: Windows Server 2003 Standard x64 Edition (5.2, Build 3790) Service Pack 2 (3790.srv03_sp2_gdr.101019-0340)
<p style=" <span lang="EN-US" style="color:#1F497D Language: English (Regional Setting: English)
<p style=" <span lang="EN-US" style="color:#1F497D System Manufacturer: HP
<p style=" <span lang="EN-US" style="color:#1F497D System Model: ProLiant DL380 G6
<p style=" <span lang="EN-US" style="color:#1F497D BIOS: Default System BIOS
<p style=" <span lang="EN-US" style="color:#1F497D Processor: Intel(R) Xeon(R) CPU X5560 @ 2.80GHz (16 CPUs)
<p style=" <span lang="EN-US" style="color:#1F497D Memory: 32768MB RAM
<p style=" <span lang="EN-US" style="color:#1F497D Page File: 2699MB used, 33211MB available
<p style=" <span lang="EN-US" style="color:#1F497D Windows Dir: C:WINDOWS
<p style=" <span lang="EN-US" style="color:#1F497D DirectX Version: DirectX 9.0c (4.09.0000.0904)

View the full article