Client Port Exhaustion Despite Using Different Destination Ports

Posted on

A server stack is the collection of software that forms the operational infrastructure on a given machine. In a computing context, a stack is an ordered pile. A server stack is one type of solution stack — an ordered selection of software that makes it possible to complete a particular task. Like in this post about Client Port Exhaustion Despite Using Different Destination Ports was one problem in server stack that need for a solution. Below are some tips in manage your linux server when you find problem about linux, networking, port, tcp, .

When reading about port exhaustion, I often see that it takes in account the quadruplet of the tcp connection (src ip, src port, dst ip, dst port). So it would seem that with a port range of 20000 (40000 to 60000) and targeting destination port 8088 and 8087 with two clients on the same machine, using the same src address and dst address, I should be able to open 40000 connections. However on my test this is not the case. Even in this configuration I am able to only open 20000 connections. I’ve dumped the output of ss -s and sort it by source port. Here is an extract :

CLOSE-WAIT  26378    0           172.24.131.110 40001     172.24.131.97 9088    
CLOSE-WAIT  25029    0           172.24.131.110 40002     172.24.131.97 9087    
CLOSE-WAIT  23840    0           172.24.131.110 40003     172.24.131.97 9087    
CLOSE-WAIT  25207    0           172.24.131.110 40004     172.24.131.97 9087    
CLOSE-WAIT  25572    0           172.24.131.110 40005     172.24.131.97 9088    
CLOSE-WAIT  26334    0           172.24.131.110 40006     172.24.131.97 9087    
CLOSE-WAIT  27089    0           172.24.131.110 40007     172.24.131.97 9087    
CLOSE-WAIT  23860    0           172.24.131.110 40008     172.24.131.97 9088    
CLOSE-WAIT  25463    0           172.24.131.110 40009     172.24.131.97 9087    
CLOSE-WAIT  26603    0           172.24.131.110 40010     172.24.131.97 9088    
CLOSE-WAIT  25436    0           172.24.131.110 40011     172.24.131.97 9087    
ESTAB       0        0           172.24.131.110 40012     172.24.131.97 9087    
CLOSE-WAIT  25042    0           172.24.131.110 40013     172.24.131.97 9087    
CLOSE-WAIT  25738    0           172.24.131.110 40014     172.24.131.97 9087    
CLOSE-WAIT  27363    0           172.24.131.110 40015     172.24.131.97 9088    
CLOSE-WAIT  25860    0           172.24.131.110 40016     172.24.131.97 9088    

It goes on until the value 60000 is reached. There is never a source port used twice, as if the pool of values is in fact common. Is this is the case or I am missing on a configuration flag somewhere ?

The kernel version is 4.15.0-144-generic.
The port range is really in control here : if I change the port range to 40000-50000 I only have 10000 possible connections.
The client and server for the tests are web polygraph tools.

Ok in fact I think I have the answer, the client I am using creates its connections with bind + connect, instead of just connect. Here is an strace dump :

bind(1490, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("172.24.131.110")}, 16) = 0
getsockname(1490, {sa_family=AF_INET, sin_port=htons(50929), sin_addr=inet_addr("172.24.131.110")}, [16]) = 0
connect(1490, {sa_family=AF_INET, sin_port=htons(9088), sin_addr=inet_addr("172.24.131.97")}, 16) = -1 EINPROGRESS (Operation now in progress)

From my understanding in this case you are in did going back to a unique pool for the different clients, as at the moment of the bind you don’t have any information on the remote side.

A useful link on this subject : https://idea.popcount.org/2014-04-03-bind-before-connect/

Leave a Reply

Your email address will not be published. Required fields are marked *