Bulk Downloads

When doing a large number of transfers, one needs to be aware of some inherit issues with data streams. When a socket is opened and then closed, the socket is left in a linger state for a period of time defined by the operating system. The socket cannot reliably be re-used until the operating system takes it out of the TIME_WAIT state. This matters because a data stream is opened when it's needed and closed as soon as that specific task is done:
  • Download File
    • Open Data Stream
      • Read bytes
    • Close Data Stream

This is not a bug in System.Net.FtpClient. RFC959 says that EOF on stream mode transfers is signaled by closing the connection. On downloads and file listings, the sockets being used on the server will stay in the TIME WAIT state because the server closes the socket when it's done sending the data. On uploads, the client sockets will go into the TIME WAIT state because the client closes the connection to signal EOF to the server.

RFC959 defines another data mode called block that allows persistent data connections but it is not implemented by this library and will not be in the foreseeable future. Support for block transfers on the server side of things is not that common. I know IIS supports this feature however I cannot name a single other server that implements MODE B.

On windows, there are registry settings that can be changed to reduce the life of the linger state and extend the number of sockets available. This will alleviate the problem for some however it can create problems not only with this library but also other network enabled applications so use it at your own risk:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"MaxUsePort"=dword:0000c350
"TcpTimedWaitDelay"=dword:0000001e

Last edited Oct 4, 2012 at 2:55 PM by jptrosclair, version 6

Comments

jptrosclair Aug 16, 2011 at 2:02 PM 
If they follow the RFC for stream mode transfers and aren't doing some kind of black magic to re-use sockets then I'd assume that they do.

braian87b Jun 8, 2011 at 1:01 AM 
All other free and commercial ftp client softwares are affected by this issue too?