This project is read-only.

FtpCommandException "Failure writing network stream" intermittent problem on downloads

Jun 13, 2014 at 1:38 PM
I'm using System.Net.FtpClient version 1.0.5200.21177

I'm getting the following Exception quite intermittently (e.g. 1 in every 100 x 15kilobyte file downloads). It's a public FTP server that's quite sluggish.
System.Net.FtpClient.FtpCommandException: Failure writing network stream.
   at System.Net.FtpClient.FtpClient.CloseDataStream(FtpDataStream stream)
   at System.Net.FtpClient.FtpDataStream.Close()
   at System.Net.FtpClient.FtpSocketStream.Dispose()
   at MyProject.DownloadBytes(String fileName, Int32 startByteOffset, Int32 byteCount)
I have set the following timeouts to 30 seconds
  • ConnectTimeout
  • DataConnectionConnectTimeout
  • DataConnectionReadTimeout
  • ReadTimeout
My download method looks like this...
public byte[] DownloadBytes(string fileName, int startByteOffset, int byteCount)
{
    byte[] byteBuffer = new byte[byteCount];
    int bytesRemainingToDownload = byteCount;
    int byteBufferOffset = 0;
    using (Stream downloadStream = ftpClient.OpenRead(fileName, FtpDataType.Binary, startByteOffset))
    {
        while (bytesRemainingToDownload > 0)
        {
            int bytesDownloaded = downloadStream.Read(byteBuffer, byteBufferOffset, bytesRemainingToDownload);
            if (bytesDownloaded < 1)
            {
                break;
            }

            byteBufferOffset += bytesDownloaded;
            bytesRemainingToDownload -= bytesDownloaded;
        }
    }

    return byteBuffer;
}
If I catch and swallow FtpCommandException "Failure writing network stream" then the contents of the byte array I get back from DownloadBytes seem to be fine.

Does anyone know what might be causing this or whether there's anything else I should be setting differently on FtpClient to avoid it?

Jimmerthy
Jun 13, 2014 at 1:57 PM
Edited Jun 13, 2014 at 1:57 PM
The fact that it's a FtpCommandException means the server is returning a temporary or permanent failure and it's probably happening when the data stream is closed after the transfer is complete. Looks like a server or network issue from what you've provided here. Additional information, including the failure, are logged using TraceListeners. Check the Debug.cs example code.
Jun 13, 2014 at 6:01 PM
Thank you for replying. I've now got the debug logging up and running which is great to see what's happening under the surface.
Disposing FtpSocketStream...
426 Failure writing network stream.
I looked up 426 as 'Connection closed; transfer aborted.' and the 4xx errors are classified as 'Transient Negative Completion reply'
I now need to do some more reading of the FtpClient docs etc to understand what action I need to take as a user of the FtpClient API to work around this. I'm seeing a bunch of other failure types too now that I've not really begun to analyse. My instinct is to write retry loops everywhere and check for disconnections and things like that but I don't really know how much of this is my responsibility and how much FtpClient will do for me, if used correctly.

With the debug log turned on I also wanted to see what effect twiddling EnableThreadSafeDataConnections and SocketKeepAlive would have. I noticed a lot of FTP site connection banner messages in the log when I had EnableThreadSafeDataConnections turned on. Turning it off greatly reduced the number these connection banners. I'm assuming the latter is more desirable as the server isn't kicking me off as much? I've seen that you've suggested turning EnableThreadSafeDataConnections off in other postings (I'm a single threaded client).
Jun 16, 2014 at 4:03 PM
The difference in Thread Safe vs. Non-Thread Safe data channels is that when you open a file transfer a new control connection to the server is established. FTP works on 2 connections, a control connection for issuing commands and receiving responses and a data connection in which the contents of file listings and files are transferred over. Multiple file transfers from a single control connection are not possible due to synchronization problems that arise (FTP wasn't designed to). To work around it System.Net.FtpClient objects can "clone" themselves and create a new control connection for the file transfer. The EnableThreadSafeDataConnections property controls this behavior. If it's false the connection is not cloned and if it's true the connection is cloned. This is the reason you are seeing multiple connect banners in the transaction logging.
Jun 16, 2014 at 4:14 PM
With regards to the 426 error above, what you should probably be doing is working off the return status of the Read() call on the stream instead of the base condition on the loop being controlled by the file size (I assume you are getting the file size and passing it in). The error message the server sent indicates, to me, that the data connection is being closed unexpectedly. I suspect that you are terminating your loop before the server sends EOF (closes its data socket). Your code should probably look more like this:
while(downloadStream.Read(byteBuffer, byteBufferOffset, bytesRemainingToDownload) > 0) {

}
Jun 17, 2014 at 12:27 AM
Thanks once again for the replies.
The download method above is used for both entire file downloads and also partial file downloads (seek to specific offset x, download n bytes from there). To confirm what you suspected; in each case I know up-front what I expect the receive byte count to be and this is passed in to the method. The FTP server contains very large binary files that are accompanied by text metadata files, which when downloaded and parsed, give byte offsets into the large binary files, allowing me to selectively download binary records of interest.
I'll update my download loop condition as suggested.