OpenRead timing out -- "Timed out trying to read data from the socket stream!"

Nov 15, 2013 at 3:21 PM
Hi,

I've keep getting the TimeoutException "Timed out trying to read data from the socket stream!" when trying to download a file.

I've tried changing the DataConnectionReadTimeout, but it doesn't seem to help.

Has anyone else had this problem and found a solution?

My code:
using (var ftp = new FtpClient())
            {
                ftp.Host = ip;

                if (string.IsNullOrEmpty(username) || string.IsNullOrEmpty(password))
                {
                    ftp.Credentials = new NetworkCredential();
                }
                else
                {
                    ftp.Credentials = new NetworkCredential(username, password);
                }

                ftp.Port = port;
                ftp.DataConnectionType = dataConnectionType.Equals("active") ? FtpDataConnectionType.AutoActive : FtpDataConnectionType.PASV;
                ftp.Connect();
                if (!ftp.IsConnected) return;

                DownloadFiles(ftp, "/" + remoteDirectoryPath, localDirectoryPath);
                ftp.Disconnect();
            }
private static void DownloadFiles(IFtpClient ftp, string remoteDirectory, string localDirectory)
        {
            var items = ftp.GetListing(remoteDirectory);
            foreach (var ftpListItem in items)
            {
                if (ftpListItem.Type == FtpFileSystemObjectType.File)
                {
                    if (!Directory.Exists(localDirectory))
                    {
                        Directory.CreateDirectory(localDirectory);
                    }
                    using (var ftpStream = ftp.OpenRead(ftpListItem.FullName))
                    {
                        Console.WriteLine("Saving file: {0} in folder: {1}", ftpListItem.Name, localDirectory);
                        Utils.SaveStreamToFile(localDirectory + "\\" + ftpListItem.Name, ftpStream);
                    }
                }
                else
                {
                    DownloadFiles(ftp, ftpListItem.FullName, localDirectory + "\\" + ftpListItem.Name);
                }
            }
        }
public static FileInfo SaveStreamToFile(string fileFullPath, Stream stream)
        {
            FileInfo newFile;
            if (stream.Length == 0) return null;

            // Create a FileStream object to write a stream to a file
            using (var fileStream = File.Create(fileFullPath, (int)stream.Length))
            {
                // Fill the bytes[] array with the stream data
                var bytesInStream = new byte[stream.Length];
                stream.Read(bytesInStream, 0, bytesInStream.Length);

                // Use FileStream object to write to the specified file
                fileStream.Write(bytesInStream, 0, bytesInStream.Length);

                newFile = new FileInfo(fileFullPath);
            }

            return newFile;
        }
Coordinator
Nov 15, 2013 at 3:27 PM
Please post the transaction log. Sounds like a firewall issue.
Nov 15, 2013 at 7:47 PM
Here's the transaction log -- seems like it's having problem with the . and .. directory names.
220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 41 of 2048 allowed.
220-Local time is now 21:40. Server port: 21.
220-Only anonymous FTP is allowed here
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
USER X
230-
230-         Welcome to the FTP archive at The University of Oslo
230-         ====================================================
230-
230-  This archive is running on a Dell PowerEdge R710 with 72GB of RAM and
230-  approx. 15 TB of diskspace.
230-
230-  For more information, see URL: http://ftp.uninett.no/
230-
230-  Please email suggestions and questions to ftp-drift@uio.no
230-
230-  ************************************************************************
230-
230 Anonymous user logged in
FEAT
211-Extensions supported:
 EPRT
 IDLE
 MDTM
 SIZE
 REST STREAM
 MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*;
 MLSD
 AUTH TLS
 PBSZ
 PROT
 UTF8
 ESTA
 PASV
 EPSV
 SPSV
 ESTP
211 End.
OPTS UTF8 ON
200 OK, UTF-8 enabled
Text encoding: System.Text.UTF8Encoding
PWD
257 "/" is your current location
TYPE I
200 TYPE is now 8-bit binary
PASV
227 Entering Passive Mode (128,39,3,170,10,161)
MLSD /debian/doc
150 Accepted data connection
type=cdir;sizd=4096;modify=20131115135207;UNIX.mode=0775;unique=fd17g140001; .
type=pdir;sizd=4096;modify=20131115163353;UNIX.mode=0755;unique=fd17g2; ..
type=file;size=995;modify=20090207003223;UNIX.mode=0664;unique=fd17g140004; 00-INDEX
type=dir;sizd=4096;modify=20130604222735;UNIX.mode=0775;unique=fd17g140002; FAQ
type=file;size=2502;modify=20130702015205;UNIX.mode=0644;unique=fd17g14000b; bug-log-access.txt
type=file;size=9174;modify=20130702015205;UNIX.mode=0644;unique=fd17g140011; bug-log-mailserver.txt
type=file;size=2955;modify=20131103015205;UNIX.mode=0644;unique=fd17g140013; bug-mailserver-refcard.txt
type=file;size=21687;modify=20130702015205;UNIX.mode=0644;unique=fd17g140012; bug-maint-info.txt
type=file;size=21178;modify=20131103015205;UNIX.mode=0644;unique=fd17g140006; bug-maint-mailcontrol.txt
type=file;size=16511;modify=20130702015205;UNIX.mode=0644;unique=fd17g140007; bug-reporting.txt
type=file;size=31126;modify=20120415084141;UNIX.mode=0644;unique=fd17g14000a; constitution.txt
type=file;size=7038;modify=19960521060000;UNIX.mode=0664;unique=fd17g14000c; debian-manifesto
type=dir;sizd=4096;modify=20090214102720;UNIX.mode=0755;unique=fd17g140003; dedication
type=file;size=67792;modify=20131114170616;UNIX.mode=0664;unique=fd17g14002f; mailing-lists.txt
type=file;size=6876;modify=20120415084141;UNIX.mode=0664;unique=fd17g14000d; social-contract.txt
type=file;size=1213;modify=19980119012702;UNIX.mode=0664;unique=fd17g14000f; source-unpack.txt
226-Options: -a -l 
226 16 matches total
Disposing FtpSocketStream...
Failed to parse file listing: type=cdir;sizd=4096;modify=20131115135207;UNIX.mode=0775;unique=fd17g140001; .
Failed to parse file listing: type=pdir;sizd=4096;modify=20131115163353;UNIX.mode=0755;unique=fd17g2; ..
Coordinator
Nov 15, 2013 at 7:50 PM
It ignores them on purpose. Are you able to do any file transfers? If file listings are working file transfers should be working. They work very similar under the hood, both open a data channel and read the data off the stream. Seems like the problem may be somewhere else.
Nov 15, 2013 at 7:59 PM
I get two files before it crashes. I'm able to get 00-INDEX and FAQ\debian-faq.en.html.tar.gz.

I'm running version 1.0.5064.19175, installed through nuget.
Coordinator
Nov 15, 2013 at 8:07 PM
For the sake of testing, set EnableThreadSafeDataChannels = false on your FtpClient object before performing transfers. That property controls how many control connections are used for a file transfer. Could be some kind of flooding/connection/throttling control the server is doing.
Coordinator
Nov 15, 2013 at 8:11 PM
For what it's worth, I'm not able to reproduce any problems with mass file transfers on my end. Everything seems to be working fine with the latest builds.
Nov 15, 2013 at 8:18 PM
EnableThreadSafeDataChannels = false got me a bit further. At least the log shows more now:
Disposing FtpSocketStream...
Failed to parse file listing: type=cdir;sizd=4096;modify=20131115135207;UNIX.mode=0775;unique=fd17g140001; .
Failed to parse file listing: type=pdir;sizd=4096;modify=20131115163353;UNIX.mode=0755;unique=fd17g2; ..
TYPE I
200 TYPE is now 8-bit binary
SIZE /debian/doc/00-INDEX
213 995
PASV
227 Entering Passive Mode (128,39,3,170,129,203)
RETR /debian/doc/00-INDEX
150 Accepted data connection
Disposing FtpSocketStream...
226-File successfully transferred
Disposing FtpSocketStream...
226 0.000 seconds (measured here), 20.20 Mbytes per second
PWD
257 "/" is your current location
TYPE I
200 TYPE is now 8-bit binary
PASV
227 Entering Passive Mode (128,39,3,170,150,72)
MLSD /debian/doc/FAQ
150 Accepted data connection
type=cdir;sizd=4096;modify=20130604222735;UNIX.mode=0775;unique=fd17g140002; .
type=pdir;sizd=4096;modify=20131115135207;UNIX.mode=0775;unique=fd17g140001; ..
type=file;size=119071;modify=20130604222729;UNIX.mode=0644;unique=fd17g140008; debian-faq.en.html.tar.gz
type=file;size=310481;modify=20130604222735;UNIX.mode=0644;unique=fd17g14000e; debian-faq.en.pdf.gz
type=file;size=253415;modify=20130604222733;UNIX.mode=0644;unique=fd17g140009; debian-f
You could try the FTP I'm using: ftp.uio.no:21 with anonymous login to see if you encounter the same problem.
Coordinator
Nov 15, 2013 at 8:39 PM
I just ran some code agains it and and I'm not able to reproduce the problem. Here's the log from where I left it running for a little while. And here's the code that I ran against the server:
       static void TestServer() {
            FtpTrace.AddListener(new TextWriterTraceListener("test.log"));

            using (FtpClient cl = new FtpClient()) {
                cl.Host = "ftp.uio.no";
                cl.Credentials = new NetworkCredential("anonymous", "anonymous");
                TestServerDownload(cl, "/debian");
            }
        }

        static void TestServerDownload(FtpClient client, string path) {
            foreach (FtpListItem i in client.GetListing(path)) {
                switch (i.Type) {
                    case FtpFileSystemObjectType.Directory:
                        TestServerDownload(client, i.FullName);
                        break;
                    case FtpFileSystemObjectType.File:
                        using (Stream s = client.OpenRead(i.FullName)) {
                            byte[] b = new byte[8192];
                            int read = 0;
                            long total = 0;

                            try {
                                while ((read = s.Read(b, 0, b.Length)) > 0) {
                                    total += read;

                                    Console.Write("\r{0}/{1} {2:p}          ",
                                        total, s.Length, (double)total / (double)s.Length);
                                }

                                Console.Write("\r{0}/{1} {2:p}       ",
                                        total, s.Length, (double)total / (double)s.Length);
                            }
                            finally {
                                Console.WriteLine();
                            }
                        }
                        break;
                }
            }
        }
Coordinator
Nov 15, 2013 at 8:46 PM
I just looked back at your code and I see you're allocating a buffer the size of the file and reading it in. Use a loop similar to the code I posted above and see if it clears up your problem. There are a number of reasons not to do it without a loop, the most important being that the remote file size may not be available and another one up there is that the file might be large.
Marked as answer by sigurdhj on 11/15/2013 at 2:17 PM
Coordinator
Nov 15, 2013 at 8:48 PM
You also need to check if the FtpListItem is a symlink, which you're not doing in your code. You're assuming that if it's not a file it must be a directory and that's not always the case.
Nov 15, 2013 at 9:09 PM
You're right, the missing loop while reading was the problem.

Thanks a lot for your help. :)
Coordinator
Nov 15, 2013 at 10:15 PM
No problem, glad that fixed it.