This project is read-only.

Download Method stucks in endless loop

Jun 28, 2013 at 11:43 AM
Hey,


I use FTPS with a localhost FileZilla Server. Connection and Authentication works basically.
Downloading over large lists produces endless loops when accepting the SSL certificate.

The situation:
Folder with ~ 300 Subfolders and ~1000+ Files

Here is the Download Method:
Public Sub Download(ByVal filename As String, ByVal localpath As String, ByRef client As FtpClient, ByVal host As String, ByVal credentials As System.Net.NetworkCredential)
        client.Host = host
        client.Credentials = credentials
        AddHandler client.ValidateCertificate, AddressOf SSLcertificateHandler       ' Zertifikate ignorieren
        Using client
            Using inputstream As Stream = client.OpenRead(filename)
                Try
                    Dim fileoutput As New FileStream(localpath, FileMode.Create, System.IO.FileAccess.ReadWrite)

                Try
                    Dim buffer(8192) As Byte
                    Dim len As Integer
                    While (InlineAssignHelper(len, inputstream.Read(buffer, 0, buffer.Length))) > 0
                        fileoutput.Write(buffer, 0, len)
                    End While
                Catch ex As Exception
                Finally
                    inputstream.Close() : fileoutput.Close()
                    End Try

                Catch unAcEx As UnauthorizedAccessException
                    File.SetAttributes(localpath, FileAttributes.Normal)

                    Dim fileoutput As New FileStream(localpath, FileMode.Create, System.IO.FileAccess.ReadWrite)

                    Try
                        Dim buffer(8192) As Byte
                        Dim len As Integer
                        While (InlineAssignHelper(len, inputstream.Read(buffer, 0, buffer.Length))) > 0
                            fileoutput.Write(buffer, 0, len)
                        End While
                    Catch ex As Exception
                    Finally
                        inputstream.Close() : fileoutput.Close()
                    End Try
                End Try
            End Using
        End Using
    End Sub
This Method catches the self signed SSL certificate:
    Public Sub SSLcertificateHandler(ByVal sender As FtpClient, ByVal e As FtpSslValidationEventArgs)
        e.Accept = True
    End Sub
In this last method it keeps rerunning.

This is a example output when the downloading gets buggy:
(002725)28.06.2013 12:42:01 - (not logged in) (::1)> Connected, sending welcome message...
(002725)28.06.2013 12:42:01 - (not logged in) (::1)> 220-FileZilla Server version 0.9.41 beta
(002725)28.06.2013 12:42:01 - (not logged in) (::1)> 220-written by Tim Kosse (Tim.Kosse@gmx.de) (002725)28.06.2013 12:42:01 - (not logged in) (::1)> 220 Please visit http://sourceforge.net/projects/filezilla/
(002725)28.06.2013 12:42:01 - (not logged in) (::1)> AUTH TLS
(002725)28.06.2013 12:42:02 - (not logged in) (::1)> 234 Using authentication type TLS
(002725)28.06.2013 12:42:02 - (not logged in) (::1)> SSL connection established
(002725)28.06.2013 12:42:02 - (not logged in) (::1)> PBSZ 0
(002725)28.06.2013 12:42:02 - (not logged in) (::1)> 200 PBSZ=0
(002725)28.06.2013 12:42:02 - (not logged in) (::1)> PROT P
(002725)28.06.2013 12:42:02 - (not logged in) (::1)> 200 Protection level set to P
(002725)28.06.2013 12:42:02 - (not logged in) (::1)> USER user
(002725)28.06.2013 12:42:02 - (not logged in) (::1)> 331 Password required for user
(002725)28.06.2013 12:42:02 - (not logged in) (::1)> PASS ****
(002725)28.06.2013 12:42:02 - user (::1)> 230 Logged on
(002725)28.06.2013 12:42:02 - user (::1)> FEAT
(002725)28.06.2013 12:42:02 - user (::1)> 211-Features:
(002725)28.06.2013 12:42:02 - user (::1)> MDTM
(002725)28.06.2013 12:42:02 - user (::1)> REST STREAM
(002725)28.06.2013 12:42:02 - user (::1)> SIZE
(002725)28.06.2013 12:42:02 - user (::1)> MLST type*;size*;modify*;
(002725)28.06.2013 12:42:02 - user (::1)> MLSD
(002725)28.06.2013 12:42:02 - user (::1)> AUTH SSL
(002725)28.06.2013 12:42:02 - user (::1)> AUTH TLS
(002725)28.06.2013 12:42:02 - user (::1)> PROT
(002725)28.06.2013 12:42:02 - user (::1)> PBSZ
(002725)28.06.2013 12:42:02 - user (::1)> UTF8
(002725)28.06.2013 12:42:02 - user (::1)> CLNT
(002725)28.06.2013 12:42:02 - user (::1)> MFMT
(002725)28.06.2013 12:42:02 - user (::1)> 211 End
(002725)28.06.2013 12:42:02 - user (::1)> OPTS UTF8 ON
(002725)28.06.2013 12:42:02 - user (::1)> 200 UTF8 mode enabled
(002725)28.06.2013 12:42:03 - user (::1)> TYPE I
(002725)28.06.2013 12:42:03 - user (::1)> 200 Type set to I
(002725)28.06.2013 12:42:03 - user (::1)> SIZE /Cloudorizer/Visual Studio 2012/Projects/Praktikum2013Feb/Praktikum2013Feb/obj/Debug/Praktikum2013Feb.xml
(002725)28.06.2013 12:42:03 - user (::1)> 213 696
(002725)28.06.2013 12:42:03 - user (::1)> EPSV
(002725)28.06.2013 12:42:03 - user (::1)> 229 Entering Extended Passive Mode (|||33318|)
(002725)28.06.2013 12:42:03 - user (::1)> RETR /Cloudorizer/Visual Studio 2012/Projects/Praktikum2013Feb/Praktikum2013Feb/obj/Debug/Praktikum2013Feb.xml
(002725)28.06.2013 12:42:03 - user (::1)> 150 Connection accepted
(002725)28.06.2013 12:42:03 - user (::1)> SSL connection for data connection established
(002725)28.06.2013 12:42:03 - user (::1)> 226 Transfer OK
(002725)28.06.2013 12:42:03 - user (::1)> QUIT
(002725)28.06.2013 12:42:03 - user (::1)> 221 Goodbye
(002725)28.06.2013 12:42:03 - user (::1)> disconnected.
(002725)28.06.2013 12:42:03 - user (::1)> disconnected.
Jun 28, 2013 at 12:10 PM
Can you provide the transaction log from System.Net.FtpClient when this occurring? The only way that event could keep firing is if the connection is being reestablished in a loop. The excerpt from FileZilla server doesn't indicate that is happening. I
need more information to go by, perhaps the call stack when it gets stuck as well so we can see what method it is hung in.
Jun 28, 2013 at 12:32 PM
The excerpt of the FileZilla log was one element in the loop. In this exact pattern one file is transmitted and the connection is then canceled and re-established for the next file.
This behaviour is totally right, as I know now, because I am using the "using client" in the download method. I deleted the "Using client" in the code and now the never ending establishing/disconnecting does not happened anymore, as desired.

Are there any improvements I could implement in my download method? I get about ~4MB as transfer rate from my local systemdrive to the local FileZilla drive on the same hard drive, but on another partition. This seems rather slow, since the Data does not even "leave" the hard drive. Are there any other problems concerning solidity of the download? I discovered that hidden files were a problem, thats why i catched this with changing the file-attribute from hidden to normal.

Greetings
Jun 28, 2013 at 1:56 PM
Looks pretty generic to me, I don't see any code that would be limiting transfer speeds. System.Net.FtpClient returns a stream that internally uses a network stream for reading and writing files so there isn't much you can change by way of performance. The only exception is that when you enable SSL internally the stream is wrapped with a SslStream object. So without encryption you are accessing a NetworkStream wrapped around a socket and with SSL you are accessing a SslStream wrapped around a NetworkStream wrapped around a socket. The only reason a custom stream implementation exists at all is so that cleanup can be done on the control connection when the stream is closed. The point is, there is nothing real fancy that would be causing massive delays in transfers going on with the Stream objects OpenRead() and OpenWrite() return. Calling Read() and Write() on the streams is just invoking Read() and Write() on the internal stream (NetworkStream or SslStream). The real difference occurs when you call Close() in which the cleanup procedures on the control connection take place, something that in no way at all would affect the speed at which data is read or written on wire.

You might give it a run without SSL just to observe the difference in speed. There is overhead with encryption on both the client and server so it's not going to be as fast as not using encryption. That might not be as noticeable on a slower link but the difference will probably show on a local transfer. In regards to the data not leaving the hard drive, it is in fact read in from disk processed through a socket on both ends and then written back out to disk. There are more steps going on here than, for example, just copying a file from one folder to another as you would with a file manager. Have you compared the speed using other clients, say windows explorer or the FileZilla client to see if the results are similar? How are you calculating the transfer speed? You should be doing something in the ballpark of (totalTransferred / totalSecondsLapsed) and you should take your samples from a large file, not small ones.
Jun 28, 2013 at 2:35 PM
Thank you for your informative text.

I get the transfer speed from the FileZilla Server.

Here are the results of the speed race:
Copy 606MB Office installation exe (hard drive internal) : ~60-70MB/S
Synchronize two folders via ftpclient: ~16 MB/s is the limit, accelerating in ~10-15 seconds from 0
Copy with FileZilla Client:
FileZilla Client says: ~10-25 MB/s
FileZilla Server says; ~13-14 MB/s

Conclusion:
netFtp is a bit faster in maximum speed, but the acceleration of FileZilla is better. As a result, FileZilla seems to be faster with large amount of small files, but a small amount of large files are preferred by netFtp.

The FTP results are FTPS only. In this particular situation the SSL overhead won't matter that match I think. But I will benchmark plain FTP also.
Jun 28, 2013 at 2:45 PM
Edited Jun 28, 2013 at 2:48 PM
FileZilla might have a better algorithm for detecting what's on the server and what type of object it is. System.Net.FtpClient, for example, prefers to get the modification time and file size using MDTM when dealing with UNIX style listings. This results in more chatter between the client and server but it also gives more accurate modification times in the FtpListItem objects GetListing() returns. Implementation details like this could be, but not necessarily are, the real difference. If MLSD (machine listing) is supported it's used over LIST and thus no extra processing is required as MLSD gives accurate and easy to parse results so System.Net.FtpClient doesn't have to make calls back to the server to get the right modification time. If you're using NLST (name list) it gets even worse than LIST. NLST returns a list of file and directory names only, no other information. After that list is retrieved System.Net.FtpClient tries to change directory to each item to see if it's a file or a directory and calls SIZE and MDTM were appropriate for each object so there is a lot of back and forth between the client and the server.

These details can make or break performance but in some cases it's necessary due to inadequacies in the FTP protocol. The approach System.Net.FtpClient takes may not always be the fastest but the goal here is accuracy with performance going to the back burner.
Jun 28, 2013 at 2:59 PM
Ok I see. Precise information about the modified times are anyway more important for my solution.

Wow I just got about 30Mb/s with Net.FtpClient. On my local configuration the hard drive workload seems to be crucial.

Anyway, I think the overall speed is sufficient for me. Just for fun I will benchmark these runs again on my big pc at home with the SSD. That will be great promo for Net.FtpClient :)