ftpclient.GetListing - How to return directories?

Mar 8, 2013 at 12:27 PM
Hi all,

The progress of my application is very good. The synchronisation between two folders (one local, one on a ftp server) via ftps works perfect for files.

The problem is that mechanism cant get through directories lower than the specified one. That means that underfolders aren't synchronized.

This is the code I am currently using for comparing the two directories:

    Public Sub sync()

        ' Lokalen Ordner erstellen wenn nicht vorhanden
        If Directory.Exists(localDir) = False Then
            Directory.CreateDirectory(localDir)
        End If

        ' Lokalen Ordner und Dateiinformationen auslesen
        Dim localDirInfo As New IO.DirectoryInfo(localDir)
        Dim localDirFileInfo As IO.FileInfo() = localDirInfo.GetFiles()

        ' FTP Zugriff
        Dim ftpclient As New FtpClient
        AddHandler ftpclient.ValidateCertificate, AddressOf SSLcertificateHandler
        ftpclient.Host = ftpHost
        ftpclient.Port = ftpPort
        ftpclient.Credentials = credentials
        ftpclient.EncryptionMode = ftpencryptmode
        ftpclient.Connect()

        If ftpclient.DirectoryExists(Form1.remoteDir) = False Then
            ftpclient.CreateDirectory(Form1.remoteDir)
        End If

'
'
' Now comes the .GetListing
' I want that files in lower directories also go into the remoteDirInfo !
'
'
'
        Dim remoteDirInfo() As FtpListItem = ftpclient.GetListing(remoteDir, FtpListOption.SizeModify)
        ftpclient.Disconnect()

        Form1.bubbleSort(remoteDirInfo, remoteDirInfo.Length)
        Form1.bubbleSort(localDirFileInfo, localDirFileInfo.Length)

        Dim localDirList As List(Of FileInfo) = localDirFileInfo.ToList
        Dim remoteDirList As List(Of FtpListItem) = remoteDirInfo.ToList


        Dim i, j, k, l As Integer
        If localDirList.Count <= remoteDirList.Count Then
            While i < localDirList.Count
                While j < remoteDirList.Count
                    If localDirList.Count <= i Then
                        Exit While
                    End If
                    ' Gleicher Name und gleiches Änderungsdatum = gleiche Datei
                    If localDirList.ElementAt(l).Name = remoteDirList.ElementAt(k).Name And Date.Compare(localDirList.ElementAt(l).LastWriteTime, remoteDirList.ElementAt(k).Modified) = 0 Then
                        localDirList.RemoveAt(i)
                        remoteDirList.RemoveAt(j)
                        'Gleicher Name aber unterschiedliche Veränderungszeiten -> älteres Löschen, damit neues synchronisiert wird
                    ElseIf localDirList.ElementAt(i).Name = remoteDirList.ElementAt(j).Name And Date.Compare(localDirList.ElementAt(i).LastWriteTime, remoteDirList.ElementAt(j).Modified) < 0 Then
                        localDirList.ElementAt(i).Delete()
                        localDirList.RemoveAt(i)
                        j = 0
                    ElseIf localDirList.ElementAt(i).Name = remoteDirList.ElementAt(j).Name And Date.Compare(localDirList.ElementAt(i).LastWriteTime, remoteDirList.ElementAt(j).Modified) > 0 Then
                        ftpclient.DeleteFile(remoteDirList.ElementAt(j).FullName)
                        remoteDirList.RemoveAt(j)
                        i += 1
                        j = 0
                    Else
                        j += 1
                    End If

                End While
                i += 1
                j = 0
            End While
        Else
            While k < remoteDirList.Count
                While l < localDirList.Count
                    If remoteDirList.Count <= k Then
                        Exit While
                    End If
                    ' Gleicher Name und gleiches Änderungsdatum = gleiche Datei
                    If localDirList.ElementAt(l).Name = remoteDirList.ElementAt(k).Name And Date.Compare(localDirList.ElementAt(l).LastWriteTime, remoteDirList.ElementAt(k).Modified) = 0 Then
                        localDirList.RemoveAt(l)
                        remoteDirList.RemoveAt(k)
                        'Gleicher Name aber unterschiedliche Veränderungszeiten -> älteres Löschen, damit neues synchronisiert wird
                    ElseIf localDirList.ElementAt(l).Name = remoteDirList.ElementAt(k).Name And Date.Compare(localDirList.ElementAt(l).LastWriteTime, remoteDirList.ElementAt(k).Modified) < 0 Then
                        localDirList.RemoveAt(l)
                        k += 1
                        l = 0
                    ElseIf localDirList.ElementAt(l).Name = remoteDirList.ElementAt(k).Name And Date.Compare(localDirList.ElementAt(l).LastWriteTime, remoteDirList.ElementAt(k).Modified) > 0 Then
                        remoteDirList.RemoveAt(k)
                        l = 0
                    Else
                        l += 1
                    End If
                End While
                k += 1
                l = 0
            End While
        End If

' All files are the same -> No synchronisation needed, the user is informed
        If localDirList.Count = 0 And remoteDirList.Count = 0 Then
            AppIcon.BalloonTipText = "Ordner bereits aktuell!"
            AppIcon.Text = AppIcon.BalloonTipText
            AppIcon.ShowBalloonTip(3000)
            Exit Sub
        End If

        'Download
        Dim uploadCount As Integer = 0
        Dim downloadCount As Integer = 0
        For Each element In remoteDirList
            Download(element.FullName, localDir + "\" + element.Name)
            downloadCount += 1
            AppIcon.BalloonTipText() = "Downloadfortschritt: " + downloadCount.ToString + "/" + remoteDirList.Count.ToString + Environment.NewLine + "Uploadfortschritt: " + uploadCount.ToString + "/" + localDirList.Count.ToString
            AppIcon.Text = AppIcon.BalloonTipText
            AppIcon.ShowBalloonTip(1000)
        Next

        'Upload
        For Each element In localDirList
            Upload(element.FullName, remoteDir + "\" + element.Name)
            uploadCount += 1
            AppIcon.BalloonTipText() = "Downloadfortschritt: " + downloadCount.ToString + "/" + remoteDirList.Count.ToString + Environment.NewLine + "Uploadfortschritt: " + uploadCount.ToString + "/" + localDirList.Count.ToString
            AppIcon.Text = AppIcon.BalloonTipText
            AppIcon.ShowBalloonTip(1000)
        Next

' sets the local files to the modified time of the remote files, for future synchronisation
            syncModifiedDate()

'Synchronisation complete Bubble
            AppIcon.BalloonTipText = "Synchronisation abgeschlossen"
            AppIcon.Text = AppIcon.BalloonTipText
            AppIcon.ShowBalloonTip(3000)

        SynchroCancel.Enabled = False

    End Sub
So what I want to have is that my fileinfo-Arrays contain every file under the specified path. Then I have to modify my download and upload-Methods to create directorys whether they are needed.
Coordinator
Mar 8, 2013 at 1:09 PM
GetListing() returns everything the server listed at the specified path. Look at the Type property of the FtpListItem objects it returned.
Coordinator
Mar 8, 2013 at 1:13 PM
And if you mean perform a recursive listing, write a recursive function. Consider the following:
function ListDirectory(FtpClient client, string path) {
    foreach(FtpListItem item in client.GetListing(path)) {
        if(item.Type == FtpFileSystemObjectType.Directory) {
             ListDirectory(client, item.FullName)
         }
         else {
              // do something with the file
         }
    }
}
Mar 8, 2013 at 1:57 PM
I want to have every file under a path to be in the listItem-Array (with files in any subdirectory). I will fiddle with the recursion next week.

Thank you for your quick answer!

jptrosclair wrote:
And if you mean perform a recursive listing, write a recursive function. Consider the following:
function ListDirectory(FtpClient client, string path) {
    foreach(FtpListItem item in client.GetListing(path)) {
        if(item.Type == FtpFileSystemObjectType.Directory) {
             ListDirectory(client, item.FullName)
         }
         else {
              // do something with the file
         }
    }
}
Mar 27, 2013 at 5:17 PM
I managed the recursive listing. Now I have the problem that some Files return wrong .Modified-Values (01.01.0001 00:00). I checked the real modified-time with FileZilla. Of course the files weren't modified on that specific date.

I need the modified-time for my synchronising. How can i fix this error?
    Public Function GetRemoteFiles(ByRef list As List(Of FtpListItem)) As List(Of System.Net.FtpClient.FtpListItem)
        Dim ftplistitem As New List(Of FtpListItem)
        For Each item In list
            If item.Type = FtpFileSystemObjectType.File Then
                ftplistitem.Add(item)
            End If
        Next
        Return ftplistitem
    End Function

    Public Function GetRemoteDirInfo(ByVal path As String) As List(Of System.Net.FtpClient.FtpListItem)
        Dim ftpclient As New System.Net.FtpClient.FtpClient
        Dim ftplistitem As List(Of FtpListItem)
        ftpclient.Host = sFtpHost
        ftpclient.Credentials = credentials
        ftpclient.EncryptionMode = FtpEncryptionMode.Explicit
        AddHandler ftpclient.ValidateCertificate, AddressOf SSLcertificateHandler
        ftpclient.Connect()
        If ftpclient.DirectoryExists(path) Then
            ftplistitem = ScanForDir(path, ftpclient)
            Return ftplistitem
        Else
            ftpclient.CreateDirectory(path)
            ftplistitem = ScanForDir(path, ftpclient)
            Return ftplistitem
        End If


    End Function

    Public Function ScanForDir(ByVal Path As String, ByRef ftpclient As System.Net.FtpClient.FtpClient) As List(Of FtpListItem)
        Dim ftplistitem As New List(Of FtpListItem)
        Dim tmpftplistitem As New List(Of FtpListItem)
        ftplistitem = ftpclient.GetListing(Path).ToList
        For Each listitem In ftplistitem
            If listitem.Type = FtpFileSystemObjectType.Directory Then
                bIsRecurse = True
                tmpftplistitem.AddRange(ScanForDir(listitem.FullName, ftpclient))
            End If
        Next
        ftplistitem.AddRange(tmpftplistitem)
        Return ftplistitem
    End Function
Coordinator
Mar 27, 2013 at 6:01 PM
I need specifics, I need to see the file listing that was returned with the correct date time values as well as what System.Net.FtpClient parsed as the date/time value. I can't even make a guess as to what's wrong without that information.
Coordinator
Mar 27, 2013 at 6:03 PM
Also, if you're not already using the latest revision you should look at downloading it. I made some bug fixes yesterday for IIS date/time errors.
Mar 27, 2013 at 8:39 PM
I use the latest revision (from yesterday). I'll try to collect some information for you.
Mar 27, 2013 at 8:58 PM
Edited Mar 27, 2013 at 8:58 PM
Image
http://s7.directupload.net/images/130327/rxngp56q.png

The first Picture is from Filezilla. The second is the Output after the marked line in picture three.
As you see all files in the first level got wrong modified and created times.

This is the final output of all files that are below the specified folder:

Image
http://s1.directupload.net/images/130327/rlr2ff94.png

I also just noticed that my code has problems with Folders that have space in their name.

Best regards
Coordinator
Mar 27, 2013 at 9:58 PM
Pass the FtpListOption.SizeModify flag as the second parameter and see if it clears up the problem.
Coordinator
Mar 27, 2013 at 10:02 PM
I'll look into the file names with spaces problems tomorrow, I suspect you're using LIST for your file listing which is problematic. If you want a good reliable file listing, at the expense of performance, use the NameList flag as well: GetListing(path, FtpListOption.SizeModify | FtpListOption.NameList)

That should use MLSD when it's available which is much faster and fall back to NLST which is slower but more reliable than LIST. I suspect that we'll never have 100% reliable parsing dealing with LIST, it's a nice idea but it's just too finicky to depend on.
Mar 28, 2013 at 12:15 AM
Edited Mar 28, 2013 at 12:26 AM
OK thank you so far!

This is very strange:
(I already set SizeModify and NameList)

Image
http://s7.directupload.net/images/130328/8gb6qbvk.png
Coordinator
Mar 28, 2013 at 12:23 AM
That's DateTime.MinValue which is the default value when the parser can't get the date. What I need to see is the transaction log from System.Net.FtpClient. Also, are you doing the listing with SizeModify and NameList? Here's the VB equivalent: ftpClient.GetListing(path, FileListOption.SizeModify Or FileListOption.NameList).
Coordinator
Mar 28, 2013 at 12:24 AM
Include the FEAT response from the server and include the file listing that you're getting DateTime.MinValue on.
Mar 28, 2013 at 10:32 AM
Edited Mar 28, 2013 at 10:33 AM
Ok it seems like the problem is at my ftp-server. The special thing is that the ftp-server that I want to use is my Router/DSL-Modem at home (AVM Fritz!Box 3370). The Ftp Server that they implemented seems not to be the best. It also only supports List, whereas my local FileZilla Server is capable of MLSD.

As a result of the more compatible local FTP-server, the output of my program is right and the synchronisation runs perfectly whereas my router-ftp-server at home is struggling with the modified times with my current code. BUT, I remember that with earlier versions of my program, the modified-times were accurate. I believe that my problem started after my last firmware update that I had to install. I contacted the support of AVM for further investigation on that.
I will send you the feat response from the server later. I will also post the response from my local server so we can compare both servers.

Greetings
Mar 28, 2013 at 10:47 AM
Output on my local server (client and server on the same host):

http://pastebin.com/CBYn452M

the file situation:
http://s14.directupload.net/images/130328/wlhrna2z.png
Image
These are the remote files. In folder abat logos there is another folder with a file in it.

Everything works fine.

I will post the output of my server at home (the router) later, when I get access again. This is also strange. When I connect from my home network to my home router with the dyndns of my router as IP, the connection with filezilla or my program is established. (The connection first goes to myfritz.net the dyndns-host and then back to my router so from a networking point of view I connect from the Internet onto my router. But now, when I am really outside of my network, I can't establish the connection to my ftp server on the router (what also ever worked). I hope the support of the manufacturer replies fast.