Speed up FileExists when using option NLST?

Oct 7, 2014 at 6:19 PM
Hi,

the current implentation of FileExists uses GetListing which returns all files and dirs in a folder.
But if the NLST is used, for each item, there is an check whether the file is a directory or not.
This is causing a lot of traffic because we only want to now this for one item.

A switch should be added to check on for this on demand (If NLST is used)):
FileExists -> Get all items -> Check if item exists in list [no check] -> Check if item is file (expensive if NLST)
Currently it is:
FileExists -> Get all items -> For all items: Check: Is File or Directory (expensive if NLST) -> Check if item exists.

Thanks.
Coordinator
Oct 8, 2014 at 2:30 AM
If you want something faster the first thing you need to do is switch to a server that supports MLSD/MLST and use GetObjectInfo(). That's the best you're going to get; if I take the check out of GetListing() then FileExists() no longer does what the name of the method implies.
Coordinator
Oct 8, 2014 at 2:37 AM
Look at the code for GetObjectInfo(), the same approach can be used in this case:
if(serverSupportsMLSD) {
    // use FtpClient.GetObjectInfo() because it's the fastest
}
else {
    // use slower GetListing() based methods because server only supports NLST or LIST.
}
Oct 8, 2014 at 1:31 PM
jptrosclair wrote:
if I take the check out of GetListing() then FileExists() no longer does what the name of the method implies.
Why? The current implementation of FileExists with GetListing() is good (same approch for several kinds of listings) but certainly an overkill: Iterating over all files/dirs just for one item is very expensive. If the file check is just performed for the desired item, the behaviour is similar.
I will write an new implementation and post a patch if you dont mind. I do not have much experience in FTP and maybe I made a mistake in my quick analysis.
Oct 8, 2014 at 3:28 PM
Edited Oct 8, 2014 at 3:30 PM
I did some performance tests and I must admit that the speed gain is minimal: Connect and Auth take most of the time.

But the performance is now stable: It does not matter how many files there are, DirectoryExists is only called 1 plus 1 and not 1 plus n times [NLST only!].

Maybe you can use it...
diff --git "a/FtpClient-d611c8a-left.cs" "b/FtpClient.cs"
index 0b3718d..24881e7 100644
--- "a/FtpClient-d611c8a-left.cs"
+++ "b/FtpClient.cs"
 
 namespace System.Net.FtpClient {
     /// <summary>
@@ -2083,6 +2085,80 @@ namespace System.Net.FtpClient {
         }
 
         /// <summary>
+        /// Checks if a file exsts on the server by using the NLST
+        /// command.
+        /// </summary>
+        /// <param name="path">The full or relative path to the file</param>
+        /// <returns>True if the file exists</returns>
+        private bool NameListFileExists(string path)
+        {
+            string ftpFileName = path.GetFtpFileName();
+            string pwd = GetWorkingDirectory();
+            string hit = null;
+
+            path = path.GetFtpPath();
+            if (path == null || path.Trim().Length == 0)
+            {
+                if (pwd != null && pwd.Trim().Length > 0)
+                    path = pwd;
+                else
+                    path = "./";
+            }
+            else if (!path.StartsWith("/") && pwd != null && pwd.Trim().Length > 0)
+            {
+                if (path.StartsWith("./"))
+                    path = path.Remove(0, 2);
+                path = string.Format("{0}/{1}", pwd, path).GetFtpPath();
+            }
+
+            try
+            {
+                m_lock.WaitOne();
+
+                Execute("TYPE I");
+
+                
+
+                // read in raw file listing
+                using (FtpDataStream stream = OpenDataStream(string.Format("NLST {0}", path.GetFtpPath()), 0))
+                {
+                    try
+                    {
+                        string buf;
+                        while ((buf = stream.ReadLine(Encoding)) != null)
+                        {
+                            if (hit == null && buf.Length > 0)
+                            {
+                                if (buf.GetFtpFileName() == ftpFileName)
+                                {
+                                    hit = buf;
+                                }
+                            }
+                        }
+                    }
+                    finally
+                    {
+                        stream.Close();
+                    }
+                }
+            }
+            finally
+            {
+                m_lock.ReleaseMutex();
+            }
+
+            if (hit != null)
+            {
+                if (DirectoryExists(hit))
+                    return false;
+                else
+                    return true;
+            }
+
+            return false;
+        }
+
+        /// <summary>
         /// Gets a file listing from the server asynchronously
         /// </summary>
         /// <param name="callback">AsyncCallback method</param>
@@ -2815,9 +2891,23 @@ namespace System.Net.FtpClient {
                 if (!DirectoryExists(dirname))
                     return false;
 
-                foreach (FtpListItem item in GetListing(dirname, options))
-                    if (item.Type == FtpFileSystemObjectType.File && item.Name == path.GetFtpFileName())
-                        return true;
+                // Reduce the overhead of GetListing in case the NLST command
+                // is used (MLSD is a good fast alternative, too)
+                if ((options & FtpListOption.NameList) == FtpListOption.NameList)
+                {
+                    return NameListFileExists(path);
+                }
+                else
+                {
+                    // Check if the file exists using the listing
+                    string ftpFileName = path.GetFtpFileName();
+                    foreach (FtpListItem item in GetListing(dirname, options))
+                    {
+                        if (item.Type == FtpFileSystemObjectType.File && item.Name == ftpFileName)
+                            return true;
+                    }
+                }
+                
             }
             finally {
                 m_lock.ReleaseMutex();