Async Uploads / Downloads.

Aug 30, 2012 at 1:59 AM

What is the recomended pattern for instantiating this class for uploading or download files in parallel?

Basically this would be like in iTunes where if you purchased an album it allowed all of the songs to download at once with progress bars for each.

Coordinator
Aug 30, 2012 at 2:35 AM

You need an instance of the FtpClient or FtpControlConnection for each transfer. The FTP protocol does not support more than 1 data stream per control connection. The code base right now doesn't do much async work, you have to do it yourself. You can easily extend FtpClient or FtpControlConnection and implement the IAsyncResult pattern for the upload and download methods or you can write your own threaded code.

I have a complete re-write I've been working on that implements the IAsyncResult pattern for nearly all methods. It's not API compatible with the current code base but it's much simpler or at least I think so. It's not feature complete but provides you with the necessary facilities for adding any missing functionality with the exception of SOCKS proxy implementations which will take diving into the actual code of System.Net.FtpClient to implement. As it stands it does SSL, downloads, uploads, file listings and a few other things including cloning the control connection for simultaneous downloads (not automatically, there is a method). If you want a copy of this code I can probably get it to you tomorrow. I'm in the southern US and we've been getting pounded by hurricane Isaac for the past few days, I'm not in a position to make it available at the moment.

Aug 30, 2012 at 7:36 AM

Thanks, Good luck weathering the storm. (stated literally)

But I am not sure I fully understand the response. I understand what you are saying  about 1 ftp connection per data stream, but does that mean I need a new instance on the entire ftpclient? In my case the ftp hostname, settings, user name and password are the same for each file I would like to transfer simultaneously. I realize if they were different I would need a new instance, but above you state that I can extend ftp client to support this. Can I simply inherit the class, set delegates ,callback methods and a BeginUpload, BeginDownload ? More specifically will .upload()/.download() create a new ftp connection each time it is called or will it try to use a connection property stored outside of the upload method. Are there any other properties that are modified in the ftpclient class when .upload() is called? I think there will be some complications around the event OnTransfer progress.

It would be interesting to see the new implementation but I am not sure if I would try to use it for my project if not completely tested yet, so no emergency on that. For now I will go ahead and set the deligates outside of the ftpclient class and create a new instance for each.

Coordinator
Aug 30, 2012 at 10:19 AM

System.Net.FtpClient does not automatically create new connections as needed.

Aug 30, 2012 at 6:02 PM

thanks, that's what i needed to know.

Sep 1, 2012 at 1:12 AM
Edited Sep 1, 2012 at 1:13 AM

I tried with the Iasync Pattern and ran into some issues with performance. switched to parralel.foreach and it is working well and is much easier to manage.

 

                //set max number of threads to execute.
                System.Threading.Tasks.ParallelOptions options = new System.Threading.Tasks.ParallelOptions();
                options.MaxDegreeOfParallelism = 5;

                //loop through ans upload all items in queue
                //foreach (string localQueuePath in m_UploadQueue.UploadQueueItems)
                Parallel.ForEach(m_UploadQueue.UploadQueueItems, options, localQueuePath =>
                {
                    //upload file
                    UploadFile(localQueuePath, FilePathRemote);
                }
                );

 

But I think something is not thread safe in DirectoryExists, from time to time i find the exception.

System.InvalidOperationException: Collection was modified; enumeration operation may not execute.   at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)   at System.Collections.Generic.List`1.Enumerator.MoveNextRare()   at System.Collections.Generic.List`1.Enumerator.MoveNext()   at System.Net.FtpClient.FtpListItem.ParseListListing(String listing)   at System.Net.FtpClient.FtpListItem.Parse(String listing, FtpListType type)   at System.Net.FtpClient.FtpListItem..ctor(String listing, FtpListType type)   at System.Net.FtpClient.FtpListItem.ParseList(String[] items, FtpListType type)   at System.Net.FtpClient.FtpClient.GetListing(String path, FtpListType type)   at System.Net.FtpClient.FtpClient.GetListing(String path)   at System.Net.FtpClient.FtpClient.GetObjectInfo(String path)   at System.Net.FtpClient.FtpClient.DirectoryExists(String path)

Coordinator
Sep 1, 2012 at 1:14 AM
Yep, looks there is a piece of code iterating on a list instead of an array. I'll see if I can dig it out.
Sep 1, 2012 at 1:27 AM

just change list to SynchronizedCollection.

Coordinator
Sep 1, 2012 at 1:27 AM

Line 90 of FtpListItem.cs looks like it could be the source based on the exception above. Try adding .ToArray() to the end of the Parsers collection in the foreach loop so that it looks like this and see if it clears up the problem:

foreach (FtpListFormatParser p in FtpListFormatParser.Parsers.ToArray()) {
Let me know the results.

Sep 1, 2012 at 1:32 AM

ok will do.

Coordinator
Sep 1, 2012 at 1:59 AM

The more I think about it I think there needs to be a mutex to synchronize access to the parsers collection. I wonder if multiple threads are accessing the Parsers list before it's been initalized. The list shouldn't change beyond initialization unless you're adding custom parsers while the threads are running. Been a long day, gonna sleep on it.

Sep 6, 2012 at 12:09 AM

sorry for the late reply, i rewrote that code as below and havent seen the exception in a few days. It may be because the contents of the directory changed during code execution.

 

        /// <summary>
        /// Gets an FTP list item representing the specified file system object
        /// </summary>
        /// <param name="path"></param>
        /// <returns></returns>
        public FtpListItem GetObjectInfo(string path) {
            if (this.HasCapability(FtpCapability.MLST)) {
                FtpReply reply;

                if ((reply = this.Execute("MLST {0}", path)).Success) {
                    return new FtpListItem(reply.InfoMessages.Split('\n'), FtpListType.MLST);
                }
            }
            else {
                string directoryName = Path.GetDirectoryName(path.Replace("\\", "/").TrimEnd('/')).Replace("\\", "/");
                string fileName = Path.GetFileName(path.Replace("\\", "/").TrimEnd('/')).Replace("\\", "/");

                //get directory listing.
                List<FtpListItem> items = new List<FtpListItem>(this.GetListing(directoryName));

                //iterate collection.
                foreach (FtpListItem l in items)
                {
                    if (l.Name == fileName) {
                        return l;
                    }
                }

            }

            return new FtpListItem();
        }