LastWriteTime is reporting the wrong year

Jan 9, 2013 at 4:54 PM

I'm looking at files in an FTP folder using:

foreach (FtpFile ftpFile in ftpClient.CurrentDirectory.Files)
            {
                FtpFileList.Add(ftpFile);
            }

This was working fine before New Year.  At some time after New Year the values returned in the LastWriteTime field has gone wrong.

For some files the year is reporting as 2013 when I can see the files are shown as Last Modified in 2012.

By looking at my database records I can see that this effect first showed up on 4th of Jan 2013 which is when we ran an analysis of the folder involved.  And again on 8th of Jan.

It is also swapping round the Month and Days.  

e.g. A file on disk marked 03/09/2012   (dd.MM.yyyy) is being returned as  09/03/2013

Can you have a look into this as my entire enterprise FTP workflow system can no longer filter by date.  :-(

Thanks

Richard

 

Coordinator
Jan 9, 2013 at 5:14 PM

Need to know more details for example, is the modify time being pulled with MDTM or is it being parsed from a LIST or MLSD format? If you can provide a transaction log it would be greatly beneficial. Generally the MDTM command is going to provide the most specific time with the exception of MLSD which should use an identical or similarly formatted file time. The MDTM and MLSD time formats have very specific format strings:

"yyyyMMddHHmmss", "yyyyMMddHHmmss.fff"
Fixing this bug might be as simple as adding a new format string, hard to say though without knowing exactly what date/time string is being parsed and failing.

Coordinator
Jan 9, 2013 at 5:19 PM

Also, if you're relying on date time being parsed from the LIST command I suggest you not do that and use the GetLastWriteTime() method of FtpClient instead. LIST formats will never be 100% reliable, there is no amount of code I can write to make it otherwise. MDTM has a date/time format explicitly defined in the RFC, there is no written specification for how a response to the LIST command should be formatted or furthermore, how a date/time value should be formatted. Judging from the bug it looks like a clear case of System.Net.FtpClient expecting a MM/dd/yyyy format rather than dd/MM/yyyy.

Coordinator
Jan 9, 2013 at 8:22 PM

It's also worth noting that System.Net.FtpClient doesn't do any kind of crazy date manipulation. It relies directly on .net's System.DateTime type for parsing so if there is an error with the date it's almost certainly a problem with the format string being passed to DateTime's parsers or it's because, as noted above, System.Net.FtpClient expects the date to be in a certain format when parsing LIST which in reality is an arbitrary format that can't be predicted. If you're parsing DOS style directory listings from IIS you'll probably want to add a format string to the FtpListFormatParser.cs's Modify property. I just looked over the code and it has formats defined for UNIX long listing styles but not IIS's DOS style. It falls back to a generic DateTime.Parse() which may be the source of the problem. If I could have a look at the transaction log where the date is wrong I could probably confirm that this is the issue.

Jan 10, 2013 at 11:51 AM
Edited Jan 10, 2013 at 11:56 AM

Here is my captured log for the ftpClient.CurrentDirectory.Files command in my code:

 

No.     Time           Source                Destination           Protocol Length Info
   8902 457.801093000  172.16.32.20          192.168.32.57         FTP      60     Request: FEAT
   8903 457.801787000  192.168.32.57         172.16.32.20          FTP      69     Response: 211-Features:
   8904 457.801788000  192.168.32.57         172.16.32.20          FTP      61     Response:  EPRT
   8905 457.801788000  192.168.32.57         172.16.32.20          FTP      61     Response:  EPSV
   8907 457.801832000  192.168.32.57         172.16.32.20          FTP      61     Response:  MDTM
   8908 457.801833000  192.168.32.57         172.16.32.20          FTP      61     Response:  PASV
   8909 457.801833000  192.168.32.57         172.16.32.20          FTP      68     Response:  REST STREAM
   8910 457.801833000  192.168.32.57         172.16.32.20          FTP      61     Response:  SIZE
   8911 457.801834000  192.168.32.57         172.16.32.20          FTP      61     Response:  TVFS
   8912 457.801834000  192.168.32.57         172.16.32.20          FTP      61     Response:  UTF8
   8913 457.801835000  192.168.32.57         172.16.32.20          FTP      63     Response: 211 End
   8915 457.803882000  172.16.32.20          192.168.32.57         FTP      62     Request: TYPE A
   8916 457.804379000  192.168.32.57         172.16.32.20          FTP      84     Response: 200 Switching to ASCII mode.
   8917 457.808279000  172.16.32.20          192.168.32.57         FTP      60     Request: EPSV
   8918 457.808883000  192.168.32.57         172.16.32.20          FTP      103    Response: 229 Entering Extended Passive Mode (|||11151|).
   8922 457.817725000  172.16.32.20          192.168.32.57         FTP      73     Request: LIST /dev/Int-Out
   8923 457.818578000  192.168.32.57         172.16.32.20          FTP      93     Response: 150 Here comes the directory listing.
   8927 457.819126000  192.168.32.57         172.16.32.20          FTP      78     Response: 226 Directory send OK.

 

I note that the command makes a FEAT request and the server does seem to support MLSD.

 Captured Data Log:

No.     Time           Source                Destination           Protocol Length Info
211 11.395429000   192.168.32.57         172.16.32.20          FTP-DATA 359    FTP Data: 305 bytes

FTP Data
[truncated] FTP Data:
-rw-r--r--    1 1035     302          5120 Nov 20 15:00 AAA-GOOD-002.MPG\r\N
-rw-r--r--    1 1035     302             0 Jul 11  2012 AAA-TEST123-001.txt\r\n
-rw-r--r--    1 1035     302          5120 Jan 09 15:25 AAA-TE

 

Note that the file AAA-GOOD-002.MPG has not returned a Year.  The file is on the server as Last Modified in Nov 20 2012 so why the server is not returning the year is a puzzle.  In the FtpFile Object it gets 2013 as the year.

Does the LIST parser assume all dates missing years are in the current year?

I tried a couple of things:

 

foreach (ftp_Client.FtpFile ftpFile in ftpClient.CurrentDirectory.Files)
            {
                DateTime dateTime = ftpClient.GetLastWriteTime(ftpFile.FullName);

 

This returns the correct date but requires 2 calls to the FTP server which I'd rather not have.Then I tried this:

 

foreach (ftp_Client.FtpFile ftpFile in ftpClient.CurrentDirectory.Files)
            {
                ftp_Client.FtpFile ftpFileNew = new ftp_Client.FtpFile(ftpClient, ftpFile.FullName);

This line puts the correct date into ftpFileNew but does not make any calls to the FTP server.  
How it works this out I do not know.
However, because FtpFile.LastWriteTime is read-only I can't correct the datatime in the object.  
I suppose I can extend FtpFile by inheritance and adding a field.
I am using FtpClient version 1.0.4563.38876 Runtime v4.0.30319. 
I cannot move to the newer version as you removed methods I use in my code.
Thanks for the assistence
Richard  

Coordinator
Jan 10, 2013 at 12:44 PM

You are right in that .net's DateTime.Parse assumes the current year when no year is provided. This is not at all uncommon when dealing with UNIX style long listing formats. When you create a new FtpFile object the LastWriteTime property is automatically loaded using the MDTM command which is effectively the same as calling GetLastWriteTime(). You can try forcing the use of MLSD, I recall maybe pure-ftpd or vsftpd (can't remember) supporting MLSD but not reporting it in FEAT. If that doesn't work you'll need to use GetLastWriteTime(). The following overloaded GetListing() method will allow you to override the command used to list files, just to see if the server will take it:

public FtpListItem[] GetListing(string path, FtpListType type)

Jan 10, 2013 at 3:16 PM

I tried :

 

ftp_Client.FtpListItem[] ftpListItems = ftpClient.GetListing(ftpClient.CurrentDirectory.ToString(),
                                                                         ftp_Client.FtpListType.MLST);

which generated a 500 Unknown Command.  So my server (vsttpd I believe) does not have it.

I can't really use:

foreach (ftp_Client.FtpFile ftpFile in ftpClient.CurrentDirectory.Files)
            {
                FTPClientFtpFileExtended ftpFileNew = new FTPClientFtpFileExtended(ftpClient, ftpFile.FullName);

because I have 1000s of files in the folders and I'm not going to hammer the servers every 10 seconds with 1000s of requests.

What I am going to do is detect any file that is from the future and decrement the year by 1.

My local UNIX sys admins tell me this LIST problem is an effect of List on a UNIX server.  Unix will show the day, month and year for any file older than 6 months.  Younger than 6 months it shows the day month and time.

Thanks for the help

 

Richard

 

Coordinator
Jan 10, 2013 at 3:19 PM

Understandable. You might want to try MLSD again using CurrentDirectory.FullName instead of ToString(), I can't recall off the top of my head that ToString() is overridden in FtpFileSystemObject's which could be the result of the 500 response. It indeed could be that it dosn't support MLSD as well, just thought I'd mention that.

Jan 10, 2013 at 5:53 PM

Tried CurrentDirectory.FullName.  Still got error 500.

Turns out I cant't add the correct datatime stamp into an extended class that inherits FtpFile like this:

internal class FTPClientFtpFileExtended : ftp_Client.FtpFile
    {
        public DateTime RealLastModifiedDateTime;

        public FTPClientFtpFileExtended(ftp_Client.FtpClient ftpClient, string path)
            : base(ftpClient, path)
        {

        }

        public FTPClientFtpFileExtended(ftp_Client.FtpFile ftpFile)
        {
            
        }
    }

Because the 2nd Constructor fails with "base class ... doesn't contain parameterless constructor".

I certainly don't want to construct a new FTPClientFtpFileExtended using the Base class constructor as that would mean the object calling to the FTP server.

Any thoughts on how I can "attach" a valid DateTime to the FtpFile class.  I'm looking to do this because FtpFile.LastWriteTime is read-only and I have used the FtpFile type in many methods so need a class that inherits from it.



Coordinator
Jan 10, 2013 at 6:30 PM

This compiles:

    public class FtpFileExtended : FtpFile {
        public override DateTime LastWriteTime {
            get {
                if (base.LastWriteTime > DateTime.Now)
                    return base.LastWriteTime.Subtract(new TimeSpan(365, 0, 0, 0, 0));
                return base.LastWriteTime;
            }
            protected set {
                base.LastWriteTime = value;
            }
        }

        public FtpFileExtended(FtpClient client, string path)
            : base(client, path) {
        }

        public FtpFileExtended(FtpFile file)
            : base(file.Client, file.FullName) {
        }
    }

Coordinator
Jan 10, 2013 at 6:33 PM

Whoops, missed that last part. Why not just grab the source from the System.Net.FtpClient_1 and change the code to subtract a year as noted above? There is no way to extend FtpFile without calling its constructors, short of adding an empty constructor to the FtpFile class which isn't a big deal, it's just not there as it stands.

Coordinator
Jan 10, 2013 at 6:37 PM

Also, calling the base constructor doesn't force it to reload everything and you should be able to set the properties such as LastWriteTIme since they are protected, not private. Here's the base constructor for FtpFileSystemObject which is the parent of FtpFile:

public FtpFileSystemObject(FtpClient client, string path) {
            this.Client = client;
            this.FullName = FtpControlConnection.GetFtpPath(path);
        }

So this code would effectively copy a FtpFile object into a FtpFileExtended object:

   public class FtpFileExtended : FtpFile {
        public override DateTime LastWriteTime {
            get {
                if (base.LastWriteTime > DateTime.Now)
                    return base.LastWriteTime.Subtract(new TimeSpan(365, 0, 0, 0, 0));
                return base.LastWriteTime;
            }
            protected set {
                base.LastWriteTime = value;
            }
        }

        public FtpFileExtended(FtpClient client, string path)
            : base(client, path) {
        }

        public FtpFileExtended(FtpFile file)
            : this(file.Client, file.FullName) {
                this.LastWriteTime = file.LastWriteTime;
                this.Length = file.Length;
        }
    }

Coordinator
Jan 10, 2013 at 6:43 PM

Also, may want to call

 

base.LastWriteTime.AddYears(-1);

 

Rather than subtract 365 days to let .net handle weird gotchas like leap years.

Jan 11, 2013 at 12:50 PM

I've never known a developer so keen to offer support.

I've got it all running now.

One other gotcha that you may want to know is don't use:

if (base.LastWriteTime > DateTime.Now)

because that relies on the FTP server and the application server having synchronised clocks.  I was getting files ingested at 11:25 being marked as from the future by an application server that was 5 minutes slow and thought it was only 11:20.

 

I used this instead:

if (base.LastWriteTime > DateTime.Now.AddDay(1).AddHour(1)

This should be enough leeway to deal with most problems including failure to deal with Sumer Time correctly.

Thanks again for all your help. 

Coordinator
Jan 11, 2013 at 12:54 PM

Not a problem, glad we were able to identify the issue and get it sorted out. 

Coordinator
Jan 11, 2013 at 1:13 PM

I noticed that it looks like you used a packet sniffer to capture the FTP transaction. In the version of System.Net.FtpClient that you're using there is a property of FtpClient called FtpLogStream that will allow you to set a stream object to the write the FTP transaction to. In the newer versions the transaction is logged to the Debug trace listener instead when DEBUG is defined at compile time. Just thought I'd mention it because it's undoubtedly easier than sniffing the network and judging by your use case, you might find it useful to log the server transactions for debugging when problems do crop up. Also worth noting that passwords are omitted automatically when the log is written to.