"Invisible" subfolders in an empty folder on a NAS

Jul 1, 2013 at 5:26 PM
Edited Jul 1, 2013 at 5:28 PM
Hi its me again.

I am trying to use my ftp solution, powered by System.Net.Ftpclient with a QNAP NAS.

I created a test folder under an empty standard folder of the NAS. I activated FTPS on the NAS and created a user for myself with sufficient permissions.

Then I connected to the NAS with the new user and opened the folder I created, using FileZilla. Everything works. The folder is empty.

After that I tried .GetListing onto the Folder in a recursive method to get all files and directories after a specific path.

What happens is that MLSD returns two folders "." and ".." and is adding them to a FtpListItem-List. Because of the recursion my method is now calling .getlisting for each folder. MLSD then returns two folders "." and ".." as Subfolders of "." and "..".
Welcome to the endless loop.

My local FileZilla-Server does not show this behaviour. The FileZilla-client also does not show the "." and ".." folder on the NAS.

This is an excerpt of the output of the TraceListener:
InterNetwork: 192.168.10.167
220 NASFTPD Turbo station 1.3.2e Server (ProFTPD) [192.168.10.167]
AUTH TLS
234 AUTH TLS successful
Time to activate encryption: 0h 0m 0s, Total Seconds: 0,754.
PBSZ 0
200 PBSZ 0 successful
PROT P
200 Protection set to Private
USER chn
331 Password required for chn
PASS <omitted>
230 User chn logged in
FEAT
211-Features:
MDTM
MFMT
UTF8
AUTH TLS
MFF modify;UNIX.group;UNIX.mode;
MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.mode*;UNIX.owner*;
LANG en-US*
PBSZ
PROT
REST STREAM
SIZE
211 End
OPTS UTF8 ON
200 UTF8 set to on
Text encoding: System.Text.UTF8Encoding
PWD
257 "/" is the current directory
CWD /Download/chn
250 CWD command successful
CWD /
250 CWD command successful
TYPE I
200 Type set to I
EPSV
229 Entering Extended Passive Mode (|||55838|)
InterNetwork: 192.168.10.167
MLSD /Download/chn
150 Opening ASCII mode data connection for MLSD
Time to activate encryption: 0h 0m 0s, Total Seconds: 0,0210021.
modify=20130701160257;perm=flcdmpe;type=dir;unique=900UAEC0001;UNIX.group=10000514;UNIX.mode=0777;UNIX.owner=10018154; .
modify=20130701160300;perm=flcdmpe;type=dir;unique=900U8E80001;UNIX.group=0;UNIX.mode=0777;UNIX.owner=0; ..
226 Transfer complete
Disposing FtpSocketStream...
TYPE I
200 Type set to I
EPSV
229 Entering Extended Passive Mode (|||56195|)
InterNetwork: 192.168.10.167
MLSD /Download/chn/.
150 Opening ASCII mode data connection for MLSD
Time to activate encryption: 0h 0m 0s, Total Seconds: 0,0060006.
modify=20130701160257;perm=flcdmpe;type=dir;unique=900UAEC0001;UNIX.group=10000514;UNIX.mode=0777;UNIX.owner=10018154; .
modify=20130701160300;perm=flcdmpe;type=dir;unique=900U8E80001;UNIX.group=0;UNIX.mode=0777;UNIX.owner=0; ..
226 Transfer complete
Disposing FtpSocketStream...
Coordinator
Jul 1, 2013 at 5:51 PM
Edited Jul 1, 2013 at 5:59 PM
The solution is pretty simple, ignore directories who's name are '.' (current directory) and '..' (parent directory). The type for these items in MLSD should be cdir and pdir respectively which are ignored by System.Net.FtpClient when parsing the listing but the type in the transcript is dir. It's an implementation problem with the server software but it's not a big deal, your recursive methods should be ignoring '.' and '..' anyway for safety from these kinds of issues. What I mean is that blindly recursing is a bad idea to begin with and your recursive methods should be aware that '.' means current directory and '..' means parent directory of the current directory. Example:

"/foo/bar/." == "/foo/bar"

and

"/foo/bar/.." == "/foo"
Coordinator
Jul 1, 2013 at 6:02 PM
Here's the relevant excerpt from RFC3659:
7.5.1.  The Type Fact

   The type fact needs a special description.  Part of the problem with
   current practices is deciding when a file is a directory.  If it is a
   directory, is it the current directory, a regular directory, or a
   parent directory?  The MLST specification makes this unambiguous
   using the type fact.  The type fact given specifies information about
   the object listed on the same line of the MLST response.

   Five values are possible for the type fact:

      file         -- a file entry
      cdir         -- the listed directory
      pdir         -- a parent directory
      dir          -- a directory or sub-directory
      OS.name=type -- an OS or file system dependent file type
Coordinator
Jul 1, 2013 at 6:16 PM
Also for further clarification, here's another issue with recursion that you need to protect yourself from so that you don't run into a similar infinite loop scenario:
$ mkdir test
$ cd test
$ ln -s . test
$ for (( i = 0; i < 5; i++ )); do cd test; pwd; done
/Users/jptrosclair/test/test
/Users/jptrosclair/test/test/test
/Users/jptrosclair/test/test/test/test
/Users/jptrosclair/test/test/test/test/test
/Users/jptrosclair/test/test/test/test/test/test
$ ls -l test
lrwxr-xr-x  1 jptrosclair  staff  1 Jul  1 12:06 test@ -> .
What we have here is symbolic link called 'test' that points at '.'; allowing me to infinitely keep changing into the 'test' directory even though I'm still really in /Users/jptrosclair/test. Point is, this could be returned in a file listing from a FTP server and cause your code to traverse the 'test' directory until you run out of stack space crashing your program.
Jul 2, 2013 at 8:58 AM
Ok thank you once more for the detailed and very fast answer. The recursive method is now aware of the current and parent directory.

I also know the symbolic link issue, but I am not quite sure on how to protect the recursive method of this. I could check for recurring subfolders with the same name as the parent dir with a counter, but that is not very elegant. Crude people could have folder situations with folder.fullname == parentFolder.fullname .

Do you know an elegant way to protect me from this issue?

Best regards
Jul 2, 2013 at 9:26 AM
I have another issue with this exact NAS.

My upload method throws a FTPCommandException "No such file or directory" when the outputstream should be created. Again, it works on the local FileZilla but not on the QNAP NAS.

Folder situation:
Sourcepath (local) contains four installation files, destination path (NAS via ftps) "download/chn" is empty .

Excerpt of TraceListener:
TYPE I
200 Type set to I
SIZE "/Download/chn/install.exe"
550 "/Download/chn/install.exe": No such file or directory
EPSV
229 Entering Extended Passive Mode (|||56170|)
InterNetwork: 192.168.10.167
STOR "/Download/chn/install.exe"
550 "/Download/chn/install.exe": No such file or directory
Eine Ausnahme (erste Chance) des Typs "System.Net.FtpClient.FtpCommandException" ist in System.Net.FtpClient.dll aufgetreten.
Upload Method:

    Public Sub Upload(ByVal filename As String, ByVal remotepath As String, ByRef client As FtpClient, ByVal host As String, ByVal credentials As System.Net.NetworkCredential)

        Dim bytesUploaded As Long
        Dim betwQuant As Long
        Using FileStream As New System.IO.FileStream(filename, FileMode.Open, FileAccess.Read), outputstream As Stream = client.OpenWrite(Chr(34) + remotepath + Chr(34))
            Do
                Dim buffer(8192) As Byte
                Dim bytesread As Long = FileStream.Read(buffer, 0, 5121)
                bytesUploaded += bytesread
                betwQuant += bytesread
                If bytesread = 0 Then
                    Exit Do
                End If
                outputstream.Write(buffer, 0, CInt(bytesread))

            Loop
        End Using

    End Sub
Coordinator
Jul 2, 2013 at 1:48 PM
Edited Jul 2, 2013 at 1:48 PM
Does the path /Download/chn/ already exist on the server? Either it doesn't exist or a permission problem are the only things I can come up with as to why it's failing.
Jul 2, 2013 at 1:53 PM
The path exists and I can upload files to that exact path using FileZilla with the same user.
Coordinator
Jul 2, 2013 at 1:56 PM
Is there more to the transcript than what you're posting? It's possible the client and server are out of sync but I can't determine that from what I see above. A complete transaction log might help.
Coordinator
Jul 2, 2013 at 2:01 PM
Edited Jul 2, 2013 at 2:02 PM
Wait a second, what's with the quotes in the file path? Is System.Net.FtpClient putting those there or are you? They shouldn't be there unless they're literally part of the file name. Here:
SIZE "/Download/chn/install.exe"
and here:
STOR "/Download/chn/install.exe"
Jul 2, 2013 at 2:15 PM
Yeah I add the quotes, as you can see in the upload method:
Chr(34) + remotepath + Chr(34)

I used this because I ran into problems with spaces in filenames. Is this wrong? Why does it work on any other ftp server?
Coordinator
Jul 2, 2013 at 2:17 PM
Edited Jul 2, 2013 at 2:18 PM
Other FTP servers might be stripping them, there is no reason to quote the path, the syntax of the command is STOR<SP>/path/file name<EOL> where <SP> is space and <EOL> is end of line. If you are running into problems with spaces in file names the problem is elsewhere.
Jul 2, 2013 at 2:21 PM
Ok thank you. I deleted them and no the upload to the NAS works.

Do you have any idea to the recursion problem you informed me about 9 comments before?
Coordinator
Jul 2, 2013 at 2:25 PM
Make sure the base name of the path isn't '.' or '..' and use a counter to limit the depth of recursion. I.e., if(counter == 50) return; so that your method won't call itself more than 50 times. You can change this number to something higher, point is
placing a limit will prevent you from overrunning the call stack if you hit some odd case where you've fallen into what would be an infinite loop.