VxWork FTP Server support.

Jun 12, 2012 at 5:48 AM
//VxWorks format directory
_listParsers.Add(new FtpListFormatParser(
  @"(\d+)\s+(\w+-\d{1,2}-\d{4}\s+\d{1,2}:\d{1,2}:\d{1,2})\s+(.*?)\s+\<\DIR\>",
   3, -1, 2, -1, -1, -1, -1, FtpObjectType.Directory));

//VxWorks format file
_listParsers.Add(new FtpListFormatParser(
  @"(\d+)\s+(\w+-\d{1,2}-\d{4}\s+\d{1,2}:\d{1,2}:\d{1,2})\s+(.*?)\s+",
  3, 1, 2, -1, -1, -1, -1, FtpObjectType.File));

in FtpList FormatParser.cs

test in VxWorks 5.5.1

Coordinator
Jun 12, 2012 at 10:59 PM

Thanks, I've added these parsers and they're now available in the latest revision!

Jun 13, 2012 at 5:31 AM

Another problem(Socket error10054 );

Download files from VxWorks  ftp server by using passive mode , ftp server will  directly close the client connection when send task finshed,

this has led to  networkstream.read (...)  throw an IOExpection (the InnerExpection is SocketException ErrorCode == 10054);

At this time the data has been already completely copied to the client socket buffer,

networkstream.read (...)  will throw an exception when taking the last packet (when read Buffer length is greater than the actual length of the last packet);

Here is a non-perfect solution:

   Dynamic calculae Buff length using file length which pass to networkstream.read (...);

Code as follows ( FtpClient.cs):

 public void Download(FtpFile remote, Stream ostream, FtpDataType datatype, long rest) {
            long size = remote.Length;
            long total = 0;
            int read = 0;
            //add by iori
            long remain = size;

            if (remote == null) {
                throw new ArgumentException("remote is null");
            }

            if (ostream == null) {
                throw new ArgumentException("ostream is null");
            }

            if (!ostream.CanWrite) {
                throw new ArgumentException("ostream is not writable");
            }

            if (rest > 0 && ostream.CanSeek) { // set reset position
                ostream.Seek(rest, SeekOrigin.Begin);
                total = rest;
                //add by iori
                remain -= rest;
            } else if (!ostream.CanSeek) {
                rest = 0;
            }

            try {

                using (FtpDataStream ch = this.OpenRead(remote.FullName, datatype, rest)) {
                    byte[] buf = new byte[ch.ReceiveBufferSize];
                    DateTime start = DateTime.Now;
                    FtpTransferInfo e = null;

                    //update by iori 
                    //(buf.Length > remain ? remain : buf.Length) will always retrun a int value
                    while ((read = ch.Read(buf, 0, (int)(buf.Length > remain ? remain : buf.Length))) > 0) {
                        ostream.Write(buf, 0, read);
                        total += read;
                        //add by iori
                        remain -= read;
                        e = new FtpTransferInfo(FtpTransferType.Download, remote.FullName, size, rest, total, start, false);

                        this.OnTransferProgress(e);
                        if (e.Cancel) {
                            break;
                        }
                    }

                    // fire one more time to let event handler know that the transfer is complete
                    this.OnTransferProgress(new FtpTransferInfo(FtpTransferType.Download, remote.FullName,
                        size, rest, total, start, true));
                }
            } finally {
                ostream.Flush();
            }
        }

I've marked the code which I updated.

 

Sorry about my poor English.

Coordinator
Jun 13, 2012 at 1:39 PM

It's normal that the server closes the stream when the last of the data is sent, what's not normal is that an exception is being thrown because the buffer size is bigger than the amount of data that's available. This is a problem because the actual size of the file is not always available (some servers don't support it) so it's not always possible to calculate the remaining bytes needed to be read to avoid the exception. I haven't seen this problem before, what OS and runtime are you running the FTP Client code on?

Coordinator
Jun 13, 2012 at 8:38 PM
If you don't mind add the following code to FtpDataStream.cs's Read
method before this.BaseStream.BeginRead(...) and see if it also
addresses the problem with VxWork's ftp server. If it does it can
potentially fix the problem without needing to know the remote file size.

if (this._socket != null && this._socket.Available > 0 &&
this._socket.Available < count) {
count = this.Socket.Available;
}

iori13 wrote:
> From: iori13
>
> Another problem(Socket error10054 );
>
> Download files from VxWorks ftp server by using passive mode , ftp
> server will directly close the client connection when send task finshed,
>
> this has led to networkstream.read (...) throw an IOExpection (the
> InnerExpection is SocketException ErrorCode == 10054);
>
> At this time the data has been already completely copied to the client
> socket buffer,
>
> networkstream.read (...) will throw an exception when taking the last
> packet (when read Buffer length is greater than the actual length of the
> last packet);
>
> Here is a non-perfect solution:
>
> Dynamic calculae Buff length using file length which pass to
> networkstream.read (...);
>
> Code as follows ( FtpClient.cs):
>
> public void Download(FtpFile remote, Stream ostream, FtpDataType datatype,long rest) {
> long size = remote.Length;
> long total = 0;
> int read = 0;
> //add by iori
> long remain = size;
>
> if (remote ==null) {
> throw new ArgumentException("remote is null");
> }
>
> if (ostream ==null) {
> throw new ArgumentException("ostream is null");
> }
>
> if (!ostream.CanWrite) {
> throw new ArgumentException("ostream is not writable");
> }
>
> if (rest> 0&& ostream.CanSeek) {// set reset position
> ostream.Seek(rest, SeekOrigin.Begin);
> total = rest;
> //add by iori
> remain -= rest;
> }else if (!ostream.CanSeek) {
> rest = 0;
> }
>
> try {
>
> using (FtpDataStream ch =this.OpenRead(remote.FullName, datatype, rest)) {
> byte[] buf =new byte[ch.ReceiveBufferSize];
> DateTime start = DateTime.Now;
> FtpTransferInfo e =null;
>
> //update by iori
> //(buf.Length> remain ? remain : buf.Length) will always retrun a int value
> while ((read = ch.Read(buf, 0, (int)(buf.Length> remain ? remain : buf.Length)))> 0) {
> ostream.Write(buf, 0, read);
> total += read;
> //add by iori
> remain -= read;
> e =new FtpTransferInfo(FtpTransferType.Download, remote.FullName, size, rest, total, start,false);
>
> this.OnTransferProgress(e);
> if (e.Cancel) {
> break;
> }
> }
>
> // fire one more time to let event handler know that the transfer is complete
> this.OnTransferProgress(new FtpTransferInfo(FtpTransferType.Download, remote.FullName,
> size, rest, total, start,true));
> }
> }finally {
> ostream.Flush();
> }
> }
>
> I've marked the code which I updated.
>
> Sorry about my poor English.
>
> Read the full discussion online
> <http://netftp.codeplex.com/discussions/359244#post848272>.
>
> To add a post to this discussion, reply to this email
> ([email removed]
> <mailto:[email removed]?subject=[netftp:359244]>)
>
> To start a new discussion for this project, email
> [email removed] <mailto:[email removed]>
>
> You are receiving this email because you subscribed to this discussion
> on CodePlex. You can unsubscribe or change your settings
> <https://netftp.codeplex.com/subscriptions/thread/project/edit> on
> codePlex.com.
>
> Please note: Images and attachments will be removed from emails. Any
> posts to this discussion will also be available online at codeplex.com
>
Jun 14, 2012 at 2:46 AM

The ftp server which I'm using is provided by VxWorks 5.5.1 for MIPS; 

http://windriver.com/products/vxworks/

Jun 14, 2012 at 3:01 AM
jptrosclair wrote:
If you don't mind add the following code to FtpDataStream.cs's Read
method before this.BaseStream.BeginRead(...) and see if it also
addresses the problem with VxWork's ftp server. If it does it can
potentially fix the problem without needing to know the remote file size.

if (this._socket != null && this._socket.Available > 0 &&
this._socket.Available < count) {
count = this.Socket.Available;
}

iori13 wrote:
> From: iori13
>
> Another problem(Socket error10054 );
>
> Download files from VxWorks ftp server by using passive mode , ftp
> server will directly close the client connection when send task finshed,
>
> this has led to networkstream.read (...) throw an IOExpection (the
> InnerExpection is SocketException ErrorCode == 10054);
>
> At this time the data has been already completely copied to the client
> socket buffer,
>
> networkstream.read (...) will throw an exception when taking the last
> packet (when read Buffer length is greater than the actual length of the
> last packet);
>
> Here is a non-perfect solution:
>
> Dynamic calculae Buff length using file length which pass to
> networkstream.read (...);
>
> Code as follows ( FtpClient.cs):
>
> public void Download(FtpFile remote, Stream ostream, FtpDataType datatype,long rest) {
> long size = remote.Length;
> long total = 0;
> int read = 0;
> //add by iori
> long remain = size;
>
> if (remote ==null) {
> throw new ArgumentException("remote is null");
> }
>
> if (ostream ==null) {
> throw new ArgumentException("ostream is null");
> }
>
> if (!ostream.CanWrite) {
> throw new ArgumentException("ostream is not writable");
> }
>
> if (rest> 0&& ostream.CanSeek) {// set reset position
> ostream.Seek(rest, SeekOrigin.Begin);
> total = rest;
> //add by iori
> remain -= rest;
> }else if (!ostream.CanSeek) {
> rest = 0;
> }
>
> try {
>
> using (FtpDataStream ch =this.OpenRead(remote.FullName, datatype, rest)) {
> byte[] buf =new byte[ch.ReceiveBufferSize];
> DateTime start = DateTime.Now;
> FtpTransferInfo e =null;
>
> //update by iori
> //(buf.Length> remain ? remain : buf.Length) will always retrun a int value
> while ((read = ch.Read(buf, 0, (int)(buf.Length> remain ? remain : buf.Length)))> 0) {
> ostream.Write(buf, 0, read);
> total += read;
> //add by iori
> remain -= read;
> e =new FtpTransferInfo(FtpTransferType.Download, remote.FullName, size, rest, total, start,false);
>
> this.OnTransferProgress(e);
> if (e.Cancel) {
> break;
> }
> }
>
> // fire one more time to let event handler know that the transfer is complete
> this.OnTransferProgress(new FtpTransferInfo(FtpTransferType.Download, remote.FullName,
> size, rest, total, start,true));
> }
> }finally {
> ostream.Flush();
> }
> }
>
> I've marked the code which I updated.
>
> Sorry about my poor English.
>
> Read the full discussion online
> <http://netftp.codeplex.com/discussions/359244#post848272>.
>
> To add a post to this discussion, reply to this email
> ([email removed]
> <mailto:[email removed]?subject=[netftp:359244]>)
>
> To start a new discussion for this project, email
> [email removed] <mailto:[email removed]>
>
> You are receiving this email because you subscribed to this discussion
> on CodePlex. You can unsubscribe or change your settings
> <https://netftp.codeplex.com/subscriptions/thread/project/edit> on
> codePlex.com.
>
> Please note: Images and attachments will be removed from emails. Any
> posts to this discussion will also be available online at codeplex.com
>

That does not work. 

            if (this._socket != null && this._socket.Available > 0 &&
                this._socket.Available < count) {
                count = this.Socket.Available;
            }

            Console.WriteLine("_socket.Available={0};count={1}"
                ,this._socket.Available,count);

I've got the infos by added code  above:

< TYPE I
> 200 Type set to I, binary mode
< EPSV
> 500 Command not recognized
< PASV
> 227 Entering Passive Mode (10,0,0,66,4,24)
< RETR /ata0/sntp/t.exe
> 150 Opening BINARY mode data connection
_socket.Available=0;count=8192
_socket.Available=8192;count=8192
_socket.Available=5120;count=5120
_socket.Available=0;count=8192
_socket.Available=2920;count=2920
_socket.Available=1460;count=1460
_socket.Available=4380;count=4380
_socket.Available=0;count=8192
_socket.Available=1460;count=1460
_socket.Available=0;count=8192
> 226 Transfer complete

Expection:

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   at System.Net.FtpClient.FtpDataStream.Read(Byte[] buffer, Int32 offset, Int32 count) in D:\Documents and Settings\Administrator\My Documents\Visual Studio 2010\Projects\System.Net.FtpClient.Test\netftp_a6ec22e2b6e7\FtpDataStream.cs:line 389
   at System.Net.FtpClient.FtpClient.Download(FtpFile remote, Stream ostream, FtpDataType datatype, Int64 rest) in D:\Documents and Settings\Administrator\My Documents\Visual Studio 2010\Projects\System.Net.FtpClient.Test\netftp_a6ec22e2b6e7\FtpClient.cs:line 1131
   at System.Net.FtpClient.FtpClient.Download(FtpFile remote, String local, FtpDataType datatype, Int64 rest) in D:\Documents and Settings\Administrator\My Documents\Visual Studio 2010\Projects\System.Net.FtpClient.Test\netftp_a6ec22e2b6e7\FtpClient.cs:line 1080
   at System.Net.FtpClient.FtpClient.Download(FtpFile remote, String local, FtpDataType datatype) in D:\Documents and Settings\Administrator\My Documents\Visual Studio 2010\Projects\System.Net.FtpClient.Test\netftp_a6ec22e2b6e7\FtpClient.cs:line 1051
   at System.Net.FtpClient.Test.Program.Main(String[] args) in D:\Documents and Settings\Administrator\My Documents\Visual Studio 2010\Projects\System.Net.FtpClient.Test\System.Net.FtpClient.Test\Program.cs:line 47

Jun 14, 2012 at 3:04 AM

By the way,I've removed the code which I Added before.

Coordinator
Jun 14, 2012 at 3:57 AM
Alright, I've implemented essentially what you had done the first time
in a way that hope doesn't affect other servers while taking care of the
issue you have discovered with VxWorks. Revision a5ee01359d01 (the
latest) has the new code. If you don't mind, please post back here and
let me know if the problem is taken care of.

Thanks
J.P.

iori13 wrote:
> From: iori13
>
> By the way,I've removed the code which I Added before.
Jun 14, 2012 at 4:43 AM
jptrosclair wrote:
Alright, I've implemented essentially what you had done the first time
in a way that hope doesn't affect other servers while taking care of the
issue you have discovered with VxWorks. Revision a5ee01359d01 (the
latest) has the new code. If you don't mind, please post back here and
let me know if the problem is taken care of.

Thanks
J.P.

iori13 wrote:
> From: iori13
>
> By the way,I've removed the code which I Added before.

Thanks for your support.

It still not work.

 

 Console.WriteLine("this.Length={0};this.Position={1};count={2};",  this.Length, this.Position, count);

 

 

I've got the infos by code above:

 

 

< TYPE I
> 200 Type set to I, binary mode
< EPSV
> 500 Command not recognized
< PASV
> 227 Entering Passive Mode (10,0,0,66,4,34)
< RETR /ata0/sntp/t.exe
> 150 Opening BINARY mode data connection
this.Length=0;this.Position=0;count=8192;
this.Length=0;this.Position=1024;count=8192;
this.Length=0;this.Position=9216;count=8192;
this.Length=0;this.Position=14336;count=8192;
this.Length=0;this.Position=15360;count=8192;
this.Length=0;this.Position=16820;count=8192;
this.Length=0;this.Position=18280;count=8192;
this.Length=0;this.Position=21200;count=8192;
this.Length=0;this.Position=22660;count=8192;
this.Length=0;this.Position=25580;count=8192;
this.Length=0;this.Position=27040;count=8192;
this.Length=0;this.Position=28672;count=8192;
> 226 Transfer complete

 

 

Exception:

 

System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   at System.Net.FtpClient.FtpDataStream.Read(Byte[] buffer, Int32 offset, Int32 count) in D:\Documents and Settings\Administrator\My Documents\Visual Studio 2010\Projects\System.Net.FtpClient.Test\netftp_a5ee01359d01\FtpDataStream.cs:line 390
   at System.Net.FtpClient.FtpClient.Download(FtpFile remote, Stream ostream, FtpDataType datatype, Int64 rest) in D:\Documents and Settings\Administrator\My Documents\Visual Studio 2010\Projects\System.Net.FtpClient.Test\netftp_a5ee01359d01\FtpClient.cs:line 1133
   at System.Net.FtpClient.FtpClient.Download(FtpFile remote, String local, FtpDataType datatype, Int64 rest) in D:\Documents and Settings\Administrator\My Documents\Visual Studio 2010\Projects\System.Net.FtpClient.Test\netftp_a5ee01359d01\FtpClient.cs:line 1082
   at System.Net.FtpClient.FtpClient.Download(FtpFile remote, String local, FtpDataType datatype) in D:\Documents and Settings\Administrator\My Documents\Visual Studio 2010\Projects\System.Net.FtpClient.Test\netftp_a5ee01359d01\FtpClient.cs:line 1053
   at System.Net.FtpClient.Test.Program.Main(String[] args) in D:\Documents and Settings\Administrator\My Documents\Visual Studio 2010\Projects\System.Net.FtpClient.Test\System.Net.FtpClient.Test\Program.cs:line 47

 

Reason:

 

        public FtpDataStream OpenRead(string path, FtpDataType datatype, long rest) {
            FtpDataStream s = this.OpenDataStream(datatype);

            //Just as you said before
            //Vxworks ftp server does not support MLST and SIZE cmd.
           //It works on Download,because the size passed by FtpFile's object.
            s.SetLength(this.GetFileSize(path));
            if (rest > 0) {
                s.Seek(rest);
            }

            if (!s.Execute("RETR {0}", path)) {
                s.Dispose();
                throw new FtpCommandException(this);
            }

            return s;
        }

 

In my opinion,there are there way to slove this problem.

1. If file Length passed by Downlad() ,just pass it to OpenRead() to create FtpDataStream;

2.Extend GetFileSize(), add a fork to get file size by FtpListType.LIST(the codes which has been added into FtpListFormatParser proved we can get filezie by  FtpListType.LIST from this server)

3.Mixed two ways. this way is performance best,but...

 

Design or function or both,I'd love to get your solution.

Jun 14, 2012 at 5:00 AM

Is there a option to block socket error 10054?

Just like this: deal with this exception as a EOF of stream by dotnet?

That will be more friendly.

Coordinator
Jun 15, 2012 at 2:25 PM
I could always catch the exception except the same exception could be thrown if the socket is closed prematurely which means a failed transfer could easily be mistaken as a successful transfer. This isn't something that I'd be willing to allow in the main distribution of the library. I'd rather figure out another way to work around the problem.

I find it odd that reading with a larger buffer than what's available on the socket would cause this exception. I believe the reason the exception happens is because the VxWorks FTP server is closing the connection before all of the data is read. Try adding this to the top of the CanRead property of FtpDataStream.cs:

if(this._socket == null || !this._socket.Connected) {
return false;
}

If that doesn't stop the exception, we can try polling the socket to see if it's still connected but care has to be taken to not cause a delay by polling too long or too much during the download. Lets and wait and see what the code above does before we go down this road.

On Jun 14, 2012, at 12:00 AM, iori13 wrote:

> From: iori13
>
> Is there a option to block socket error 10054? Just like this: deal with this exception as a EOF of stream by dotnet? That will be more friendly.