Should Encoding property of FtpClient be settable for ftp server which does not support UTF-8.

Feb 12, 2014 at 9:03 AM
Hi, thanks for the great project. I've encountered a problem downloading files with non-ASCII file name from a FTP server which does not support UTF-8.

I noticed that the setter of Encoding property is private, and in Connect method the m_textEncoding will be set to Encoding.ASCII.

Besides, even if I modify encoding field by reflection, every time a client is cloned, the Encoding property will be lost again.

I'm wondering is there any reason do not allow developer to assign the preferred encoding, or is there any solution to solve this problem.

Thanks in advance.
San
Coordinator
Feb 12, 2014 at 12:43 PM
Please post the transaction log from the server, at least include the FEAT command and its response and the initial connection setup which tells which encoding System.Net.FtpClient uses. The encoding is setup automatically based on the server response. If the server supports UTF8 (advertised in the FEAT response) then the encoding property is changed accordingly, otherwise it's left at the default which is ASCII. Those are the only 2 encodings I've ever known to be standard with FTP so I need to see some more details to decide what might be going on.
Coordinator
Feb 12, 2014 at 12:51 PM
I should have mentioned that you can see an example of logging the session transactions in the Examples\Debug.cs file. System.Net.FtpClient uses TraceListeners for logging, you just need to add your own instance of class derived from TraceListener (such as ConsoleTraceListener as seen in the example).
Feb 13, 2014 at 5:09 AM
===================================
The code of my first try is as below.
ftp.Host = host;
ftp.Credentials = new NetworkCredential(userName, password);
var stream = ftp.OpenRead("/中文.txt");
And the log from the server is
230 User xxx logged in.
FEAT
211-FEAT
    SIZE
    MDTM
211 END
Text encoding: System.Text.ASCIIEncoding
TYPE I
200 Type set to I.
SIZE /中文.txt
550 /中文.txt: The filename, directory name, or volume label syntax is incorrect.
...
===================================
Second try: reset encoding through reflection after ftp.Connect() was called, still failed because the encoding of the cloned connection is still Encoding.ASCII.
The server log is the same as first try.
ftp.Host = host;
ftp.Credentials = new NetworkCredential(userName, password);

ftp.Connect();
// reset encoding private field after connect using reflection
var big5Encoding = Encoding.GetEncoding(950);
ftp.GetType().GetField("m_textEncoding", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(ftp, big5Encoding);
var stream = ftp.OpenRead("/中文.txt");
===================================
Third try: besides encoding, also set EnableThreadSafeDataConnections to false to prevent connection from cloning, finally succeeded.
ftp.Host = host;
ftp.Credentials = new NetworkCredential(userName, password);
ftp.EnableThreadSafeDataConnections = false;

ftp.Connect();
// reset encoding private field after connect using reflection
var big5Encoding = Encoding.GetEncoding(950);
ftp.GetType().GetField("m_textEncoding", BindingFlags.NonPublic | BindingFlags.Instance).SetValue(ftp, big5Encoding);
var stream = ftp.OpenRead("/中文.txt");
The server log:
230 User xxx logged in.
FEAT
211-FEAT
    SIZE
    MDTM
211 END
Text encoding: System.Text.ASCIIEncoding
TYPE I
200 Type set to I.
SIZE /中文.txt
213 341
...
Coordinator
Feb 13, 2014 at 11:06 AM
Alright, I'll look into adding a way to manually set the encoding today. By the way, the reason the encoding is lost when you set it with reflection is because the control connection is cloned and a new connection to the server is made for file transfers. If you set the EnableThreadSafeDataConnections property to false once you connect you should be able to override the Encoding property using Reflection like you're doing in your code above.
Coordinator
Feb 13, 2014 at 1:33 PM
Alright, the latest revision allows you to manually set the encoding property. It overrides any thing detected in the FEAT response which in this case there is nothing advertised which is why it's stuck in ASCII. This should be a straight forward way for you to change the encoding.
Feb 14, 2014 at 3:00 AM
Thank you for your fast fixing, you're awesome!!!
Feb 14, 2014 at 4:47 AM
Edited Feb 14, 2014 at 4:48 AM
I've done the testing with the code
ftp.Host = host;
ftp.Credentials = new NetworkCredential(userName, password);
ftp.Encoding = Encoding.GetEncoding(950);
ftp.Connect(); // this line must be here or it will fail.

var stream = ftp.OpenRead("/中文.txt");
Because the setter of Encoding has the following lines
// line 217 of FtpClient.cs
if (IsConnected)
    m_textEncoding = value;
I have to call ftp.Connect() explicitly to ensure that it will use the given encoding, is this by design?

thank you for your help again.
Coordinator
Feb 15, 2014 at 1:57 PM
Edited Feb 15, 2014 at 1:58 PM
You don't have to call Connect explicitly, whatever you set to the encoding property is what's used. All that statement does is say if it's already connected to go ahead and change the text encoding. Another reference to the encoding is stored on the following line to ensure it's re-used upon reconnection (clone ftp connections for file transfers). I've tested this code and to the best I can tell it works as expected; the text encoding I specify is always used.
Feb 16, 2014 at 1:12 PM
Although the setter saves the given encoding in the m_overrideEncoding variable, but if it was not connected, the m_textEncoding won't be touched. Considering the code in the CloneConnection method and Encoding getter.
internal FtpClient CloneConnection() {
    ....
    prop.SetValue(conn, prop.GetValue(this, null), null);
public Encoding Encoding {
    get {
        return m_textEncoding;
    }
    ...
}
When prop.GetValue is called on Encoding property, the getter returns m_textEncoding which is still ASCII.

Try the following test code:
var ftp = new FtpClient();
ftp.Encoding = Encoding.GetEncoding(950);
var clonedFtp = ftp.CloneConnection();
Debug.Assert(clonedFtp.Encoding.CodePage == 950);
Coordinator
Feb 16, 2014 at 1:29 PM
Yes but the encoding is set when a connection is made. Look in the Connect method. You can verify the encoding from the transaction log.

Sent from my iPhone

Coordinator
Feb 16, 2014 at 1:43 PM
I think that you're over analyzing the code, the Encoding to be used is setup upon connection (Connect method). It defaults to ASCII and when you set the encoding manually it's assigned to a variable other than m_textEncoding which is then used in the Connect() method rather than using ASCII or UTF8. Show me a transaction log where the encoding you manually set is not maintained. The encoding being used is logged to the trace listeners in FtpTrace upon connection so that is the way to verify that the proper encoding is being used.

The reason it's done this way is because any encoding other than ASCII and UTF8 are non standard. System.Net.FtpClient automatically handles the encoding in standard situations. The value that you assign to the Encoding property goes unused until a real connection has been made in which case the Encoding property should properly depict what you specified. This holds true to cloned connections. In other words, a new instance of FtpClient that is not connected to a server will always return ASCII from the Encoding property no matter what you specify. When you Connect, if you specified an encoding to use regardless of standards, then that encoding is assigned to m_textEncoding and used unless you change it again.
Feb 17, 2014 at 1:47 AM
OK, I see, thanks for your detailed explanation. But the exception is still happened.

Here is the code which will fail,
using (FtpClient cl = new FtpClient())
{
    cl.Host = "host";
    cl.Credentials = new NetworkCredential("uuu", "passwd");
    cl.Encoding = Encoding.GetEncoding(950);

    var stream = cl.OpenRead("/中文.txt");
}
If OpenRead(), OpenWrite() or OpenAppend() was called immediately after setup, it will fail.
Feb 17, 2014 at 1:49 AM
The full transaction log of code above.
'Tests.vshost.exe' (CLR v4.0.30319: Tests.vshost.exe): Loaded.
InterNetwork: xxx.xxx.xxx.xxx
220 Microsoft FTP Service
USER uuu
331 Password required for uuu.
PASS <omitted>
230 User uuu logged in.
FEAT
211-FEAT
    SIZE
    MDTM
211 END
Text encoding: System.Text.DBCSCodePageEncoding
InterNetwork: xxx.xxx.xxx.xxx
220 Microsoft FTP Service
USER uuu
331 Password required for uuu.
PASS <omitted>
230 User uuu logged in.
Text encoding: System.Text.ASCIIEncoding
PWD
257 "/" is current directory.
CWD /
250 CWD command successful.
TYPE I
200 Type set to I.
SIZE /中文.txt
550 /??.txt: The filename, directory name, or volume label syntax is incorrect. 
EPSV
500 'EPSV': command not understood
PASV
227 Entering Passive Mode (203,242,169,32,15,173).
InterNetwork: xxx.xxx.xxx.xxx
RETR /中文.txt
550 /??.txt: The filename, directory name, or volume label syntax is incorrect. 
QUIT
221  
Disposing FtpClient object...
Disposing FtpSocketStream...
A first chance exception of type 'System.Net.FtpClient.FtpCommandException' occurred in System.Net.FtpClient.dll
Disposing FtpClient object...
QUIT
221  
Disposing FtpSocketStream...
Coordinator
Feb 17, 2014 at 2:08 AM
Alright thanks for posting the transaction log. I will dig deeper in to the problem tomorrow and let you know what I come up with. I kinda feel that the code behind this is getting too tricky so I might just take the time to rethink the way the encoding is handled all together to remove the surprises like this one.

Sent from my iPhone

Coordinator
Feb 17, 2014 at 8:58 PM
Edited Feb 17, 2014 at 9:01 PM
Alright, I've pushed up a new revision. Here's the transaction from my tests using Encoding.Default which defaults to SBCSCodePageEncoding on my win 7 64 machine; let me know how it works for you:
InterNetwork: 10.0.1.2
220 xyz FTP server (Version 6.00LS) ready.
USER ftptest
331 Password required for ftptest.
PASS <omitted>
230 User ftptest logged in, access restrictions apply.
FEAT
211- Extensions supported:
 MDTM
 REST STREAM
 SIZE
211 End.
--> __Text encoding: System.Text.SBCSCodePageEncoding__
InterNetwork: 10.0.1.2
220 xyz FTP server (Version 6.00LS) ready.
USER ftptest
331 Password required for ftptest.
PASS <omitted>
230 User ftptest logged in, access restrictions apply.
--> __Text encoding: System.Text.SBCSCodePageEncoding__
PWD
257 "/" is current directory.
CWD /
250 CWD command successful.
TYPE I
200 Type set to I.
SIZE test.txt
213 0
EPSV
229 Entering Extended Passive Mode (|||63046|)
InterNetwork: 10.0.1.2
STOR test.txt
150 Opening BINARY mode data connection for 'test.txt'.
226 Transfer complete.
QUIT
221 Goodbye.
Disposing FtpClient object...
Disposing FtpSocketStream...
Disposing FtpSocketStream...
Disposing FtpClient object...
QUIT
221 Goodbye.
Disposing FtpSocketStream...
Feb 18, 2014 at 6:07 AM
Didn't see the commit you mentioned, the latest commit is still d298615.
Coordinator
Feb 18, 2014 at 12:34 PM
Sorry, forgot to push, it's there now.
Marked as answer by bigsan on 2/18/2014 at 11:05 PM
Feb 19, 2014 at 6:07 AM
It works great, thank you.