Line ending with file stream

[Fabian Cenedese]

Hi

I have a question about writing native text files. I simply want to write
text files that have native line endings on Windows or Linux. Trying
several wx classes (wxFile, wxTextFile with Write("\n") or
wxFFileOutputStream, wxTextOutputStream with << endl)
all lead to the same error. The written file on disk has 0x0D 0x0D 0x0A
instead of 0x0D 0x0A for Windows.

I traced it down into the crt of MSVC 2010. In crt/src/write.c
there's this code/comment:

} else if ( _osfile(fh) & FTEXT ) {
/* text mode, translate LF's to CR/LF's on output */

So the c library itself translates 0x0A to 0x0D 0x0A.
Together with e.g. this code:

void wxTextOutputStream::WriteString(const wxString& string)
...
if ( c == wxT('\n') )
{
switch ( m_mode )
{
case wxEOL_DOS:
out << wxT("\r\n");
continue;

we end up with double-D's :) I'm wondering now if nobody else has
this problem. Does that mean that we have to use binary mode even
for text files? What's the best way to write native line endings on big
text files? wxTextFile should only be used for smaller files.

Thanks

bye Fabi

[Vadim Zeitlin]

On Wed, 24 Sep 2014 16:30:45 +0200 Fabian Cenedese wrote:

FC> Does that mean that we have to use binary mode even for text files?

If you use wxTextFile &c you must use binary mode as it's impossible to
create Unix-style EOLs under Windows otherwise. If you use wx[F]File you
can use text mode perfectly well.

Regards,
VZ

[Fabian Cenedese]

Post by Vadim Zeitlin
FC> Does that mean that we have to use binary mode even for text files?
If you use wxTextFile &c you must use binary mode as it's impossible to
create Unix-style EOLs under Windows otherwise. If you use wx[F]File you
can use text mode perfectly well.

I don't want to have Unix-EOL on Windows, just normal native EOLs.
However wxFile::Open always uses binary mode, so writing
Write("\n"); just gives 0x0A, good for Unix but not for Windows.
So I would have to make the OS distinction myself which I'd like
not to do, that's why I'm using wxWidgets.

wxTextFile could be used for text mode but preferably small files.

For writing large files wxFFileOutputStream/wxTextOutputStream
might be the right thing to use as here the EOL is used
depending on the system. But interestingly they also use
binary mode, maybe exactly because of the problem I got
if creating the file in text mode.

This solution seems to work for me (at least on Windows,
need to test Linux as well). I just thought that a little
hint in the docs might be useful how a file or stream
must be created to get the desired result, especially
as it's not obvious to use binary mode to write native
line endings in text files. Text mode doesn't work on
Windows (at least with VC2010) because of the additional
conversion in the crt as described in my last mail.

Thanks

bye Fabi

[Vadim Zeitlin]

On Wed, 24 Sep 2014 17:23:48 +0200 Fabian Cenedese wrote:

FC> At 16:43 24.09.2014, Vadim Zeitlin wrote:
FC> >On Wed, 24 Sep 2014 16:30:45 +0200 Fabian Cenedese wrote:
FC> >
FC> >FC> Does that mean that we have to use binary mode even for text files?
FC> >
FC> > If you use wxTextFile &c you must use binary mode as it's impossible to
FC> >create Unix-style EOLs under Windows otherwise. If you use wx[F]File you
FC> >can use text mode perfectly well.
FC>
FC> I don't want to have Unix-EOL on Windows, just normal native EOLs.

Yes, I understand this. I was explaining why you had to use binary mode
files with any classes that allow you choosing the EOL kind.

FC> However wxFile::Open always uses binary mode,

Yes, sorry for forgetting this, it does, so you shouldn't use it. You
still should/could use all the rest mentioned in my previous email.

FC> So I would have to make the OS distinction myself which I'd like
FC> not to do, that's why I'm using wxWidgets.

Or just use wxFFile. Or std::ofstream for that matter. Both of them do
what you want just fine, there is really no need to do something
complicated.

FC> wxTextFile could be used for text mode but preferably small files.

wxTextFile is worth using if you want to handle different EOLs when
reading the file. If you're just interested in writing it, it's useless for
you.

FC> For writing large files wxFFileOutputStream/wxTextOutputStream
FC> might be the right thing

They work too but you don't really need them just to create a text file.
If you prefer stream API, just use std::ofstream. If you prefer C-like API,
use wxFFile.

FC> This solution seems to work for me (at least on Windows,
FC> need to test Linux as well). I just thought that a little
FC> hint in the docs might be useful how a file or stream
FC> must be created to get the desired result, especially
FC> as it's not obvious to use binary mode to write native
FC> line endings in text files. Text mode doesn't work on
FC> Windows (at least with VC2010) because of the additional
FC> conversion in the crt as described in my last mail.

Sorry, I think you're confused somewhere (although I don't really see
where exactly). Text mode works just fine. And so does binary. You get
exactly what you would expect from both of them. The only thing which
indeed does not work is using wxFile for text files.

Regards,
VZ

[Fabian Cenedese]

Post by Vadim Zeitlin
FC> For writing large files wxFFileOutputStream/wxTextOutputStream
FC> might be the right thing
They work too but you don't really need them just to create a text file.
If you prefer stream API, just use std::ofstream. If you prefer C-like API,
use wxFFile.

I ended up using wxFFile in text mode which indeed does what I
need, I don't know what went wrong when I first tried.

The problems I had came from using wxFFile in text mode
together with wxFFileOutputStream and wxTextOutputStream
so the EOL conversion was made twice (once in wx, once in crt).
That only worked correctly with wxFFile in binary mode which
didn't seem obvious to me. But then again that's maybe not
how these classes should be used.

Thanks

bye Fabi

[Vadim Zeitlin]

On Thu, 25 Sep 2014 10:40:21 +0200 Fabian Cenedese wrote:

FC> The problems I had came from using wxFFile in text mode
FC> together with wxFFileOutputStream and wxTextOutputStream
FC> so the EOL conversion was made twice (once in wx, once in crt).
FC> That only worked correctly with wxFFile in binary mode which
FC> didn't seem obvious to me. But then again that's maybe not
FC> how these classes should be used.

Things do work by default as wxFFileOutputStream default mode is "wb"
and so wxTextOutputStream with its default wxEOL_NATIVE mode works as
expected. But it's true that it's not very nice that things break if you
use "w" (or "wt") with wxFFileOutputStream instead. I'm not sure what, if
anything, can be done about this though. We can't really detect the mode of
the underlying stream in wxTextOutputStream (it might not be
wxFFileOutputStream at all), so the best we could do probably would be to
document that you can use wxFFileOutputStream("wb") +
wxTextOutputStream(wxEOL_NATIVE) or wxFFileOutputStream("w") +
wxTextOutputStream(wxEOL_UNIX) but that the other combination doesn't work.
If you feel like writing this up for wxTextOutputStream description in
interface/wx/txtstream.h, please don't hesitate to submit a patch to the
documentation.

Regards,
VZ