Printing latin characters not working

[George Thompson]

I have a program which works perfectly using desktop java. It prints strings which include Latin characters such as  "û"  "ý"   "þ"  "ÿ"
I have modified the program to run in Tomcat but now characters line these are simply printing as "?", each one.
I've checked the web.xml files and everything seems to be set to UTF-8
I've been using System.out.println to write to the tomcat log and FileWriter to write to files, both give the same result

George

[Tim Holloway]

UTF-8 can handle letters such as thorn. Your problem is almost certainly in what program you display the generated text with. .If you don't have an appropriate font selected, yes, you'll almost certainly see boxes, "?", or other replacement glyphs.

And for the record, I þink it's high time we brought back þe þorn! Too long has the English-printing world been tyrannized by the limitations of Shakespearean-era printing.

Just to be pedantic (again), thorn isn't actually a "Latin" character. It's English. But it's part of most Latin font sets. Also, System.out.println may be OK for crude problem solving, but I do hope you use proper logging when you get seerious about the app. It's not going to be any different in what it displays. but System.out is not officially supported in Enterprise Java and it's not as civilized as a true logger, anyway.

[George Thompson]

found a clue online:
The answer is that a java.io.FileWriter doesn't use UTF-8 encoding by default. Instead use the following code to create the writer:
def writer = new OutputStreamWriter(new FileOutputStream(outputFile),"UTF-8")
Hat tip to http://www.malcolmhardie.com/weblogs/angus/2004/10/23/java-filewriter-xml-and-utf-8/ for the answer.

[Tim Holloway]

You might find this informative: https://www.baeldung.com/java-char-encoding

Actually, in most cases, I'd be using a PrintWriter rather than a FileWriter for outputting text. The PrintWriter has the advantage of supporting OS-independent output of end-of-line characters and automatic buffer flushing of lines (which is important for console output in interactive apps).

[Campbell Ritchie]

George Thompson wrote:. . . java.io.FileWriter doesn't use UTF-8 encoding by default. . . ..

The documentation (Java14 version) says it uses

. . . either a specified charset or the platform's default charset.

This has changed since the comment about it in the blog: Java6 version of documentation.
I am a bit worried that the blog you quoted suggested mixing XYZWriters and XYZStreams.

. . . new OutputStreamWriter(new FileOutputStream(path),"UTF-8");

This is what its documentation says:-

. . . FileOutputStream is meant for writing streams of raw bytes such as image data. For writing streams of characters, consider using FileWriter. . . .

You can probably convert that to a byte stream with the output stream writer, but nowadays I would use a Formatter or this sort of code to get a buffered writer.

try
(BufferedWriter writer = Files.newBufferedWriter(
      Paths.get("~/Documents/myFile", Charset.forName("UTF-8"))))
{
   // etc
}
catch (...)
{
   //  ...
}

Plain simple

[Tim Holloway]

Actually, my go-to is a PrintWriter constructed on a FileOutputStream. It's very rare that I'd be outputting nothing but text and not have it arranged in lines.

[Mike Simmons]

For what it's worth, for some time now (JDK 1.5+) PrintWriter has had constructors that allow you to directly specify both an output file and a character encoding.   With a BufferedWriter in between for good measure. You don't have to go explicitly make 2-3 streams chained together anymore, most of the time.  But, there's not a constructor that includes autoflush with the file and encoding - if you want autoflush, you still need at least two streams.  Presumably that's why Tim still does it...

[Campbell Ritchie]

Tim, please explain more about autoflush.

Yesterday, I wrote:Plain simple

I think I was supposed to follow that with something else, but I have forgotten what.

[Tim Holloway]

Campbell Ritchie wrote:Tim, please explain more about autoflush.

Yesterday, I wrote:Plain simple

I think I was supposed to follow that with something else, but I have forgotten what.

Java borrows heavily from C, as we all know. One of the characteristics of the C printf function was that whenever a newline is encountered in the process of rendering output, the flush() function would be automatically invoked. This is why we have:

int main(int argc, char**argv) {
  printf("Hello, World!\n");
  return 0;
}

Instead of this:

int main(int argc, char**argv) {
  printf("Hello, World!\n");
  flush(stdout);
  return 0;
}

The Java PrintWriter class also conforms to this convention. As I said, it's a lot more convenient when you're writing conversational terminal applications if you don't have to explicitly invoke flush() after every println() in order to get the latest response or prompt to display.

[Campbell Ritchie]

Thank you You don't notice you ae using it all the time on standard out.

[Mike Simmons]

Well, you're probably not using autoflush all the time... unless you explicitly set it up.  PrintStream and PrintWriter have a variety of constructors - those that accept a boolean are the ones that allow you to use that boolean to force autoflush, by setting the boolean to true.  All the other constructors, however, do not enable autoflush - the default is false.

As for System.out and System.err, I don't think it's documented whether those have autoflush enabled or not.  It may be platform dependent, I think.  But I suspect they are not, in most or all cases.  In any event, you should never assume it's enabled.  If you want it, set it.  (You can even use System.setOut() to ensure it's true for all subsequent uses of System.out.)

However, be aware that you don't necessarily want autoflush enabled all the time.  It has the potential to degrade performance, especially if you're writing a long file with a lot of short lines.  That's part of why they encouraged us to include a BufferedOutputStream or BufferedWriter, back in the days before it could come bundled automatically.  If you're writing a "conversational" app as Tim says, you almost certainly do want autoflush - but if you're writing a high volume of lines to files, you probably don't.

Also, note that there's a small difference between PrintWriter and PrintStream, with respect to autoflush.  While PrintStream does ensure that with autoflush on, flushing occurs after any \n character, PrintWriter does not.  Instead, it only enforces flushing after a println, printf, or format method call.  You can still bypass it using a print or write call.  Usually that's fine, since most of us use println, printf or format most of the time anyway.  But the difference is there nonetheless...