Trouble removing spaces in ArrayList

[Mike London]

Hello,

I have an array list with these string elements:

"     1"
"2"
"3"
"4    "

But, running this code does not remove the spaces:

 List<String> listWithoutSpaces =  list.stream()
                .map(String::trim)
                .collect(Collectors.toList());

Even in the debugger, when this code runs, the spaces are still present.

The lists are not "final" or any other problem cause I can think of.

Suggestions?

Thanks,

-- mike

[Mike Simmons]

Your code really looks like it should remove the spaces.  Is it possible you're still looking at the original list, rather than the new one?  If not, can you show more code of where this is called and how you know it's not working?

[Ron McLeod]

The code looks fine.  Are you sure that the blank characters are actually spaces?

var list = List.of("     1", "2", "3", "4    ");
List<String> listWithoutSpaces = list.stream()
        .map(String::trim)
        .collect(Collectors.toList());
listWithoutSpaces.forEach(item -> System.out.format("!%s!%n", item));

!1! !2! !3! !4!

[Ron McLeod]

Ron McLeod wrote:Are you sure that the blank characters are actually spaces?

For example:

var nbsp = "\u00a0";
var list = List.of(nbsp + "     1", "2", "3", "4    ");
List<String> listWithoutSpaces = list.stream()
        .map(String::trim)
        .collect(Collectors.toList());
listWithoutSpaces.forEach(item -> System.out.format("!%s!%n", item));

!      1! !2! !3! !4!

[Mike London]

Maybe it's how I'm creating the list that is the problem. It "looks" like spaces but this is in a SpringBoot.

Below is a section of the code that builds the UTF-8 list from either of two at the top:

 if (encoding.toUpperCase().matches("ISO_8859_2|ISO_8859_1"))
            {
                lines = Files.readAllLines(
                                filePath, StandardCharsets.ISO_8859_1)
                        .stream()
                        .filter(line -> !line.trim().matches("^\\s*$"))   // filter out empty lines
                        .collect(Collectors.toList());

                // In this case, convert and add to list in UTF-8 format
                for (String line : lines)
                {
                    utf8Lines.add(new String(
                            line.getBytes(),
                            StandardCharsets.UTF_8));
                }

This must be where the problem is.

If I just add an ArrayList in code, and add values to it, the stream() API works fine.

Suggestions?

Thanks,

- mike

[Mike London]

Here's the value of  "       1"       (that refuses to trim)

after this code executes:

utf8Lines = Files.readAllLines(
                               filePath, StandardCharsets.UTF_8)
                       .stream()
                       .filter(line -> !line.matches("^\\s*$"))     // skip blank lines
                       .collect(Collectors.toList());

[-1, -2, 32, 0, 32, 0, 32, 0, 32, 0, 32, 0, 32, 0, 49, 0]

Doesn't seem to be UTF-8, right?

Confusing...

Thanks,

-- mike

[Paul Clapham]

I recently spent quite a while trying to find why two strings from two totally different external sources appeared to be identical but were actually not. The strings were only two words long, how could they be different? It turned out that one of them had a non-breaking space between the two words -- took me a long time to find that.

[Mike London]

Paul Clapham wrote:I recently spent quite a while trying to find why two strings from two totally different external sources appeared to be identical but were actually not. The strings were only two words long, how could they be different? It turned out that one of them had a non-breaking space between the two words -- took me a long time to find that.

It looks like my "      1" string isn't the expected value after my (attempted) "UTF-8" conversion.

Not sure what's off.

-- mike

[Paul Clapham]

Mike London wrote:Doesn't seem to be UTF-8, right?

More like UTF-16, with some kind of marker at the beginning.

[Ron McLeod]

If you can determine which characters are causing you grief, you could remove them before using trim.  U+00A0, U+2007, and U+202F are the likely ones.

lines = Files.readAllLines(filePath, StandardCharsets.ISO_8859_1)
        .stream()
        .map(item -> item.replaceAll("\\x{00A0}|\\x{2007}|\\x{202F}", ""))
        .filter(line -> !line.trim().matches("^\\s*$"))   // filter out empty lines
        .collect(Collectors.toList());

[Mike London]

Ron McLeod wrote:If you can determine which characters are causing you grief, you could remove them before using trim.  U+00A0, U+2007, and U+202F are the likely ones.

lines = Files.readAllLines(filePath, StandardCharsets.ISO_8859_1)
        .stream()
        .map(item -> item.replaceAll("\\x{00A0}|\\x{2007}|\\x{202F}", ""))
        .filter(line -> !line.trim().matches("^\\s*$"))   // filter out empty lines
        .collect(Collectors.toList());

That's a good idea, I'll investigate ... but here's the thing... this should work (famous last words, I know).

In BBEdit, I created a simple text file:

   1
2
3
4

In a standalone Java program -> should convert to UTF-8, so why the Unicode characters?:

public class ConvertEncoding
{
    public static void main(String[] args)
    {
        List<String> list = new ArrayList<>();
        try (BufferedReader reader = new BufferedReader(new InputStreamReader(
                new FileInputStream("path_here_to_text_file"), StandardCharsets.UTF_8)))
        {
            String line;
            while ((line = reader.readLine()) != null)
            {
                list.add(line.trim());
            }
        }

        catch (IOException e)
        {
            e.printStackTrace();
        }

        System.out.println();
    }
}

Putting the break point on the sop, I get:

0 = "    1"
1 = "2"
2 = "3"
3 = "4"

WTF?

[Ron McLeod]

Can you share a file (attachment to post, not text in post) which is problematic?

[Mike London]

Ron McLeod wrote:Can you share a file (attachment to post, not text in post) which is problematic?

Sure, attached.

Just set the path to the text file (list1.txt) on your system, set a breakpoint on the SOP and note that no trimming occurred of the first value.

Look forward to hearing your results.

Thanks Ron.

-- mike

[Ron McLeod]

It looks like that file is UTF-8 with a BOM sequence.

I tried this and it did work.  It helps show what the problem is, but is a bit of a hack and a better solution should be used.

final var bom = new String(new byte[] { (byte) 0xef, (byte) 0xbb, (byte) 0xbf });
var filePath = Path.of("/tmp/list1.txt");
var list = Files.readAllLines(filePath, StandardCharsets.UTF_8)
        .stream()
        .map(item -> item.replace(bom, ""))
        .filter(line -> !line.trim().matches("^\\s*$")) // filter out empty lines
        .collect(Collectors.toList());

[Mike London]

Ron McLeod wrote:It looks like that file is UTF-8 with a BOM sequence.

I tried this and it did work.  It helps show what the problem is, but is a bit of a hack and a better solution should be used.

final var bom = new String(new byte[] { (byte) 0xef, (byte) 0xbb, (byte) 0xbf });
var filePath = Path.of("/tmp/list1.txt");
var list = Files.readAllLines(filePath, StandardCharsets.UTF_8)
        .stream()
        .map(item -> item.replace(bom, ""))
        .filter(line -> !line.trim().matches("^\\s*$")) // filter out empty lines
        .collect(Collectors.toList());

Thanks. Weird. I just created two new BBEdit files and they worked, too.

Don't you love Encoding issues?

Thanks very much.

-- mike

[Campbell Ritchie]

Ron McLeod wrote:. . . remove them before using trim.  U+00A0, U+2007, and U+202F are the likely ones. . . .

It is a long time since I used trim() and I might be mistaken, but I believe it doesn't remove such characters as hard space (\u00a0). Try String#strip() instead.

[Mike London]

Campbell Ritchie wrote:

Ron McLeod wrote:. . . remove them before using trim.  U+00A0, U+2007, and U+202F are the likely ones. . . .

It is a long time since I used trim() and I might be mistaken, but I believe it doesn't remove such characters as hard space (\u00a0). Try String#strip() instead.

Cool tip, thank you! Will experiment.

-- mike

[Campbell Ritchie]

Mike London wrote:. . . a better solution should be used. . . .

Try the isBlank() method without bothering to trim or strip anything.

Don't you love Encoding issues? . . .

They are better than beer, aren't they

[Campbell Ritchie]

I tried printing all the chars in that text file and got this:-

\ufeff \u0020 \u0020 \u0020 \u0020 \u0031
\u0032
\u0033
\u0034

I think you are right that the byte order mark is the real problem.

[Campbell Ritchie]

Output from my program, as before:-

~/java$ java EncodingDemo.java /run/media/campbell/TOSHIBA/Trimming-issue/list1.txt
showChars() method
\ufeff \u0020 \u0020 \u0020 \u0020 \u0031
\u0032
\u0033
\u0034

Tried a few tweaks. All kludgy to the worst degree./* Replace byte order mark with hard space */ import java.io.IOException; import java.io.BufferedReader; import java.nio.file.Paths; import java.nio.file.Files; class EncodingDemo {    public static void main(String... args)    {        new EncodingDemo(args[0]).showChars();    }    private EncodingDemo(String fileName)    {        this.fileName = fileName;    }    private final String fileName;    private void showChars()    {        try (BufferedReader reader = Files.newBufferedReader(Paths.get(fileName)))        {            System.out.println("showChars() method");            String line;            while ((line = reader.readLine()) != null)            {                if (line.charAt(0) >= 0xfeff)                {                    line = "\u00a0" + line.substring(1);                }                char[] chars = line.toCharArray();                for (char c : chars)                {                    System.out.printf("\\u%04x ", +c);                }                System.out.println();            }        }        catch (IOException ex)        {            ex.printStackTrace();        }    } }

java$ java EncodingDemo.java /run/media/campbell/TOSHIBA/Trimming-issue/list1.txt
showChars() method
\u00a0 \u0020 \u0020 \u0020 \u0020 \u0031
\u0032
\u0033
\u0034

/* In addition, use strip() */ import java.io.IOException; import java.io.BufferedReader; import java.nio.file.Paths; import java.nio.file.Files; class EncodingDemo {    public static void main(String... args)    {        new EncodingDemo(args[0]).showChars();    }    private EncodingDemo(String fileName)    {        this.fileName = fileName;    }    private final String fileName;    private void showChars()    {        try (BufferedReader reader = Files.newBufferedReader(Paths.get(fileName)))        {            System.out.println("showChars() method");            String line;            while ((line = reader.readLine()) != null)            {                if (line.charAt(0) >= 0xfeff)                {                    line = ("\u00a0" + line.substring(1)).strip();                }                char[] chars = line.toCharArray();                for (char c : chars)                {                    System.out.printf("\\u%04x ", +c);                }                System.out.println();            }        }        catch (IOException ex)        {            ex.printStackTrace();        }    } }Output unchanged Even better, guess what I got from JShell!

jshell> Character.isWhitespace((char)0xa0)
$1 ==> false

So the hard space doesn't seem to count as whitespace./* Omit the hard space */ import java.io.IOException; import java.io.BufferedReader; import java.nio.file.Paths; import java.nio.file.Files; class EncodingDemo {    public static void main(String... args)    {        new EncodingDemo(args[0]).showChars();    }    private EncodingDemo(String fileName)    {        this.fileName = fileName;    }    private final String fileName;    private void showChars()    {        try (BufferedReader reader = Files.newBufferedReader(Paths.get(fileName)))        {            System.out.println("showChars() method");            String line;            while ((line = reader.readLine()) != null)            {                if (line.charAt(0) >= 0xfeff)                {                    line = line.substring(1).strip();                }                char[] chars = line.toCharArray();                for (char c : chars)                {                    System.out.printf("\\u%04x ", +c);                }                System.out.println();            }        }        catch (IOException ex)        {            ex.printStackTrace();        }    } }Now we have got what you want

/java$ java EncodingDemo.java /run/media/campbell/TOSHIBA/Trimming-issue/list1.txt
showChars() method
\u0031
\u0032
\u0033
\u0034

[Carey Brown]

Following along with all of this I get...

import java.nio.file.Path;
import java.nio.file.Files;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.stream.Collectors;

public class TrimmingIssue
{
	static final int	BOM = 0xfeff;
	
	public static void main( String... args ) throws IOException
	{
		var filePath = Path.of( "data/Trimming-issue.txt" );
		var list = Files.readAllLines( filePath, StandardCharsets.UTF_8 )
				.stream()
				.map( line -> ( line.charAt(0) >= BOM
					? line.substring(1)
					: line
					).strip() )
				.filter( line -> ! line.isBlank() )
				.collect( Collectors.toList() );
		
		list.forEach( s -> System.out.println( "!" + s + "!" ) );
	}
}

[Ron McLeod]

In the example file that ML posted, the BOM sequence was ef bb bf (UTF-8), not fe ff (UTF-16).
od -tx1 ./list1.txt 0000000 ef bb bf 20 20 20 20 31 0a 32 0a 33 0a 34

[Carey Brown]

Yes, I tried that BOM replacement and it didn't work but my above code did. And I just ran 'od' on my download of the file and I got the same thing you did.

[Ron McLeod]

Ron McLeod wrote:In the example file that ML posted, the BOM sequence was ef bb bf (UTF-8), not fe ff (UTF-16).
od -tx1 ./list1.txt 0000000 ef bb bf 20 20 20 20 31 0a 32 0a 33 0a 34

char uses UTF-16, and BOM is represented in UTF-16 as 0xfeff, so comparing to 0xfeff is correct when work with a char.

[Ron McLeod]

Here's another approach (although with more code) using a BufferedReader and peeking at the first char in the buffer.

void removeSpacesTest() throws IOException {
    var filePath = Path.of("/tmp/list1.txt");
    var reader = Files.newBufferedReader(filePath, StandardCharsets.UTF_8);
    var list = removeBom(reader)
            .lines()
            .filter(line -> !line.isBlank())
            .map(String::strip)
            .collect(Collectors.toList());

    list.forEach(item -> System.out.format("!%s!%n", item));
}

BufferedReader removeBom(BufferedReader reader) throws IOException {
    final var bom = Character.toChars(0xfeff);
    final var bLen = bom.length;
    reader.mark(bLen);
    var cbuf = new char[bLen];
    var rLen = reader.read(cbuf, 0, bLen);
    if (rLen < bLen || Arrays.compare(bom, cbuf) != 0) {
        reader.reset(); // no BOM - reset stream back to beginning
    }
    return reader;
}