Byte[] to String and then again to Byte[]

[Partheban Udayakumar]

Hi,

I want to store a byte[] in a database and then read it back as byte[]. I have tried the following:

Converting byte[] to string:

the byte array contains the following when i tried to print it as string : [B3af456

String byteString = bytAr.toString;

Then to convert String to byte Array I tried this:

byte[] retByteAr = byteString. getBytes("UTF-8");

But I get different byte arrays. What can I do to retrive the same byte array?

[Ulf Dittmer]

bytAr.toString

That's not how you convert a byte array to a string. The fact that the resulting string is much shorter than the original array (for anything but short arrays) should tip you off.

You need to use something like base-64 encoding to convert a byte[] to a storable string representation, and later back to the original array.

[Partheban Udayakumar]

Ulf,

Thanks. I knew that Base 64 encoded byte array can be used as String, I forgot. Just know remembered it. Sorry. Do we have any other way to convert it?

I have one more request, take a look at this.

I need more clarification on my answer. It will be better if experts like you answer it.

[Ulf Dittmer]

What should this other way of converting do differently? Or, to put it another way, how does using base-64 not solve the problem adequately?

[Partheban Udayakumar]

Ulf,

Base 64 conversion solves the problem. I am just interested in knowing about the other ways.

I use Base 64 encoder provided by bouncy castle as I am developing a JME app. When I use it to encode the bytes, it gives the same output. As the bytes which it encodes is also auto generated, I have no idea whether the bytes are same or this encoder gives the same encoded value for all the bytes. Can you help me with this?

[Ulf Dittmer]

What stops you from comparing the bytes and/or strings to see whether they're the same? That aside, base-64 is a standard algorithm, it better be the same across implementations.

[Richard Tookey]

I would have thought that one should use a BLOB to store the bytes in the DB then no encoding is necessary!

ASCii85 is another method of encoding bytes to ASCII. Base64 inflates the size by a factor of about 4/3 but ASCii85 with the two prefix characters removed inflates by only 5/4 .

[Partheban Udayakumar]

Ulf,

I checked with the random generated inputs, they are different but the encoder always gives the same encode and the decoded output is also different but it is shortened by one or two characters.

First Set:

Input: [B@ffed4784
Output: [B@1fb15e

Second Set:

Input: [B@49ea68f
Output: [B@1c968f

Richard,

Can you enlighten me on ASCii85 encoders and decoders? Have you used it in JME?

[Richard Tookey]

Partheban Udayakumar wrote:Ulf,

I checked with the random generated inputs, they are different but the encoder always gives the same encode and the decoded output is also different but it is shortened by one or two characters.

First Set:

Input: [B@ffed4784
Output: [B@1fb15e

Second Set:

Input: [B@49ea68f
Output: [B@1c968f

The toString() method of an array does not reflect the content of the array but produces some pseudo address of the array. To look at the content you should use one of the static toString(byte[]) methods in the Arrays class.

Richard,

Can you enlighten me on ASCii85 encoders and decoders? Have you used it in JME?

As you are aware I know little or nothing about JME but I have written a set of ASCii85, Base64 and Hex encoders that use nothing but bytes and Strings which I'm pretty sure are available in JME. If you Google for Java ASCii85 you should find implementations.

But why do you think you need to encode anything? Surely you just need to ship the bytes as raw bytes (I do hope you are not sending everything as characters/Strings)!

[Ulf Dittmer]

Partheban Udayakumar wrote:I checked with the random generated inputs, they are different but the encoder always gives the same encode and the decoded output is also different but it is shortened by one or two characters.

I had mentioned in my first post that the byte[].toString method is not the right approach. Any further work spent on that is futile.

[Partheban Udayakumar]

Ulf and Richard,

I am not able to use java.util.Arrays.toString() method. I tried using Arrays class in org.bouncycastle.util.Arrays but it doesn't have toString() implementation.

[Ulf Dittmer]

I still don't understand why the solutions offered so far -encoding via base-64 or similar, or storing as BLOB in the DB- would not solve the problem. What else are you looking for?

[Partheban Udayakumar]

Ulf and Richard,

Both the suggestions worked. Thank You.

But I have a new question for you.

I tried the following code. Please take a look at this:

String AESkeygen() {
        SecretKey sk;
        String r = null;
        byte[] iv = new byte[16];
        try {
            KeyGenerator keygen = KeyGenerator.getInstance("AES");
            SecureRandom sr = new SecureRandom();
            keygen.init(256);
            sr.nextBytes(iv);
            sk = keygen.generateKey(); 
            byte[] aKey = sk.getEncoded();
            comString = Arrays.toString(aKey) + "<;" + Arrays.toString(iv); 
            System.out.println("Combined String: "+ comString);
            String[] red = comString.split("<;");
            System.out.println("Red[0]: "+ red[0]);
            System.out.println("Red[1]: "+ red[1]);
            byte[] retKey = red[0].getBytes();
            byte[] retIv = red[1].getBytes();
            System.out.println("Original KEY Set");
            System.out.println("Key: "+ Arrays.toString(aKey));
            System.out.println("IV: "+Arrays.toString(iv));
            System.out.println("Transported Key Set");
            System.out.println("Key: "+ Arrays.toString(retKey));
            System.out.println("IV: "+Arrays.toString(retIv));
        } catch (Exception ex) {
            ex.printStackTrace();
        }
   return comString;
}

I don't get errors but the bytes printed in the output (Original Key Set and Transported Key Set) are different. But after split both remain same. I think the problem is with getBytes("UTF-8"). Do you guys have any idea over this issue.

The problem here is I am generating keys in one class and then combining the key and iv as a string and sending it to the servlet to send it to the client. Can we send two or more byte[] to the class? In that way the conversion can be avoided.

[Ulf Dittmer]

You can't treat binary data like you would treat character data, like converting it to strings the way Arrays.toString does.

What are you trying to accomplish?

[Campbell Ritchie]

Not a beginner's problem: moving to another forum.

[Partheban Udayakumar]

Ulf,

I am generating AES key in my server (because I couldn't generate it in JME which is due to the absence of Secure Random class) and sending it to client to encrypt the data. Then I send back the key and encrypted message to store it in a database which can be retrieved by the receiver to decrypt the message. I solved the key transfer problem but the problem now in hand is, I store the received message in a vector, the only collection I could find in JME. Now I cannot convert the object which it returns to an byte array. I tried using ObjectOutputStream to convert it to byte[] but it is not available in JME.

[Richard Tookey]

Partheban Udayakumar wrote:

            byte[] aKey = sk.getEncoded();
            comString = Arrays.toString(aKey) + "<;" + Arrays.toString(iv); 
            System.out.println("Combined String: "+ comString);
            String[] red = comString.split("<;");
            System.out.println("Red[0]: "+ red[0]);
            System.out.println("Red[1]: "+ red[1]);
            byte[] retKey = red[0].getBytes();
            byte[] retIv = red[1].getBytes();

This is just silly! You are using Arrays.toString() to create the String and then expecting red[0].getBytes() to get back the original bytes. Just look at the result of the Arrays.toString() and you will see that it create a comma separated list of decimal values with a prefix and suffix. How do you expect red[0].getBytes() to reverse that? The way to reverse that would be create a simple parse but why would you want to use this approach anyway?

Some simple rules I use when dealing with binary data (crypto keys, ciphertext and IV values are binary data) -

1) Try to keep binary data as bytes.
2) When you write binary data to a channel (file or socket or whatever) first write as a prefix the length and then the bytes. DataOutputStream and DataInputStream assist with this.
3) When storing binary data in a database use a BLOB.
4) When you absolutely must have a string representation of binary data then use Base64 or Hex or ASCii85 encoding. Never ever ever ever use new String(binary data).

[Ulf Dittmer]

It seems we've gone over all this before; Arrays.toString is not the way to convert binary data to strings - something like base-64 is.

[Richard Tookey]

Partheban Udayakumar wrote:Ulf,

I am generating AES key in my server (because I couldn't generate it in JME which is due to the absence of Secure Random class) and sending it to client to encrypt the data. Then I send back the key and encrypted message to store it in a database which can be retrieved by the receiver to decrypt the message.

So you are still sending the key over an insecure channel!

I solved the key transfer problem

If you are sending the key over an insecure channel you have most definitely not solved the transfer problem! Just the opposite!

but the problem now in hand is, I store the received message in a vector, the only collection I could find in JME. Now I cannot convert the object which it returns to an byte array. I tried using ObjectOutputStream to convert it to byte[] but it is not available in JME.

I would say to use DataOutputStream but since the whole approach is just plain wrong I won't !

[Partheban Udayakumar]

Richard Tookey wrote:
So you are still sending the key over an insecure channel!

I am using HTTPS channel to transfer data.

Richard Tookey wrote:
If you are sending the key over an insecure channel you have most definitely not solved the transfer problem!

Ya I am not able to retrieve the byte[] (which I stored in database as VarBinary) using Input Stream and Byte Array Output Stream

Richard Tookey wrote:
I would say to use DataOutputStream but since the whole approach is just plain wrong I won't !

There is neither DataOutputStream.writeBytes() method in JME nor any method to convert object to byte[] in JME

Ulf Dittmer wrote:
It seems we've gone over all this before; Arrays.toString is not the way to convert binary data to strings

I just used it to compare the byte[] sent and received. I am not using it in my program

Out of curiosity, what is the way to do that? Base 64 encodes that and then we store it as a String. What if I want to convert a byte[] to String without encoding?

[Richard Tookey]

Partheban Udayakumar wrote:

Richard Tookey wrote:
So you are still sending the key over an insecure channel!

I am using HTTPS channel to transfer data.

Then I am totally lost! This all makes no sense at all! Why why why do you think you need to AES encrypt the data if you are already encrypting the channel using HTTPS?

[Partheban Udayakumar]

Richard,

My client wants it. I don't know for what reasons. But he still says he wants it.

Do you have any ideas on how to retrieve the byte[] stored in a db. I used the following code:

int n;
        byte[] a = {} ;
        InputStream is;   //I actually use is = ResultSet.getBinaryStream(1);
        ByteArrayOutputStream bos = new ByteArrayOutputStream();                
        try {
            System.out.println("InputStream: "+ is.available());  //This returns 44
            is.read(a,0,a.length);                                
                bos.write(a);
                        
        } catch (IOException ex) {
            Logger.getLogger(Retr1.class.getName()).log(Level.SEVERE, null, ex);
        }        
        byte[] retByteArr= bos.toByteArray();

On printing, it shows [0,0,0,0....]

[Tim Holloway]

Partheban Udayakumar wrote: What if I want to convert a byte[] to String without encoding?

Meaningless question. ALL strings have an encoding. It's why "A" in ASCII isn't the same byte value as "A" in EBCDIC. The standard encoding for java.lang.String is Unicode, but to convert an array of bytes to a java.lang,String, a code conversion has to be applied, since bytes (as defined by Java) are 8 bits long, but a Unicode is a 16-bit code having the same values for the lowest 256 values as ASCII (more or less - ASCII was originally a 7-bit code. The IBM PC hijacked the parity bit for an additional 128 values).

You can see this in action in that the String constructor for an array of bytes requires a translation parameter, although an older version that assumed ASCII translation exists in deprecated form.

Going the other direction, the byte.toString() returns a string representation of the byte's bit value as a decimal number and not a character value (which is what languages like BASIC usually did).

Beyond that, 64-bit encoding is about 1 step below a plastic Cap'n Crunch decoder ring in the realm of encryption. If you want to store encrypted values in a database as byte objects, use the Java encryption facilities. They'll also handle most of the byte/String issues. Note that for security reasons, encryption methods usually work with byte or character arrays, since Strings are immutable and cannot be wiped from memory. Instead they hang around until garbage-collected, which means that potentially an invader could sift through them looking for goodies.

[Winston Gutkowski]

Partheban Udayakumar wrote:My client wants it. I don't know for what reasons. But he still says he wants it.

Then I suggest you find out, because it sounds to me as though he is not only telling you what he wants, but how to do it - and that's almost always a recipe for disaster (too many cooks, etc).

Who is the programmer here?

Winston

[Richard Tookey]

Partheban Udayakumar wrote:Richard,

My client wants it. I don't know for what reasons. But he still says he wants it.

Then it is your task to convince your client that it is a waste of time. This statement conflicts with one of your threads in the Cryptography forum when you said you had been given the task of evaluating methods of encrypting the transfer of data between a mobile client and a server. In that thread it was suggested by me and by others that you just need to use JSSE/HTTPS but you rejected this (I can't remember your exact words) saying that you could not use JSSE/HTTPS. Now your solution relies on JSSE/HTTPS anyway and for some obscure ridiculous reason you want to add more encryption!

The extra encryption over above HTTPS is not needed. I say it again - the extra encryption over and above HTTPS is not needed.

Do you have any ideas on how to retrieve the byte[] stored in a db. I used the following code:

int n;
        byte[] a = {} ;
        InputStream is;   //I actually use is = ResultSet.getBinaryStream(1);
        ByteArrayOutputStream bos = new ByteArrayOutputStream();                
        try {
            System.out.println("InputStream: "+ is.available());  //This returns 44
            is.read(a,0,a.length);                                
                bos.write(a);
                        
        } catch (IOException ex) {
            Logger.getLogger(Retr1.class.getName()).log(Level.SEVERE, null, ex);
        }        
        byte[] retByteArr= bos.toByteArray();

On printing, it shows [0,0,0,0....]

Every time you post code you show that your basic Java knowledge and your cryptography knowledge is not up to any part of the task you have been given and if you are a professional you should tell your client that you don't think you can handle it.

[Partheban Udayakumar]

Richard,

I am sorry but I can't talk to my clients my management does that. My management wants me to use http with AES Encryption. What can I do? I am a fresher as you say and I have no control over it. I am sorry. Let's get to the problem please. Please help me get this.

[Richard Tookey]

Partheban Udayakumar wrote:Richard,

I am sorry but I can't talk to my clients my management does that. My management wants me to use http with AES Encryption. What can I do? I am a fresher as you say and I have no control over it. I am sorry. Let's get to the problem please. Please help me get this.

Then talk to your line manager and explain to him that if you have to use HTTPS for secure key exchange then there is no point in using any further encryption ! As I see it your company are just wasting the client's money.

Sorry I cannot help with this ridiculous project any further.

[Ulf Dittmer]

byte[] a = {} ;

is.read(a,0,a.length);

Two comments: a) what value does "a.length" have?, and b) do not ignore the return values of I/O methods.

I agree that at this point you should have told your manager that you're struggling mightily with this project, and that you need support from a more experienced developer.