Reading variable size buffer into an byte array

[Mike Bates]

I am reading a buffer from a serial port that is consistently variable in size, for example, some times 8 bytes, 40 bytes, or 191 bytes. The byte array (record is 99 bytes)
with no indicator of an end point and the start is always LOO or (0x4c4f4f). Also, the very first response from the serial port is an ACK (0x06) that I need to drop.

Here is an example of the data output:

064c4f4f00009f01 Bytes: 8      <--Note I do not care about the first byte and 
                                  the line below another record starts in the middle someplace.
11777e0221b80000014a00ffffffffffffffffffffffffffffff59ffffffffffffff0000004a000000ffff0000 _ 
                                     1200400400005100fa09ffffffffffffff0000000000000000000
000000000000000002e030609c10223070a0d99954c4f4f00009f0111777e0221b80000 _ 
     014a00ffffffffffffffffffffffffffffff59ffffffffffffff0000004a000000
ffff00001200400400005100fa09ffffffffffffff000000000000000000000000000000000000 _ 
                                         2e030609c10223070a0d9995 Bytes: 191
4c4f4f0000a00111 Bytes: 8
777e0221b80000014a00ffffffffffffffffffffffffffffff59ffffffffffffff0000004a000000 Bytes: 40 
ffff00001200400400005100fa09ffffffffffffff0000000000000000000000000000000000002e Bytes: 40 
030609c10223070a0d5d94 Bytes: 11
4c4f4f0000a00112 Bytes: 8 
777e0221b80000014a00ffffffffffffffffffffffffffffff59ffffffffffffff0000004a000000 Bytes: 40 
ffff00001200400400005100fa09ffffffffffffff0000000000000000000000000000000000002e Bytes: 40 
030609c10223070a0d5309 Bytes: 11
4c4f4f0000a00112 Bytes: 8
777e0221b80000014a00ffffffffffffffffffffffffffffff59ffffffffffffff0000004a000000 Bytes: 40
ffff00001200400400005100fa09ffffffffffffff0000000000000000000000000000000000002e Bytes: 40
030609c10223070a0d5309 Bytes: 11
4c4f4f0000a00112 Bytes: 8
777e0221b90000014a00ffffffffffffffffffffffffffffff59ffffffffffff Bytes: 32                                   
ff0000004a000000ffff00001200400400005100fa09ffffffffffffff00000000000000000000000000000 _
                                                                   00000002e Bytes: 48   
030609c10223070a0d5e06 Bytes: 11                                                                             

The code I started with is:

                case SerialPortEvent.DATA_AVAILABLE:
                        try {
                              byte[] readBuffer = new byte[1024];
                                while (inputStream.available() > 0) {
                                        numBytes = inputStream.read(readBuffer);
                                }
                                byte byteResult[] = new byte[numBytes];
                                System.arraycopy(readBuffer,0,byteResult,0,numBytes);
                                String result  = new String(readBuffer,0,numBytes);

In reality, I want to append only 99 bytes to byteResult[] and forward that on to a record processor to parse the data.
I know I need to just process this at the byte level but have not been able to figure out an approach.

I am looking for guidance in approach (yes, an example would be great but guidance so I can push myself on coding it would be great as well.)
I know if I can get this figured out the solution can be used in other places as well.

Thanks
Mike

[Steve Luke]

You might want to look into using a ByteBuffer to collect data into. For example read from the source in chunks and copy the data to the ByteBuffer. Then, if the ByteBuffer is >99 bytes long:
-- consume the first 99 bytes
-- remove any of the unwanted bytes
-- replace any removed bytes with remaining bytes from the buffer
-- -- may need to re-read more bytes if not enough left in the buffer for the replacement
-- compact the buffer so leftover bytes are the beginning of the next record
-- send the 99 bytes back as a record

If you read up on ByteChannels and the NIO packages, then you might be able to more directly link your ByteBuffer to the source.

[Campbell Ritchie]

Too difficult for "beginning". Moving thread.

[Mike Bates]

Okay,

I took the idea and started coding (mix of pseudo code and code), it has not been tested yet.

Am I on the right track?


Riche, I did not realize this was to complex for beginning java and will assess before posting again there.

Thanks
Mike

import java.nio.ByteBuffer;
ByteBuffer loopDataIn = new ByteBuffer.allocate[99];

case SerialPortEvent.DATA_AVAILABLE:
                        try {
                              byte[] readBuffer = new byte[1024];
                                while (inputStream.available() > 0) {
                                        numBytes = inputStream.read(readBuffer);
                                }
                        buildLoopDataIn(readBuffer, numBytes);  // load information to ByteBuffer


buildLoopDataIn(byte[] data, int num) {
       int capacity = loopDataIn.capacity();
       int position = loopDataIn.position();

       if (position + num > capacity) {
            int byteArrayPosition = 0;
            for(new int i = position; i++; i<capacity+1) {
                loopDataIn.put(data[0]);
                byteArrayPosition++;
             }
             if (loopDataIn.capacity() = 0) { // handle records larger than byte array
                parseData();                         // parse the data
                loopDataIn.clear();                // reset the ByteBuffer to refill
                // Copy what is left of byte array, reset num  Pass back to buildLoopDataIn

                }
       } elsif  (capacity = 0) {
            parseData();  // parses the ByteBuffer
            loopDataIn.clear();     // reset ByteBuffer to refill
        }else {
             loopDataIn.limit( loopDataIn.position() + data);
             loopDataIn(data, 0, num);
          }
}

[Steve Luke]

I don't think am following what you are doing. But I am pretty sure it isn't correct. First let's start with the loop where you are reading from the InputStream:

                        try {
                              byte[] readBuffer = new byte[1024];
                                while (inputStream.available() > 0) {
                                        numBytes = inputStream.read(readBuffer);
                                }
                        buildLoopDataIn(readBuffer, numBytes);  // load information to ByteBuffer

It is possible to continue to read through the entire stream this way, and then when the loop is completed numBytes would be -1. What happens if you first read 500 bytes from the Stream in one chunck, and then read 10 bytes from the stream, after which there are no more available bytes? Then the numBytes is set to 10, you lost the first 10 bytes read from the Stream and you have to reconstruct the rest by reading bytes 11->490 first, then reading 1->10. The best way to read from an input stream is like this:

        try
        {
            byte[] readBuffer = new byte[1024];
            int numBytes = -1;
            while ((numBytes = inputStream.read(readBuffer) != -1)
            {
                //Do something with readBuffer[0] to readBuffer[numBytes]
            }
        }

This way, you will read in up to 1024 byte chunks. The actual amount read in each chunk is recorded in the numBytes variable. Inside the loop you can work with 0-numBytes to process every byte read. The loop knows to stop when numBytes = -1, which signals end of stream. So in your code, it might look something like this:

        try
        {
            byte[] readBuffer = new byte[1024];
            int numBytes = -1;
            while ((numBytes = inputStream.read(readBuffer) != -1)
            {
                buildLoopDataIn(readBuffer, numBytes);  // load information to ByteBuffer
            }
        }

Now, I can't follow what you were trying to do in the rest of the code you posted. Here is what I would do though (and why I would go with ByteBuffer)

1) You have a number of bytes you read, which could be any amount from 0 to 1024. You could have read a small part of a 'report' of 99 bytes you need, or you may have read up to 10 'reports' with some extra bytes. So make sure you keep track of all the extra bytes that can come between chunks, dump everything into the ByteBuffer. This would mean that the ByteBuffer should be larger than your readBuffer just to make sure that no overflow occurs from a partial read immediately followed by a complete read:

        ByteBuffer reportBuffer = ByteBuffer.allocate(2 * 1024);

        //... Then in the 'filling' method
            reportBuffer.put(readBuffer, 0, numBytes);

2) The ByteBuffer has a bulk get() method that will fill an array. When building the report-sized array I would use this method. Something like:

byte[] getReportArray(ByteBuffer source)
{
    byte[] report = new byte[99];
    source.get(report); //fill the report
    return report;
}

3) Then it is just a matter of getting from the 'filling' stage in 1) to the 'reporting' stage in 2). This is just a matter of preparing the ByteBuffer for reading, rather than writing, then reading as many 'reports' as are present in the buffer, then making the buffer ready again for writing. Methods that will help you:
flip() - prepare for reading
remaining() - Check number of remaining bytes (if remaining() >= 99 then make report, for example)
compact() - prepare for writing

That last step is just a bit of glue, and you would probably end up calling the 'getRportArray()' method in a loop to handle cases when one read chunk reads > 99 bytes of data.

Any kind of byte checking to remove un-wanted bytes would be done on top of this (for example you can consume the InputStream up to the point where the ACK signal is reached BEFORE calling the stream reading code I provided up top).

[Steve Luke]

p.s. I edited your first post so we don't have to scroll sideways a lot to view the post.