aspose file tools*
The moose likes I/O and Streams and the fly likes Merged files? (can't think of a better name) Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "Merged files? (can Watch "Merged files? (can New topic
Author

Merged files? (can't think of a better name)

Daniel Gen Li
Ranch Hand

Joined: Jan 02, 2010
Posts: 32
Say I have an image file that's a spritesheet. and I have a text file with the indexes and locations of each sprite on the spritesheet. How could I merge those two into one and when I read the file I can re-read the image and the indexes? Or for example: games like starcraft have their soundeffects packed into a single file. how can I pack the sound files and read them out individually?

P.S. if anyone can think of a better name for this thread please tell me.


Li
Gerardo Tasistro
Ranch Hand

Joined: Feb 08, 2005
Posts: 362
How about zipping them? and opening that in memory, parse and dispose. Not sure if those files are really big, but that could be an option since Java handles zips very well.
Daniel Gen Li
Ranch Hand

Joined: Jan 02, 2010
Posts: 32
Not a bad idea at all. I've done zipping before, but how exactly could I load the extracted content into memory and read them as files? Perhaps load them into a byte buffer? because I'm making a game and I don't want people to just change the images so I want to place them all into a big file that's hard to decipher. lol
Gerardo Tasistro
Ranch Hand

Joined: Feb 08, 2005
Posts: 362
Daniel Gen Li wrote:Not a bad idea at all. I've done zipping before, but how exactly could I load the extracted content into memory and read them as files? Perhaps load them into a byte buffer? because I'm making a game and I don't want people to just change the images so I want to place them all into a big file that's hard to decipher. lol


Zip utils have a way to handle zip entries and get a stream from that. You could work from there on.
Daniel Gen Li
Ranch Hand

Joined: Jan 02, 2010
Posts: 32
one thing though... because I'm going to use this for a game, would unzipping take way too long? because i will need to be streaming music data files and i don't know how steaming from a zip entry is going to work out. what if the rate of streaming if far greater than the rate of decompressing? Zipping isn't a bad idea for images because they are meant to be pre-loaded into the memory. but for large files such as background music, it would be impractical to load the whole piece of music into memory before playing it.

which is the reason why I'm asking for tips and suggestions on just simply reading from a non-zipped data file.

take a simple spritesheet format for example. the first chunk of the file would be the locations and indexes of individual sprites on the sprite sheet. the second chunk of the file would be the actual image data. how could I read those two in and process them separately?
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19762
    
  20

You can set the compression level to uncompressed.


SCJP 1.4 - SCJP 6 - SCWCD 5 - OCEEJBD 6
How To Ask Questions How To Answer Questions
Daniel Gen Li
Ranch Hand

Joined: Jan 02, 2010
Posts: 32
aaahhhh. so would that be just equal to 2 files placed next to eachother in the same file?
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19762
    
  20

More or less. There is some overhead, to indicate the filenames, sizes etc, but that's about it.
Daniel Gen Li
Ranch Hand

Joined: Jan 02, 2010
Posts: 32
So I just did exactly what the above discussed. and I tested out the efficiency. I read a text file and then an image file and it took 16 milliseconds. I also read the same two files except this time they were put into a zip with a compression level of 0, and it took 356 milliseconds. that's a pretty big draw back... and I think it has to do with using the zip file thing.

Can I just place the contents of two files into one and read both of them out separately? All I need is a way to distinguish between the contents of the two files.... how would I know I've reached the end of the contents of one file and direct the rest of the input somewhere else?
Gerardo Tasistro
Ranch Hand

Joined: Feb 08, 2005
Posts: 362
Daniel Gen Li wrote: how would I know I've reached the end of the contents of one file and direct the rest of the input somewhere else?


More important than that. After you've implemented that logic what will the performance be? And what if it's near the 300 ms mark?
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19762
    
  20

If you don't need the file names it's actually not that hard to do this. The separators could be the file size, as 8 bytes. You can use a DataInputStream / DataOutputStream for that. In short:
Reading is then just as easy:
You can use DataInputStream.readFully if you have the memory for the file. Otherwise you must ensure that you don't read too many bytes.

There is one problem with this - you can't use a Reader to read the text file. The Reader may try to read a little ahead and so consume bytes that don't belong to the text file.

If the names are required then you'll need even more meta-data besides the file size. I think you could use writeUTF / readUTF for that. If not then you'll need to write the file name size (as an int) followed by the name chars. You can use writeChars() / readChar() for that.

I'd like to see your performance results for this solution.
Daniel Gen Li
Ranch Hand

Joined: Jan 02, 2010
Posts: 32
@ Gerardo

My theory is that it shouldn't. Because what's the difference between reading two files when they're spearate and when they're together? aside from the fact that they don't appear to be separate filesystem?

@ Rob

Well... you see the files that I'm trying to mush together are list of image and text files, and therefore their size is unknown. Inside the file could look like this:

==start====
TEXT
IMAGE
TEXT
IMAGE
TEXT
IMAGE
==end of file===

except that the TEXT and IMAGE are replaced with actual text and image information (the "start" and "end of file" are just there for your understanding).

and about reading the text, I could read the information byte by byte and concatenate it to the end of an empty string, but that would mean as many iterations as the amount of bytes in that one text file. which wouldn't be very practical... But even if that DOES work, I couldn't know about the size of the image =\

It WOULD probably work however, if there are only two file contents in the file. that way I COULD place the text file first and after finish reading it I'll let the rest of the input go to Image loading things.
Gerardo Tasistro
Ranch Hand

Joined: Feb 08, 2005
Posts: 362
Daniel Gen Li wrote:@ Gerardo

My theory is that it shouldn't. Because what's the difference between reading two files when they're spearate and when they're together? aside from the fact that they don't appear to be separate filesystem?



I don't know. What I do know is there is no free lunch. So something is taking up that difference. It might be inefficient code or just processes needed to handle it properly. Before venturing into a development like that vs using a proven tool I'd test to see how significant that difference is. Are you loading this once every so often, then the impact isn't so great and you could have your application running much sooner. If on the other hand accessing these files could be a serious bottle neck(many reads per minute) then yes you'd have to optimize.

Did you do a series of reads and then average the times? If so how many tests did you do to come up with those values?
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Merged files? (can't think of a better name)