posted 12 years ago
Welcome to the Ranch, Karthik!
Since the files themselves aren't all that big, Java itself probably isn't going to be the limiting factor. More important is going to be the directory structure and filesystem that you use.
Not all filesystems even support multi-Terabytes of data. Some do, but should be considered carefully. I once worked with one that was projected to require multiple days to repair itself in the event of a major crash.
Different filesystems also vary in how they treat directories. Some limit the total number of files in a directory. Some don't technically limit them, but have problems with limitations in the support utilities. Some can handle many files, but the time required to open a file is dependent on the speed of a linear search of a directory. Others may use hashed directories, which generally makes them faster at opening files... provided that you don't end up with too many hash collisions and long overflow chains.
Your Mileage May Vary, in other words. I recommend setting up a pilot study.
The secret of how to be miserable is to constantly expect things are going to happen the way that they are "supposed" to happen.
You can have faith, which carries the understanding that you may be disappointed. Then there's being a willfully-blind idiot, which virtually guarantees it.