Java Read/Write on huge file system

[Karthik Karnan]

I am working on a J2EE application where the user can upload/read/delete files. There is going to be a separate file server (SAN drive) for this.

Here is the worst case.

File Server size = 100TB
Maximum allowed size per file = 5MB
This leads to the maximum number of files = 20971520

The files will be stored in a specific folder structure but I don't know how much it will impact the performance of file search/read/write/delete in my java program for this large number of files. I thought of splitting the 100TB into multiple drives but still I have no idea what are the problems I am going to face at the worst case.

I appreciate any kind of suggestions. Thanks a lot in advance.

[Tim Holloway]

Welcome to the Ranch, Karthik!

Since the files themselves aren't all that big, Java itself probably isn't going to be the limiting factor. More important is going to be the directory structure and filesystem that you use.

Not all filesystems even support multi-Terabytes of data. Some do, but should be considered carefully. I once worked with one that was projected to require multiple days to repair itself in the event of a major crash.

Different filesystems also vary in how they treat directories. Some limit the total number of files in a directory. Some don't technically limit them, but have problems with limitations in the support utilities. Some can handle many files, but the time required to open a file is dependent on the speed of a linear search of a directory. Others may use hashed directories, which generally makes them faster at opening files... provided that you don't end up with too many hash collisions and long overflow chains.

Your Mileage May Vary, in other words. I recommend setting up a pilot study.