• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Java Read/Write on huge file system

 
Greenhorn
Posts: 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am working on a J2EE application where the user can upload/read/delete files. There is going to be a separate file server (SAN drive) for this.

Here is the worst case.

File Server size = 100TB
Maximum allowed size per file = 5MB
This leads to the maximum number of files = 20971520


The files will be stored in a specific folder structure but I don't know how much it will impact the performance of file search/read/write/delete in my java program for this large number of files. I thought of splitting the 100TB into multiple drives but still I have no idea what are the problems I am going to face at the worst case.

I appreciate any kind of suggestions. Thanks a lot in advance.
 
Saloon Keeper
Posts: 27763
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Welcome to the Ranch, Karthik!

Since the files themselves aren't all that big, Java itself probably isn't going to be the limiting factor. More important is going to be the directory structure and filesystem that you use.

Not all filesystems even support multi-Terabytes of data. Some do, but should be considered carefully. I once worked with one that was projected to require multiple days to repair itself in the event of a major crash.

Different filesystems also vary in how they treat directories. Some limit the total number of files in a directory. Some don't technically limit them, but have problems with limitations in the support utilities. Some can handle many files, but the time required to open a file is dependent on the speed of a linear search of a directory. Others may use hashed directories, which generally makes them faster at opening files... provided that you don't end up with too many hash collisions and long overflow chains.

Your Mileage May Vary, in other words. I recommend setting up a pilot study.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic