Is there any free libraries avaialble for file splitting usiong java??

[Ramakanta Sahoo]

Hi All,

I want to split a file of 10GB or so into ten 1GB files using java. Is there any freeware reliable libraries are available then please point me to there.

[Rob Spoor]

How about the core API, with FileInputStream and FileOutputStream? Read from the FileInputStream and copy to the FileOutputStream. Then, when you've written 1GB, close the FileOutputStream and open a new one.

Pasting them together is similar. Copy all contents from the FileInputStream to the FileOutputStream. Then, when the FileInputStream has been read, close it and open the next one, writing to the same FileOutputStream.

[Ramakanta Sahoo]

Hi Thanks,

The program is like below but it hangs when ever i want to split any file of 2gb or beyond.Even if the folder am not able to do a ls -ltr after running the splitme pro.
I know something is foolishly wrong can somebody please take a look at it.

  
import java.io.RandomAccessFile;
import java.io.File;


public class Splitme {


	public static void main(String args[])
	{
		//Checking the presents of argument.
		if(args.length==0)
		{
			System.out.println("Type -h for getting help on this splitter");
		}
		else if (args.length==1)
		{
			//if the paramenter is -h, the system displays the syntax for using this utility.
			if(args[0].equals("-h"))
			{
				System.out.println("-----------------------------------");
				System.out.println("For Splitting..");
				System.out.println("java Splitme -s <filepath> <Split file size in bytes>");
				System.out.println("-----------------------------------");
				System.out.println("For Joining..");
				System.out.println("java Splitme -j <filepath>");
				System.out.println("-----------------------------------");
			}
			// Displaying the error if parameter required are not given.
			else if(args[0].equals("-s") || args[0].equals("-j"))
			{
				System.out.println("Parameters missing, type -h for help.....");
			}
		}
		//if the parameter is -s then splitting of file happens.
		else if(args[0].equals("-s"))
		{
			if(args.length==3)
			{
				String FilePath="";
				FilePath=args[1];
				File filename=new File(FilePath);
				long splitFileSize=0;
				if(filename.exists())
				{
					try
					{
						splitFileSize=Long.parseLong(args[2]);
						Splitme spObj=new Splitme();
						spObj.split(FilePath,(int) splitFileSize);
						spObj=null;
					}
					catch(Exception e)
					{
						e.printStackTrace();
					}
				}
				else
				{
					System.out.println("File Not Found....");
				}
				filename=null;
				FilePath=null;
			}
			else
			{
				System.out.println("Parameters missing, type -h for help.....");
			}
		}
		//if the parameter is -s then joining of file happens.
		else if(args[0].equals("-j"))
		{
			String FilePath="";
			FilePath=args[1];
			File filename=new File(FilePath.trim());
			if(filename.exists())
			{
				FilePath=FilePath.substring(0,FilePath.length()-4);
				filename=new File(FilePath);
				if(filename.exists())
				{
					System.out.println("\""+ FilePath+ "\" File Already Exist....");
				}
				else
				{
					Splitme spObj=new Splitme();
					spObj.join(FilePath);
					spObj=null;
				}
			}
			else
			{
				System.out.println(FilePath+filename);
				System.out.println("File Not Found....");
			}
			filename=null;
			FilePath=null;
		}
		System.out.println();
		System.out.println();
		}

	public void join(String FilePath)
	{
		long leninfile=0,leng=0;
		int count=1,data=0;
		try
		{
			File filename=new File(FilePath);
			RandomAccessFile outfile = new RandomAccessFile(filename,"rw");
			while(true)
			{
				filename=new File(FilePath + count + ".sp");
				if (filename.exists())
				{
					RandomAccessFile infile = new RandomAccessFile(filename,"r");
					data=infile.read();
					while(data!=-1)
					{
						outfile.write(data);
						data=infile.read();
					}
					leng++;
					infile.close();
					count++;
				}
				else
				{
					break;
				}
			}
			outfile.close();
		}
		catch(Exception e)
		{
			e.printStackTrace();
		}
	}

	public void split(String FilePath,int splitlen)
	{
		long leninfile=0,leng=0;
		int count=1,data;
		try
		{
			File filename=new File(FilePath);
			RandomAccessFile infile = new RandomAccessFile(filename,"r");
			data=infile.read();
			while(data!=-1)
			{
				filename=new File(FilePath + count + ".sp");
				RandomAccessFile outfile = new RandomAccessFile(filename,"rw");
				while( data!=-1 && leng<splitlen)
				{
					outfile.write(data);
					leng++;
					data=infile.read();
				}
				leninfile+=leng;
				leng=0;
				outfile.close();
				count++;
			}
		}
		catch(Exception e)
		{
			e.printStackTrace();
		}
	}
}


  

Thanks,
-Ricky>

[Ramakanta Sahoo]

somebody help me

[Rob Spoor]

http://faq.javaranch.com/java/PatienceIsAVirtue

And never ever ever use that many smilies in one place again. Ugh!

[Rob Spoor]

2GB is 2 * 1024 * 1024* 1024 bytes = 2,147,483,648 bytes. Integer.MAX_VALUE is 2,147,483,647, one smaller. So when you read the long value and cast it to int, it will wrap around and become -2,147,483,648 instead. As such, the check "leng<splitlen" will never be true, the inner while loop will never execute and you never read any data anymore. As a result, data will keep the value of the first byte forever.

Instead of limiting your splitlen to int, make it long. Also, disallow negative values; if someone types in -1 the same will occur.>

[Ramakanta Sahoo]

Thanks Rob,

Sorry for too many similes. I changed to long its working but the speed of splitting is very slow almost takes 3 and 1/2hrs to split a 3GB file in 2GB slices.
Is there anything I can do to make it faster. Any suggestions for Better performance??

[Ramakanta Sahoo]

Hi All,

Does somebody knows how i will be able to tweak the program for optimal performance.

Now its taking 3hr:30mins to split a 3GB file into 2GB slices.

Please give your valuable suggestions


Thanks

[Rob Spoor]

Switch from RandomAccessFile to InputStream and OutputStream, then use BufferedInputStream and BufferedOutputStream. So your code changes as this (leaving out unchanged code):

//RandomAccessFile outfile = new RandomAccessFile(filename,"rw");
OutputStream outfile = new BufferedOutputStream(new FileOutputStream(filename));

//RandomAccessFile infile = new RandomAccessFile(filename,"r");
InputStream infile = new BufferedInputStream(new FileInputStream(filename));  

Since InputStream and OutputStream also have the read() / write() and close() methods that should cause your code to compile without any problems.

[Ramakanta Sahoo]

Thanks, noe its much faster than before.

1 Question What is the major difference between RandomAccess and InputStream ?? Why so much of performance diff ??

[Rob Spoor]

I don't know if it is the InputStream that is making the difference; I think it's the BufferedInputStream. The RandomAccessFile really reads one byte at a time, whereas the BufferedInputStream reads chunks of multiple bytes, even if you only call the read() method that returns one byte. Reading multiple bytes (usually entire "pages" of several kilobytes) is usually much more efficient.