Jerry Kreps

Greenhorn
+ Follow
since Jun 16, 2003
Cows and Likes
Cows
Total received
0
In last 30 days
0
Total given
0
Likes
Total received
0
Received in last 30 days
0
Total given
0
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Jerry Kreps

Hi, Stephen --
I was in your shoes not too long ago, looking for a way to do FTP operations in Java. I do not recommend the sun.net.ftp package. It is not intended for public use and is not documented very well. There are several different ftp tools in Java, but the one that I use is the open source Apache Commons Net package that is really slick. You can download it at
this location.
Here is a procedure that I wrote that uses this tool to perform ftp file transfers of files in a specified source directory and to rename the source files afterward. As you can see, it is pretty simple. However, in getting it to work I did find that one of the object methods in the package had a bug for which I had to find a work-around. For further info on this, see my thread in this forum.

Hope this helps
[ August 04, 2003: Message edited by: Jerry Kreps ]
[ August 04, 2003: Message edited by: Jerry Kreps ]
[ August 04, 2003: Message edited by: Jerry Kreps ]
Well, I found my own work-around that gets the job done.
My code in the original message used the FTPClient.listFiles() method to retrieve the array of FTPFile objects and then looped through the array using the getName() method of each object to pass the file name to the FTPClient.retrieveFile() method. But the trouble was that the FTPClient.listFiles() method always returned null even though the working directory clearly contained files that I wanted to download.
So I decided to go through the back door.
I noticed that the FTPClient.listNames() method returned an array of string objects that were the names of all the files in the source directory that I wanted to download. So then just I looped through the string array and I passed each file name string to the FTPClient.retrieveFile() method and it worked like a charm.
The only question is: Why did the FTPClient.listFiles() method always return null when the FTPClient.listNames() method was able to identify the files I wanted?
Well, I hope this helps someone else who comes across the same problem. If any of you know that answer to my puzzle, please fill me in. Thanks!
I LOVE JavaRanch!!!
18 years ago
Well, I found my own work-around that gets the job done.
My code in the original message used the FTPClient.listFiles() method to retrieve the array of FTPFile objects and then looped through the array using the getName() method of each object to pass the file name to the FTPClient.retrieveFile() method. But the trouble was that the FTPClient.listFiles() method always returned null even though the working directory clearly contained files that I wanted to download.
So I decided to go through the back door.
I noticed that the FTPClient.listNames() method returned an array of string objects that were the names of all the files in the source directory that I wanted to download. So then just I looped through the string array and I passed each file name string to the FTPClient.retrieveFile() method and it worked like a charm.
The only question is: Why did the FTPClient.listFiles() method always return null when the FTPClient.listNames() method was able to identify the files I wanted?
Well, I hope this helps someone else who comes across the same problem. If any of you know that answer to my puzzle, please fill me in. Thanks!
I LOVE JavaRanch!!!
Well, I found my own work-around that gets the job done.
My code in the original message used the FTPClient.listFiles() method to retrieve the array of FTPFile objects and then looped through the array using the getName() method of each object to pass the file name to the FTPClient.retrieveFile() method. But the trouble was that the FTPClient.listFiles() method always returned null even though the working directory clearly contained files that I wanted to download.
So I decided to go through the back door.
I noticed that the FTPClient.listNames() method returned an array of string objects that were the names of all the files in the source directory that I wanted to download. So then just I looped through the string array and I passed each file name string to the FTPClient.retrieveFile() method and it worked like a charm.
The only question is: Why did the FTPClient.listFiles() method always return null when the FTPClient.listNames() method was able to identify the file I wanted?
Well, I hope this helps someone else who comes across the same problem. If any of you know that answer to my puzzle, please fill me in. Thanks!
I LOVE JavaRanch!!!
As a PS to the above message:
I've put this message on the "Other Open Source APIs" forum, since I thought it might apply to that too. If this violates forum rules, I apologize and I will be more careful where to put my messages in the future.
Hi, Everybody!
Listed below is a method I have written that is designed to connect to my remote FTP server, change the working directory on the FTP server to to a certain subdirectory (ftpSrcPath) that contains files that I want to FTP to my local directory (targetPath). I want to get an array of FTPFiles and FTP only those whose names do not start with "x_" (previously sent files). The variables loginSuccess, rootDir, cwdSuccess, and workingDir are there only so that I can see their values when I am debugging in my IDE.
When I run it through my debugger, I make the connection successfully (loginSuccess = true), my initial working directory on the remote FTP server is the root directory (rootDir = "/"), then I change the working directory successfully (cwdSuccess = true) and I can see the new working directory on the remote FTP server (workingDir = "/srcdir"). But when I try to get the array of FTPFile objects, the FTPClient.listFiles() method returns null.
What am I doing wrong? My initial suspicion is that the DefaultFTPFileListParser is not handling the names of the files in my source directory and that I need to make my own FTPFileListParser. But how? I've been able to find very little documentation on the web other than the API JavaDocs, and they don't say much about it.
Thanks in advance for your help!
Here's the code:
private void ftpFileTransfer()
throws DwhGenericException
{
try
{
FTPClient ftpClient = new FTPClient();
ftpClient.connect("99.999.9.999");
boolean loginSuccess = ftpClient.login(
"user",
"password");
String rootDir = ftpClient.printWorkingDirectory();
boolean cwdSuccess = ftpClient.changeWorkingDirectory(ftpSrcPath);
String workingDir = ftpClient.printWorkingDirectory();
// **** The following operation returns null. ****
FTPFile[] ftpFiles = ftpClient.listFiles();
for (int i = 0; i < ftpFiles.length; i++)
{
String fileName = ftpFiles[i].getName();
if (fileName.startsWith("x_"))
{
continue;
}
File newFile = new File(targetPath + "\\" + fileName);
FileOutputStream fos = new FileOutputStream (newFile);
ftpClient.retrieveFile(fileName,fos);
fos.close();
ftpClient.rename(fileName, "x_" + fileName);
}
ftpClient.logout();
ftpClient.disconnect();
ftpClient = null;
}
catch (Exception e)
{
JobLog.storeError(
"Error - ftpFileTransfer()",
e);
throw new DwhGenericException();
}
}
As a PS to the above message:
I've put this message on the "Sockets and Internet Protocols" forum, since I thought it might apply to that too. If this violates forum rules, I apologize and I will be more careful where to put my messages in the future.
Hi, Everybody!
Listed below is a method I have written that is designed to connect to my remote FTP server, change the working directory on the FTP server to to a certain subdirectory (ftpSrcPath) that contains files that I want to FTP to my local directory (targetPath). I want to get an array of FTPFiles and FTP only those whose names do not start with "x_" (previously sent files). The variables loginSuccess, rootDir, cwdSuccess, and workingDir are there only so that I can see their values when I am debugging in my IDE.
When I run it through my debugger, I make the connection successfully (loginSuccess = true), my initial working directory on the remote FTP server is the root directory (rootDir = "/"), then I change the working directory successfully (cwdSuccess = true) and I can see the new working directory on the remote FTP server (workingDir = "/srcdir"). But when I try to get the array of FTPFile objects, the FTPClient.listFiles() method returns null.
What am I doing wrong? My initial suspicion is that the DefaultFTPFileListParser is not handling the names of the files in my source directory and that I need to make my own FTPFileListParser. But how? I've been able to find very little documentation on the web other than the API JavaDocs, and they don't say much about it.
Thanks in advance for your help!
Here's the code:
private void ftpFileTransfer()
throws DwhGenericException
{
try
{
FTPClient ftpClient = new FTPClient();
ftpClient.connect("99.999.9.999");
boolean loginSuccess = ftpClient.login(
"user",
"password");
String rootDir = ftpClient.printWorkingDirectory();
boolean cwdSuccess = ftpClient.changeWorkingDirectory(ftpSrcPath);
String workingDir = ftpClient.printWorkingDirectory();
// **** The following operation returns null. ****
FTPFile[] ftpFiles = ftpClient.listFiles();
for (int i = 0; i < ftpFiles.length; i++)
{
String fileName = ftpFiles[i].getName();
if (fileName.startsWith("x_"))
{
continue;
}
File newFile = new File(targetPath + "\\" + fileName);

FileOutputStream fos = new FileOutputStream (newFile);
ftpClient.retrieveFile(fileName,fos);
fos.close();
ftpClient.rename(fileName, "x_" + fileName);
}
ftpClient.logout();
ftpClient.disconnect();
ftpClient = null;
}
catch (Exception e)
{
JobLog.storeError(
"Error - ftpFileTransfer()",
e);
throw new DwhGenericException();
}
}
Hi, Everybody!
Listed below is a method I have written that is designed to connect to my remote FTP server, change the working directory on the FTP server to to a certain subdirectory (ftpSrcPath) that contains files that I want to FTP to my local directory (targetPath). I want to get an array of FTPFiles and FTP only those whose names do not start with "x_" (previously sent files). The variables loginSuccess, rootDir, cwdSuccess, and workingDir are there only so that I can see their values when I am debugging in my IDE.
When I run it through my debugger, I make the connection successfully (loginSuccess = true), my initial working directory on the remote FTP server is the root directory (rootDir = "/"), then I change the working directory successfully (cwdSuccess = true) and I can see the new working directory on the remote FTP server (workingDir = "/srcDir"). But when I try to get the array of FTPFile objects, the FTPClient.listFiles() method returns null. What am I doing wrong? Any suggestions? Thanks in advance.
Here is my code:
private void ftpFileTransfer()
throws DwhGenericException
{
try
{
// Make ftp connection to the ndrftp server.
FTPClient ftpClient = new FTPClient();
ftpClient.connect("99.999.9.999");
boolean loginSuccess = ftpClient.login(
"user",
"password");
String rootDir = ftpClient.printWorkingDirectory();
boolean cwdSuccess = ftpClient.changeWorkingDirectory(ftpSrcPath);
String workingDir = ftpClient.printWorkingDirectory();
FTPFile[] ftpFiles = ftpClient.listFiles();
for (int i = 0; i < ftpFiles.length; i++)
{
String fileName = ftpFiles[i].getName();
if (fileName.startsWith("x_"))
{
continue;
}
File newFile = new File(targetPath + "\\" + fileName);
FileOutputStream fos = new FileOutputStream(newFile);
ftpClient.retrieveFile(
fileName,
fos);
fos.close();
ftpClient.rename(
fileName,
"x_" + fileName);
}
ftpClient.logout();
ftpClient.disconnect();
ftpClient = null;
}
catch (Exception e)
{
JobLog.storeError(
"Error - ftpFileTransfer()",
e);
throw new DwhGenericException();
}
}
18 years ago
Thanks to everyone for your suggestions.
Sorry that I was slow in responding, but I was out of the office this morning.
David, thanks for your suggestion about the server version of JIT. I'll check into that.
Ilja and Thomas, you made some very good points on how to approach the whole issue of OOP vs "coding for performance". I'll take your suggestions to heart if I should have to tweek my app for better performance in the future.
Stan, thanks for your suggestion about batching. At first I didn't batch my updates, but when I finally did it cut the time by almost half. So you're right, batching was a big plus. Right now, I'm just fiddling around with the batch size to optimize peformance.
Let me just say that I think Java Ranch is fantastic. This is one great source of info/encouragement for new code hands like me.
The Java Ranch community is great! I appreciate everyone's help! Happy code punching!
18 years ago
Thanks, David.
Here is a more detailed explanation of what I am trying to do:
I use a BufferedReader object to read the flat file line by line. Before I start reading the file, however, I create an ArrayList of what I call "DataField" objects. Each DataField object corresponds to a column on the DB table that I want to load from the current flat file I am about to read. Each class of DataField object returns a particular data type, depending on the data type of the DB column to which it corresponds, and it contains attributes (starting position, field length, data type, default value, left trim character, etc.) that describe how to parse the line of text and extract the data.
So, when I get a line from the BufferedReader, I send it to a routine which loops through the ArrayList of DataField objects, passing to each one the line of data. Each DataField object extracts its data and loads it into a parameter of a PreparedStatement object. When the entire ArrayList has been traversed, I have loaded the data for one row, so I insert it into the DB table via the PreparedStatement.
Common to all DataField objects is a base class which contains methods that are used by all the various types of DataField Objects. So when I instantiate a DataField object, I pass to the base class constructor all the parsing paramters (these never change) and it loads them into private fields of the base class. So, since the base class already has all the parsing information it needs, all the child DataField object has to do when getting an inputline from the BufferedReader is to pass the line in a call to the appropriate base class method which will extract the data type it needs.
Of course, this routine traverses the ArrayList of DataField objects for each line of data, sometimes processing millions of records in a file.
Currently, my process can import a 190MB file containing over 1,600,000 records in about 48 minutes.
I'm trying to squeeze as much performance out of my app as possible.
I know that calling a static method is much faster then calling methods across class hierarchies. But by using a static method, I would have to keep passing the same parsing attributes as parameters in the call to the static method that extracts the data from the input line. Would this negate any increase in performance obtained from using a static method?
I know that the usually answer to this type of question is "It depends" but is their a genereal rule that applies?
Thanks again.
18 years ago
I'm relatively new to Java, and I am writing a server side Java app that parses data from fixed length ascii files and loads them into tables on our db. I want to keep my app as object orientedly pure as possible but performance is more important. So here goes:
Which is more efficient (i.e. faster)?
(1) Create a bunch of child classes from a base class. During construction each child object passses unique attributes as parameters to the constructor of the base class which stores them in private variables. These attributes are later used by the base class when executing commonly used methods called by the child object.
This decreases the number of parameters that must be passed by each call to a base class method.
OR.....
(2) Ditch the base class and convert all the commonly used methods to static methods. That will require a larger number of parameters in each method call than in scenario (1), but it doesn't have the overhead of parent-child object interaction and the static mehtods would be faster.
Related question:
Would it be faster to convert the methods into functions and have the results returned by the function calls or keep them as methods and have them change either protected variables (scenario 1) or public static variables (scenario 2) which can be accessed later by the calling method?
I hope my question is clear. Thanks for any suggestions you all might have.
18 years ago