Reading log files from various unix servers through Java program on windows

[Arthur Ford]

Hi,

I have few production unix servers. For debugging purposes, I have to login into 5-6 servers and check for logs in each of the servers. Is it possible to automate this task? Can I write a java program which will connect to each of the servers and read the log files. I cannot run any programs in the unix boxes. Anything that has to happen has to happen in the window box. Any suggestions or ideas?

Thanks!

[Sushil Sharma]

Get org.apache.commons package from http://jakarta.apache.org/site/downloads/downloads_commons-net.cgi

import java.io.*;
import java.text.*;
import java.util.*;
 

import org.apache.commons.net.ftp.*;
 

/**
 *
 * @author shussain
 * @version 0.1
 */
public class FtpTestClient {
    
    public final static int DEFAULT_FTP_PORT = 21;
    
    private FTPClient ftpc;
        
    public static void main(String[] args) {
    
        
        String[] host_names = {"host_1", "10.1.1.2", "host.mycompany.com"};
        String[] user_names = {"user_1", "superuser", "hostuser"};
        String[] passwords =  {"password_1", "superpass", hostpass"};
        String ftp_src_folder = "/home/log";
        String ftp_src_file = "log_1.txt";
        
        for (int i = 0; i < host_names.length; i++) {
            FtpTestClient.getFile(host_names[i], user_names[i],
                passwords[i], ftp_src_folder, ftp_src_file);
        }
    }
        
    public FtpTestClient(String host, String login, String passwd)
    throws Exception
    {
        this(host, String.valueOf(FtpTestClient.DEFAULT_FTP_PORT), login, passwd);
    }
    /**
     * Creates a new instance of FtpTestClient
     * 
     * 
     * @param host Server host name
     * @param port Server port number
     * @param login FTP login name
     * @param passwd FTP login password
     * @throws Exception if client create fails
     */
    public FtpTestClient(String host, String port, String login, String passwd)
    throws Exception
    {
        try {
            ftpc = new FTPClient();
        } catch (Exception e) {
            throw new Exception(e.getMessage());
        }
    }
    
    /**
     * get a single file from the server. Destination filename is same as that
     * on server.
     *
     * @param server ftp server name
     * @param username ftp server login name
     * @param password ftp server password
     * @param folder source folder on the ftp server
     * @param fname file to be transferred
     * @return true if successful
     */
    public static boolean getFile(String server, String username,
            String password, String folder, String fname)
    {
        boolean status = false;
        
        FTPClient ftp = null;
        try {
            ftp = new FTPClient();
            
            ftp.connect( server );
            ftp.login( username, password );
            
            ftp.changeWorkingDirectory( folder );
            
            File file = new File(fname);
            FileOutputStream fos = new FileOutputStream( file );
            
            ftp.retrieveFile(fname, fos);
            fos.close();
            
            status = true;
            
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            try {
                ftp.logout();
                ftp.disconnect();
            } catch (Exception e) {}
        }
        
        return status;
    }
    
    public static void getAllFiles( String server,
            String username,
            String password,
            String folder,
            String destinationFolder,
            Calendar start,
            Calendar end ) {
        try {
            // Connect and logon to FTP Server
            FTPClient ftp = new FTPClient();
            ftp.connect( server );
            ftp.login( username, password );
            System.out.println("Connected to " +
                    server + ".");
            System.out.print(ftp.getReplyString());
            
            // List the files in the directory
            ftp.changeWorkingDirectory( folder );
            FTPFile[] files = ftp.listFiles();
            System.out.println( "Number of files in dir: " + files.length );
            DateFormat df = DateFormat.getDateInstance( DateFormat.SHORT );
            for( int i=0; i<files.length; i++ ) {
                Date fileDate = files[ i ].getTimestamp().getTime();
                if( fileDate.compareTo( start.getTime() ) >= 0 &&
                        fileDate.compareTo( end.getTime() ) <= 0 ) {
                    // Download a file from the FTP Server
                    System.out.print( df.format( files[ i ].getTimestamp().getTime() ) );
                    System.out.println( "\t" + files[ i ].getName() );
                    File file = new File( destinationFolder +
                            File.separator + files[ i ].getName() );
                    FileOutputStream fos = new FileOutputStream( file );
                    ftp.retrieveFile( files[ i ].getName(), fos );
                    fos.close();
                    file.setLastModified( fileDate.getTime() );
                }
            }
            
            // Logout from the FTP Server and disconnect
            ftp.logout();
            ftp.disconnect();
            
        } catch( Exception e ) {
            e.printStackTrace();
        }
    }
    
    /**
     * Gets all files which matches a specified file name template. The template
     * is a regular expression. 
     *
     * @param server ftp server name
     * @param username ftp server login name
     * @param password ftp server password
     * @param folder source folder on the ftp server
     * @param ftempl regular expression for file name
     * @return true if successful
     * @throws Exception if unable to connect and other problems.
     */
    public static boolean getFilesOfType(String server, String username,
            String password, String folder, String ftempl)
            throws Exception 
    {
        FTPClient ftp = null;
        boolean status = false;
        
        try {
            // Connect and logon to FTP Server
            ftp = new FTPClient();
            ftp.connect( server );
            ftp.login( username, password );
            
            // List the files in the directory
 
            ftp.changeWorkingDirectory( folder );
            FTPFile[] files = ftp.listFiles();
        
            // get files which match name template
            // at present just check for prefix
            for (int i = 0; i < files.length; i++) {
                /*
                 if (files[i].matches(ftempl)) {
                    
                }*/
            }
        } catch (Exception e) {
            throw e;
        }
        
        finally {
            // close connection
            if (ftp != null && ftp.isConnected()) {
                ftp.disconnect();
            }
        }
        
        return status;
    }
    
    /**
     * Upload a single file to the server. Destination filename is same as that
     * on server.
     *
     * @param server ftp server name
     * @param username ftp server login name
     * @param password ftp server password
     * @param folder source folder on the ftp server
     * @param fname file to be transferred
     * @return true if successful
     */
    public boolean putFile(String server, String username,
            String password, String folder, String fname)
    throws Exception {
        return false;
    }
    
    public void setMode(int type) {
        
    }
    
    public int getMode() {
        return -1;
    }
    
    
}

[/CODE]

[Arthur Ford]

Hi ,
Thanks a lot for the reply. But unfortunately, FTP is disabled in all the servers.

[Joe Ess]

So what do you have to work with? That drives the solution and without knowing those parameters, we're kind of in the dark.

[Senad Cimic]

Hi,
I'm experiencing similar issue. In my case I have Java application that searches multiple files on few different Unix servers for user specified search terms, but the problem is that FTPing all those files simply takes too much time making application unusable (we have log files that are over 100 MB in size). I'm trying to modify existing application to perform search on Unix servers rather than FTPing every single file. Did anyone did similar thing before? Does anyone have any suggestions? It is possible to write ruby script and manually execute this, but is it possible to incorporate this as part of Java application (perhaps using ssh4j or jruby)?

Thanks in advance.

[Joe Ess]

Have you tried file compression? A zipped-up text file is a fraction (10% or less) of the size of an uncompressed file.

I'm trying to modify existing application to perform search on Unix servers rather than FTPing every single file.

That would be another good solution.