File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Meaningless Drivel and the fly likes cat|cut|cut|sort|uniq|sort > file Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Other » Meaningless Drivel
Bookmark "cat|cut|cut|sort|uniq|sort > file" Watch "cat|cut|cut|sort|uniq|sort > file" New topic
Author

cat|cut|cut|sort|uniq|sort > file

David O'Meara
Rancher

Joined: Mar 06, 2001
Posts: 13459

So it turns out running this on 24Gb takes a while.

Who knew?
David O'Meara
Rancher

Joined: Mar 06, 2001
Posts: 13459

actually I've been running the following on the combined log files for the month from 100 client sites. Personally I think the problem is too much logging.

cat *.log* |cut -d' ' -f7|cut -d'?' -f1|sort |uniq -c|sort -n>file.count
David O'Meara
Rancher

Joined: Mar 06, 2001
Posts: 13459

Ho hum.

I was hoping to have a look at the results, but I might go to bed.
Doug Slattery
Ranch Hand

Joined: Sep 15, 2007
Posts: 294
Maybe this will save you a second or two . (Warning, I didn't test it, but I think you get the idea...)

sed -e 's/^\([^ ]* \)\{6\}\([^?]*\)?/\2/' < *.log* > reallybig.log; sort -n reallybig.log | uniq -c > file.count

Aloha,
Doug

-- Nothing is impossible if I'mPossible
Pat Farrell
Rancher

Joined: Aug 11, 2007
Posts: 4658
    
    5

When N is big, even N ln(N) is big
Doug Slattery
Ranch Hand

Joined: Sep 15, 2007
Posts: 294
yes, but at least now, it's not 2N

Aloha,
Doug

-- Nothing is impossible if I'mPossible
David O'Meara
Rancher

Joined: Mar 06, 2001
Posts: 13459

keeping in mind we're in MD and not Linux, I'll just say "One day I need to look at sed"
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: cat|cut|cut|sort|uniq|sort > file