SCRIPT TO FIND MULTIPLE ENTRIES

[Gopu Akraju]

I tried in oerl but due to extensive search it is taking quite a long time and hence I thought I will give a try with shell scripts.

My data file looks as below with thousands of records. The basic columns are sa below:

   
+   test   NS008_456_R0030_3008   67   223  
+   ghi    NS008_456_R0030_3678   17   678 
+   ggl    NS008_456_R0030_3678   17   270 
+   ghi    NS008_456_R0030_3672   17   209 
+   ghi    NS008_456_R0030_3690   17   280 
+   ghi    NS008_456_R0030_3690   15   267 
 

My objective is to find the records which has multiple enteries on the 3rd column. For example in the above case,

   
+   ghi    NS008_456_R0030_3678   17   678 
+   ggl    NS008_456_R0030_3678   17   270 
 

and

  
+   ghi    NS008_456_R0030_3690   17   280 
+   ghi    NS008_456_R0030_3690   15   267 
  

are the candidate record which I am interested.

Hence I need the records whose 3rd column has more than one record as above. Please let me know.

[Joe Ess]

Gopu Akraju wrote:I tried in oerl but due to extensive search it is taking quite a long time

This sounds like a non-trivial operation and with thousands of records, is bound to take a lot of time. I don't think shell scripts are faster than perl, especially for this kind of operation which is not well-suited for scripting. You may gain some speed with Java or C, but probably not much (speculation on my part).
Maybe if you showed us your perl, we could help you tweak it (well, maybe someone else. I'm not a perl guy).