How to remove duplicate from ArrayList?

[jay lai]

Anyone can help me with remove duplicate from ARrayList?
I need to add item into this ArrayList , and remove all the duplicated records if it exists.

I have an POJO object call MyClass

 

 
public MyClass {
 
private String userid;
private String userName;
 
//including getter/setter for those userid, userName
 
}
 

ArrayList dupslist = new ArrayList();
Set finalSet = new HashSet();
for (int x=0; x< dataList.size(); x++)
{
duplist.add( new MyClass(useridvalue, usernamevalue);
 
}
 
finalSet.addAll(duplist);
duplist.clear();
duplist.addAll(finalSet);
 
//but it seems does not work, 

Please help, i really need this to be resovled. Greatly appreciated with your help

[Jesper de Jong]

The way you show us here most likely does not work because you have not added hashCode() and equals() methods to your class MyClass. You must add these methods so that HashSet can properly detect when two MyClass objects are equal.

See the description of the methods hashCode() and equals() in the API documentation of class java.lang.Object for details on how you should implement those methods in MyClass.

[Maneesh Godbole]

need to add item into this ArrayList , and remove all the duplicated records if it exists.

Wouldn't it be simpler to check before you add?
for (int x=0; x< dataList.size(); x++){
MyClass myClass=new MyClass(useridvalue, usernamevalue);
if(!(dupList.contains(myClass))
duplist.add(myClass);
}

Hope this helps.
Best of luck

[jay lai]

Thank you for a quick reply:
This is what I added in Myclass

public boolean equals(Object obj)
{
 if (obj instanceof MyClass)
{
 MyClass m = (MyClass) obj;
 return (userid == m.getUserId() && username == m.getUserName());
}
 return false;
}
 
public int hashCode()
{
  // what do I implement here?
}
 

Again thank and greatly apprciated for your help

[jay lai]

Thanks Maneesh Godbole , I did try your suggestion, but still not eliminate dups from the ArrayList

Any other suggestion, please help.
regards

[Justin Chu]

Use a.equals(b) instead of == sign. (check for null pointer first)

[Tony Docherty]

Originally posted by jay lai:
Thanks Maneesh Godbole , I did try your suggestion, but still not eliminate dups from the ArrayList.

The code posted by Maneesh doesn't remove duplicates it only checks to see if the ArrayList already contains a duplicate object or not and only adds the new object if there isn't already a duplicate object in the ArrayList. If you need to remove all duplicates and then add the new object you could do something like:

for (int x=0; x< dataList.size(); x++){
MyClass myClass=new MyClass(useridvalue, usernamevalue);
 
// loop until all duplicates are removed
while(dupList.remove(myClass)) {}
 
duplist.add(myClass);
}

If the list is only ever filled in this loop then there will never be more than 1 duplicate so you don't need to the while loop just the the line

dupList.remove(myClass);

BTW I used the same for loop as shown in your code for clarity but it doesn't make sense. You are adding dataList.size() number of identical MyClass objects (they are identical because you are creating them all with the same initial values of id and name which your equals method uses to test for equality) so you'll only ever have 1 object in dupList.

[jay lai]

Hi Thanks for your suggestion
But I did what you suggest, it not work either.
Thanks for the help

for (int x=0; x< dataList.size(); x++){
MyClass myClass=new MyClass(useridvalue, usernamevalue);
 
// loop until all duplicates are removed
while(dupList.remove(myClass)) {} //[B] it is a close braclet here, do nothing here?[/B]
 
duplist.add(myClass);
}
 

[Tony Docherty]

Originally posted by jay lai:
But I did what you suggest, it not work either.

In what way didn't it work?

EDIT: I've just looked at your equals method and there's definitely a problem there which maybe why it's not working. The line

return (userid == m.getUserId() && username == m.getUserName());

should be (assuming they are both Objects and not primitive types)

return (userid.equals(m.getUserId()) && username.equals(m.getUserName()));

and you need to check for null values as well as Chu Tan previously suggested.
[ August 07, 2007: Message edited by: Tony Docherty ]

[jay lai]

Hi Again:



I did implement like this


In equals methods:

return (if (userid !=null && userid.equals(m.getUserId()) && (if this.username !=null && username.equals(m.getUserName()));
 

how about in public int hashcode() method? what do I have to do,
thanks very much

[Nicholas Jordan]

You should not have to remove them, unless that is a specific program constraint that one search for and remove duplicates.

package demo;
 
import java.util.Map;//
 
/* Note: Recompile with -Xlint:unchecked for details.
 * A map cannot contain duplicate keys.
 * Each key can map to at most one value.
 */
 
public class PreventDuplicate extends Object
{
    Map m;
    // Every class has Object as a superclass.
    PreventDuplicate ()
    {
        super();//
    }
    //
    public boolean add( PlainOldJavaObject pojo )
    {
        if(m.containsKey((Object) pojo.userid))
        {
            return false;//
        }
        else
        {
            PlainOldJavaObject added =
            (PlainOldJavaObject) m.put(pojo.userid, pojo);//
            if (added == null){return true;}
            return false;// object already existed.
        }
    }
    //public Object get(Object key)
    public Object get(String userid)
    {
        if(m.containsKey((Object) userid))
        {
              return (PlainOldJavaObject) m.get((Object) userid);//  public Object
        }
        return null;//
    }
 
    // OO question here as to whether to indicate map held value when asked to remove.
    public boolean remove(String userid)
    {
        if(m.containsKey((Object) userid))
        {
            m.remove((Object) userid);// Removes the mapping for this key from this map
            return true;//
        }
        return false;//
    }
    //
    public boolean find(String userid)
    {
        return m.containsKey((Object) userid)  ;//
    }
}
 
class PlainOldJavaObject
{
    String userid;//
    String username;//
    PlainOldJavaObject( String uid, String name)
    {
        userid = new String(uid);//
        username = new String(name);//
    }
}

There is more than one way to do this, what this code does is not add if the Object is there already. You can write update methods to update a record that already exists.
[ August 07, 2007: Message edited by: Nicholas Jordan ]

[jay lai]

thank you so much for posted and responded.

One question: So I just instantiate this PrevenDuplicates and call add(), remove() accordingly?

 
ArrayList dupslist = new ArrayList();
Set finalSet = new HashSet();
[I]PreventDuplicate pd = new PreventDuplicate();[/I]
for (int x=0; x< dataList.size(); x++)
{
MyClass mc = new MyClass(dataList.getUserId(), dataList.getUsername());
pd.add(mc); //?????
 
}
 
 

[jay lai]

Hi:
I kind of figure out, but now the method remove duplication but it remove everything and only left 1 record


For example
In my dupList Array: it stored
userid username
123 john
345 mike
345 mike
456 lisa

the method remove everything and keep only
userid username
123 john

it should keep
userid username
123 john
345 mike
456 lisa

What is wrong with this method?

for (int i=0; i< dataList.size(); i++)
{
 OtherJavaClass ojc  = (OtherJavaClass) dataList.get(i);
MyClass mc = new MyClass(ojc.getUserId(), ojc.getUsername());
duplist.add(mc)
 
}
 
// then invoke the duplicate method
removeDuplicate(duplist);
 
public static void removeDuplicate(ArrayList arlList)
 {
 Set set = new HashSet();
 List newList = new ArrayList();
 Iterator iter = arlList.iterator();  
 while(iter.hasNext()) {
 Object element = iter.next();
   if (set.add(element))
      newList.add(element);
    }
    arlList.clear();
    arlList.addAll(newList);
 

Again always thanks for all the help and suggestion

[Jesper de Jong]

Originally posted by jay lai:
how about in public int hashcode() method? what do I have to do,
thanks very much

Did you look at the Javadoc documentation of the method hashCode() in class java.lang.Object? It explains in detail how you should implement this method.

Note, if you write an equals() method, then you are also required to write a hashCode() method. One of the requirements of hashCode() is that two objects are equal, the hashCode() method must return the same value when called on those two objects. If it doesn't, then containers like HashSet can get confused when you add objects of that type to them.
[ August 08, 2007: Message edited by: Jesper Young ]

[Nicholas Jordan]

[jay lai :]One question: So I just instantiate this PrevenDuplicates and call add(), remove() accordingly?

The way I wrote it, you do not need any of the ArryaList and other code you have in your post of posted Tuesday, August 07, 2007 8:03 PM but be dilligent in reading the code, and testing it many times. Look for things like what Chu Tan suggests. I just wrote 9,000 characters of code, just to discuss an idea with some of the ranchers. It is useful to write small, tight code - but any code must be thought through many times.

For example:

public int hashCode()
{
// Implement nothing here, unless there is some reason you want a hash code
}

What do you need the hash code for ? You need it because you have been reading about hash codes and you think you need it. Eventually you will see that I keyed the map on the string called user id. Well, userid's should be unique - this is where code reuse comes in, the Java Collections have worked on this problem so I just used their work. I just wrapped it in a class that included your comments and accessor methods.

Use it like this;

PreventDuplicate pd = new PreventDuplicate();

if(pd.add( PlainOldJavaObject pojo )).... code ....;//

PlainOldJavaObject pojo = pd.get(String userid);

if(pd.remove(String userid)).... code ....;//

if(pd.find(String userid)).... code ....;//


It wraps everything you need, you do not need any of the externals that you wrote. You do need the stuff that Jesper Young is talking about, it goes in the PlainOldJavaObject, and would be written thus:

    /**
     * Compares two strings lexicographically.
     * The comparison is based on the Unicode value of each character in
     * the strings. The character sequence represented by this
     * String object is compared lexicographically to the
     * character sequence represented by the argument string. The result is
     * a negative integer if this String object
     * lexicographically precedes the argument string. The result is a
     * positive integer if this String object lexicographically
     * follows the argument string. The result is zero if the strings
     * are equal; compareTo returns 0 exactly when
     * the equals(Object) method would return true.
     */
 boolean equals(Object obj)
{
     PlainOldJavaObject pojo = (PlainOldJavaObject) obj;
     // you can do this in several lines if you want
     // other posters may provide clearer coding constructs
     return  (obj.userid.compareTo(this.userid) == 0x00)?true:false;
}
 

Comparator states:
/* Compares its two arguments for order. Returns a negative integer,
* zero, or a positive integer as the first argument is less than, equal
* to, or greater than the second.
*/
and is written:

public int compare(Object o1, Object o2)
{
return o1.userid.compareTo(o2.userid);
}

OR

public int compare(PlainOldJavaObject o1, PlainOldJavaObject o2)
{
return o1.userid.compareTo(o2.userid);
}

depending on what the current release takes as it's idea of an object


The way I would write the hascode method is just to call Sring.hashCode on user id - this will more than suffice for any collection that needs a hash code to operate:

// this goes in the pojo
public int hashCode()
{
return userid.hashCode();//Returns a hash code for userid.
}
[ August 12, 2007: Message edited by: Nicholas Jordan ]

[Rambo Rolling]

Simple Way to do this .. I am sure this can help..

public ArrayList removeDups()
{
ArrayList list = new ArrayList();
list = addList();
for(int i = 0 ; i < list.size(); i++)
{
NqDlvrOutOfCompliance nqDlvrOutOfCompliance = (NqDlvrOutOfCompliance) list.get(i);
String waybill = nqDlvrOutOfCompliance.getWaybill();
for(int j = i+1; j < list.size(); j++)
{
NqDlvrOutOfCompliance nqDlvrOutOfCompliance1 = (NqDlvrOutOfCompliance) list.get(j);
if(waybill.trim().equalsIgnoreCase(nqDlvrOutOfCompliance1.getWaybill()))
list.remove(j);
}
}

return list;
}


public ArrayList addList()
{
ArrayList list = new ArrayList();
NqDlvrOutOfCompliance nqDlvrOutOfCompliance;
for (int i = 0 ; i < 3 ; i++)
{
nqDlvrOutOfCompliance = new NqDlvrOutOfCompliance();
nqDlvrOutOfCompliance.setWaybill("1234");
list.add(nqDlvrOutOfCompliance);
}
for (int i = 0 ; i < 3 ; i++)
{
nqDlvrOutOfCompliance = new NqDlvrOutOfCompliance();
nqDlvrOutOfCompliance.setWaybill("ABCD");
list.add(nqDlvrOutOfCompliance);
}
for (int i = 0 ; i < 3 ; i++)
{
nqDlvrOutOfCompliance = new NqDlvrOutOfCompliance();
nqDlvrOutOfCompliance.setWaybill("XYZ");
list.add(nqDlvrOutOfCompliance);
}
return list;
}




public class NqDlvrOutOfCompliance {

String waybill;

/**
* author xyz
* @return
*
*/
public String getWaybill() {
return waybill;
}

/**
* author xyz
* @param string
*
*/
public void setWaybill(String string) {
waybill = string;
}
}


Call the removeDups method from any class ... This method can be modified to accept the arraylist (in which the dups have to be removed) as the input.

[Rob Spoor]

"Rambo",

Please read your private messages.

Also:
http://faq.javaranch.com/java/UseCodeTags
http://faq.javaranch.com/java/DontWakeTheZombies. This thread was almost 2 years old.

[Gyanesh Kumar]

package rnd.collections;

/*Gyanesh Kumar Pandey
* DuplicateArray
*
*
*/

public class DuplicateArray
{
public static final int NUM_ITEMS = 50;

// Return true if array a has duplicates; false otherwise
public static boolean duplicates( Object [ ] a )
{
for( int i = 0; i < a.length; i++ )
for( int j = i + 1; j < a.length; j++ )
if( a[ i ].equals( a[ j ] ) )
return true; // Duplicate found

return false; // No duplicates found
}





// Test the duplicates method
public static void main( String [ ] args )
{
Integer [ ] a = new Integer[ NUM_ITEMS ];
for( int i = 0; i < a.length; i++ )
a[ i ] = new Integer( i );

System.out.println( "Should be false: " + duplicates( a ) );

a[ NUM_ITEMS - 1 ] = a[ 0 ];

System.out.println( "Should be true: " + duplicates( a ) );
}

public void rearr(int aa[])
{
int l=aa.length;

for (int j =l-1;j<l/2;j--)
{



}

}

}
@Gyanesh

[zahid zubair]

Very simple. Use

Set uniqueSet = new HashSet(duplicateList);

[Campbell Ritchie]

Maybe simple, but it is likely the original poster is no longer reading this thread. Have a look at this.

[Jevgeni Pogorelov]

One possible solution in case we nead remove dublicates in List by one or more field condition.

public class SomeObject () {

private String field1;

private String field2;

public String getField1() {
return field1;
}

public void setField1(String field1) {
this.field1 = field1;
}

public String getField2() {
return field2;
}

public void setField2(String field2) {
this.field2 = field2;
}

public List<SomeObject> removeDublicated(List<SomeObject> someObjectsList) {

List<SomeObject> ret = new ArrayList<SomeObject>();

for (SomeObject aSomeObjectsList : someObjectsList) {

String field1 = aSomeObjectsList.getField1();
String field2 = aSomeObjectsList.getField2();

boolean exist = false;

for (SomeObject someObject : ret) {

if (field1.trim().equalsIgnoreCase(someObject.getField1())
&& field2.trim().equalsIgnoreCase(someObject.getField2())) {

exist = true;

}
}

if (!exist) {

ret.add(aSomeObjectsList);
}

}

return ret;
}

}

[Paul Clapham]

It is STILL likely the original poster is no longer reading this thread. Have a look at this.

[Rob Spoor]

This?

[Jevgeni Pogorelov]

My answer was for all user who try to find answer in the internet and got link to this forum from search engines. Like me:-) I hope it helpful for all, not only for original poster.

[prathap venkata naga yelugula]

Hi Friends,

ArrayList l=new ArrayList();
l.add("sasi");
l.add("sasi1");
l.add("sasi2");
l.add("sasi3");
l.add("sasi4");
l.add(0);
l.add(0);

l.add(null);
l.add(null);
l.add(null);
System.out.println(l);
HashSet hs=new HashSet(l);
l.clear();
l.add(hs);
System.out.println(l);



Cheers!
Prathap.

[sudheer kiran]

oh lot of replies for this question.there are good answers.let me give a final explanation.
here you want to add objects to a collection and remove duplicates.
but how do java knows that two objects are equal.for that it uses "==" operator.
meaning for that operator is if both are referencing to same memory location.

Object's equal method uses this operator,String and wrapper classes implemented the equals n hash code methods.
so if you are comparing your own objects like MyClass objects then that class must implement equals and hashcode methods.sp just add these lines to your code in MyClass.java

  public boolean equals(Object obj)   
{   
 if (obj instanceof MyClass)   
{   
 MyClass m = (MyClass) obj;   
 return (userid == m.getUserId() && username == m.getUserName());   
}   
 return false;   
}   
    
public int hashCode()   
{   
return   userId;
}   
  

now HashCannot nontain duplicates of Myclass Objects.

[xsunil kumar]

Why you not using Set. Set does not allow duplicates.

[Harshana Dias]

see the problem here is not override the hashCode..so basically even the two object has same attributes (eg:123 john) they have different hash codes.