Remove Python Console Duplicates from Lists

by Dinesh 2012-08-27 12:56:35

Remove Python Console Duplicates from Lists

If you want to remove duplicates from a list, just put every element into a dict as a key (e.g. with None as value) and check dict.keys(). I found this optimized version in the WWW:

from operator import setitem
def distinct(l):
d = {}
map(setitem, (d,)*len(l), l, [])
return d.keys()

This makes use of the fact that map fills up shorter lists (in this case the empty one) with None. Newer Python versions allow for an even more concise formulation of the above:*

def distinct(l):
return dict.fromkeys(l).keys() # works with python 2.3+

def distinct24(l):
return list(set(l)) # needs python 2.4 or later

These are clearly concise enough to be used in-place. Note that all variants so far have two limitations:
The elements are returned in arbitrary order.
Lists with unhashable elements (e.g. sub-lists) may not be processed.

You must LOGIN to add comments