lsdup

lsdup is a command line tool which can assist in finding and removing files with identical contents. It has a comparison mode which lets you tell lsdup to only read the files in the particular directory and do not consider any for duplicates. lsdup also has normal deduplication where if given a single directory, it will sort the originals from the duplicates.

lsdup works by first looking through all of the directories you specified, and creates a text file which contains all of the originals and duplicates. You can look through and modify the list in your favorite text editor. Finally, feed the file back into lsdup to move or delete the files as needed.