Dedup

Java command line utility to removes duplicate lines from text files.

DeDup is a Java command-line utilty for removing duplicate lines in text files. The dedup process compares adjacent lines only. It does not sort first. The comparison is case sensitive. It removes adjacent indentical lines. It replaces the original file, so do a backup first. You can dedup 1 to N files in one execution, just put their names on the command line. DeDup deletes blank lines from both the beginning and end of the file. It also deletes trailing blanks from each line. It also cleans up line terminators \r \n \r\n and converts them all to platform standard. If the file is already deduped, it does not change the date. Use it with: java.exe -jar dedup.jar myfile.txt anotherfile.txt yetanotherfile.txt or java.exe -jar dedup.jar myfile.txt anotherfile.txt yetanotherfile.txt or, if you have the jar association set up, even even dedup.jar myfile.txt anotherfile.txt yetanotherfile.txt It can handle a list of files, or an os-expanded wildcard. It does not -s to process subdirectories or the use of directory names to process entire directories.

Changes: add icon and pad file

Minimum requirements: Java 1.7+

Operating system: Java,Linux,Mac OS X,Unix,Win2000,Win7 x32,Win7 x64,WinServer,WinVista,WinVista x64,WinXP

Program URL: http://mindprod.com/products1.html#DEDUP

Dedup screenshot

...

Special permissions

You may include Dedup any CD compilation. You may not sell Dedup. It is free.

EULA

This program is for non-military use only. For details on what that licencing restriction means and why I restricted this software that way, see http://mindprod.com/contact/nonmil.html If you incorporate any of my code in your programs, they too must be marked for non-military use only. However it is not your responsibility to enforce that restriction.

Read more