Removing Duplicate Rows From Sorted Text File With EsProc
Remove duplicates from sorted text file: write script p1.dfx in esProc, import data with @i option, perform distinct on ordered sequence & export result.
Problem description & analysis Below is data in text file txt.txt: Each row in the text file contains a number. There are duplicates and rows are already sorted. We are trying to trim the file to remove the duplicates and generate a new ordered file, as shown below: Solution Write the following script p1.dfx in esProc: Explanation: A1 Import the txt data; @i option enables returning a sequence when the result set has only one column. A2 Perform distinct on A1’s ordered sequence and export result to result.txt. Explanation: A1 Import the txt data; @i option enables returning a sequence...