How DeDup 1.5.3 finds duplicate videos on Android
The video scan in DeDup 1.5.3 reads the video list from your phone's media library, then groups videos two ways: files with exactly the same size, and videos whose sampled frames look alike. It saves its progress on the phone, so a later scan can skip videos it has already handled.
What the scan reads
DeDup asks Android's media library for each video's name, size, duration, type and file path, newest added first. It keeps only files with a known video extension, such as mp4, mkv, mov, avi or webm. The code lists 25 extensions.
For each video with a duration, DeDup then takes three still frames, at 10%, 50% and 90% of the way through. It shrinks each frame to a tiny grid of brightness comparisons and stores the result as a 64-bit fingerprint. Up to three videos are processed at the same time. If a video takes longer than 8 seconds, or its frames can't be read, DeDup keeps it without fingerprints.
How duplicates are detected
Each new video is checked against the ones already seen, in this order:
- Same size. If its size in bytes matches another video's (and isn't zero), it joins that group.
- Similar frames. Otherwise, if it has fingerprints, DeDup compares them with other videos' fingerprints. A frame counts as matching when no more than 3 of the 64 bits differ. Two videos are grouped when at least two of the three frames match.
Same size alone is enough. The code doesn't compare the contents of two videos that have identical sizes. A video with no fingerprints can only be grouped by size.
What's saved for next time
DeDup writes to a database on the phone after every 5 newly scanned videos, and again when the scan ends, even if you stop it. It stores each video's details and fingerprints, plus a group key for videos that landed in a group.
When you reopen the screen, saved groups appear straight away. A banner reads "Previous scan saved" and offers "Clear". The main button then says "Resume", and an extra "Rescan" button appears.
- Resume skips any video whose address is already saved. A line says how many were skipped. The scan code matches by address alone.
- Rescan clears the saved data and starts from scratch.
What the results screen offers
The header counts groups, and while scanning it shows how many videos have been scanned. Each group is titled "Duplicate Group (N items)" and shows its videos two to a row, five groups per page. Every video has a thumbnail, a checkbox, its duration, size and file name, and a Play button. The first video in each group is labelled KEEP. That's the first one in the group, and I found no quality ranking for videos.
Tap a video to select it; a trash icon then appears in the top bar. On Android 11 and newer, deleting goes through the same Android confirmation as photos. Guests are asked to sign in first. A sort and filter menu offers largest, smallest and most copies.
Scope of this post
Checked against: DeDup 1.5.3 source code (Android).
Not tested on a device: how long a scan takes, how accurate the frame matching is, how the screen looks, or what happens with unusual video formats. I read the code for each of these but didn't run it.