Photo Library Deduplication

Create an array of distinct photo fingerprints leveraging imagededup hashing algorithms (PHash, AHash, WHash, DHash), EXIF data, and metadata

Setup

poetry install
poetry run python src/main.py [library_paths]

Example

poetry run python src/main.py library.photoslibrary ~/Pictures /Volumes/ExternalSSD/Photos

Usage

usage: main.py [-h] [-d DB_PATH] [-v] [--dry-run] [path ...]

Deduplicate photo albums

positional arguments:
  path                  path to .photoslibrary or photo directory

optional arguments:
  -h, --help            show this help message and exit
  -d DB_PATH, --db_path DB_PATH
                        database file path where results persist (defaults to assets/duplicates.db)
  -v, --verbose         verbose logging
  --dry-run             do not write or encode. list what operations would be performed

Goals

• Scan photo directory, Apple .photoslibrary or photos.db with osxphotos
• Create hashes of each photo
  • JPEG, PNG, BMP, GIF formats
  • HEIC, HEIF formats (will try pyheif)
  • Leverage ImageMagick's identification signature
  • Google/Android Photos with Motion
  • Burst Photos
  • Apple Live Photos
  • Video
• Log results to JSON
• Persist photo hashing scores in SQLite db
• Partition and group photos based on hashing scores
• Create new Apple .photolibrary

References

• idealo/imagededup
• RhetTbull/osxmetadata
• JohannesBuchner/imagehash
• devedge/imagehash
• david-poirier-csn/pyheif