How to correctly use immich-go to remove deplicates of HEIC / JPG files

[vw-kombi]

I am a bit late to the party with immich and have used immich-go to inport the following in this order - all to the one user in immich :

1 - My photos/video files from my server (this only goes up to about 3 years back)
2 - My icloud (from icloudpd) - started using icloud about 1 year ago
3 - My wifes icloud (from icloudpd) - started using icloud about 1 year ago
4 - My google photos - started using google photos about 3 years ago and stopped when icloud took over
5 - My wifes google photos - started using google photos about 3 years ago and stopped when icloud took over

I have found I have >20,000 duplicates reported in immich - seems to be mostly a copy of HEIC and a JPG. Likely from 3 years ago when my workflow was me connecting the phones and using software to copy the photos off and converting HEIC to JPG's as it processed. But as the HEIC's were still on the phones/icloud, now many duplicates of these.

The immich-go has the cool duplicate option - which I ran like so -
./immich-go -server http://192.168.1.175:8080 -key xxx duplicate –yes
Then I ran like this to get rid of more -
./immich-go -server http://192.168.1.175:8080 -key xxx duplicate -ignore-tz-errors -yes

And now its just the pesky HEIC/JPG's left I believe.

And then I found some obscure references to the issue I now had left - the HEIC/JPG copies and the -ignore-extension flag.
Ideally, I would like to say 'keep the HEIC versions' - as they are smaller with no quality loss. Cant find anywhere to do that (maybe an enhancement) so ran it as below like such regardless - but its not actually doing anything -
./immich-go -server http://192.168.1.175:8080 -key xxx duplicate -ignore-tz-errors -yes -ignore-extension

What am I doing wrong ?

[vw-kombi]

Further to this, I have another question - but this may be more immich than immich-go - I often see the duplicates reported in immich are showing the enhanced version and the standard version of the HEIC photo (where iphone has an enhanced image (but iphone keeps both), so one is named IMG_E8211.HEIC and one is named IMG_8211.HEIC - they are often the same size but not always. I guess this is a manual compare and pick, but I would think the 'enhanced' one should be the default winner (or why enhance it in the first place). This may be another future enhancement for immich-go - a flag to check for the E in the file name, and deleted the non 'E' version.

[simulot]

Good to know thanks. I'll come back on this later

[vw-kombi]

One other possible enhancement - is there a way of adding an option to not import screenshots ? The wife takes tons and they are for her phone to remind her of stuff, not really things for our photo collection. If there was an option to not import those, then that would be a big enhancement to the workflow on my end. I have thousands to remove as it is.....

[stevenmcastano]

And then I found some obscure references to the issue I now had left - the HEIC/JPG copies and the -ignore-extension flag.
Ideally, I would like to say 'keep the HEIC versions' - as they are smaller with no quality loss. Cant find anywhere to do that (maybe an enhancement) so ran it as below like such regardless - but its not actually doing anything -

This isn't actually about immich-go... but I've been having the exact same problem! So... and let me preface this buy saying I'm not a developer... at all... but I've been giving it my best shot!

I've been working on messing around with an app called Immich Duplicate Finder which was created by @vale46n1 and I've been loving it so far... check it out, it's not exactly the same... no stacking option, and it only compares 2 images at a time... but I've been hacking away on the python code, one of the only languages I know decently well... and I've gotten it to the point there it will detect if there's an HEIC and it's extracted JPG and show you in the UI so you can better pick which one to toss.

No auto delete yet, not much in the way of config yet... but I'm getting there. If anyone would be willing to help... so show me how to contribute similar code to this project, I'd be happy to pitch in more... and if it's something you're interesting in testing, I'll do what I can to package up the pretty hacky code and get a fork posted!

Normally it highlight differences in a color with green being the "better" one. When it finds HEICs and JPGs and notices they are dups, it usually only highlights the size and makes the larger one green thinking its got more data so it must be better even though everything else matches.

This is the original output from the code in the current repo:

Now, with my new code you can see I added the "GPS Data" section to you can tell if you're delete a copy with GPS data when the other one doesn't have any, I also added the "Camera Info" sections which it puts together from EXIF data in the image, not just the "lens" tag... then the big win for me is that it now can tell if it's an HEIC pair, colors the HEIC file green, make the JPG file red and changes the size coloring to yellow if they're different so you can still see there's a difference, but you'd probably still want to go with the HEIC anyway. Here's what that looks like:

P.S. The "EXT_DIFF_HEIC" text can be turned off... that's just running in a debug mode so I can more easily find examples while testing!