convmv error

[woncheol-kim]

convmv error

Hi. convmv is a perl script which can be found at:

https://github.com/skangas/home-bin/blob/master/convmv

This is a file renaming tool and at first it seemed to work fine in a-Shell, but I noticed that --replace option does not actually work.

The best solution for now that I found is to use --exec "pbcopy mv #1 #2.new" option instead of using --replace and manually run the command which is copied to the clipboard. FYI, --exec "pbcopy mv #1 #2" did not work because mv command did not recognize the difference of the two file names.

Now I'm wondering if you can look into the perl code and identify what causes the problem in a-Shell.

For your convenience, please find below a part of the entire code:

sub renameit() {
	my $oldfile=shift;
	my $newname=shift;
	my $cmd;
	$newname=encode_utf8($newname) if ($to_is_utf8);
	if ($opt_exec) {
				$cmd = $opt_exec;
				$cmd =~ s/\#2/\000f8d9hqoäd\#2/g; # make the #2 unique so that file names may contain "#2"
				$cmd =~ s/\#1/\Q$oldfile\E/g;
				$cmd =~ s/\000f8d9hqoäd\#2/\Q$newname\E/g;
				print "$cmd\n";
	} else {
		#print is_utf8($oldfile) ? 1 : 0;
		#print is_utf8($newname) ? 1 : 0;
		&print_ask ("mv \"". &$from_print($dir.$oldfile)."\"\t\"".&$from_print($dir).&$to_print($newname)."\"",$opt_i) or return;
	}
	if (-e $newname and !$opt_exec) {
		if ($opt_replace and !&compare($oldfile,$newname)) {
			if ($opt_notest) {
				unlink $newname or print STDERR "Error: $!\n";
				rename ($oldfile, $newname) or print STDERR "Error: $!\n";
			}
		} else {
			print STDERR &$to_print($newname)," exists and differs or --replace option missing - skipped\n";
		}
	} else {
		if ($opt_notest) {
			if ($opt_exec) {
				system($cmd);
			} else {
				rename ($oldfile, $newname) or print STDERR "Error: $!\n";
			}
		}
	}
}

[holzschu]

Hi,
I'm running tests on my iPad. I tried some simple commands. When I try with --replace, I get the error " exists and differs or --replace option missing - skipped". Is that what you mean with "the --replace option does not actually work"? Or do you have a different error message?

[woncheol-kim]

I'm using a-Shell in Apple Vision Pro, and there's no error message. This caused confusion until I found out that it does not actually work. My latest shell script using convmv is as below:

#!/usr/bin/env sh
convmv -f utf8 -t utf-8-strict --nfc --notest --exec "mv #1 #2.convmv" ./*.*
# Loop through all files with the *.convmv pattern
for file in *.*.convmv; do
  # Check if the file exists to avoid errors
  if [ -e "$file" ]; then
    # Create the new filename by removing the .convmv part
    new_name="${file%.convmv}"
    
    # Rename the file
    mv "$file" "$new_name"
    echo "Renamed: $new_name"
  fi
done

[holzschu]

I'm sorry, but you need to walk me through the steps more slowly, and with greater details. I don't currently understand enough to see what your problem is.

Your first message mentionned an issue with the --replace option. Can you give me an example of a command with the --replace option that doesn't work?

(Also, when you're posting shell scripts in your answer, it's easier to understand if you enclose them within "```sh" (shell script) "```" . It will prevent the sh comments (# comment) from being interpreted as section titles by GitHub markdown interpreter. )

[woncheol-kim]

I believe that convmv with —replace option tries to rename the original file into newly encoded file name, but a-Shell seems to interpret the two names are identical.

An example of the convmv command is the 2nd line of the shell script above, except that it uses —exec option instead of —replace option, and the remaining part of the shell script is a solution that I’ve found:

using —exec option instead of —replace and rename the original file into newly encoded file name plus additional extension ‘.convmv’
and subsequently rename all the files with .convmv extension into ones without .convmv extension.

My issues are thus solved with this shell script, but please aware that a-Shell does not differentiate the original file name and the newly encoded file name, unlike as in standard macOS or linux terminal.

[holzschu]

Thank you for the explanations, I understand better now. I'm still trying to reproduce the issue (in order to, maybe, fix it). Could you provide me with an example of a file name where convmv should actually do something? (with my limited utf8 knowledge, all I'm producing is filenames that are actually identical in utf8 and utf-8-strict).

Just to be clear, the command you want to run is:

convmv -f utf8 -t utf-8-strict --nfc --notest --replace <filename>

and there were no error messages at all?

[woncheol-kim]

Below is a file name that I’ve downloaded from Youtube into my local storage and renamed with Files app:

2024년 외국 영화 Top 10 [Qbu2jjEoo7g].webm

[woncheol-kim]

Apology. I’ve just tried the command with —replace option again, and there is indeed the error message which is the same as you and the file are just deleted.

[holzschu]

Thank you for all the help and explanations. I was able to reproduce your problem:

• I re-downloaded it from youtube using yt-dlp https://www.youtube.com/watch?v=Qbu2jjEoo7g, in order to have the non strict-utf-8 characters in the filename.
• With the --nfc option, convmv does not do anything (which means it thinks the file name is already compatible with normalization form C).
• With the --nfd option, it first removes the file, then fails to rename it (because the file has been removed).

Since calls to the Perl functions unlink and rename are direct calls to system functions, I think that what this tells us is that the system in iOS treats non-strict UTF-8 filenames differently than OSX does. The safest option, given this, is indeed to use your script with --exec "mv #1 #2.convmv". As you found out, --exec "mv #1 #2" does not work, for the same reason that Perl --rename didn't work: it ends up calling the system function rename(), which thinks #1 and #2 are the same (when in fact they are not), and thus doesn't do anything but also doesn't return an error.

Short version: iOS internal system calls tend to convert non-strict-UTF8 characters in file names into their strict UTF8 version, which affects the behaviour of several functions (convmv, mv, others?). The issue happens deep inside the standard library, and thus would be very difficult to fix.