msys2/clang64 unicode paths

[eddiezato]

I recently started building in msys2/clang64 environment and ran into a problem with unicode paths in some apps. F.e., the latest git-version of exiv2 has it.

In an attempt to investigate the problem, I made a simple test program (not an expert in c++):

#include <iostream>
#include <filesystem>
#include <string>

namespace fs = std::filesystem;
using namespace std;

template <typename T> string f(T in) {
    if (fs::exists(in))
        return "exists          ";
    else
        return "doesn't exists  ";
}

int main() {
    setlocale(LC_CTYPE, ".utf8");
    
    string ss = "Darth-Vader-SW-Персонажи-Звездные-Войны-фэндомы-2304085.jpg";
    fs::path ps = ss;
    fs::path pu = fs::u8path(ss);
    
    wstring ws = L"Darth-Vader-SW-Персонажи-Звездные-Войны-фэндомы-2304085.jpg";
    fs::path pw = ws;

    cout << "string              " << f(ss) << ss << endl;
    cout << "path from string    " << f(ps) << ps << endl;
    cout << "u8path from string  " << f(pu) << pu << endl << endl;

    cout << "wstring             " << f(ws); wcout << ws; cout << endl;
    cout << "path from wstring   " << f(pw) << pw << endl;
}

Then compiled it in clang64 and ucrt64 environments:

user@host CLANG64 ~/123
$ clang++ -Wall -std=c++17 123.cpp -o 123clang

user@host UCRT64 ~/123
$ clang++ -Wall -std=c++17 123.cpp -o 123ucrt

Output:

D:\msys2\home\user\123> 123clang.exe
string              doesn't exists  Darth-Vader-SW-Персонажи-Звездные-Войны-фэндомы-2304085.jpg
path from string    doesn't exists  "Darth-Vader-SW-Персонажи-Звездные-Войны-фэндомы-2304085.jpg"
u8path from string  exists          "Darth-Vader-SW-

wstring             exists          Darth-Vader-SW-
path from wstring   exists          "Darth-Vader-SW-

D:\msys2\home\user\123> 123ucrt.exe
string              exists          Darth-Vader-SW-Персонажи-Звездные-Войны-фэндомы-2304085.jpg
path from string    exists          "Darth-Vader-SW-Персонажи-Звездные-Войны-фэндомы-2304085.jpg"
u8path from string  exists          "Darth-Vader-SW-Персонажи-Звездные-Войны-фэндомы-2304085.jpg"

wstring             exists          Darth-Vader-SW-Персонажи-Звездные-Войны-фэндомы-2304085.jpg
path from wstring   exists          "Darth-Vader-SW-Персонажи-Звездные-Войны-фэндомы-2304085.jpg"

Is this some kind of bug or intended behavior that needs to be adjusted in the code before building in msys2/clang64?

[mati865]

Unicode support in libc++ is not great, for more info see mstorsjo/llvm-mingw#145

[eddiezato]

I'm the least worried about cout and more concerned about this:

string ss = "Darth-Vader-SW-Персонажи-Звездные-Войны-фэндомы-2304085.jpg";
fs::path ps = ss;
...
fs::exists(ss); // false
fs::exists(ps); // false

This code can't find the file when compiling against libc++ and can against libstdc++.
And rewriting code with u8path here and there doesn't make much sense, since u8path is deprecated in c++20.