If the makeotf command path contains Hangul, a UnicodeDecodeError is raised.

[sandoll]

I installed AFDKO using pip, and the version of makeotf is as follows. : makeotf.py v2.6.0 Dec 7 2018
I am using OneDrive and the account name includes "(주)산돌커뮤니케이션", so I get the following error when using makeotf.
If you use makeotfexe in the same path, you can make fonts without problems.

Traceback (most recent call last):
File "c:\python27\lib\runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "c:\python27\lib\runpy.py", line 72, in run_code
exec code in run_globals
File "C:\Python27\Scripts\makeotf.exe_main.py", line 9, in
File "c:\python27\lib\site-packages\afdko\makeotf.py", line 2683, in main
getOptions(makeOTFParams, args)
File "c:\python27\lib\site-packages\afdko\makeotf.py", line 1504, in getOptions
checkInputFile(makeOTFParams)
File "c:\python27\lib\site-packages\afdko\makeotf.py", line 1520, in checkInputFile
convertFontIfNeeded(makeOTFParams)
File "c:\python27\lib\site-packages\afdko\makeotf.py", line 1936, in convertFontIfNeeded
report = fdkutils.runShellCmd(commandString)
File "c:\python27\lib\site-packages\afdko\fdkutils.py", line 45, in runShellCmd
return tounicode(stdoutdata, encoding='utf-8')
File "c:\python27\lib\site-packages\fontTools\misc\py23.py", line 131, in tounicode
return s.decode(encoding, errors)
File "c:\python27\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc1 in position 41: invalid start byte

[miguelsousa]

@sandoll what command line options did you use?

[sandoll]

what command line options did you use?

The command line I used was:
makeotf -f cidfont.ps -o output.otf -fi cidfontinfo -ff output_feature.txt -mf .\FontMenuNameDB -cs 3 -cl 23 -ch UniAKR-UTF32-H -nshw -r

[miguelsousa]

I haven't been able to reproduce this problem, so here's a few more questions:

1. Which version of Windows are you on?
2. What result do you get when you type the command chcp?
3. Does the problem happen with this command makeotf -f cidfont.ps -r?

[sandoll]

In addition, makeotf_commands.bat file exists in the path of "F:\OneDrive - (주)산돌커뮤니케이션[AK-9]fontgen", and the content of this batch file is "makeotf -f cidfont.ps -o output.otf -fi cidfontinfo -ff output_feature.txt -mf .\FontMenuNameDB -cs 3 -cl 23 -ch UniAKR-UTF32-H -nshw -r".'

1. I am using Windows 10 Pro.
2. I tried the chcp command on drive F and it says "활성 코드 페이지: 949". The translation is "Active code page: 949".
3. command makeotf -f cidfont.ps -r also produces a UnicodeDecodeError.

Running makeotf_commands.bat on the "D:\OneDrive - (S)sandoll[AK-9]fontgen" path will produce the font without any problems,
Running makeotf_commands.bat on the "D:\OneDrive - (주)sandoll[AK-9]fontgen" path will result in a UnicodeDecodeError.

UnicodeDecodeError occurs when the absolute path simply contains Hangul.

[kenlunde]

Keep in mind that modern hangul syllables, such as U+C8FC 주, decompose into sequences of two or three combining jamo, such as <U+110C,U+116E> in this case. File systems often apply Normalization Form NFD, and I think that this explains the errors.

[readroberts]

This is certainly a bug that needs to be fixed. I suspect I need to install the necessary language pack in order to reproduce the bug, as on my English Windows 10 system , I can create a directory named '주', but the Python os.path methods show this as only Unicode 63, '?".

[miguelsousa]

@sandoll can you install the AFDKO on Python 3 and test if this problem still persists?

[miguelsousa]

@sandoll I believe b092bc5 fixes the UnicodeDecodeError you reported.
I've issued a pre-release build v2.8.5a0. Please give it a try once the files are up on pypi.

[miguelsousa]

To reproduce this I had to change my Windows 10 system locale to Korean, via Control Panel > Region > Administrative (screenshot below)

and then run this command

makeotf -f 산돌커뮤니케이션\cidfont.ps -cs 3 -cl 23 -ch 산돌커뮤니케이션\UniAKR-UTF32-H -nshw -r

Running the makeotf command from inside the directory 산돌커뮤니케이션 (i.e. makeotf -f cidfont.ps -cs 3 -cl 23 -ch UniAKR-UTF32-H -nshw -r) did not produce the UnicodeDecodeError.

[miguelsousa]

In the process, I learned that on Python 2.7 Windows sys.argv (which makeotf uses for loading command line arguments) is unable to handle characters outside the system's encoding. The only workarounds are,

• switch to Python 3
• use getopt
• use argparse

[sandoll]

@miguelsousa I downloaded b092bc5 as a temporary fix and replaced fdkutils.py on the C:\Python27\Lib\site-packages\afdko path. I will soon switch to Python 3 as the only workaround you have noticed.
thank you :-D.

[sandoll]

@miguelsousa
I switched to Python 3.7.2 (32bit), but I still get UnicodeDecodeError.
I installed AFDKO using pip, and the version of makeotf is as follows. : makeotf.py v2.7.3 Mar 5 2019
I downloaded b092bc5, the method I solved last time, and replaced fdkutils.py in the c:\users\sanga\appdata\local\programs\python\python37-32\lib\site-packages\afdko path, but that did not solve the problem.

Traceback (most recent call last):
File "c:\users\sanga\appdata\local\programs\python\python37-32\lib\site-packages\afdko\fdkutils.py", line 90, in get_shell_command_output
str_output = tounicode(bytes_output, encoding='utf-8')
File "c:\users\sanga\appdata\local\programs\python\python37-32\lib\site-packages\fontTools\misc\py23.py", line 131, in tounicode
return s.decode(encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc1 in position 27: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\users\sanga\appdata\local\programs\python\python37-32\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "c:\users\sanga\appdata\local\programs\python\python37-32\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Users\SangA\AppData\Local\Programs\Python\Python37-32\Scripts\makeotf.exe_main.py", line 9, in
File "c:\users\sanga\appdata\local\programs\python\python37-32\lib\site-packages\afdko\makeotf.py", line 2712, in main
getOptions(makeOTFParams, args)
File "c:\users\sanga\appdata\local\programs\python\python37-32\lib\site-packages\afdko\makeotf.py", line 1557, in getOptions
checkInputFile(makeOTFParams)
File "c:\users\sanga\appdata\local\programs\python\python37-32\lib\site-packages\afdko\makeotf.py", line 1573, in checkInputFile
convertFontIfNeeded(makeOTFParams)
File "c:\users\sanga\appdata\local\programs\python\python37-32\lib\site-packages\afdko\makeotf.py", line 2024, in convertFontIfNeeded
psName = get_font_psname(filePath)
File "c:\users\sanga\appdata\local\programs\python\python37-32\lib\site-packages\afdko\makeotf.py", line 2032, in get_font_psname
'tx', '-dump', '-0', font_path], std_error=True)
File "c:\users\sanga\appdata\local\programs\python\python37-32\lib\site-packages\afdko\fdkutils.py", line 93, in get_shell_command_output
encoding=sys.getfilesystemencoding())
File "c:\users\sanga\appdata\local\programs\python\python37-32\lib\site-packages\fontTools\misc\py23.py", line 131, in tounicode
return s.decode(encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc1 in position 27: invalid start byte

[miguelsousa]

@sandoll can't reproduce it. Please uninstall the afdko (pip3 uninstall afdko) and re-install it. Version 2.8.7 worked fine for me, as you can see below.

C:\Users\labuser\Desktop>makeotf -f z산돌커뮤니케이션\cidfont.ps -cs 3 -cl 23 -ch z산돌커뮤니케이션\UniAKR-UTF32-H -nshw -r
makeotf [Warning] Could not find default features file. Font will be built without any layout features.
makeotf [Warning] Could not find FontMenuNameDB file. Font will be built with menu names derived from PostScript name.
makeotf [Warning] Could not find an Adobe CMAP Mac encoding mapping file. If you want a Mac cmap subtable, please specify the Mac CMAP encoding file with the '-cm' option.
makeotf [Warning] Could not find a Unicode Variation Sequence file in the expected 'c:\users\labuser\appdata\local\programs\python\python37-32\lib\site-packages\afdko\resources' subdirectory.
makeotfexe [WARNING] FontMenuNameDB file was not specified or not found. [SourceSerif-Test]
makeotfexe [WARNING] <SourceSerif-Test> cmap{plat=1,script=3,lang=23}: empty encoding
makeotfexe [WARNING] <SourceSerif-Test> Macintosh Adobe CMap not seen
makeotfexe [WARNING] <SourceSerif-Test> Unicode variation Selector File not seen
Built release mode font 'SourceSerif-Test.otf' Revision 1.000

C:\Users\labuser\Desktop>python --version
Python 3.7.2

C:\Users\labuser\Desktop>pip3 list
Package           Version
----------------- -----------
afdko             2.8.7
appdirs           1.4.3
booleanOperations 0.8.2
cu2qu             1.6.5
defcon            0.6.0
fontMath          0.4.9
fontParts         0.8.5
fontPens          0.2.4
fonttools         3.38.0
fs                2.4.4
lxml              4.3.2
MutatorMath       2.1.2
pip               19.0.3
psautohint        1.9.2b2
pyclipper         1.1.0.post1
pytz              2018.9
setuptools        40.6.2
six               1.12.0
ufonormalizer     0.3.6
ufoProcessor      1.0.4

[sandoll]

@miguelsousa I found the option below and solved the problem!!!!
Control Panel > Region > Administrative > Change system locale > Beta: use unicode utf-8 for worldwide language support > checkbox enable > Restart the computer

Thank you so much for checking :)