Add capability to use custom fonts that include characters from the Unicode Basic Multilingual Plane

[Bodmer]

UTF-8 decoder added to print stream so the UTF-8 encoded Unicode strings produced by the compiler are rendered correctly on the graphical displays.

Example for ILI9341 TFT showing printing Hiragana characters to screen here:

hiragana_example.zip

This update makes glyphs from the entire 16 bit Unicode Basic Multilingual Plane available. See comments in the example. This example uses ~45% of FLASH on an UNO so it should run fine on any hardware. The included font files were generated by fontconvert tool with the Unicode Hiragana code point range of 12353 - 12435.

[Bodmer]

Screenshot from test sketch:

[Bodmer]

I suspect that to get around the lack of UTF-8 decoding in the library folk are having to use a work-around and bypass the print class by calling the write() member function directly.

So, the current pull request needs to have a UTF-8 on/off switch included. The cp437() on/off switch function does not have a a generic name so, unfortunately, a further function will need to be added somehow. Perhaps a gfxConfig() function is the best approach as that can also be used for future switchable extensions too, perhaps associated with graphics, stream and character set handling options.

[pljakobs]

Hi Bodmer, I'm glad to see someone taking on the extended character set topic.
I had been thinking about this for years but never gotten around to actually doing it.
One of my concerns was the required flash space for a full 16 bit slice so I thought it would be more apropriate to implement iso8859 codepages.
I will pull this into my experimental fork and have a look at it.

[Bodmer]

OK, I have added a switch function to my local copy so will push that to Git in the next few hours.

My thoughts are:

1. To make the minimum changes needed and (importantly) to make complete use of existing library features, but not preclude functional expansion at a future time (e.g.21 bit code point)
2. No definitive numerical point code boundary constraints or size limts on the glyph block start and end (no power of 2 constraints on boundaries). But start with a 16 bit Unicode code point variable.
3. Allow any number of separate contiguous point code blocks, initially from the Basic Multilingual Plane, to be loaded as arrays (one at a time) from arrays in FLASH
4. Make the library user repsonible for restricting the range/size, number of glyph blocks and selection of the needed array, according to the hardware they are using and it's capabilities.
5. Support UTF-8 (aka use encoded Unicode) on the print stream only, via the write() function
6. Add a UTF-8 on/off switch (default is UTF-8 off for legacy sketch support) to alow direct use of 8 bit write calls to access the top 128-255 characters of the GLCD legacy font.
7. Keep the UTF-8 decoder simple and not try to manage corruptted data streams
8. Make the drawChar function volatile, so users can create extension libraries that for example: add diacritical marks to characters, write characters right to left or up to down, add their own font formats, use alternative storage media for the fonts etc etc. Thie drawChar will be fed decoded Unicode code points.

This means the update can be transparent to existing users until deliberately switched on but that the rich character/glypg/grapheme set from the Basic Multilingual plane is available if wanted. The Arduino IDE/GCC compiler already uses UTF-8 encoding for the print stream so this is the obvious way forward to me. The GFX "free fonts" will thus be still availble in the basic 32-127 range and in blocks anywhare in the 16 bit code point space. The code size impact is minimal.

[Bodmer]

I have uploaded the latest changes.

The copy of the ILI9341 library here has two new examples that demo the added features. Adapting for other display hardware should be straightforward.

[idea--list]

@Bodmer That is simply awesome!

Would pretty much produce my own font.h file. Installed MinGW, newest Freetype, etc. Can generate fonts (fontconvert myfont.ttf 12 32 512 > myfont), however characters beyond 0xFF simply do not render if using my own fonts. While your hiragana fonts work ok.
If i check my own fonts with this tool, beyond 0xff i find hex ids that consist of only 3 digits like 0x100 etc. While UTF-8 fonts should have a four digit hex identifier if i am right.

So my question is how can i generate fonts.h that has special character identifiers in the proper hex format?

[Bodmer]

The font data has been created following the instructions here:
https://www.youtube.com/watch?v=L8MmTISmwZ8

Note that online converters for Adafruit_GFX compatible fonts are
available but these typically only use characters in the range 32-127,
and thus do not include the accented characters. These online converters
can however still be used with this sketch but the example characters
used must be changed.

[idea--list]

First thanks for the quick reply!

Yes i saw that video yesterday and right now i also found this link of yours.
There you say you used this command to generate the hiragana.h font:
fontconvert C:\Windows\Fonts\sourceFont.ttf 12 12353 12435 > hiragana12.h

just did the same with my arial.ttf font and now even my glyphs have hex ids like 0x3041.
I can confirm it by checking it here. of course in arial.ttf there was no such character but i can redraw it in that tool and save the modified arial.h file. Now by pasting 0x3041 hex id in the arduino sketch the display renders exactly the glyph i just drawed.

However if i build the font with this command:
fontconvert C:\Windows\Fonts\arial.ttf 12 32 512 > arial.h
I get a file that contains a lot of real glyphs (as arial.ttf is a UTF8 font) however this time most characters have a hex id like 0x123. If i paste such hex ids in the arduino sketch then something weird gets displayed despite the font.h file contains the glyph.

For me it seems as a possible bug in GFX library, i mean it seems to be able to lookup 1 byte characters that have a hex id between 0x00 and 0xff. It can also look up 2 byte characters with hex ids between 0x0000 and 0xffff (like in your hiragana font). But it seems to fail finding glyphs correctly if their hex id is between 0x000 and 0xfff.

While converting font.ttf files shipped with windows, after 0xff (=255 dec) the next glyph id is always 0x100 (=256 dec) but those do not get rendered right while the font.h contains most special fonts exactly in that range...

Can you please confirm this on your side or tell me what i might be missing?

[Bodmer]

Create an example complete with font file for me to test. Post it here by making a zip file and dragging to the Git Write window.

[idea--list]

Here it is:

UTF8_test.zip

[Bodmer]

There are quite a few problems with your sketch.

1. A char array should be zero terminated otherwise you will likely get garbage characters at the end of the string. So add a zero thus:
   char testch_ff[] = {'A','b', 0x21, 0x26, 0x4d, 0x7e, 0xff, 0x00};

2. A char array can only hold 8 bit values, so if you try to load it with a larger value then only the least significant byte will be stored. So your line thus:
   char testch_ffff[] = {0x3041, 0x3042, 0x3043, 0x3044, 0x3045};
   Will actually create an array:
   char testch_ffff[] = {0x41, 0x42, 0x43, 0x44, 0x45};

3. To use UTF8 encoded characters you must switch on UTF8 decoding with a line thus:
   display.setAttribute(UTF8_ENABLE , true);
   To use a font with only characters in the range 0-0xff turn it off:
   display.setAttribute(UTF8_ENABLE , false);

4. To use 16 bit Unicode characters you have to use either the character itself in a string, e.g "こんにちは" or you must use the universal character name format e.g.:
   display.println("\u3041\u3042\u3043\u3044\u3045");
   display.println("Works: \u0100\u0106\u0124\u015c\u00ff"); // ALWAYS define as 16 bit hex

Make these changes to your sketch and see if it works.

[idea--list]

@Bodmer i could not stop and i think i got it working now ;-)

Here you have a modified version of v1.3.6 of the GFX library, but that does not compile on my board.

So now i updated v1.5.6 with your modifications (will open a PR in your branch) and now it compiles to my board and i can get any custom characters displayed if i pass them as char array or as String.

Passing by hex code still does not work however, but i think i can live without that :-)

[idea--list]

@Bodmer Thanks for taking your time and for the hints!

[idea--list]

@Bodmer just created PR #244 where i describe everything and mention your work and help.
Thanks again!

[MOSHEMEL]

HI bodmer I built fonts as you said using the fontconvert app for hebrew letters and I checked it using the Adafruit GFX Pixel font customiser and I can see the font from 0x5d0 -0x5ea but my problem is when using tft.print("\u05D0\u05D1"); I do not see different letter but the same letter
Do I need to take your Adafruit_gfx libray?

[MOSHEMEL]

or I need to do something else?

[Bodmer]

You would need to download the adapted library here.

Note that the adapted library is a temporary proof of concpt library and that Adafruit have not adopted the changes needed in their master library and I will not be able to resolve any issues you have with it.

I would use this library to extend the fonts and permit use of UTF8 encoded strings. That library is well written and is designed to be used with the original Adafruit_GFX library.

[Jarvis785]

Hi @Bodmer and @idea--list
I came across your Adafruit-GFX-Library fork you did a great job !
But when I tried your Hiragana example on Adafruit_NeoMatrix but it didn't worked can you please help me ?
if this example works the i will use my converted "devanagari" font. Any help is appreciated !
`#include <Adafruit_GFX.h>
#include <Adafruit_NeoMatrix.h>
#include <Adafruit_NeoPixel.h>

#include "hiragana10.h"
#include "hiragana12.h"
#include "hiragana16.h"

// Easily remembered names for the fonts
#define HIRAGANA10 &hiragana10pt8b
#define HIRAGANA12 &hiragana12pt8b
#define HIRAGANA16 &hiragana16pt8b

#define PIN 8

Adafruit_NeoMatrix matrix = Adafruit_NeoMatrix(12, 14, PIN,
NEO_MATRIX_BOTTOM + NEO_MATRIX_LEFT +
NEO_MATRIX_ROWS + NEO_MATRIX_ZIGZAG,
NEO_RGB + NEO_KHZ800);

void setup() {
matrix.begin();
matrix.setBrightness(70);
matrix.setTextColor(matrix.Color(255,0, 0));//Red
matrix.setAttribute(UTF8_ENABLE, true);
matrix.setAttribute(CP437_ENABLE, true);
matrix.setFont(HIRAGANA12);

matrix.drawRect(0,0,10,12, matrix.Color(0,255, 0));// Test draw green rectangle (Working)
matrix.show();
delay(1000);

}

void loop() {
matrix.fillScreen(0);//Black background
matrix.setTextColor(matrix.Color(255,0, 0));//Red
matrix.setCursor(0, 14);
matrix.println(("さようなら"));
// matrix.println(F("AB"));//This is working perfecty fine when i remove setFont(HIRAGANA12);

matrix.show();
delay(100);
}`
Hiragana_Neomatrix_Test.zip

[colesnicov]

can I write Cyrillic?

[DoomHammer]

@Bodmer can you resolve the merge conflicts?

[Bodmer]

@DoomHammer
I think that would be nugatory effort given that this pull request is 3 years old!

[DoomHammer]

I believe this year to be the year of UTF-8 on microchips.

[colesnicov]

I believe this year to be the year of UTF-8 on microchips.

I've used mcufont.