In reply to twb.

Ok, well I guess that technically works, but getting an 8x12 pixel font, even for Latin1, is somewhat suboptimal.

In reply to jwz.

This works for me, I get meaningful characters for both scripts:
sudo apt install fonts-unifont xscreensaver-gl
/usr/libexec/xscreensaver/unicrud --block Katakana,Arabic --font Unifont-120

Omitting the font size (--font Unifont) makes it segfault instead of defaulting to 12pt.

strace -etrace=openat shows it is opening DejaVuSans-Bold.ttf even with --no-titles.

--font System-UI-300 is choosing Cantarell-VF.otf, which is consistent with fontconfig:

$ for i in Monospace Serif Sans Sans\\-Serif System\\-UI; do fc-match "$i" file; done
:file=/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf
:file=/usr/share/fonts/truetype/dejavu/DejaVuSerif.ttf
:file=/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf
:file=/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf
:file=/usr/share/fonts/opentype/cantarell/Cantarell-VF.otf

$ fc-list : family | sort -u
Cantarell
DejaVu Sans
DejaVu Sans Mono
DejaVu Serif
FontAwesome
Noto Color Emoji
OpenSymbol
Quicksand
Quicksand Light
Quicksand Medium
Terminus
Unifont
Unifont CSUR
Unifont Sample
Unifont Upper
Unifont\-JP

In reply to twb.

xlsfonts is not in the picture, that's for XLoadFont. I am no longer using XLFD, I'm just doing XftNameParse ("sans serif-300:style=Bold")

Anyway further speculation is not helpful unless you've installed/uninstalled a thing and made "unicrud --block Katakana,Arabic --font X" work on a Linux system and seen it with your own lying eyes.

In reply to jwz.

Ah, sorry, I wasn't clear on "try [this]".
I am *absolutely* guessing here, because I don't know Xft, but...

You mentioned XftFontOpenXlfd("-*-sans serif-bold-r-*-*-*-180-*-*-*-*-*-*") earlier.
Try "sudo apt install fonts-unifont xfonts-unifont" then... ugh, restart X I guess?
Then try XftFontOpenXlfd("-*-unifont-*-*-*-*-*-120-*-*-*-*-*-*").
If you get actual characters instead of empty boxes for English, Russian, Japanese, and Arabic, then that is one layer of bullshit eliminated from the bullshit cake.
Then you "just" have to find a font foundry that provides all the scripts (languages) you want, in one file, that doesn't look fugly.  (I mentioned Iosevka before, but Iosevka doesn't do Japanese OR Arabic, so ignore Iosevka).

I was just testing it here and after installing all the unifont packages, I still can't convince xlsfonts or xfontsel to "see" Unifont, and I can't be arsed restarting X right now.  I definitely have Unifont working normally in emacs-pgtk and firefox-esr with only fonts-unifont installed (xfonts-unifont and psf-unifont and unifont NOT installed).
The whole -*-*-*-*- stuff for me was a long long time ago and I don't remember too good beyond 1 week ago.

You shouldn't need to remove any Noto packages.
I don't think asking for "Sans Serif" is was ever supposed to work, it would have been "Sans", "Serif", "System-UI" (with the - escaped in various crazy ways), or "Monospace".

What I'm actually personaly using day-to-day is DejaVu + Unifont.
DejaVu covers all the alphabets (latin, cyrillic, greek, georgian, etc) and some abjads (hebrew, arabic).  Unifont is Good EnoughTM for me to get by in Korean and I don't read Japanese or any abugidas well enough to care, and it means my font list is only like 12 files instead of Noto's 2000+.

In reply to twb.

I think what you're saying is "uninstall package noto, install package X and it will work" but I am unclear in what X is for values of Debian.

So I just noticed the original post is using NotoSans-Bold.ttf where the first response is using NotoSerifCJK-Regular.ttc.
Google choose to break up Noto into one file per script (except for en-US-Latn exceptionalism).
If you want all languages, and your font engine can only use one file at a time, then you can't use Noto.

$ fc-scan /usr/share/fonts/truetype/noto/NotoSans-Regular.ttf | grep lang: | grep ja
[no hits -- Japanese NOT supported by this file]

$ fc-scan /usr/share/fonts/opentype/noto/NotoSerifCJK-Regular.ttc | grep lang: | grep ar
[this file supports Chinese/Japanese/Korean but not Arabic/Farsi]

GNU Unifont looks fugly but at least packs all the languages into a single file (unifont.otf).
Iosevka packs everything into one .ttc file and doesn't look ugly, but it's like a 700MB file.
I don't know how to do this stuff in plain Xft, sorry.
I've only ever fought this stuff at the high level -- GTK3 and CSS4 (browsers).

In reply to jwz.

Hmm, it works fine for me with no changes, so if it's not working for you, that'll be down to what fonts you have available. Run it with FC_DEBUG=1 and see what got matched.

I get family: "Noto Color Emoji"(s) "Noto Color Emoji"(w) "Apple Color Emoji"(w) "Segoe UI Emoji"(w) "Twitter Color Emoji"(w) ... and so on for 30 font families, clearly I have too many emoji fonts. It ultimately picks Noto Color Emoji.

In reply to Kyzer.

Interesting, thanks!

Oddly, while it can display many languages, it doesn't seem to be able to display emojis or blackulas:  "👨🧛🏿💩🏃‍🚀"

In reply to jwz.

Pango comes with an example program featuring every bell and whistle it has, called pango-view

Here's a quick and dirty example instead. It will render arbitrary UTF-8 in argv[1]:

/* pangotest.c */
#include <X11/Xlib.h>
#include <pango/pangocairo.h>
#include <cairo/cairo-xlib.h>

int main(int argc, char *argv[]) {
    Display *dpy = XOpenDisplay(NULL);
    int screen = DefaultScreen(dpy);
    Window win = XCreateSimpleWindow(dpy, RootWindow(dpy, screen), 100, 100, 400, 200, 1, BlackPixel(dpy, screen), WhitePixel(dpy, screen));
    XSelectInput(dpy, win, ExposureMask | KeyPressMask);
    XMapWindow(dpy, win);

    cairo_surface_t *surface = cairo_xlib_surface_create(dpy, win, DefaultVisual(dpy, screen), 400, 200);
    cairo_t *cr = cairo_create(surface);

    PangoFontDescription *font = pango_font_description_from_string("Sans Bold 40");

    for (;;) {
       XEvent ev;
        XNextEvent(dpy, &ev);
        if (ev.type == Expose) {
            cairo_set_source_rgb(cr, 0.0, 0.0, 0.0);
            cairo_paint(cr);

            PangoLayout *layout = pango_cairo_create_layout(cr);
            pango_layout_set_text(layout, argv[1], -1);
            pango_layout_set_font_description(layout, font);

            cairo_set_source_rgb(cr, 0.0, 1.0, 0.0);
            cairo_move_to(cr, 50, 80);
            pango_cairo_show_layout(cr, layout);

            g_object_unref(layout);
        }
        else if (ev.type == KeyPress) {
            break;
        }
    }

    pango_font_description_free(font);
    cairo_destroy(cr);
    cairo_surface_destroy(surface);
    XCloseDisplay(dpy);
    return 0;
}

Compile and run like so:

$ gcc -o pangotest pangotest.c $(pkgconf --cflags --libs pango cairo pangocairo) -lX11
$ ./pangotest "مرگ بر آمریکا"

In reply to Joseph Brenner.

It's worse than that - the character shapes you use for Chinese can be really fucking political, because they're different in the PRC than they are in Taiwan.

In reply to flatrute.

What puzzles me is there was a way to embed a locale in Unicode, which looked to me like a good way of providing a hint as to which CJK font to use, but for some reason it was deprecated (around the time of Unicode 5.0 if I remember right).  

Anyway, yeah-- the original sin of Unicode is they were originally intent on squeezing all the world's characters into 16 bits (you see, it was now a 16 bit world!  We've got 16 bit microprocessors now! And it's important to have codepoint boundaries line up with the microprocessor's words because *mumble*).   There's absolutely no way that could be done with separate code points for Chinese and Japanese so they had to share ("the Han Unification", ha ha).  The CJKV block is over 40% of the BMP (the bottom 64k of the codepoints, and the original scheme for unicode).   I gather that the Chinese and Japanese can kind-of read each other's fonts, but they really don't like to-- so the original question jwz asked about displaying a character without knowing the language first, maybe you can find a way, but anything you do is going to be half-assed.

In reply to Stephen Thorne.

If someone were to say: "to render an arbitrary UTF-8 string without knowing what language it is in ahead of time, you should replace your calls to XftFontOpenName("sans serif-300") and XftDrawStringUtf8() with [...this Pango code...]", that might be a helpful response.

So far nobody has said anything like that, and I'm not even sure it's true.

In reply to jwz.

It's all right.  I have it on good authority that your dear leader is soon to issue an executive order forbidding any character set other than ASCII in American software.

Apparently it has slightly held up by an argument amongst the faithful that EBCDIC should be mandated instead.

In reply to jwz.

This sucks. But I'm wondering about the use-case. You say it's for locations on a map. Perhaps you can use pango (or whatever hacked up tool works) to pre-compute the right parameters for the calls.

This would only be possible if you had a dictionary of map labels ahead of time, but the rate at which the names of countries and towns in the world changes is quite low, so perhaps this is a better tradeoff than making xscreensaver run untrustworthy fuzzymatching libraries at runtime.

I'm suggesting you use pango as either a manual step during preparing your release, or make it part of the build-deps during compilation.

In reply to Kyzer.

This is a clever, if quite manual, hack.

Easier than maintaining a mapping between countries and their replacement letters, though, would be to just use ligatures (which the free FontForge supports). When the font encounters the full name of a country and a ligature for that combination of letters exists, it could swap in an outlined SVG of the localized name....

In reply to twb.

It's well-known pain, we call this this problem Ye Olde¹ Thorn².

1: https://en.wikipedia.org/wiki/English_articles#Ye_form
2: https://en.wikipedia.org/wiki/Thorn_(letter)

K3n.

In reply to jwz.

I *think* it’s that freetype is a font engine (render 1 codepoint correctly, or 2 with kerning), and pango/harfbuzz is a text engine (render whole words correctly).
Freetype alone can only pretend to be a text engine for baby scripts where you just lay each grapheme out one after another, like Suomi or Ukrainian but not Farsi or Punjabi or Bangla – which never worked properly on Linux before harfbuzz.
I don’t know why xft is failing on Japanese because – other than rubigana – that only needs basic one-after-another layout and the glyphs don’t change the shape of nearby glyphs. And you probably aren’t caring about Hanja unification issues yet.

Ok I guess technically e.g. Latin has different glyphs for /u/ depending on where it is in a word (turning into a “v”) and English has sharp s in older manuscripts, but most fonts just ignore that stuff. It’s an optional opentype setting in EB Garamond, and trying to use it makes you realise how shitty fontconfig is. Browsers can do it via css3 and hard coding which codepoints use which font files, which was tedious but worked.

In reply to Kyzer.

FontForge is free, available for macOS, and can import SVGs! I haven't used it, but the guy who made the Bad Apple font notes that FontForge has a command line interface, and he wrote a script to import all his SVGs as glyphs.

In reply to jwz.

There's this horrible line where text rendering crosses from easy to insanely complex. Xft lives on one side of that line. Pango et al live on the other side.

By needing a single word using complex text layout, you damn yourself.

Pango is just GNOME's solution to this, it's nice they didn't hardwire it into GTK. Qt uses a different library (WorldType), but even then, if you say your Qt app needs any complex text layout, it brings in a whole other library just for that.

My wacky idea would be to produce your own font with pre-composed characters, each character being the name of a country. Then you can skip all the hard bits about doing text shaping for every language in the world as code.

The hard bit then becomes the manual labour of turning county names first into SVGs using some "text to SVG" website and a specific font, then making a font out of these SVGs (a cursory web search suggests GlyphSmith can easily do this but it costs $29 if you want more than 10 glyphs in the font). Then you can print "A" for ભારત, "B" for ایران, "C" for ᠮᠤᠩᠭᠤᠯ
ᠤᠯᠤᠰ and so on.

In reply to Kyzer.

So xscreensaver could then be included into a font (as wasm) and then be displayed by xscreensaver...

In reply to moof.

Look, here is the problem that I am currently trying to solve that led me down this godforsaken rathole:

• Here is the name of a country, in the official spelling in its native language.
• I have a UTF-8 string that unambiguously represents those characters.
• I would like to use that text as a label on a map.

Linux has made this god damned impossible.

No amount of telling me "well, but it's really hard, there could be all of these edge cases" is going to make this sound any less ridiculous. It is patently absurd that I can't load a font and have those characters show up as something other than a blank rectangle.

It manages to find the glyph for a goddamned poop with eyeballs ferchrissakes, it should work with a random Chinese glyph too!

In reply to jwz.

if you want to be able to emit utf-8 sequences that aren't just "emit one independent glyph after another, left-to-right", then you're going to have to use harfbuzz or some other character shaping library. producing Arabic - or even worse, Mongolian - is a non-trivial task, and there are still edge conditions in many renderers that get it wrong. (See https://www.unicode.org/notes/tn57/utn57-mong-4.pdf for a delightful 80 page PDF that goes into the details and phases of character shaping and joining for Mongolian.)

In reply to moof.

Hardcoding in support for one particular language is not the goal here.

In reply to raveguide.

fwiw, https://github.com/hrs-allbsd/kterm does japanese text without pango or cairo - but it also doesn't use Xft, instead just using XDrawString (and Xaw), and requiring a bunch of fiddly config specifying whether it's using utf-8 or euc-jp or shift-jis or whatever.

In reply to Elladan.

Yep and it made quite an amount of hardcore computer users in those East Asian countries pissed off if I am not wrong. There is a workaround by inserting dedicated character select control codes but it does not work for all of them.

In reply to #?.info.

For the record, X11-versus-Wayland has nothing to do with this. Xft is a thin wrapper over FontConfig, which is the same in both universes, so Wayland won't save you.

In reply to jwz.

what… and I can’t be more emphatic about this: the fuck?!

In reply to Szymon Sokół 🇵🇱🇪🇺🇺🇦.

Apparently the X11 font rendering system requires you specify a language as well as a font, then gets the glyths for that language/font combination.

Which is pretty ugly. Honestly, this is another reason I think X11 should be...

...fixed and not replaced with something worse.

In reply to Szymon Sokół 🇵🇱🇪🇺🇺🇦.

don't you also need to know what CJK language the text is written in due to Han Unification, or something?

In reply to jwz.

is it because no font contains all existing Unicode glyphs, or is there another reason?