Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.forth > #58442 > unrolled thread

Xft and complex unicode rendering

Back to article view | Back to comp.lang.forth

Contents

  Xft and complex unicode rendering krishna.myneni@ccreweb.org - 2017-05-01 08:13 -0700
    Re: Xft and complex unicode rendering krishna.myneni@ccreweb.org - 2017-05-01 08:27 -0700
      Re: Xft and complex unicode rendering krishna.myneni@ccreweb.org - 2017-05-06 09:21 -0700
    Re: Xft and complex unicode rendering anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-02 13:54 +0000
      Re: Xft and complex unicode rendering krishna.myneni@ccreweb.org - 2017-05-02 14:54 -0700
        Re: Xft and complex unicode rendering anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-04 10:51 +0000
          Re: Xft and complex unicode rendering krishna.myneni@ccreweb.org - 2017-05-04 04:47 -0700
            Re: Xft and complex unicode rendering krishna.myneni@ccreweb.org - 2017-05-04 05:02 -0700
      Re: Xft and complex unicode rendering krishna.myneni@ccreweb.org - 2017-05-02 19:24 -0700
        Re: Xft and complex unicode rendering anton@mips.complang.tuwien.ac.at (Anton Ertl) - 2017-05-04 10:50 +0000
          Re: Xft and complex unicode rendering krishna.myneni@ccreweb.org - 2017-05-04 04:42 -0700
            Re: Xft and complex unicode rendering Alex <alex@rivadpm.com> - 2017-05-04 13:21 +0100
    Re: Xft and complex unicode rendering Bernd Paysan <bernd@net2o.de> - 2017-05-02 23:14 +0000
      Re: Xft and complex unicode rendering krishna.myneni@ccreweb.org - 2017-05-02 18:31 -0700
        Re: Xft and complex unicode rendering Bernd Paysan <bernd@net2o.de> - 2017-05-06 01:25 +0000

#58442 — Xft and complex unicode rendering

Now that Unicode and its encoding as UTF-8 are supported with the xchars addition to the Forth standard, one would expect that complex text rendering should be relatively easy to do in Forth with an appropriate library. 

First, there is a lack of Unicode support for terminals for rendering composite characters in languages with multiple code points, e.g. Telugu. The Gnome terminal, under Linux, certainly doesn't support it, and a search on the web does not seem to turn up anything. Does a decent terminal under Linux which supports complex text exist?

Second, I succeeded in using the kForth library bindings to the Xft library to render the base Telugu alphabet within an X-window, using the XftDrawStringUtf8 library function. However this does not work at all for composite characters using diacritics with the base characters. Very frustrating. It appears that the Xft library can't handle complex character representation, or I'm not using the appropriate library functions -- the documentation for this library is also not that great. Obviously other libraries which do handle complex character composition exist, as many X applications such as gedit and LibreOffice handle these languages just fine.

Have any Forthers succeeded in rendering UTF-8 encoded strings in other (non-western) languages with complex character composition? If so, which libraries did you use for the rendering?

Krishna

[toc] | [next] | [standalone]

#58443

On Monday, May 1, 2017 at 10:13:23 AM UTC-5, krishna...@ccreweb.org wrote:

> ... 
> Second, I succeeded in using the kForth library bindings to the Xft library to render the base Telugu alphabet within an X-window, using the XftDrawStringUtf8 library function. ...

Here's the code, in very rough form at this point. I plan to turn it into a program for teaching the Telugu alphabet, with a little refinement of layout, user interface, and synthesized speech.

\ demo-xft-telugu.4th 
\
\ Demonstrate use of Xft library for UTF-8 telugu alphabet rendering.
\
\ Krishna Myneni, Creative Consulting
\   for Research & Education,  http://ccreweb.org
\
\ Revisions:
\   2017-05-01  km  based on kForth's libs/x11/demo-xft.4th

include ans-words
include asm
include strings
include lib-interface
include libs/x11/libX11
include libs/x11/libXft

Also X11
Also Xft

1 constant DEFAULT_BDWIDTH  \ border width

\ utf-8 encodings
z" అఆఇఈఉఊఋఌఎఏఐఒఓఔౠౡకఖగఘఙచఛజఝఞటఠడఢణతథదధనపఫబభమయరఱలళవశషసహ" 
ptr telugu_alphabet

z" కఁకంకఃకాకికీకుకూకృకౄకెకైకొకోకౌక్కౕకౖ" ptr telugu_diacritics 
\ Font
z" Lohit Telugu-18" ptr font_name

0 value   mainW
0 value   screen
0 value   root_win
0 value   visual
0 value   colormap
0 value   font
0 value   ftdraw
0 value   fgpix
0 value   bgpix
0 value   Done

variable  dpy
variable  x
variable  y
variable  w
variable  h

XRenderColor xrcolor
XftColor     txtcolor
XGlyphInfo   extents
XEvent       event

: alloc-ftcolor ( red green blue alpha acolor -- )
    >r
    xrcolor XRenderColor->alpha w!
    xrcolor XRenderColor->blue  w!
    xrcolor XRenderColor->green w!
    xrcolor XRenderColor->red   w!
    dpy @  visual colormap xrcolor r>  XftColorAllocValue drop    
;

: draw-utf8 ( x y az u -- )
    2>r 2>r
    ftdraw txtcolor font 2r> 2r> XftDrawStringUtf8  ;

: draw-window ( -- )
    dpy @ mainW XClearWindow drop
    10 x !  h @ 2+ y !
    4 0 DO
      x @  
      y @ h @ 2* I 1+ * + 
      telugu_alphabet I 30 * + 30 draw-utf8
    LOOP
;

: main ( -- )

   \ open connection to X display
   0 XOpenDisplay dup dpy !
   0= IF  cr ." cannot connect to X server" cr ABORT  THEN
 
   dpy @ XDefaultScreen to screen
   dpy @ screen
   2dup XDefaultVisual   to visual
   2dup XRootWindow      to root_win
   2dup XBlackPixel      to fgpix
   2dup XWhitePixel      to bgpix
   2dup XDefaultColormap to colormap
   2dup font_name  XftFontOpenName to font
   2drop

   dpy @  font  telugu_alphabet 3 extents XftTextExtentsUtf8
   extents XGlyphInfo->height w@ h !
   extents XGlyphInfo->width  w@ w !
     
   \ create top level window
   dpy @  root_win 10 10 w @ 20 * h @ 2* 8 *
   DEFAULT_BDWIDTH fgpix bgpix  XCreateSimpleWindow  to mainW

   \ Xft draw context
   dpy @ mainW  visual colormap  XftDrawCreate  to ftdraw

   \ allocate text color
[ HEX ]
\  red  green  blue  alpha
   0     0     255   ffff  txtcolor  alloc-ftcolor \ Xft text color
[ DECIMAL ]
   
   \ select inputs
   dpy @ mainW  ExposureMask ButtonPressMask or 
   XSelectInput drop
   
   \ make window visible
   dpy @ mainW XMapWindow drop
   ." click on the window to exit" cr
 
   \ retrieve and process events
   BEGIN
     Done invert
   WHILE 
      dpy @ event XNextEvent drop
      event XAnyEvent->window @ mainW =  IF
         event @
         CASE
            Expose      OF  draw-window  ENDOF
            ButtonPress OF  true to Done  ." good-bye!" cr  ENDOF
         ENDCASE
      THEN
   REPEAT

   \ close connection to display

   ftdraw  XftDrawDestroy 
   dpy @ visual colormap txtcolor  XftColorFree
   dpy @ mainW XDestroyWindow drop
   dpy @ XCloseDisplay drop         
;

main

[toc] | [prev] | [next] | [standalone]

#58737

On Monday, May 1, 2017 at 10:27:54 AM UTC-5, krishna...@ccreweb.org wrote:
> On Monday, May 1, 2017 at 10:13:23 AM UTC-5, krishna...@ccreweb.org wrote:
> 
> > ... 
> > Second, I succeeded in using the kForth library bindings to the Xft library to render the base Telugu alphabet within an X-window, using the XftDrawStringUtf8 library function. ...
> 
> Here's the code, in very rough form at this point. I plan to turn it into a program for teaching the Telugu alphabet, with a little refinement of layout, user interface, and synthesized speech.
> ...

A more polished version of the program, which displays the characters in a regular, resizable grid, is available here. This program does not require the xchar wordset because it is limited to just the base Telugu alphabet characters. For display of actual telugu text, in a terminal or in an X window, the xchar words will be very useful.

ftp://ccreweb.org/software/kforth/examples/libs/x11/demo-xft-telugu.4th

A picture of the output of the program is here:

http://ccreweb.org/software/kforth/telugu-alphabet.png

Krishna

[toc] | [prev] | [next] | [standalone]

#58496

krishna.myneni@ccreweb.org writes:
>First, there is a lack of Unicode support for terminals for rendering compo=
>site characters in languages with multiple code points, e.g. Telugu. The Gn=
>ome terminal, under Linux, certainly doesn't support it, and a search on th=
>e web does not seem to turn up anything. Does a decent terminal under Linux=
> which supports complex text exist?

I dimly remember that I used an uxterm (and that functionality has
supposedly been absorbed into xterm when you use the appropriate
settings (LANG etc.)) for my experiments with Unicode and xchars, and
also successfully experimented with composing characters: E.g., take a
look at Figure 1 in
<http://www.complang.tuwien.ac.at/anton/euroforth/ef05/ertl-paysan05.pdf>.
I think that the contents of the ." in the line after the "pad 9 type"
uses composite characters.

Some years later I tried to repeat the experiments (using either the
Gforth history or another file containing the input I used in my first
experiments, but had problems getting all (or even the majority of)
these characters to display properly; apparently the then-available
fonts I tried (I tried several) no longer included the more exotic
characters:-(.  In particular, I think that they did not include the
characters of the part with the composing characters.

I hope things are getting better again after getting worse.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2017: http://www.euroforth.org/ef17/

[toc] | [prev] | [next] | [standalone]

#58539

On Tuesday, May 2, 2017 at 9:11:48 AM UTC-5, Anton Ertl wrote:
> krishna.myneni@...org writes:
> >First, there is a lack of Unicode support for terminals for rendering compo=
> >site characters in languages with multiple code points, e.g. Telugu. The Gn=
> >ome terminal, under Linux, certainly doesn't support it, and a search on th=
> >e web does not seem to turn up anything. Does a decent terminal under Linux=
> > which supports complex text exist?
> 
> I dimly remember that I used an uxterm (and that functionality has
> supposedly been absorbed into xterm when you use the appropriate
> settings (LANG etc.)) for my experiments with Unicode and xchars, and
> also successfully experimented with composing characters: E.g., take a
> look at Figure 1 in
> <http://www.complang.tuwien.ac.at/anton/euroforth/ef05/ertl-paysan05.pdf>.
> I think that the contents of the ." in the line after the "pad 9 type"
> uses composite characters.
> 

Thanks for the link. That paper will be useful in making use of the xchars words in my own code. I've been studying them on the forth-standard.org website. At present, the Gnome Terminal does not render Telugu characters appropriately.

> Some years later I tried to repeat the experiments (using either the
> Gforth history or another file containing the input I used in my first
> experiments, but had problems getting all (or even the majority of)
> these characters to display properly; apparently the then-available
> fonts I tried (I tried several) no longer included the more exotic
> characters:-(.  In particular, I think that they did not include the
> characters of the part with the composing characters.
> 
> I hope things are getting better again after getting worse.
> 

I've found some academic papers on modifying the underlying vte library used by Gnome Terminal to support Indian languages. The Cairo library, or the higher level Pango library might be needed, though. 

Krishna

[toc] | [prev] | [next] | [standalone]

#58628

krishna.myneni@ccreweb.org writes:
>Thanks for the link. That paper will be useful in making use of the xchars words in my own code.

Note that the paper describes a first draft of what later became the
xchar wordset, and there were some changes along the way.

There have been suggestions to add more rationale explaining the
purpose to the xchars wordset part of the standard.  If that's what
you find useful about the paper, maybe you or I should add examples or
comments in the appropriate sections in forth-standard.org.

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2017: http://www.euroforth.org/ef17/

[toc] | [prev] | [next] | [standalone]

#58632

On Thursday, May 4, 2017 at 6:04:54 AM UTC-5, Anton Ertl wrote:
...
> There have been suggestions to add more rationale explaining the
> purpose to the xchars wordset part of the standard.  If that's what
> you find useful about the paper, maybe you or I should add examples or
> comments in the appropriate sections in forth-standard.org.

Examples would be an excellent addition to the xchars rationale. I'm planning to use the xchars words to simplify my X window example for teaching the Telugu alphabet. I think we can come up with some useful examples, in a couple of different languages, to add. Counting the number of characters in a UTF-8 encoded string is one of the most useful words.

Krishna

[toc] | [prev] | [next] | [standalone]

#58636

On Thursday, May 4, 2017 at 6:47:17 AM UTC-5, krishna...@ccreweb.org wrote:
...
> Counting the number of characters in a UTF-8 encoded string is one of the most useful words.
 
Sorry, I should have said "code points", not "characters", in the above sentence.

Krishna

[toc] | [prev] | [next] | [standalone]

#58547

On Tuesday, May 2, 2017 at 9:11:48 AM UTC-5, Anton Ertl wrote:
> krishna...org writes:
> >First, there is a lack of Unicode support for terminals for rendering composite characters in languages with multiple code points, e.g. Telugu. 
> 
> I dimly remember that I used an uxterm (and that functionality has supposedly been absorbed into xterm when you use the appropriate settings (LANG etc.)) for my experiments with Unicode and xchars, ...

Problem solved. Emacs (24.3.1 using GTK+) provides a terminal (term) which renders complex text correctly, including Telugu. Running Forth under this terminal allows experimenting with UTF-8 text output and xchar implementations.

Krishna

[toc] | [prev] | [next] | [standalone]

#58627

krishna.myneni@ccreweb.org writes:
>Problem solved. Emacs (24.3.1 using GTK+) provides a terminal (term) which renders complex text correctly, including Telugu.

Thanks.  Which font are you using?

- anton
-- 
M. Anton Ertl  http://www.complang.tuwien.ac.at/anton/home.html
comp.lang.forth FAQs: http://www.complang.tuwien.ac.at/forth/faq/toc.html
     New standard: http://www.forth200x.org/forth200x.html
   EuroForth 2017: http://www.euroforth.org/ef17/

[toc] | [prev] | [next] | [standalone]

#58631

On Thursday, May 4, 2017 at 5:51:44 AM UTC-5, Anton Ertl wrote:
> krishna...org writes:
> >Problem solved. Emacs (24.3.1 using GTK+) provides a terminal (term) which renders complex text correctly, including Telugu.
> 
> Thanks.  Which font are you using?
> 

That's an interesting question. Within Forth running in the emacs terminal, I simply copy and paste a utf-8 string and type it to the console, e.g.

s" తెలుగు" type
తెలుగు ok

I don't know how the terminal determines which font to use. This is the same as in the gedit text editor, in which I can enter Telugu text, but there's no font setting of which I'm aware. I'm not sure what resources or configuration files these programs use to select a font appropriate for the utf-8 encoded characters. In contrast, when I enter Telugu text into LibreOffice, I select the "Lohit Telugu" font which appears in the font selection list (there may be more than one Telugu font, however).

Even Gnome Terminal selects an appropriate font, but the code points do not always form a composite character and are sometimes output as separate characters. For example, the word, "telugu" is written as three composite characters: three base characters, "ta", "la", "ga", and three diacritic marks, following each base character, which modify the sound of each base character. Thus, there are 6 code points (U+xxxx codes), which, in UTF-8, end up as a three byte sequence per code point, resulting in an 18 byte string:

"te": E0  B0  A4  E0  B1  86
"lu": E0  B0  B2  E0  B1  81
"gu": E0  B0  97  E0  B1  81

Krishna

[toc] | [prev] | [next] | [standalone]

#58641

On 5/4/2017 12:42, krishna.myneni@ccreweb.org wrote:
> On Thursday, May 4, 2017 at 5:51:44 AM UTC-5, Anton Ertl wrote:
>> krishna...org writes:
>>> Problem solved. Emacs (24.3.1 using GTK+) provides a terminal
>>> (term) which renders complex text correctly, including Telugu.
>> 
>> Thanks.  Which font are you using?
>> 
> 
> That's an interesting question. Within Forth running in the emacs
> terminal, I simply copy and paste a utf-8 string and type it to the
> console, e.g.
> 
> s" తెలుగు" type తెలుగు ok

My newsreader renders this; but the question is, does it
do so correctly? I don't know. Thunderbird on Windows is using Calibri, 
so I presume this font contains
  an extensive set of glyphs. It is 3 characters according to the string 
functions that manage the cursor.

-- 
Alex

[toc] | [prev] | [next] | [standalone]

#58543

Am Mon, 01 May 2017 08:13:22 -0700 schrieb krishna.myneni:
> Have any Forthers succeeded in rendering UTF-8 encoded strings in other
> (non-western) languages with complex character composition? If so, which
> libraries did you use for the rendering?

I plan to use a combination of Freetype and Harfbuzz for that, but so 
far, I'm just using Freetype.  Essentially, you first use Harfbuzz to 
convert your string into the correct order with the correctly selected 
ligatures, and then you use Freetype to render the individual glyphs.

That's not yet tried, so far, but it's fairly easy, because you only need 
the buffers part of Harfbuzz: https://behdad.github.io/harfbuzz/harfbuzz-
Buffers.html

-- 
Bernd Paysan
"If you want it done right, you have to do it yourself"
net2o ID: kQusJzA;7*?t=uy@X}1GWr!+0qqp_Cn176t4(dQ*
http://bernd-paysan.de/

[toc] | [prev] | [next] | [standalone]

#58546

On Tuesday, May 2, 2017 at 6:17:39 PM UTC-5, Bernd Paysan wrote:
> Am Mon, 01 May 2017 08:13:22 -0700 schrieb krishna.myneni:
> > Have any Forthers succeeded in rendering UTF-8 encoded strings in other
> > (non-western) languages with complex character composition? If so, which
> > libraries did you use for the rendering?
> 
> I plan to use a combination of Freetype and Harfbuzz for that, but so 
> far, I'm just using Freetype.  Essentially, you first use Harfbuzz to 
> convert your string into the correct order with the correctly selected 
> ligatures, and then you use Freetype to render the individual glyphs.
> 
> That's not yet tried, so far, but it's fairly easy, because you only need 
> the buffers part of Harfbuzz: https://behdad.github.io/harfbuzz/harfbuzz-
> Buffers.html

From what I've read so far, the hb_shape() function in Harfbuzz is the critical piece to determine how all glyphs are positioned relative to each other. Pango library functions apparently call hb_shape(), and provide the rendering function(s) as well. 

I'm guessing it should be possible to XftDrawGlyph() functions to use the output from hb_shape() to render the text as well. What are the advantages of using the freetype library over Xft? 

Krishna

[toc] | [prev] | [next] | [standalone]

#58724

Am Tue, 02 May 2017 18:31:50 -0700 schrieb krishna.myneni:

> I'm guessing it should be possible to XftDrawGlyph() functions to use
> the output from hb_shape() to render the text as well. What are the
> advantages of using the freetype library over Xft?

I don't use X to render anymore, I use OpenGL.  It's available on X and 
it's available when you don't have X, like on Android, Wayland, MacOS/iOS, 
Windows, etc.  And Xft is just the X frontend to freetype.

-- 
Bernd Paysan
"If you want it done right, you have to do it yourself"
net2o ID: kQusJzA;7*?t=uy@X}1GWr!+0qqp_Cn176t4(dQ*
http://bernd-paysan.de/

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.forth

csiph-web