Unix Fonts and AbiWord
Tomas Frydrych <tomas@frydrych.uklinux.net>
Table of Contents
Fonts in the Unix/X Environment
Needs of a WYSIWYG Word Processor
Adding and Removing AbiWord's Fonts
My Fonts for Encoding XXX Print as Circles but Display Correctly
When I Type I Get Small Circles on the Screen
"Abiword Messes Up My Fonts" Problem
Introduction
The way AbiWord handles fonts in the Unix environment has its own peculiarities, and is the single most common source of confusion, difficulties and frustration among the AbiWord users. The AbiWord developers are planning to redesign the font handling mechanism, but as this is a non-trivial task, it will only be done after the release of version 1.0. In the meantime, this document will outline the main aspects of AbiWord's font handling, the reasons behind it, and provide detailed instructions on font customization.
AbiWord's Font Requirements
The way AbiWord uses fonts stems from two sets of limitation: the needs of a wysiwyg word processors and the nature of font management in the Unix/X environment.
Fonts in the Unix/X Environment
For historical reasons, the way a Unix application displays its data on the screen, and the way it sends it to the printer are entirely different. The screen is accessed through the X display system, which provides a highly abstract way of handling fonts, allowing the application to use fonts of a variety of different types without having to know anything at all about them. This makes displaying text on the screen rather simple, but has some other drawbacks, as we will see in a moment.
In contrast, to print the application translates the document into the PostScript programming language and sends the PostScript program to the printer (or a PostScript interpreter that will translate it into code that the printer understands). While the PostScript language is extremely powerful, the way it handles fonts has nothing at all to do with the way the X system handles fonts. Basically, each font that the document requires needs to be embedded into the PostScript representation of the document, which means that the font must be in a format the PostScript language understands. Furthermore, because the PostScript language is designed to produce high quality printed documents, it works with a much higher resolution than the screen. In order to be able to produce the layout in the PostScript output, the application has to be able to retrieve the dimensions of the individual characters directly from the font (or more precisely, from the metrics file associated with it); it cannot just scale the dimensions it got from the X server for drawing on the screen, since this would result in unacceptable rounding errors.
Consequently, an application such as AbiWord can only use fonts that are both understood by the X display system and the PostScript language. In practice this means using either PostScript fonts (with the extensions .pfa or .pfb), or, with some additional preprocessing, TrueType fonts (the standard fonts in the Windows world, with the extension .ttf). It further has to be able to access the font files directly; unfortunately the abstract way in which X handles fonts hides the font details from the application, so that the application does not have direct access to these files, and needs to provide its own way of ensuring it can get to them. In AbiWord this is done by insisting that we will only use fonts that reside in a directory of our own choice (and AbiWord is not the only application that takes this path).
While having all the fonts in our font directory means that we can access them directly, in order to be able to use them for drawing on screen, we also have to register them with the X system. We have two options here: (1) we could insist that the user does this manually and permanently when s/he installs AbiWord; (2) we can do that automatically when AbiWord is run, for the length of the AbiWord session. The original developers choose the latter option, presumably to make AbiWord easier to install and use. Unfortunately this has some pitfalls. For one, registering the fonts with the X system is not that simple, and the automatic process can easily fail, and what's worse, what works on on one *nix, does not work on another. Further, if a font formally similar to the one AbiWord uses is already registered on the system, it can happen that when AbiWord is running other applications will start using the font supplied by AbiWord in its place. This has been a source of copious complaints from users, and the solution to this problem will be discussed later.
Needs of a WYSIWYG Word Processor
A layout application such as AbiWord has other requirements on its fonts that further narrow the fonts it can use and how it uses them.
- AbiWord needs that its fonts would be fully scalable
- As a Unicode based application, AbiWord needs a way of handling fonts that contain more glyphs than the standard Unix 8-bit encoding provides, so that it can accommodate users who want to create multi-lingual documents. This requirement means that some font-handling techniques used by pure 8-bit applications cannot be used by AbiWord.
- Because as the result of the above requirements AbiWord cannot use just any font, and because on a Unix/X based system we are not guaranteed to have fonts that AbiWord can make use of, AbiWord has to ensure it has at least basic fonts available. Since most word processor documents are created by people using MS Windows, and since Windows provides a standard set of fonts called Times New Roman, Arial, and Courier New, this was adopted as the basic AbiWord font set, and AbiWord provides versions of these fonts.
Adding and Removing AbiWord's Fonts
As was explained in the previous section, AbiWord limits the fonts it makes available to the user to those located in its fonts directory (typically /usr/share/AbiSuite/fonts or /usr/local/AbiSuite/fonts, unless you chose different installation location) and in the locale-specific sub-directories of this directory, and it installs a basic set of fonts into this directory. This does not mean that you are limited to using these fonts; you can add new fonts and remove fonts you do not like or use. The fonts you can use with AbiWord are, however, limited to PostScript Type 1 fonts (pfa/pfb) and TrueType fonts (ttf).
The Elementary Procedure
If you wish to make an additional font available to AbiWord, you need to do the following:
(1) Place the font into the fonts directory
(2) Update the fonts.dir and fonts.scale files.
Similarly, if you wish to remove a font, you simple remove it from the AbiWord font directory, and remove the entries from the fonts.dir and fonts.scale files.
The detailed procedure for adding pfa/pfb fonts and for ttf fonts is slightly different and is described below. Also note that the procedure is slightly more involved for non-Latin1 locales, with the details discussed later on, and if you are installing fonts for other than Latin 1 locale, you should read Locales Other than Latin 1 first.
pfa / pfb fonts
Either copy the font file to the fonts directory, or create a symbolic link to it, if it is located elsewhere on your system. If your font came with a vendor provided afm file, you should copy or symlink it alongside the pfa/pfb file. If not AbiWord can generate the afm file automatically, but using a vendor provided afm file is preferable. Also, if any of the users who will be using AbiWord do not have write permission for the fonts directory, you have to provide the afm files (your sysadmin can generate them using the pfa2afm utility).
To update the fonts.dir file, you should add an entry for the new font into fonts.scale and then either run mkfontdir, or add exactly the same to fonts.dir and increase the font count at the very top of the file accordingly. Your fonts probably came with a sample fonts.dir or fonts.scale file, but if you do not know what the entry in the fonts.scale should be, you can generate it using the utility ttmkfdir (please do not email the developer list asking how to use ttmkfdir, see the documentation that comes with that program).
VERY IMPORTANT !!!: Make sure that you not only modify the fonts.dir file, but that you also make appropriate changes to fonts.scale. On some systems the mkfontdir utility can be executed automatically, and since this utility does not handle scalable fonts, it will truncate your fonts.dir file to 0 unless it can find fonts.scale, in which case it will just copy the contents of this file into the new fonts.dir.
TTF fonts
** Currently TTF fonts are not supported by GnomePrint; if using Gnome
** build you have to print via 'Print Directly'.
In order to be able to use TTF fonts with AbiWord, two conditions have to be satisfied: (1) your X server has to be able to handle TTF fonts and has to be configured to do so. If it does not, you should consult your distribution documentation on how to set it up, please do not email the developer's list with questions how to do this. (2) Your PostScript interpreter has to be able to handle TTF fonts. Recent versions of GhostScript have this capability, but you need to enable it at compile time, and may need to make some changes to the GhostScript configuration file (see the GhostScript documentation). If you are using a genuine PostScript printer, then you need to consult your printer's documentation to see if it can handle Type 42 fonts; more recent printers should have this capability.
The following instructions assume that the two conditions above are satisfied and that you are installing the fonts directly into the fonts directory. You should only do this if you are using Latin-1 locale; if you use a different locale, you should first see the section Locales Other than Latin 1 below.
(1) Change into the AbiSuite directory, usually, /usr/share/AbiSuite, or /usr/local/AbiSuite
# cd /usr/share/AbiSuite
(2) This directory contains a number of subdirectories, one of which is called fonts -- change into it.
# cd fonts
(3) All fonts you want to use with AbiWord have to be copied or symbolically linked into here. In my case they are actually located on a Windows partition of my drive which is mounted under the directory /windows, so for the font Times New Roman I do
# ln -s /windows/windows/fonts/times.ttf times.ttf
You have to do this for all fonts and all variants of each font you want use (i.e., timesb.ttf for the bold Times New Roman, etc).
(4) Now you need to set up the fonts.scale and fonts.dir files for this directory. You will need the program ttmkfdir for that, and you will find details of how to do that in the program's documentation and your distribution's documentation. Once these files have been generated, you should open them in a plain text editor. You will see that for each font there are multiple entries that differ only in the encoding specified at the very end of each line. Delete all lines that do not have the encoding of your locale, i.e., in our case delete all lines that do not end in iso8859-1 (note that there is a slight difference between the look of your LANG encoding [ISO-8859-2] and the encoding format used by the fonts.dir and fonts.scale files [iso8859-1]; this is nothing to worry about).
VERY IMPORTANT: Make sure that you not only modify the fonts.dir file, but that you also make appropriate changes to fonts.scale. On some systems the mkfontdir utility can be executed automatically, and since this utility does not handle scalable fonts, it will truncate your fonts.dir file to 0 unless it can find fonts.scale, in which case it will just copy the contents of this file into the new fonts.dir.
(5) Now change to the bin subdirectory of the AbiSuite directory, and execute the following command:
# cd /usr/share/AbiSuite/bin
# ./ttftool -e print
You will get back a list of encodings known to the ttftool program; check if your encoding is there (the encoding of your LANG and the encoding used by ttftool must correspond exactly, i.e., in our case we are looking for ISO-8859-1). If your encoding is not there, all is not lost. Drop me an email, and I will add it to the program, this is very simple to do. If your encoding is there, then proceed to the next step.
(6) Now execute the following command, supplying the directory and encoding that are appropriate for your set up
# ./ttfadmin.sh /usr/share/AbiSuite/fonts ISO-8859-1
The script ttfadmin.sh will call the program ttftool for each TTF font in your directory; there should not be any errors, providing you have the necessary permissions for the directory (but since you just created it, you should).
(7) Switch to font directory and have a look at the contents
# cd /usr/share/AbiSuite/fonts
# ls
You should see three new files for each font file, one with extension .afm, one with extension .t42 and one with extension .u2g. That's it. You are done. Start AbiWord, not forgetting to set the LANG variable beforehand if necessary.
Locales Other than Latin-1
If you are installing a font for a different encoding than iso8859-1, things are slightly more complicated. First of all, you should be installing the font into a subdirectory of the fonts directory named after the locale encoding. For instance if you are going to use Hebrew locale, you create a directory ISO-8859-8 inside the AbiWord fonts directory. The name of this directory has to correspond exactly to the encoding reported by your locale variables. If you are using Bash, try
# echo $LANG
You should see something like
# iw_IL.ISO-8859-8
It is important that the encoding (the part after the dot) is specified explicitly, i.e., if you see something like
# iw_IL
it will not work. Setting the LANG variable prior to starting AbiWord is easy, in Bash you do something like
# LANG=iw_IL.ISO-8859-8
# export $LANG
You do not have to make a system-wide modification of your $LANG setting, all that is needed is that the variable is set prior to running AbiWord. I personally use a short script to run Czech localization of AbiWord which I call abicz and which looks something like this:
#!/bin/sh
LANG=cs_CZ.ISO-8859-2
export $LANG
abiword
This way I do not have to change my normal LANG setting, and do not have to remember modifying it manually when I want to run Czech localization of AbiWord.
If you are adding a TTF type of font, then all you need to do now is to follow the instructions in the TTF installation guide above, except instead of the fonts directory, you install the fonts into the locale specific directory.
If you are using pfa or pfb fonts, you too need to follow the instructions in the pfa/pfb section above, but also do a couple of extra things. First, when adding the entry for the font to fonts.scale (and fonts.dir) make sure that the font encoding, i.e., the last two parts of the line separated by a dash are set to adobe-fontspecific. Then you have to create a file called fonts.alias in where you alias the font to the actual encoding it is for. For instance if your font called BLAH is for iso8859-8, the entry in fonts.scale/fonts.dir should look something like:
blah.pfa -...-BLAH-.........-adobe-fontspecific
(the dots stand for whatever makes up your specific entry). You then have to create a fonts.alias file, which should contain the following line:
"-...-BLAH-.........-iso8858-8" "-...-BLAH-.........-adobe-fontspecific"
As far as the fonts are concerned, that's all. However, you may want to have a look at the section Keyboard Related Problems that deals with some other problems that you may face. Otherwise you can start AbiWord, and the new fonts should appear in the font combo box.
Unicode peculiarities
While AbiWord does not provide a complete support for the utf-8 locale, it is capable of handling documents containing arbitrary characters from the UCS-2 character set if the locale encoding is set to utf-8. However, it can only do so with Unicode fonts. To make the X system to treat a font as a Unicode font, you need to specify the encoding in the XLFD (the entry in the fonts.dir/fonts.scale file) as iso10646-1. To see if your font is treated as a Unicode font run
#xlsfonts -ll -fn font_name
This will dump out lot of information about your font, and somewhere among it should be values min_byte1 and max_byte1; if both of these are 0, then your font is being treated as 8-bit only font. In my experience it is not possible to use a pfa Type 1 or 2 font under XFree86 4 as a proper Unicode font (if you know how to make XFree treat a pfa font as a Unicode font, please let me know so that I can update this document), which means that you will be limited to TTF fonts only.
Trouble in Paradise
My Fonts for Encoding XXX Print as Circles but Display Correctly
Symptoms: you installed a custom pfa/pfb font for some other encoding than Latin-1 (commonly ISO-8859-2) and while everything works fine on the screen you get circles in place of some of the (accented) characters when you print the document out. The font works fine with other applications. (NB: if you are seeing the circles on the screen, than that is a different problem.)
Diagnosis: While this problem can sometimes be keyboard-related most likely it is caused by your font. To decide which it is, you should carry out the following test. Create a new AbiWord document and type couple of letters from your language (those that appear printed as circles). Save the document in the abw format and then open it with a plain text editor. At the start of the document there is a kind of a header, and then later on you will find your characters in between a pair of <c></c> tags. In a normal AbiWord document created under an 8-bit locale each of these characters should be represented by a single byte, and most likely you will be able to read them. If this is the case, then read on. However, if you find instead that the characters are represented by &#xXXXX tokens (where XXXX is the Unicode value of the character), then your problem is keyboard related and you should see Keyboard Related Problems below.
Cause: your font is broken. It was probably created by modifying an original Latin-1 font, and while the person doing the modification changed the glyphs, s/he did not change the glyph names. It works with other (8-bit) applications because they use 8-bit character codes to access the glyphs, assuming that the first 256 glyphs in the font and its metric file are ordered according to the encoding of the current locale. This assumption is unsafe and AbiWord cannot afford to make it, because we support other than 8-bit locales and fonts that contain many more glyphs than the 256 that a 8-bit application would use (for instance the MS Times New Roman font contains around 1200 glyphs). The only reliable way we can retrieve the metrics information about the glyph from the font's metric file is by using the name of the glyph. In case of a pfa/pfb font, Adobe specifies an unambiguous conversion between its standard glyph names and Unicode, and we use this to translate the Unicode value into a glyph name. If the glyphs in your font are misnamed, the glyph which according to the Adobe's PostScript standard we should be looking for is not found in the font, and we draw a small circle in the place.
Solution: while this is not AbiWord's fault, we have provided a workaround for such broken fonts. You need to create a u2g file with a name identical with the name of the font, i.e., if your font is called amaretto.pfa, you need a file called amaretto.u2g. Here you have to specify the mapping from glyph names to Unicode values that your font uses. Any lines in the u2g file starting with # will be considered a comment and ignored, the first non-comment line must contain the count of the map entries (i.e., lines) to follow. Each individual entry consists of the glyph name followed by a comma and then a hexadecimal Unicode value. Each entry must be on a single line, and they have to be sorted alphabetically by the glyph name. You can use adobe-full.u2g, located in the fonts directory as a sample. If you have multiple fonts that all use the same wrong glyph names, you only need to create one such file, and just create symbolic links to it for the rest of the fonts. If all fonts in your directory used the same wrong glyph names, you do not need to create symbolic links, just call the file locale.u2g.
When I Type I Get Small Circles on the Screen
Symptoms: as you type, instead of some/all letters you only get to see small circles on the screen.
Diagnosis: this problem can be either keyboard-related, or it is caused by your font. To decide which it is, you should carry out the following test. Create a new AbiWord document and type couple of letters from your language (those that appear printed as circles). Save the document in the abw format and then open it with a plain text editor. At the start of the document there is a kind of a header, and then later on you will find your characters in between a pair of <c></c> tags. In a normal AbiWord document created under an 8-bit locale each of these characters should be represented by a single byte, and most likely you will be able to read them. If this is the case, then read on. However, if you find instead that the characters are represented by &#xXXXX tokens (where XXXX is the Unicode value of the character), then your problem is keyboard related and you should see Keyboard Related Problems below.
Cause: the small circle is an indication that the requested glyph was not found in the current font. You might be, for instance, asking for a character from Latin 2 set while you have only the default Latin 1 fonts installed.
Solution: If you want to use AbiWord with a locale that requires different encoding than Latin 1 you have provide your own fonts. You need to follow the procedure described in Adding and Removing AbiWord's Fonts, including the instructions found in the section Locales Other than Latin 1.
"Abiword Messes Up My Fonts" Problem
Symptoms: when I start up application XXX while AbiWord is running, its fonts go pear-shape; in the more extreme case, this happens even when AbiWord is not running.
Cause: there are fonts on your system that have names identical to the names of some of AbiWord's fonts, and the application XXX is using the AbiWord fonts instead of the fonts it used to use.
Solution: If you do not want the other applications to use AbiWord fonts, you have to make AbiWord use the fonts that are already available on your system; AbiWord can handle Type1 and TrueType fonts. Here come the step-by-step instructions:
(1) Change to the AW font directory, usually /usr/share/AbiSuite/fonts or /usr/local/AbiSuite/fonts.
(2) Open the file fonts.dir in your favorite text editor.
(3) Identify the offending font(s). Say, for instance, that you have the nice MS Arial font installed, but AbiWord is causing your applications to use some other Arial you think is much uglier. In that case locate the line following lines:
n019003l.pfa -AbiSource-Arial-regular-r-normal--0-0-0-0-p-0-iso8859-1
n019004l.pfa -AbiSource-Arial-bold-r-normal--0-0-0-0-p-0-iso8859-1
n019023l.pfa -AbiSource-Arial-regular-i-normal--0-0-0-0-p-0-iso8859-1
n019024l.pfa -AbiSource-Arial-bold-i-normal--0-0-0-0-p-0-iso8859-1
and change them to whatever is appropriate for your font (since you have the font installed somewhere, there is also a fonts.dir file in that directory with all the information you need); in the Arial case should be something like:
arial.ttf -monotype-Arial-medium-r-normal--0-0-0-0-p-0-iso8859-1
arialb.ttf -monotype-Arial-bold-r-normal--0-0-0-0-p-0-iso8859-1
ariali.ttf -monotype-Arial-medium-i-normal--0-0-0-0-p-0-iso8859-1
arialbi.ttf -monotype-Arial-bold-i-normal--0-0-0-0-p-0-iso8859-1
(Rather than copying this, you should find the fonts.dir file on your system in which the font is listed and copy its contents.)
(4) Create symbolic links called arial.ttf, arialb.ttf, etc., in the AbiWord fonts directory, pointing to where you have the actual font files. If the font in question is a Type1 (pfa/pfb) font, you also have to create a symbolic link to the font's afm file, which should be located in the same directory as the font itself; if the font in question is a TrueType font, as in the example above, you have to follow the instructions in the first section of this document and run the ttfadmin.sh script.
(5) You can do this for all the fonts in the AbiWord fonts.dir file, although replacing the symbol fonts might create problems with the bulleted lists in AbiWord. Also, note that if AbiWord cannot find a font requested by a document, it will try to use Times New Roman instead, and if it cannot find it either, it will take the first font it finds. So, if you get rid off the Times New Roman entry, make sure that the very first font listed in the fonts.dir file is some sensible font (e.g., not a symbol font).
(6) If you are not using any of the original fonts, i.e., all the fonts listed in your fonts.dir are physically located somewhere else then in the AbiWord font directory, and are already accessible to your applications through X, it is not necessary or desirable to modify the X font path; to stop AbiWord from doing so, go to Tools->Preferences and on the Preference Schemes tab clear the Modify Unix Font Path checkbox, and further comment out the the xset fp lines in the AbiWord wrapper script (located in AbiSuite/bin).
Keyboard Related Problems
Strictly speaking, this section does not belong into a file dealing with font issues, but because of the manifestation of the problem described below, the vast majority of the users will consider this a font problem, and so this is a proper place to deal with the issue.
Symptoms: (1) You are getting small circles instead of your characters on screen, or you see correct characters on the screen but when you print the document you get small circles for some, many or even all characters. (2) If you try to open your document, created with AbiWord running under your Linux distribution in AbiWord running on Windows, you see garbage.
Diagnosis: there is a simple test to carry out. Create a new AbiWord document and type couple of letters from your language. Save the document in the abw format and then open it with a plain text editor. At the start of the document there is a kind of a header, and then later on you will find your characters in between a pair of <c></c> tags. In a normal AbiWord document each of these characters should be represented by a single byte, and you should be able to read them. However, if you find instead that the characters are represented by &#xXXXX tokens (where XXXX is the Unicode value of the character), than you are having the problem this section is dealing with, otherwise, you may want to see My Fonts for Encoding XXX Print as Circles but Display Correctly.
Cause: It appears that many of the out-of-the-box Linux distributions do not have the keyboard setup correctly if you are using a language other than English. The X system supports a large number of characters, and it has a name for each of them. For instance the Unicode character u05d0, which is the Hebrew letter Aleph, is known to the XFree server as hebrew_aleph. When you want to type Hebrew Aleph, the keyboard has to send this specific character to the X server. The association of a particular key with a particular character is done via what is called xmodmap. This file specifies what character each key on the keyboard generates. For instance, it associates the key which would have 't' on the English keyboard with the letter Aleph on Hebrew keyboard.
The proper way to do this is to assign to this key (key no. 28) the character name hebrew_aleph. When an 8-bit application, such as the terminal requests character input from the X server, the server translates the code hebrew_aleph into the character set the terminal is using, probably ISO-8859-8, and so it will send the terminal code 0x00e0. A Unicode application, in contrast, will request the keycode as is and translate it internally into a Unicode value (either by some extra calls to the X system, or, as AbiWord does, by lookup in internal tables). In our case it translates the code hebrew_aleph into the Unicode value 0x05d0.
Unfortunately, often instead of using the X character names, the xmodmap assigns the key a numerical value of the character from the 8-bit character set of the language, e.g., in our example the key 28 gets associated with value 0x00e0, i.e., the code for Aleph in ISO-8859-8 character set. The problem with this is that for the X server 0x00e0 is not an Aleph, but the character agrave. Now when an 8-bit application request input from the server, it will send it this numerical value, and everything will appear to be fine. However, when a Unicode application such as AbiWord requests the input, it will also send it this numerical value, which will be internally translated into Unicode, leaving us with Unicode character 0x00e0. Since this character is not found in the ISO-8859-8 fonts which you are probably using if you are wanting Hebrew, you will see a small circle instead (sometimes, if your fonts are misconfigured as well, you will see your text on screen fine, but get only the small circles when trying to print).
Solution: You have to fix your keyboard map. How precisely you do this depends on your setup. If you are using some keyboard utility, such as the kikbd that comes with KDE, you will need to fix the keyboard definition files of the utility, if you are using xmod, you need to fix the xmodmap file that contains the keyboard definition. As far as AbiWord is concerned, you have two options; either you can use the character names as described above (you will find the names in the X server's documentation), or you can assign the key a value 0x0100XXXX where XXXX is the Unicode code of the character (e.g., 0x010005d0 for Hebrew Aleph). This latter method has the advantage that you can assign even characters that X system does not have names for, but you should be aware that not all applications are capable of handling keyboard defined in this way.