@kaldari I think this is fallout of https://phabricator.wikimedia.org/T97758#2436310. I have ran &action=purge on MediaWiki_SVG_fonts.svg after fixing it and the new version at https://upload.wikimedia.org/wikipedia/commons/thumb/2/2c/MediaWiki_SVG_fonts.svg/860px-MediaWiki_SVG_fonts.svg.png seems sane to me. Let me know if you find any further occurrences, but I think it should be fixed now.

@MoritzMuehlenhoff: https://upload.wikimedia.org/wikipedia/commons/thumb/2/2c/MediaWiki_SVG_fonts.svg/860px-MediaWiki_SVG_fonts.svg.png is broken for me, i.e. Times, Helvetica, and several other fonts are not anti-aliased.

Please keep in mind that Linux implements font-substitution and has many fonts not installed. There is a fc-list, but not sure if it is valid now. Font-substitution is configuration dependant.

T40010 mentions issues with Pango... Don't know whats good or bad yet.

@Menner: Regardless of whether the font is substituted, it should always be anti-aliased. Otherwise it is very difficult to read at small sizes, for example in maps. It does appear that all of the fonts that are not being anti-aliased are fonts which are missing from the fc-list. Before the upgrade, all fonts were being anti-aliased regardless of whether they were substituted.

Yeah I agree, something seems broken with the font setup since the upgrade.

I wonder if this is related to Ghostscript, as most of the fonts that are broken are fonts that are typically substituted with other (non-proprietary) fonts by Ghostscript.

Hmm, for some reason Debian jessie (frozen in November 2014) has an older version of Ghostscript than Ubuntu trusty (released in April 2014)! As such the migration to the more recent jessie actually brought us an older Ghostscript (trusty has 9.10 and jessie has 9.06). I can build 9.10 for jessie next week and check whether it makes a difference.

@MoritzMuehlenhoff: That would be good to try. The sooner the better.

@MoritzMuehlenhoff: Please consider remarks about Pango in T40010 as well. To me they sound more plausible.

Yeah, sounds like we should make sure the whole stack is up to date: Pango, Cairo, Harfbuzz, Ghostscript.

Seeing complaints on the Commons' village pump: https://commons.wikimedia.org/wiki/Commons:Village_pump#Font_rendering.

I've built backports of Ghostscript 9.19 and Pango 1.40.1 for jessie, but to test this on a depooled server I need a proper test case. The error shown in https://phab.wmfusercontent.org/file/data/ybfgneclaia6wc47obuh/PHID-FILE-lxfzaccwd3ugjoq6xekd/anti-aliasing.png does _not_ appear with what's currently rendered on the scalers: https://upload.wikimedia.org/wikipedia/commons/thumb/2/2c/MediaWiki_SVG_fonts.svg/860px-MediaWiki_SVG_fonts.svg.png, e.g. Terminal is rendered just fine?
I've copied a screenshot to https://people.wikimedia.org/~jmm/svg/terminal-font-current-scalers.png

@MoritzMuehlenhoff: The bug definitely shows up for me at https://upload.wikimedia.org/wikipedia/commons/thumb/2/2c/MediaWiki_SVG_fonts.svg/860px-MediaWiki_SVG_fonts.svg.png (in an incognito window with browser cache cleared). Terminal, Times, etc. are not anti-aliased for me, same as in the original screenshot. @TheDJ: Does the bug appear for you at that URL? Here's my X-Cache header in case that helps: X-Cache
cp1048 hit/2, cp2017 miss, cp4007 miss, cp4005 hit/6.

Times, esp. italics of it look extremely jagged on that PNG to me. Terminal might 'seem' a bit better compared to the very original shot, but mostly because the most original shot seems to indicate the Terminal font wasn't actually installed previously and it was falling back to another (antialiased) font.

Status update: Using Times as the sample and scaling to only 675 I can reproduce this. https://people.wikimedia.org/~jmm/svg/alias-trusty.png is the result with trusty and https://people.wikimedia.org/~jmm/svg/alias-jessie-old.png represents the current scaler config. I depooled an image scaler and installed the Ghostscript 9.19 backport, this doesn't make a difference: https://people.wikimedia.org/~jmm/svg/alias-jessie-ghostscript9.19.png. I'll try Pango 1.40 next, the backport mentioned earlier needs more work.

Status update: This does not appear to be related to the upgrade of librsvg, but to some other difference between trusty and jessie: I downgraded to the standard librsvg in jessie (2.40.5) and the result is at https://people.wikimedia.org/~jmm/svg/alias-jessie-stock.png

I also tried upgrading Pango to 1.40.1, but that doesn't make a difference either, the result is at https://people.wikimedia.org/~jmm/svg/alias-jessie-pango1.40.png

Status update:
The SVG is rendered fine with Debian unstable: https://people.wikimedia.org/~jmm/svg/alias-debian-unstable.png
I did some further testing and found that it works fine on my personal jessie desktop: https://people.wikimedia.org/~jmm/svg/jessie-workstation.png
For comparison the result one a (depooled) jessie image scaler with the librsvg default version in jessie: https://people.wikimedia.org/~jmm/svg/1298.png

I'll compare installed packages between those two jessie systems next.

Besides Pango the library Harfbuzz is mention in the librsvg bugreport on Gnome, too.

@Menner On the jessie system which is anti-aliased Harfbuzz is installed in the same version as on the jessie image scaler where it doesn't work, so it doesn't seem to be a matter of a an outdated or missing Harfbuzz.

From outside it is difficult to estimate when fallback fonts are use. Maybe this is something to look at since fallback fonts seem to be affected only: https://wiki.archlinux.org/index.php/Font_configuration#EmbeddedBitmap

I've install a Debian Jessie from netinstall on a local virtual machine and added Wikimedia repositories for backport: https://wikitech.wikimedia.org/wiki/APT_repository

With this installation even without GUI desktop installed I get good results for librsvg 2.40.16 backport. (Actually dependencies install lots of GUI stuff)

https://commons.wikimedia.org/wiki/File:Test.svg

My console:

root@debian-jessie:~# fc-match Times
DejaVuSerif.ttf: "DejaVu Serif" "Book"
root@debian-jessie:~#
root@debian-jessie:~# fc-match "DejaVu Serif"
DejaVuSerif.ttf: "DejaVu Serif" "Book"

BTW: How do you invoke rsvg-convert / librsvg on image scalers?

@Menner: The invocation of librsvg is rather straightforward, e.g.

/usr/bin/rsvg-convert -w 1097 -h 3942 -o /tmp/transform_f1580b2b27d5.png /tmp/svg_4cb72b1ebcf5f40d7cb5eefe/localcopy_a9feba0a7a0d.svg

But fc-match is a very useful hint!

On the new scalers which misrender:
$ fc-match Times
timR12-ISO8859-1.pcf.gz: "Times" "Regular"

On the old scalers which are fine:
$ fc-match Times
n021003l.pfb: "Nimbus Roman No9 L" "Regular"

On my Debian unstable system which is fine:
$ fc-match Times
n021003l.pfb: "Nimbus Roman No9 L" "Regular"

timR12-ISO8859-1.pcf.gz is from one of those the xfonts-75dpi/xfonts-100dpi package, these are also installed on the trusty scalers, but possibly the precedence in Fontconfig needs to be fixed (so I suppose this is based somewhat on installation order?)

I ran

FC_DEBUG=1024 fc-match "Times"

on the trusty and jessie-based image scalers and that revealed that the fontconfig-config package in trusty ships a /etc/fonts/conf.d/10-antialias.conf with the following content:

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<!--  Use the Antialiasing -->
  <match target="font">
    <edit name="antialias" mode="assign"><bool>true</bool></edit>
  </match>
</fontconfig>

This file is no longer available in jessie/unstable and there's no mention of it in the changelog.

I've added such a file locally to one of the new jessie-based scalers and that seems to fix it, result is at:
https://people.wikimedia.org/~jmm/svg/alias-jessie-new.png

I'll puppetise the config file next.

Judging from http://www.ceus-now.com/weird-font-hinting-in-firefox-4/
I suspect this was removed, because this file overrides options at a 'global' level of fontconfig, where some of these are more appropriately configured inside apps etc.

We might want to check if the hinting of fonts was similarly removed from global config.

This probably needs to be specified somewhere inside of cairo or librsvg, for a 'proper' solution, but if restoring the config files works, then I think that is a good quick fix.

Change 299131 had a related patch set uploaded (by Muehlenhoff):
Provide fontconfig configuration which forces antialiasing

https://gerrit.wikimedia.org/r/299131

And so the problem starts. Librsvg expects these decisions on global level not per application/desktop environment.

Just to make sure, don't deliberately copy font configs from one ditribution to another. You need to know what you are doing. Some are distro specific tweak.

@Menner: On these systems rsvg-convert is the only application processing fonts. As such adding https://gerrit.wikimedia.org/r/#/c/299131/2/modules/mediawiki/files/fontconfig-antialias.conf seems fine to me (and it fixes the problem in my test case). Let me know if you see a problem with that change.

In T139543#2465507, @MoritzMuehlenhoff wrote:

I've added such a file locally to one of the new jessie-based scalers and that seems to fix it, result is at:
https://people.wikimedia.org/~jmm/svg/alias-jessie-new.png

I'll puppetise the config file next.

I don't understand how 10-antialias.conf fixes the problem. I cannot follow your steps.

It provides a system-wide configuration stanza for fontconfig to force antialiased fonts. Without that config change fontconfig ends up using the non-aliased Times X font. See https://wiki.archlinux.org/index.php/Font_configuration/Examples#Enable_anti-aliasing_only_for_bigger_fonts for a configuration example and the section "Font properties" from https://www.freedesktop.org/software/fontconfig/fontconfig-user.html :

antialias Bool Whether glyphs can be antialiased

There's several other recommendations/configs in a similar manner:
http://forums.debian.net/viewtopic.php?f=6&t=46550#p265654
https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/media-libs/fontconfig/files/fontconfig-2.11.1-conf-d.patch

This is also consistent with your statement that Librsvg expects these decisions on global level not per application/desktop environment, while a browser like Chromium or Firefox enables antialiasing on the per-application level. One example is xmonad, an X11 window manager:
https://github.com/hallettj/dot-xmonad/blob/master/home/.config/fontconfig/conf.d/10-antialias.conf

And since we execute rsvg-convert directly, the /etc/fonts/conf.d directory in fontconfig comes into effect. But fontconfig in jessie is a little more recent than trusty and no longer provides 10-antialias.conf. There's no reason specified in the debian/changelog, but http://www.ceus-now.com/weird-font-hinting-in-firefox-4/ indicates that this is intentionally done since otherwise the global fontconfig configuration overrides desktop-specific settings as done in GNOME or KDE. But as our scalers don't use any desktop components that's really not a problem for us.

I'm away next week, but Giuseppe volunteered to review/merge this next week. Thanks!

Change 299131 merged by Giuseppe Lavagetto:
Provide fontconfig configuration which forces antialiasing

https://gerrit.wikimedia.org/r/299131

I merged the patch, and it was correctly applied on all imagescalers that run Debian jessie.

I am not sure this solved the issue, though, as my current tests show it didn't.

Still looks broken to me :(

What is the result of fc-match?

FC_DEBUG=1024 fc-match "Times"

Look here for an obviuous example for testing: File:Test_T139543.svg

What is particularly baffling to me is that @MoritzMuehlenhoff has reported success in generating the fonts.svg correctly after adding the configuration for antialiasing, while now even my local test on the servers show that this change has no effect.

@Menner fc-match reports

timR12-ISO8859-1.pcf.gz: "Times" "Regular"

and looking at the log with debug shows the new file gets loaded:

FC_DEBUG=1024
	Loading config file /etc/fonts/fonts.conf
	Scanning config dir /etc/fonts/conf.d
	Loading config file /etc/fonts/conf.d/10-antialias.conf
	Loading config file /etc/fonts/conf.d/10-scale-bitmap-fonts.conf
	Loading config file /etc/fonts/conf.d/11-lcdfilter-default.conf
	Loading config file /etc/fonts/conf.d/20-unhint-small-dejavu-lgc-sans-mono.conf
	Loading config file /etc/fonts/conf.d/20-unhint-small-dejavu-lgc-sans.conf
	Loading config file /etc/fonts/conf.d/20-unhint-small-dejavu-lgc-serif.conf
	Loading config file /etc/fonts/conf.d/20-unhint-small-dejavu-sans-mono.conf
	Loading config file /etc/fonts/conf.d/20-unhint-small-dejavu-sans.conf
	Loading config file /etc/fonts/conf.d/20-unhint-small-dejavu-serif.conf
	Loading config file /etc/fonts/conf.d/20-unhint-small-vera.conf
	Loading config file /etc/fonts/conf.d/25-arphic-ukai-render.conf
	Loading config file /etc/fonts/conf.d/25-arphic-uming-render.conf
	Loading config file /etc/fonts/conf.d/25-wqy-zenhei.conf
	Loading config file /etc/fonts/conf.d/30-0-google-crosextra-caladea-fontconfig.conf
	Loading config file /etc/fonts/conf.d/30-0-google-crosextra-carlito-fontconfig.conf
	Loading config file /etc/fonts/conf.d/30-metric-aliases.conf
	Loading config file /etc/fonts/conf.d/30-urw-aliases.conf
	Loading config file /etc/fonts/conf.d/35-arphic-ukai-aliases.conf
	Loading config file /etc/fonts/conf.d/35-arphic-uming-aliases.conf
	Loading config file /etc/fonts/conf.d/40-nonlatin.conf
	Loading config file /etc/fonts/conf.d/41-arphic-ukai.conf
	Loading config file /etc/fonts/conf.d/41-arphic-uming.conf
	Loading config file /etc/fonts/conf.d/45-latin.conf
	Loading config file /etc/fonts/conf.d/49-sansserif.conf
	Loading config file /etc/fonts/conf.d/50-user.conf
	Loading config file /etc/fonts/conf.d/51-local.conf
	Loading config file /etc/fonts/conf.d/57-dejavu-sans-mono.conf
	Loading config file /etc/fonts/conf.d/57-dejavu-sans.conf
	Loading config file /etc/fonts/conf.d/57-dejavu-serif.conf
	Loading config file /etc/fonts/conf.d/58-dejavu-lgc-sans-mono.conf
	Loading config file /etc/fonts/conf.d/58-dejavu-lgc-sans.conf
	Loading config file /etc/fonts/conf.d/58-dejavu-lgc-serif.conf
	Loading config file /etc/fonts/conf.d/60-latin.conf
	Loading config file /etc/fonts/conf.d/64-01-tlwg-kinnari.conf
	Loading config file /etc/fonts/conf.d/64-02-tlwg-norasi.conf
	Loading config file /etc/fonts/conf.d/64-11-tlwg-waree.conf
	Loading config file /etc/fonts/conf.d/64-12-tlwg-loma.conf
	Loading config file /etc/fonts/conf.d/64-13-tlwg-garuda.conf
	Loading config file /etc/fonts/conf.d/64-14-tlwg-umpush.conf
	Loading config file /etc/fonts/conf.d/64-15-laksaman.conf
	Loading config file /etc/fonts/conf.d/64-21-tlwg-typo.conf
	Loading config file /etc/fonts/conf.d/64-22-tlwg-typist.conf
	Loading config file /etc/fonts/conf.d/64-23-tlwg-mono.conf
	Loading config file /etc/fonts/conf.d/64-arphic-uming.conf
	Loading config file /etc/fonts/conf.d/64-wqy-zenhei.conf
	Scanning config dir /etc/fonts/conf.d/65-0-fonts-beng-extra.conf
	Loading config file /etc/fonts/conf.d/65-0-fonts-beng-extra.conf/65-0-fonts-beng-extra.conf
	Scanning config dir /etc/fonts/conf.d/65-0-fonts-deva-extra.conf
	Loading config file /etc/fonts/conf.d/65-0-fonts-deva-extra.conf/65-0-fonts-deva-extra.conf
	Loading config file /etc/fonts/conf.d/65-0-fonts-gubbi.conf
	Scanning config dir /etc/fonts/conf.d/65-0-fonts-gujr-extra.conf
	Loading config file /etc/fonts/conf.d/65-0-fonts-gujr-extra.conf/65-0-fonts-gujr-extra.conf
	Scanning config dir /etc/fonts/conf.d/65-0-fonts-guru-extra.conf
	Loading config file /etc/fonts/conf.d/65-0-fonts-guru-extra.conf/65-0-fonts-guru-extra.conf
	Scanning config dir /etc/fonts/conf.d/65-0-fonts-orya-extra.conf
	Loading config file /etc/fonts/conf.d/65-0-fonts-orya-extra.conf/65-0-fonts-orya-extra.conf
	Loading config file /etc/fonts/conf.d/65-0-fonts-smc.conf
	Loading config file /etc/fonts/conf.d/65-0-fonts-taml-tscu.conf
	Loading config file /etc/fonts/conf.d/65-0-fonts-telu-extra.conf
	Loading config file /etc/fonts/conf.d/65-culmus.conf
	Loading config file /etc/fonts/conf.d/65-fonts-persian.conf
	Loading config file /etc/fonts/conf.d/65-khmer.conf
	Loading config file /etc/fonts/conf.d/65-nonlatin.conf
	Loading config file /etc/fonts/conf.d/66-lohit-assamese.conf
	Loading config file /etc/fonts/conf.d/66-lohit-bengali.conf
	Loading config file /etc/fonts/conf.d/66-lohit-devanagari.conf
	Loading config file /etc/fonts/conf.d/66-lohit-gujarati.conf
	Loading config file /etc/fonts/conf.d/66-lohit-kannada.conf
	Loading config file /etc/fonts/conf.d/66-lohit-oriya.conf
	Loading config file /etc/fonts/conf.d/66-lohit-punjabi.conf
	Loading config file /etc/fonts/conf.d/66-lohit-tamil-classical.conf
	Loading config file /etc/fonts/conf.d/66-lohit-tamil.conf
	Loading config file /etc/fonts/conf.d/66-lohit-telugu.conf
	Loading config file /etc/fonts/conf.d/69-droid-sans-fallback.conf
	Loading config file /etc/fonts/conf.d/69-manchufont.conf
	Loading config file /etc/fonts/conf.d/69-unifont.conf
	Loading config file /etc/fonts/conf.d/75-arphic-ukai-select.conf
	Loading config file /etc/fonts/conf.d/80-delicious.conf
	Loading config file /etc/fonts/conf.d/89-tlwg-garuda-synthetic.conf
	Loading config file /etc/fonts/conf.d/89-tlwg-kinnari-synthetic.conf
	Loading config file /etc/fonts/conf.d/89-tlwg-laksaman-synthetic.conf
	Loading config file /etc/fonts/conf.d/89-tlwg-loma-synthetic.conf
	Loading config file /etc/fonts/conf.d/89-tlwg-umpush-synthetic.conf
	Loading config file /etc/fonts/conf.d/89-tlwg-waree-synthetic.conf
	Loading config file /etc/fonts/conf.d/90-arphic-ukai-embolden.conf
	Loading config file /etc/fonts/conf.d/90-arphic-uming-embolden.conf
	Loading config file /etc/fonts/conf.d/90-fonts-linux-libertine.conf
	Loading config file /etc/fonts/conf.d/90-fonts-unfonts-core.conf
	Loading config file /etc/fonts/conf.d/90-fonts-unfonts-extra.conf
	Loading config file /etc/fonts/conf.d/90-synthetic.conf
timR12-ISO8859-1.pcf.gz: "Times" "Regular"

One more thing. Did you reboot? Otherwise try:

fc-cache

Additionally you may look at the following links. They mention similar issues:
https://github.com/wkhtmltopdf/wkhtmltopdf/issues/2193
or
https://wiki.archlinux.org/index.php/Font_configuration#EmbeddedBitmap

But on Ubuntu I can't find a package with the bogus font and have no Debian in reach.

@Menner I already ran fc-cache; in fact the no-bitmap configuration

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<!-- Reject bitmap fonts -->
 <selectfont>
  <rejectfont>
   <pattern>
     <patelt name="scalable"><bool>false</bool></patelt>
   </pattern>
  </rejectfont>
 </selectfont>
</fontconfig>

but not on the jessie ones; I have confirmed that its addition solves the issue; I am going to prepare a quick patch and possibly deploy it today (although it's very late here).

Change 299586 had a related patch set uploaded (by Giuseppe Lavagetto):
mediawiki::packages::fonts: reject bitmap fonts on jessie

https://gerrit.wikimedia.org/r/299586

Change 299586 merged by Giuseppe Lavagetto:
mediawiki::packages::fonts: reject bitmap fonts on jessie

https://gerrit.wikimedia.org/r/299586

Change 299587 had a related patch set uploaded (by Giuseppe Lavagetto):
mediawiki: use puppet-provided fontconfig everywhere

https://gerrit.wikimedia.org/r/299587

Change 299587 merged by Giuseppe Lavagetto:
mediawiki: use puppet-provided fontconfig everywhere

https://gerrit.wikimedia.org/r/299587

Ok; I think my last two commits actually fixed this - @Menner @kaldari can you confirm as well?

Looks good from here. The only thing I'm curious now is that Times is not substitued by DejaVu Serif as on my Debian system last friday (not at my hands yet). On Wikimedia it is obviously not:
https://commons.wikimedia.org/wiki/File:Test_T139543.svg

As a hotfix definitly good to go until final investigation. The font timR12-ISO8859-1.pcf.gz still confuses me. Got to get a Debian to reproduce the issue.

Yay! Looks great to me! Is there a way we can automatically regenerate all the SVG thumbs now?

Also, I'm happy that Times is not substituted by DejaVu Serif. Hopefully it's being substituted by Nimbus Roman No9 L or Liberation Serif (which actually look similar to Times).

In T139543#2472383, @kaldari wrote:

Yay! Looks great to me! Is there a way we can automatically regenerate all the SVG thumbs now?

Not that I know of, without going through all of swift; I honestly don't know if it's worth the effort vs purging images whenever we see one. What we could do is gather a list of SVGs created in the last few weeks and regenerate the thumbs for those.