Unicode 8.0 Emoji

[pento]

Unicode 8 was finalised in June, making the new emoji from there official. We should add support for the new emoji.

Twemoji hasn't added support yet, and there's currently ​no indication of when they will. So, our options are:

• Reach out to any contacts we have within Twitter to see if they can share their plans.
• Update the Twemoji images ourselves, or contract the original artists, ​The Iconfactory.
• Evaluate other Open Source emoji sets, such as ​Emoji One or ​emojidex.

We'll probably need to look at checking what browsers support the various parts of Unicode 8, too.

[pento]

@joen - given your original work on emoji, do you have opinions on these options?

[Joen]

Replying to pento:

@joen - given your original work on emoji, do you have opinions on these options?

I do, and thank you for the ping!

Drawing the images ourselves is not only a lot of work for very skilled artists, but will very likely be duplicate work as I'm confident Unicode 8 emoji will eventually hit Twemoji — emoji are simply too popular for the set not to be expanded to include the cool new stuff.

Reaching out to Twitter would be great if anyone has contacts, though I'm unsure what'll come of that. Seems like the expanded set will probably drop at an unknown time when Twitter considers it a priority, and I'm unsure whether us pinging them will move the needle.

Filling in the 8.0 range with temporary replacements from Emoji One or Emojidex I'm not personally a fan of — while both sets are gorgeous on their own, they feel visually incompatible with Twemoji, though of the two Emoji One is definitely closer if we're antsy.

"Wait and see" is a lazy thing to say, I know, but at the moment that's what I'd personally do. That being said, I think improvements to the current set could be super worthwhile. There was a diversity upgrade recently, which I don't seem to remember us supporting yet — when emoji are inserted via iOS, the emoji shows up with a skin-color square next to it. Seems like getting that to work could be a nice iteration while waiting for the expanded set, maybe.

[pento]

I wrote an experimental modification to svg.sh to generate skintone variations of Twemoji:

skintone() {
  local f
  local key
  local tonecode
  local tone
  local content

  local tonecodes=(1f3fb 1f3fc 1f3fd 1f3fe 1f3ff)
  local tones=(ffe0c4 e2b98f dfa675 d79e84 5b0000)

  local base=(1f385 1f466);

  for f in "${base[@]}"; do
    for key in "${!tonecodes[@]}"; do
      tonecode="${tonecodes[$key]}"
      tone="${tones[$key]}"

      content=$(
        cat "svg/$f.svg" |
        sed "s/#d79e84/#$tone/g"
      )

      echo "svg/$f-$tonecode.svg"
      echo "$content" >"svg/$f-$tonecode.svg"
    done
  done
}

Given diverse-santa.png​, it obviously needs tweaking, but it's a start.

Unfortunately, that doesn't generate the PNG files, which are created from the AI files. We could probably figure out how to create them as a stop-gap until the official Twemoji update comes.

My concern with this option is that I'm not aware of any Twitter contacts that anyone has - we may be waiting some time.

[pento]

I'm not wild about the tones I used. So I don't have to keep searching for them, here are the tones that Apple uses:

#fadcbc
#e0bb95
#bf8468
#9b643d
#594539

[pento]

diverse-santa.png​ uses Apple's skin tones, and changes the eye colour for the darkest skin tone. Still needs tweaking, but it shows that we can tweak these things on a per-tone basis.

skintone() {
  local f
  local key
  local tonecode
  local tone
  local content

  local tonecodes=(1f3fb 1f3fc 1f3fd 1f3fe 1f3ff)
  local tones=(fadcbc e0bb95 bf8468 9b643d 594539)

  local base=(1f385);

  for f in "${base[@]}"; do
    for key in "${!tonecodes[@]}"; do
      tonecode="${tonecodes[$key]}"
      tone="${tones[$key]}"

      content=$(
        cat "svg/$f.svg" |
        sed "s/#d79e84/#$tone/g"
      )

      if [ $tone == "594539" ]
          then
            content=$(
              echo "$content" |
              sed "s/#662113/#101010/g"
            )
      fi

      echo "svg/$f-$tonecode.svg"
      echo "$content" >"svg/$f-$tonecode.svg"
    done
  done
}

[pento]

svg-diversity.zip​ contains a first pass at converting all of the "diversifiable" emoji. It needs a lot of tweaking. :-)

svg.sh​ does the conversion.

[pento]

Things that definitely need attention:

• Skin-tone highlights (for example, in thumbsup) need to change for each skin tone.
• In policeman, the badge is changing colour instead of the skin.
• Blue eye colour needs to change on the darkest 2-3 tones.
• Blond hair needs to change on the darkest 2-3 tones.
• Hair/eyes/mouth on the lightest and darkest baby need tweaking.
• Red lips need tweaking on the darkest 2-3 tones.
• Scream hands and head need a lot of work, same with sad and worried head.
• The purple in the devil faces needs changing.

Suggestions are welcome. :-)

[pento]

In [svg-diversity.2.zip] and [svg.2.sh]:

• Add preview.html, for easy viewing
• Add Simpsons Yellow default skin tone
• First pass at skin tone highlights
• Fixed policeman
• Fixed baby

[Joen]

Replying to pento:

Suggestions are welcome. :-)

It looks like you're already doing some one-off conditionals to tune individual emoji that need attention in certian skintones. It seems to me like this is a good thing, and probably something we should do for all the edgecases you mention, like thumbs up highlights and police badges. If the script knows what SVG elements to look for and how to color them, we won't risk overriding changes downstream.

[pento]

Replying to Joen:

If the script knows what SVG elements to look for and how to color them

It's not that smart. :-)

It just search and replaces #aabbcc colour values, it doesn't try to figure out what element their in. So far, there hasn't been much in the way of crossed wires.

we won't risk overriding changes downstream.

I'm not really concerned about that. The next update to twemoji, whenever it comes, will almost certainly include Unicode 8, so we don't need to be forwards compatible.

[mkaz]

Replying to Joen:

@joen, @pento - I've reached out to our contact at Twitter.
I'll update when I hear back on their plans with Unicode 8 Emoji.

[pento]

No status update from me.

@mkaz, any update from your Twitter contacts?

[pento]

In 35596:

Emoji: Add some new flags, and a handful from Unicode 8.0.

Including the following that should be used as often as possible:

U+1F1E6 U+1F1FA Flag for Australia
U+1F3CF Cricket Bat and Ball
U+1F32F Burrito

See #33592.

[pento]

I'm (probably) not going to get diversity emoji done for 4.4, but I need to add a test for Unicode 8.0 emoji to wp-emoji-loader.js.

[pento]

In 35606:

Emoji: Use twemoji in browsers that don't support Unicode 8 emoji.

Some less advanced browsers are yet to add support for the important advances made in Unicode 8. Let's make ensure that their users can experience emoji in their full glory.

See #33592.

[pento]

Okay, we can continue kicking this can down the road.

[pento]

Twitter just landed their Unicode 8 implementation, they've confirmed plans to update the GitHub repo.

[pento]

In 36126:

Emoji: Update emoji support to Unicode 8.0, including the Diversity set.

Fixes #33592.

[pento]

Not quite finished, investigating whether the s.w.org CDN can be flushed (some of the images were updated), or if we need to move the images to a new path.

[nacin]

We also need to check if the browser supports diversity, a la [35606].

And then, we should backport this to 4.4.1.

[jorbin]

This feels fairly borderline to be backported into 4.4.1. I would be fine if something can happen in the next few days, but this might also be something that needs to slip to a potential 4.4.2

[kraftbj]

Quick drive-by report while still on vacation: On Trunk, at least some native emoji is being replaced. OS X on latest stable Chrome. Example "Snowman Without Snow" (U+26C4) is being rendered natively, then replaced by Twiemoji on my prod site ( ​https://kraft.im/2016/01/snow-day/ ).

In another post, I can't save the hot dog emoji without it being saved in the db as an img tag. I can debug these a bit more (and spin off into a new ticket if needbe) next week when I'm back in the saddle.

[pento]

In 36160:

Emoji: Add a test to the emoji loader, to see if the browser supports emoji diversity. If it doesn't, fall back to twemoji.

See #33592.

[pento]

Chrome should just be replacing everything - they still haven't fixed rendering colour emoji when the font-weight is greater than 500.

I wasn't able to reproduce the hot dog emoji being saved as an img tag.

[pento]

In 36161:

Emoji: Add Emoji Diversity support, and fall back to twemoji if the browser doesn't support diverse emoji.

Merge of [36126] and [36160] to the 4.4 branch.

See #33592.

[pento]

Leaving this ticket open, pending feedback from Systems as to whether we can purge the CDN or not.

[ocean90]

Shouldn't wp_encode_emoji() be updated as well?

[pento]

Nope. That regex covers (rather lazily, admittedly) the entire character space of emoji, rather than just the individual characters supported by twemoji.

There's probably a good argument that we should figure out how to keep the twemoji.js regex in sync with the wp_encode_emoji() regex.

[ocean90]

Should we clarify this in docs of wp_encode_emoji()? I was confused by the "Currently, only Unicode 7 emoji are supported." line.

[pento]

Actually, I just had another look at the regex, and it's missing a bunch of edge cases. I opened #35293 to fix it.

[pento]

CDN has been purged, the magic is happening live.