Event Platform/EventStreams HTTP Service

EventStreams is a web service that exposes continuous streams of structured event data. It does so over HTTP using chunked transfer encoding following the Server-Sent Events (SSE) protocol. EventStreams can be consumed directly via HTTP, but is more commonly used via a client library.

EventStreams provides access to arbitrary streams of data, including MediaWiki RecentChanges. It replaced RCStream, and may in the future replace irc.wikimedia.org. EventStreams is backed by Apache Kafka.

Note: Often SSE and EventSource are often used interchangeably as the names of this web technology. This document refers to SSE as the server-side protocol, and EventSource as the client-side interface.

When not to use EventStreams

The public EventStreams service is intended for use by small scale external tool developers. It should not be used to build production services within Wikimedia Foundation. WMF production services that react to events should directly consume the underlying Kafka topic.

Examples

JavaScript

Node.js ESM (with wikimedia-streams)

import WikimediaStream from "wikimedia-streams";

// "recentchange" can be replaced with any valid stream
const stream = new WikimediaStream("recentchange");

stream.on("open", () => { 
    console.info('Opened connection.');
});
stream.on("error", (event) => { 
    console.error('Encountered error', event);
});
stream.on("recentchange", (data, event) => {
    // Edits from the English Wikipedia
    if (data.wiki === "enwiki") {
        // Output the page title
        console.log(data.title);
    }
});

Node.js (with eventsource)

const EventSource = require('eventsource');

// "recentchange" can be replaced with any valid stream
const url = 'https://stream.wikimedia.org/v2/stream/recentchange';
const eventSource = new EventSource(url);

eventSource.onopen = () => {
    console.info('Opened connection.');
};
eventSource.onerror = (event) => {
    console.error('Encountered error', event);
};
eventSource.onmessage = (event) => {
    // event.data will be a JSON string containing the message
    const data = JSON.parse(event.data);
    // Edits from the English Wikipedia
    if (data.wiki === "enwiki") {
        // Output the page title
        console.log(data.title);
    }
};

Server side filtering is not supported. You can filter client-side instead, for example to listen for changes to a specific wiki only:

var wiki = 'commonswiki';
eventSource.onmessage = function(event) {
    // event.data will be a JSON string containing the message event.
    var change = JSON.parse(event.data);
    if (change.wiki == wiki)
        console.log(`Got commons wiki change on page ${change.title}`);
};

TypeScript

Node.js (with wikimedia-streams)

import WikimediaStream from "wikimedia-streams";

// "recentchange" can be replaced with any valid stream
const stream = new WikimediaStream("recentchange");

stream.on("recentchange", (data: MediaWikiRecentChangeEvent, event) => {
    // Edits from the English Wikipedia
    if (data.wiki === "enwiki") {
        // Output the page title
        console.log(data.title);
    }
});

Python

Using python-sseclient. There is also a more asynchronous-friendly version called aiosseclient (requires Python 3.6+).

import json
from sseclient import SSEClient as EventSource

url = 'https://stream.wikimedia.org/v2/stream/recentchange'
for event in EventSource(url):
    if event.event == 'message':
        try:
            change = json.loads(event.data)
        except ValueError:
            pass
        else:
            print('{user} edited {title}'.format(**change))

The standard SSR protocol defines ways to continue where you left after a failure or other disconnect. We support this in EventStreams as well. For example:

import json
from sseclient import SSEClient as EventSource

url = 'https://stream.wikimedia.org/v2/stream/recentchange'
for event in EventSource(url, last_id=None):
    if event.event == 'message':
        try:
            change = json.loads(event.data)
        except ValueError:
            pass
        else:
            if change.user == 'Yourname':
                print(change)
                print(event.id)

# - Run this Python script.
# - Publish an edit to [[Sandbox]] on test.wikipedia.org, and observe it getting printed.
# - Quit the Python process.
# - Change last_id=None to last_id='[{"topic":"…"},{…}]', as taken from the last printed line.
# - Publish another edit, while the Python process remains off.
# - Run this Python script again, and notice it finding and printing the missed edit.

Server-side filtering is not supported. To filter for something like a wiki domain, you'll need to do this on the consumer side side. For example:

wiki = 'commonswiki'
for event in EventSource(url):
    if event.event == 'message':
        try:
            change = json.loads(event.data)
        except ValueError:
            continue
        if change['wiki'] == wiki:
            print('{user} edited {title}'.format(**change))

Pywikibot is another way to consume EventStreams in Python. It provides an abstraction that takes care of automatic reconnection, easy filtering, and combination of multiple topics into one stream. For example:

>>> from pywikibot.comms.eventstreams import EventStreams
>>> stream = EventStreams(streams=['recentchange', 'revision-create'],
		          since='20190111')
>>> stream.register_filter(server_name='fr.wikipedia.org', type='edit')
>>> change = next(iter(stream))
>>> print('{type} on page "{title}" by "{user}" at {meta[dt]}.'.format(**change))
edit on page "Véronique Le Guen" by "Speculos" at 2019-01-12T21:19:43+00:00.

Command-line

With curl and jq Set the Accept header and prettify the events with jq.

curl -s -H 'Accept: application/json'  https://stream.wikimedia.org/v2/stream/recentchange | jq .

Setting the Accept: application/json will cause EventStreams to send you newline delimited JSON objects, rather than data in the SSE format.

API

The list of streams that are available will change over time, so they will not be documented here. To see the active list of available streams, visit the swagger-ui documentation, or request the swagger spec directly from https://stream.wikimedia.org/?spec. The available stream URI paths all begin with /v2/stream, e.g.

"/v2/stream/recentchange": {
    "get": {
      "produces": [
        "text/event-stream; charset=utf-8"
      ],
      "description": "Mediawiki RecentChanges feed. Schema: https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/mediawiki/recentchange"
    }
  },
"/v2/stream/revision-create": {
      "get": {
        "produces": [
          "text/event-stream; charset=utf-8"
        ],
        "description": "Mediawiki Revision Create feed. Schema: https://github.com/wikimedia/mediawiki-event-schemas/tree/master/jsonschema/mediawiki/revision/create"
      }
    }

Stream selection

Streams are addressable either individually, e.g. /v2/stream/revision-create, or as a comma separated list of streams to compose, e.g. /v2/stream/page-create,page-delete,page-undelete.

See available streams: https://stream.wikimedia.org/?doc

Historical Consumption

Since 2018-06, EventStreams supports timestamp based historical consumption. This can be provided as individual assignment objects in the Last-Event-ID by setting a timestamp field instead of an offset field. Or, more simply, a since query parameter can be provided in the stream URL, e.g. since=2018-06-14T00:00:00Z. since can either be given as a milliseconds UTC unix epoch timestamp or anything parseable by Javascript Date.parse(), e.g. a UTC ISO-8601 datetime string.

When given a timestamp, EventStreams will ask Kafka for the message offset in the stream(s) that most closely match the timestamp. Kafka guarantees that all events after the returned message offset will be after the given timestamp. NOTE: The stream history is not kept indefinitely. Depending on the stream configuration, there will likely be between 7 and 31 days of history available. Please be kind when providing timestamps. There may be a lot of historical data available, and reading it and sending it all out can be compute resource intensive. Please only consume the minimum of data you need.

Example URL: https://stream.wikimedia.org/v2/stream/revision-create?since=2016-06-14T00:00:00Z.

If you want to manually set which topics, partitions, and timestamps or offsets your client starts consuming from, you can set the Last-Event-ID HTTP request header to an array of objects that specify this. E.g.

[{"topic": "eqiad.mediawiki.recentchange", "partition": 0, "offset": 1234567}, {"topic": "codfw.mediawiki.recentchange", "partition": 0, "timestamp": 1575906290000}]

Response Format

All examples here will consume recent changes from https://stream.wikimedia.org/v2/stream/recentchange. This section describes the format of a response body from a EventStreams stream endpoint.

Requesting /v2/stream/recentchange will start a stream of data in the SSE format. This format is best interpreted using an EventSource client. If you choose not to use one of these, the raw stream is still human readable and looks as follows:

event: message
id: [{"topic":"eqiad.mediawiki.recentchange","partition":0,"timestamp":1532031066001},{"topic":"codfw.mediawiki.recentchange","partition":0,"offset":-1}]
data: {"event": "data", "is": "here"}

Each event will be separated by 2 line breaks (\n\n), and have event, id, and data fields.

The event will be message for data events, and error for error events. id is a JSON-formatted array of Kafka topic, partition and offset|timestamp metadata. The id field can be used to tell EventStreams to start consuming from an earlier position in the stream. This enables clients to automatically resume from where they left off if they are disconnected. EventSource implementations handle this transparently. Note that the topic partition and offset|timestamp for all topics and partitions that make up this stream are included in every message's id field. This allows EventSource to be specific about where it left off even if the consumed stream is composed of multiple Kafka topic-partitions.

Note that offsets and timestamps may be used interchangeably SSE id. WMF runs stream.wikimedia.org in a multi-DC active/active setup, backed by multiple Kafka clusters. Since Kafka offsets are unique per cluster, using them in a multi DC setup is not reliable. Instead, id fields will always use timestamps instead of offsets. This is not as precise as using offsets, but allows for a reliable multi DC service.

You may request that EventStreams begins streaming to you from different offsets by setting an array of topic, partition, offset|timestamp objects in the Last-Event-ID HTTP header.

Filtering

EventStreams does not have $wgServerName (or any other) server side filtering capabilities. You'll need to do your filtering client side, e.g.

/**
 * Calls cb(event) for every event where recentchange event.server_name == server_name.
 */
function filterWiki(event, server_name, cb) {
    if (event.server_name == server_name) {
        cb(event);
    }
}

eventSource.onmessage = function(event) {
    // Print only events that come from Wikimedia Commons.
    filterWiki(JSON.parse(event.data), 'commons.wikimedia.org', console.log);
};

Architecture

SSE vs. WebSockets/Socket.IO

RCStream was written for consumption via Socket.IO, so why not continue to use it for its replacement?

WebSockets doesn't use HTTP, which makes it different than most of the other services that Wikimedia runs. It is especially powerful when clients and servers need a bi-directional pipe to communicate with each other asynchronously. EventStreams only needs to send events from the server to clients, and is 100% HTTP. As such, it can be consumed using any HTTP client out there, without the need for programming several RPC like initialization steps.

We did originally build a Kafka -> Socket.io library (Kasocki), but after doing so we decided that SSE was a better fit and built KafkaSSE.

KafkaSSE

KafkaSSE is a library that glues a Kafka Consumer to a connected HTTP SSE client. A Kafka Consumer is assigned topics, partitions, and offsets, and then events are streamed from the consumer to the HTTP client in chunked-transfer encoding. EventStreams maps stream routes (e.g /v2/stream/recentchanges) to specific topics in Kafka.

Kafka

WMF maintains several internal Kafka clusters, producing hundreds of thousands of messages per second. It has proved to be highly scalable and feature-ful. It is multi producer and multi consumer. Our internal events are already produced through Kafka, so using it as the EventStreams backend was a natural choice.

Kafka allows us to begin consuming from any message offset (that is still present on the backend Kafka cluster). This feature is what allows connected EventStreams clients to auto-resume (via EventSource) when they disconnect.

WMF Administration

EventStreams/Administration

Notes

Server side enforced timeout

WMF's HTTP connection termination layer enforces a connection timeout of 15 minutes. A good SSE / EventSource client should be able to automatically reconnect and begin consuming at the right location using the Last-Event-ID header.

See this Phabricator discussion for more info.

Uses

Check out the Powered By page for a list of EventStreams client uses.