Why Microsoft needed to make Windows run Linux software

Perhaps the biggest surprise to come from Microsoft's Build developer conference last week was the Windows Subsystem for Linux (WSL).

The system will ship as part of this summer's Anniversary Update for Windows 10. WSL has two parts; there's the core subsystem, which is already included in Insider Preview builds of the operating system, and then a package of software that Canonical will provide. The core subsystem is what provides the Linux API on Windows, including the ability to natively load Linux executables and libraries. Canonical will provide bash and all the other command-line tools that are expected in a Linux environment.

Microsoft is positioning WSL strictly as a tool for developers, with a particular view to supporting Web developers and the open source software stacks that they depend on. Many developers are very familiar with the bash shell, with building software using make and gcc, and editing text in vi or emacs. WSL will give these developers versions of these tools that are equal in just about every regard to the ones you get on Linux, because they'll be the ones you get on Linux running unmodified on Windows.

With that developer focus, Microsoft isn't supporting WSL as a deployment platform. It might be possible to run, for example, the Apache Web server under WSL, and it might even be useful to do so for development, but the intent is not that applications would ever be run in production with this configuration.

Making lemonade

WSL was derived from Project Astoria, the effort to enable Android apps to run on Windows 10 Mobile; we'd expect that, initially at least, its support is strongest for those APIs that Android needed and that it will be weaker in other areas. This means that while application support should be extensive, Windows won't be able to run every Linux program under the sun. There's no Linux kernel here, so this won't bring support for filesystems such as ZFS or btrfs to Windows. Applications demanding low-level access to hardware devices, such as Wireshark, seem similarly unlikely to run and will continue to need Windows ports. WSL also doesn't include X Windows or any other graphical subsystem; it's strictly for command-line applications only.

Microsoft deciding to support Linux software on Windows is a surprising move, and the obvious question is, why? Superficially, the answer is easy—developers like these tools, so supporting them makes developers happy—but the story runs a little deeper than that. Supporting Linux apps in Windows is a significant move for a company that has spent decades promoting its own APIs and application platform, and it's unlikely that the company would make such a move just to provide a few creature comforts to developers. The need goes deeper than that.

Wind the clock back 15 years and Windows was the only serious platform for software developers. Linux was already an important consideration for servers, but on the desktop was even less of a concern than it is today, reserved only for the most hardcore fans. OS X was in its infancy, and only ran on weird, expensive, underpowered PowerPC hardware. This made Windows the development platform of choice by default. There simply wasn't any good alternative.

This in turn had consequences for software, where solid Windows support could drive server-side usage. A prime example of this is the MySQL database server. While it is better today than it once was, MySQL has always been regarded as second-best when it comes to important considerations such as technical capabilities and data integrity, with PostgreSQL generally held to be the much better database. In spite of this, MySQL won considerable market- and mind-share because it was a much better development platform. It had an easy Windows installer, it had good Windows software for managing databases and writing queries, and it had good ODBC drivers. PostgreSQL's developers, on the other hand, didn't appear to prioritize any of these things, and while it wasn't impossible to run the database on Windows, doing so meant forfeiting the creature comforts that MySQL offered.

It's no great surprise, then, that MySQL proliferated in a way that PostgreSQL didn't.

But things don't work that way any more. Setting up a Ruby development environment on Windows is a wretched experience. Getting Ruby running well is awkward on any platform, but Windows is arguably the worst. The popular node.js environment was born on OS X and Linux, and for a long time could not be effectively run on Windows. Salvatore Sanfilippo, developer of top NoSQL data store redis, has refused to accept patches to make the software run on Windows, not out of any particular hostility towards Microsoft, but because he saw no need for it. He supported the idea of forks of the software that supported Windows, but nothing that would impede development of the core product.

It's still not the year, but it's also not not the year

Windows certainly hasn't disappeared completely from view, but it's no longer the essential, must-have platform that it once was. Why not? Because those two non-contenders in 2000 are more or less viable today. Linux for various reasons still may not be the most comfortable desktop platform (especially for anyone wanting to use it on a brand-new laptop), but it's much more livable than it used to be. And OS X, thanks to a combination of the switch to x86 and Apple's fine hardware design, has become an appealing option for a great many developers.

Indeed, for a number of years, it wasn't a huge exaggeration to say that Apple made the only x86 laptops that were both reasonably affordable and pleasant to use. The MacBook Air, in particular, set a new standard for size and weight, and while the initial release was expensive, subsequent iterations were much less so. The traditional PC world had little that could match, and even when PC OEMs did build good systems—as Lenovo often did—they tended to be high-priced "corporate" machines. Apple truly led the way when it came to offering that mix of price, power, and portability, and it did so with an operating system that just happened to use a large portion of FreeBSD's code. There are still, of course, differences between OS X's FreeBSD and Linux running on a server, but they're a great deal smaller than the differences between either of those and Windows.

What Apple did was to give every computer science and software engineering student a three-pound Unix workstation for not very much money, and unsurprisingly, they proliferated. This effect was compounded further by Apple's enduring popularity in San Francisco and Silicon Valley, and, related, the growth of iOS as an application platform. Smart kids fresh out of university (whether they dropped out or otherwise) weren't thrust into the Windows-centric world of corporate America. They could instead thrive in a culture that revered Apple and equipped them all with shiny new MacBook Pros. Windows wasn't reviled the way that the die-hard open source advocates reviled it; it just wasn't part of their life.

This is a problem that Microsoft has been slow to recognize. Microsoft is generally good at addressing the needs of Microsoft's own existing development community, but this came with a kind of myopia. Anything outside this constituency was ignored. The Visual Studio C and C++ compiler, for example, still lacks full support for C99, the version of the C language that was standardized about 17 years ago. In spite of repeated requests to the company that it add C99 support, there was always pushback. The rationale I was given more than once was that since Windows developers weren't writing C99 code, there was no reason for C99 support.

That Windows developers couldn't write C99 code because the main Windows C and C++ compiler couldn't compile it was apparently beside the point. And that the open source world had embraced C99, and was now producing code that simply couldn't be compiled on Windows—code that Windows developers might well want to use, if only it would work—was similarly ignored. The people writing this code weren't part of the Windows development constituency, and so addressing their needs wasn't a priority. In the last couple of years, there does seem to have been something of a realization within Redmond that it's missing out, and adherence to the newest iterations of the C++ specification has been made a priority, but full C99 support still hasn't been implemented, and it's not clear it ever will be.

The same is broadly true of Unix compatibility in general, and the Unix shell environment and workspace in particular. A hot young developer coming out of a top university is going to be comfortable with bash and make and all those other tools. Visual Studio may be great—Visual Studio is great—but it's irrelevant to this way of working. Microsoft was not completely blind to this problem—I've heard from a few people inside the company that there has to be a not insignificant education effort for new hires, because coming to Microsoft is literally the first time they're exposed to the Windows way of software development—but until last week's announcement, it seems that little was being done to address it in a systematic way.

What we had instead was individual porting efforts. And it's important not to discount these; Microsoft contributed money and developer time to node.js to help port it to Windows. The software was rearchitected to accommodate this—node.js relies heavily on asynchronous I/O, and the optimal approach for this is different on every platform—and I think most people would agree that the result is that node.js has become better software (it also turns out that Windows' approach to asynchronous I/O is really good). The asynchronous I/O parts were broken out into their own library, libuv, which is useful in its own right, and node.js is a client of that library. Microsoft has similarly developed a port of redis that runs natively on Windows.

These efforts were valuable, but it's not an effort that scales well. Microsoft can't port every project, and while Ryan Dahl, creator of node.js, recognized that supporting Windows was valuable and was willing to make considerable changes to node.js to support Windows, that's not true of every developer.

Better hardware calls for broader horizons

The hardware gap that Apple once enjoyed has largely gone away. Devices like the HP Spectre x360 and Dell XPS 13 are fine systems; there's a range of diverse, high quality, affordable PC hardware options that really didn't exist five years ago. There's also an interesting degree of diversity; you can get traditional laptops, or laptops with 360 degree hinges, or convertibles like the Surface Pro 4, or exotica such as the Surface Book. These are all solid choices. But the hardware alone isn't enough any more; Windows doesn't offer the development user experience that is now so commonplace. Microsoft needs to do something to appeal to this developer community, and maybe even win it over to Windows.

Windows Subsystem for Linux fixes that by bringing a large part of that user experience to Windows. Redis, for example, will work on WSL. Not a forked port of redis that's been altered to work around the differences between Windows and Unix systems; real redis. The Ruby experience, well, it won't ever be good, because it's a pain on every platform. But it will be better. The same is broadly true for most of the open source stack that powers a huge proportion of Web and cloud development these days. It'll work on Windows, just the same as it already does on Linux. Microsoft is working with Canonical initially, so all of these things will be an apt-get away just like they are on Ubuntu. It wouldn't be tremendously surprising to one day see other Linux flavors, too.

As WSL matures, it doesn't feel beyond the realm of possibility that there will be pressure on Microsoft to treat it as something more than just a development tool and address the desire to deploy onto WSL on Windows Server. Especially for small deployments, the ability to run redis within an otherwise Windows-based deployment without spinning up a virtual machine feels obviously desirable.