はてなブックマーク

As we continue to focus our AI research and development on solving increasingly complex problems, one of the most significant and challenging shifts we’ve experienced is the sheer scale of computation required to train large language models (LLMs). Traditionally, our AI model training has involved a training massive number of models that required a comparatively smaller number of GPUs. This was th

Threads has entered the fediverse! As part of our beta experience, now available in a few countries, Threads users aged 18+ with public profiles can now choose to share their Threads posts to other ActivityPub-compliant servers. People on those servers can now follow federated Threads profiles and see, like, reply to, and repost posts from the fediverse. We’re sharing how we’re continuing to integ

Marking a major investment in Meta’s AI future, we are announcing two 24k GPU clusters. We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training. We are strongly committed to open compute and open source. We built these clusters on top of

On July 5, 2023, Meta launched Threads, the newest product in our family of apps, to an unprecedented success that saw it garner over 100 million sign ups in its first five days. A small, nimble team of engineers built Threads over the course of only five months of technical work. While the app’s production launch had been under consideration for some time, the business finally made the decision a

Meta’s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing dead code. SCARF combines static and dynamic analysis of programs to detect dead code from both a business and programming language perspective. SCARF automatically creates change requests that delete the dead code identified from the program analysis, minimizing developer costs. In our last blo

Earlier this year, a small team of engineers at Meta started working on an idea for a new app. It would have all the features people expect from a text-based conversations app, but with one very key, distinctive goal – being an app that would allow people to share their content across multiple platforms. We wanted to build a decentralized (or federated) app that would enable people to post content

Fixit is dead! Long live Fixit 2 – the latest version of our open-source auto-fixing linter. Fixit 2 allows developers to efficiently build custom lint rules and perform auto-fixes for their codebases. Fixit 2 is available today on PyPI. Python is one of the most popular languages in use at Meta. Meta’s production engineers (PEs) are specialized software engineers (SWEs) who focus on reliability,

Buck2, our new open source, large-scale build system, is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient. In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1. Buck2, Meta’s open source large-scale build system, is now publicly available via the Buc

Facebook for iOS (FBiOS) is the oldest mobile codebase at Meta. Since the app was rewritten in 2012, it has been worked on by thousands of engineers and shipped to billions of users, and it can support hundreds of engineers iterating on it at a time. After years of iteration, the Facebook codebase does not resemble a typical iOS codebase: It’s full of C++, Objective-C(++), and Swift. It has dozens

UPM is our internal standalone library to perform static analysis of SQL code and enhance SQL authoring. UPM takes SQL code as input and represents it as a data structure called a semantic tree. Infrastructure teams at Meta leverage UPM to build SQL linters, catch user mistakes in SQL code, and perform data lineage analysis at scale. Executing SQL queries against our data warehouse is important to

Move faster, wait less: Improving code review time at Meta Code reviews are one of the most important parts of the software development process At Meta we’ve recognized the need to make code reviews as fast as possible without sacrificing quality We’re sharing several tools and steps we’ve taken at Meta to reduce the time waiting for code reviews When done well, code reviews can catch bugs, teach

Sapling is a new Git-compatible source control client. Sapling emphasizes usability while also scaling to the largest repositories in the world. ReviewStack is a demonstration code review UI for GitHub pull requests that integrates with Sapling to make reviewing stacks of commits easy. You can get started using Sapling today. Source control is one of the most important tools for modern developers,

MemLab: An open source framework for finding JavaScript memory leaks We’ve open-sourced MemLab, a JavaScript memory testing framework that automates memory leak detection. Finding and addressing the root cause of memory leaks is important for delivering a quality user experience on web applications. MemLab has helped engineers and developers at Meta improve user experience and make significant imp

The leap second concept was first introduced in 1972 by the International Earth Rotation and Reference Systems Service (IERS) in an attempt to periodically update Coordinated Universal Time (UTC) due to imprecise observed solar time (UT1) and the long-term slowdown in the Earth’s rotation. This periodic adjustment mainly benefits scientists and astronomers as it allows them to observe celestial bo

Transparent memory offloading: more memory at a fraction of the cost and power Transparent memory offloading (TMO) is Meta’s data center solution for offering more memory at a fraction of the cost and power of existing technologies In production since 2021, TMO saves 20 percent to 32 percent of memory per server across millions of servers in our data center fleet We are witnessing massive growth i

Meta Open Source is transferring Jest to the OpenJS Foundation Meta Open Source is officially transferring Jest, its open source JavaScript testing framework, to the OpenJS Foundation. With over 17 million weekly downloads and over 38,000 GitHub stars, Jest is the most used testing framework in the JavaScript ecosystem and is used by companies of all sizes, including Amazon, Google, Microsoft, and

The components of an integration test. The testing infrastructure provides the foundation on top of which engineers write their tests, and the execution platform for running them. Enabling developers to prototype, test, and iterate on new features quickly is important to Facebook’s success. To do this effectively, it’s key to have a stable infrastructure that doesn’t introduce unnecessary friction

Now that our platforms are up and running as usual after yesterday’s outage, I thought it would be worth sharing a little more detail on what happened and why — and most importantly, how we’re learning from it. This outage was triggered by the system that manages our global backbone network capacity. The backbone is the network Facebook has built to connect all our computing facilities together, w

To all the people and businesses around the world who depend on us, we are sorry for the inconvenience caused by today’s outage across our platforms. We’ve been working as hard as we can to restore access, and our systems are now back up and running. The underlying cause of this outage also impacted many of the internal tools and systems we use in our day-to-day operations, complicating our attemp

Facebook engineers have built and open-sourced an Open Compute Time Appliance, an important component of the modern timing infrastructure. To make this possible, we came up with the Time Card — a PCI Express (PCIe) card that can turn almost any commodity server into a time appliance. With the help of the OCP community, we established the Open Compute Time Appliance Project and open-sourced every a

How we built a general purpose key value store for Facebook with ZippyDB ZippyDB is the largest strongly consistent, geographically distributed key-value store at Facebook. Since we first deployed ZippyDB in 2013, this key-value store has expanded rapidly, and today, ZippyDB serves a number of use cases, ranging from metadata for a distributed filesystem, counting events for both internal and exte

MySQL, an open source database developed by Oracle, powers some of Facebook’s most important workloads. We actively develop new features in MySQL to support our evolving requirements. These features change many different areas of MySQL, including client connectors, storage engine, optimizer, and replication. Each new major version of MySQL requires significant time and effort to migrate our worklo

What the research is: The Ribbon filter is a new data structure that is more space-efficient than the popular Bloom filters that are widely used for optimizing data retrieval. One of the ways that Bloom, and now Ribbon, filters solve real engineering problems is by providing smooth configurability unmatched by other filters. Bloom filters work by overapproximating a set of keys associated with som

Facebook is embracing Rust, one of the most loved and fastest-growing programming languages available today. In addition to bringing new talent to its Rust team, Facebook has announced that it is officially joining the nonprofit Rust Foundation. Alongside fellow members including Mozilla (the creators of Rust), AWS, Microsoft, and Google, Facebook will be working to sustain and grow the language’s

People upload hundreds of millions of videos to Facebook every day. Making sure every video is delivered at the best quality — with the highest resolution and as little buffering as possible — means optimizing not only when and how our video codecs compress and decompress videos for viewing, but also which codecs are used for which videos. But the sheer volume of video content on Facebook also mea

A high-level overview of FOQS. The arrows show the flow of data. We will be hosting a talk about our work on Scaling a Distributed Priority Queue during our virtual Systems @Scale event at 11 am PT on Wednesday, February 24, followed by a live Q&A session. Please submit any questions to [email protected] before the event. The entire Facebook ecosystem is powered by thousands of distributed sys