Some panics cause hangs despite --hard-fail, under load

[SimonSapin]

In #24871 we have a WPT test that causes a panic because of a failing debug assertion. Usually this causes the process to exit with an error code, wptrunner to register the test outcome as a CRASH, and all is well (as far as this issue is concerned). Intermittently however, the process hangs and wptrunner (eventually) registers a TIMEOUT.

I wonder if this is the cause of many of our expected (CC #24880) or intermittent timeouts.

wptrunner passes --hard-fail to Servo, which is supposed to exit on panics, avoiding hangs. However it can only do so when this method is called:

servo/components/constellation/constellation.rs

Lines 2427 to 2437 in b0f573b

	fn handle_panic(
	&mut self,
	top_level_browsing_context_id: TopLevelBrowsingContextId,
	reason: String,
	backtrace: Option<String>,
	) {
	if self.hard_fail {
	// It's quite difficult to make Servo exit cleanly if some threads have failed.
	// Hard fail exists for test runners so we crash and that's good enough.
	println!("Pipeline failed in hard-fail mode. Crashing!");
	process::exit(1);

Steps to reproduce this particular crash/hang:

• Check out commit 858bc5a
• Build with ./mach build --release --with-layout-2020 --with-debug-assertions
• Run for i in $(seq 100); (echo ============= Run $i; RUST_BACKTRACE=1 target/release/servo tests/wpt/web-platform-tests/css/CSS2/box-display/block-in-inline-relpos-002.xht --hard-fail --exit -z)
• While it’s running add load to the system, for example by running ./mach test-wpt --release --layout-2020

Expected output (crash, here without CPU load):

============= Run 100



assertion failed: self.font_size.map_or(true, |f| f == font_size) (thread LayoutThread PipelineId { namespace_id: PipelineNamespaceId(1), index: PipelineIndex(1) }, at components/style/rule_cache.rs:29)
stack backtrace:
   0: servo::main::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:468
   2: std::panicking::begin_panic
   3: style::values::specified::length::FontRelativeLength::to_computed_value
   4: style::values::computed::length::<impl style::values::computed::ToComputedValue for style::values::specified::length::LengthPercentage>::to_computed_value
   5: style::properties::longhands::margin_left::cascade_property
   6: style::properties::cascade::Cascade::apply_properties
   7: _ZN5style10properties7cascade13cascade_rules17h96faf43c84170c77E.llvm.7578349383558476548
   8: style::stylist::Stylist::cascade_style_and_visited
   9: style::style_resolver::StyleResolverForElement<E>::cascade_primary_style
  10: style::style_resolver::StyleResolverForElement<E>::resolve_style_with_default_parents
  11: style::traversal::compute_style
  12: <layout::traversal::RecalcStyle as style::traversal::DomTraversal<E>>::process_preorder
  13: style::driver::traverse_dom
  14: layout_thread::LayoutThread::handle_reflow
  15: profile_traits::time::profile
  16: layout_thread::LayoutThread::handle_request_helper
  17: layout_thread::LayoutThread::start
  18: profile_traits::mem::ProfilerChan::run_with_memory_reporting
  19: std::sys_common::backtrace::__rust_begin_short_backtrace
  20: _ZN3std9panicking3try7do_call17ha66e0d91de444c54E.llvm.1363856088101530016
  21: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:81
  22: core::ops::function::FnOnce::call_once{{vtable.shim}}
  23: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/1bd30ce2aac40c7698aa4a1b9520aa649ff2d1c5/src/liballoc/boxed.rs:942
  24: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/1bd30ce2aac40c7698aa4a1b9520aa649ff2d1c5/src/liballoc/boxed.rs:942
      std::sys_common::thread::start_thread
             at src/libstd/sys_common/thread.rs:13
      std::sys::unix::thread::Thread::new::thread_start
             at src/libstd/sys/unix/thread.rs:79
  25: start_thread
  26: __clone

[2019-11-27T09:51:40Z ERROR servo] assertion failed: self.font_size.map_or(true, |f| f == font_size)
Redirecting call to abort() to mozalloc_abort

Stack trace for thread "LayoutThread PipelineId { namespace_id: PipelineNamespaceId(1), index: PipelineIndex(1) }"
stack backtrace:
   0: servo::install_crash_handler::handler
   1: _ZL15WasmTrapHandleriP9siginfo_tPv
             at /home/simon/.cargo/git/checkouts/mozjs-fa11ffc7d4f1cc2d/5906588/mozjs/js/src/wasm/WasmSignalHandlers.cpp:969
   2: <unknown>
   3: _ZL22AnnotateMozCrashReasonPKc
             at target/release/build/mozjs_sys-c4b353f67dc6d2ba/out/dist/include/mozilla/Assertions.h:38
      mozalloc_abort
             at /home/simon/.cargo/git/checkouts/mozjs-fa11ffc7d4f1cc2d/5906588/mozjs/memory/mozalloc/mozalloc_abort.cpp:33
   4: abort
             at /home/simon/.cargo/git/checkouts/mozjs-fa11ffc7d4f1cc2d/5906588/mozjs/memory/mozalloc/mozalloc_abort.cpp:79
   5: __libunwind_Unwind_Resume
   6: style::stylist::Stylist::cascade_style_and_visited
   7: <unknown>
   8: <unknown>

Intermittent output (hang, here with CPU load):

============= Run 9



assertion failed: self.font_size.map_or(true, |f| f == font_size) (thread LayoutThread PipelineId { namespace_id: PipelineNamespaceId(1), index: PipelineIndex(1) }, at components/style/rule_cache.rs:29)
stack backtrace:
   0: servo::main::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:468
   2: std::panicking::begin_panic
   3: style::values::specified::length::FontRelativeLength::to_computed_value
   4: style::values::computed::length::<impl style::values::computed::ToComputedValue for style::values::specified::length::LengthPercentage>::to_computed_value
   5: style::properties::longhands::margin_left::cascade_property
   6: style::properties::cascade::Cascade::apply_properties
   7: _ZN5style10properties7cascade13cascade_rules17h96faf43c84170c77E.llvm.7578349383558476548
   8: style::stylist::Stylist::cascade_style_and_visited
   9: style::style_resolver::StyleResolverForElement<E>::cascade_primary_style
  10: style::style_resolver::StyleResolverForElement<E>::resolve_style_with_default_parents
  11: style::traversal::compute_style
  12: <layout::traversal::RecalcStyle as style::traversal::DomTraversal<E>>::process_preorder
  13: style::driver::traverse_dom
  14: layout_thread::LayoutThread::handle_reflow
  15: profile_traits::time::profile
  16: layout_thread::LayoutThread::handle_request_helper
  17: layout_thread::LayoutThread::start
  18: profile_traits::mem::ProfilerChan::run_with_memory_reporting
  19: std::sys_common::backtrace::__rust_begin_short_backtrace
  20: _ZN3std9panicking3try7do_call17ha66e0d91de444c54E.llvm.1363856088101530016
  21: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:81
  22: core::ops::function::FnOnce::call_once{{vtable.shim}}
  23: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/1bd30ce2aac40c7698aa4a1b9520aa649ff2d1c5/src/liballoc/boxed.rs:942
  24: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
             at /rustc/1bd30ce2aac40c7698aa4a1b9520aa649ff2d1c5/src/liballoc/boxed.rs:942
      std::sys_common::thread::start_thread
             at src/libstd/sys_common/thread.rs:13
      std::sys::unix::thread::Thread::new::thread_start
             at src/libstd/sys/unix/thread.rs:79
  25: start_thread
  26: __clone

[2019-11-27T09:20:26Z ERROR servo] assertion failed: self.font_size.map_or(true, |f| f == font_size)
Redirecting call to abort() to mozalloc_abort

^C

[SimonSapin]

@jdm Have we considered compiling with panic = abort for WPT CI?

[jdm]

No, but that sounds worth investigating.

[gterzian]

It doesn't look like it's the constellation handle_panic that is called when the crash does come through, from the stacktrace it appears that it's rather this signal handler:

servo/ports/glutin/main2.rs

Line 51 in 2ad6e94

fn install_crash_handler() {

It also looks like this particular panics calls into

https://github.com/servo/mozjs/blob/99952b4e799a9b2d7647176d6f2d94d78e642b7e/mozjs/memory/mozalloc/mozalloc_abort.cpp#L23

and

https://github.com/servo/mozjs/blob/99952b4e799a9b2d7647176d6f2d94d78e642b7e/mozjs/js/src/wasm/WasmSignalHandlers.cpp#L927

which then I guess calls into the handler setup in Servo.

Maybe the system is slow to deliver/forward the signal(s) under load, or something in mozjs is stuck waiting on some system resource?

(Also, the signal handler wouldn't be called if tests are run with multiprocess Servo, since the handler is setup in the "main process", not each content process)

[SimonSapin]

compiling with panic = abort for WPT CI

I tried it: https://community-tc.services.mozilla.com/tasks/groups/SzXIvJXxTHueVBEbv079QA

There’s a high number of crashes. Could be worth looking into some more, but I’m not going to today.

[jdm]

Interestingly in those results, the log of known intermittents doesn't display any crashes. It's all panics in threads that are either not monitored by the hard-fail panic monitoring, or panics during shutdown.

[SimonSapin]

It also looks like this particular panics calls into

https://github.com/servo/mozjs/blob/99952b4e799a9b2d7647176d6f2d94d78e642b7e/mozjs/memory/mozalloc/mozalloc_abort.cpp#L23

Now I’m very confused.

According to the Redirecting call to abort() to mozalloc_abort message being printed we’re going through here:

https://github.com/servo/mozjs/blob/99952b4e799a9b2d7647176d6f2d94d78e642b7e/mozjs/memory/mozalloc/mozalloc_abort.cpp#L59-L86

The comment says that extern "C" void abort(void) overrides the system abort. So presumably some Rust code calling libc::abort goes here.

This is what calls mozalloc_abort, which then calls MOZ_CRASH()

https://github.com/servo/mozjs/blob/99952b4e799a9b2d7647176d6f2d94d78e642b7e/mozjs/mfbt/Assertions.h#L259-L293

… which in turn calls MOZ_REALLY_CRASH()

https://github.com/servo/mozjs/blob/99952b4e799a9b2d7647176d6f2d94d78e642b7e/mozjs/mfbt/Assertions.h#L244-L257

… which calls abort. How is this not infinite recursion?

[SimonSapin]

… which calls abort.

I was wrong. The line before that causes a segfault. So the question is why we hang after a segfault.