Discovering a JDK Race Condition, and Debugging It in 30 Minutes with Fray

Summary

Discovering a JDK Race Condition, and Debugging it in 30 Minutes with Fray I’ve been adding more integration tests for Fray recently. To ensure Fray can handle different scenarios, I wrote many creative test cases. Many of them passed as expected, while some failures led to epic fixes in Fray. Then something unexpected happened: Fray threw a deadlock exception while testing the following seemingly innocent code: 1private void test() { 2 ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(1); 3 // Shutdown thread. 4 new Thread(() -> { 5 executor.shutdown(); 6 }).start(); 7 try { 8 ScheduledFuture<?> future = executor.schedule(() -> { 9 Thread.yield(); 10 }, 10, TimeUnit.MILLISECONDS); 11 try { 12 future.get(); 13 Thread.yield(); 14 } catch (Throwable e) {} 15 } catch (RejectedExecutionException e) {} 16} This code creates a ScheduledThreadPoolExecutor, schedules a task, and shuts down the executor in another thread. Initially, I suspected a bug in Fray, but after investigation, I discovered that the deadlock was actually caused by a bug in the JDK itself. Debugging this issue was straightforward thanks to Fray’s deterministic replay and schedule visualization. To understand the deadlock, let’s first take a look of the implementation of ScheduledThreadPoolExecutor: 1public class ScheduledThreadPoolExecutor extends ThreadPoolExecutor implements ScheduledExecutorService { 2 public Future<?> schedule(Runnable command, long delay, TimeUnit unit) { 3 // ... 4 RunnableScheduledFuture<?> task = decorateTask(...); 5 // delayedExecute method 6 super.getQueue().add(task); 7 8 // addWorker method 9 for (int c = ctl.get();;) { 10 if (runStateAtLeast(c, SHUTDOWN) 11 && (runStateAtLeast(c, STOP) 12 || firstTask != null 13 || workQueue.isEmpty())) 14 return ...; 15 // add task to a worker thread 16 } 17 return task; 18 } 19 20 public void shutdown() { 21 // tryTerminate method 22 int c = ctl.get(); 23 if (isRunning(c) || 24 runStateAtLeast(c, TIDYING) || 25 (runSt...