Fixing Sysbench Parser After Criterion Removal

The Unexpected Breakage

Have you ever updated a system, only to find that a crucial tool suddenly stops working? That's precisely the situation we found ourselves in recently when the Sysbench results parser started failing. For those unfamiliar, Sysbench is a powerful tool for benchmarking database performance, and a parser is essential for making sense of its output. Unfortunately, a recent change, specifically commit 9121c69b, removed Criterion from our benchmarks. Criterion, a well-regarded benchmarking framework, was instrumental in providing a standardized output format that our process_results.py script relied upon. With its removal and the subsequent adoption of a custom harness, the output format changed dramatically, leaving our trusty parser completely bewildered. This led to a rather unhelpful error message: "Error: No benchmark results found," which, while technically true from the parser's perspective, didn't offer much in the way of a solution. The immediate impact was that we could no longer accurately analyze the performance metrics from Sysbench tests, hindering our ability to track improvements and identify regressions. This situation underscores the importance of maintaining compatibility when making significant changes to underlying systems, especially those involved in testing and analysis.

How to See the Problem for Yourself

Reproducing this issue is quite straightforward, making it easy to verify the fix once it's implemented. If you're looking to observe the broken Sysbench parser in action, simply execute the following command in your terminal:

make benchmark-smoke

After the benchmarks complete, you'll notice a series of messages indicating the progress of result processing. The critical output you'll be looking for, which signals the parser's failure, appears after the line "Processing SYSBENCH benchmark results...". At this point, instead of seeing the parsed data integrated into the database, you'll be greeted with the aforementioned error: "Error: No benchmark results found". This clear indication confirms that the process_results.py script is unable to interpret the output generated by the new custom harness. It's a simple but effective way to demonstrate the incompatibility that has arisen. This diagnostic step is crucial for developers and testers to confirm the bug before and after applying any potential solutions, ensuring that the fix has indeed resolved the parsing issue. The benchmark-smoke target is designed to run a representative subset of tests, providing a quick way to trigger the failure without needing to run a full, time-consuming benchmark suite. Therefore, this reproduction step is both efficient and reliable for confirming the bug.

Unpacking the Root Cause: A Tale of Two Formats

To truly understand why the Sysbench results parser is broken, we need to delve into the root cause: the fundamental difference in output formats between the old Criterion-based benchmarks and the new custom harness. Our scripts/process_results.py script, specifically the SysbenchParser class located between lines 700 and 787, was meticulously crafted to understand and extract data from a very specific structure. This structure, dictated by Criterion, typically looked something like this:

sysbench_point_select/vibesql/10000
time: [1.23 µs 1.45 µs 1.67 µs]

Notice how Criterion provided the workload name, client details, and then a clear line indicating the time with associated latency values. The parser was programmed to find these patterns, extract the relevant numbers, and store them. However, the commit 9121c69b that switched to a custom harness changed everything. The new harness adopted a much more human-readable, table-based output format. Here’s an example of what the new output looks like:

--- VibeSQL Results ---
Workload               Client   Operations     Avg Latency      Ops/sec
Point Select              all      2825478         0.00 us    402891487
Insert                    all       328254         5.09 us       196596
...

As you can see, the new format presents information in columns, with headers like Workload, Client, Operations, Avg Latency, and Ops/sec. While this is arguably better for quick human inspection, our existing parser, expecting the line-based, pattern-specific Criterion output, simply couldn't find any recognizable markers. It looked for sysbench_point_select/vibesql/10000 and the time: keyword, but these were nowhere to be found in the new table structure. Consequently, the parser concluded that no results were present, leading to the "Error: No benchmark results found" message. The core issue, therefore, is a mismatch between the parser's expectations and the actual data format produced by the benchmark execution.

The Solution: Adapting the Parser

Now that we understand the problem – the Sysbench parser is expecting the old Criterion output format while the benchmarks are now generating a new, table-based format – the fix becomes clear. We need to update the SysbenchParser.parse() method within scripts/process_results.py to gracefully handle this new table structure. This involves refactoring the parsing logic to recognize the column headers and extract data accordingly. Instead of searching for specific lines like time: [...], the parser will now need to identify the table rows and parse the values under the relevant headers, such as Avg Latency and Ops/sec. This might involve using libraries or custom logic to parse tabular data, potentially by splitting lines and aligning values based on the header row. The goal is to transform the new, human-friendly table output into the structured data format that the rest of our system expects for analysis and storage. This update ensures that we can continue to leverage Sysbench for performance monitoring, even after significant changes to the underlying benchmarking infrastructure. It's a classic case of adapting legacy code to modern requirements, ensuring that valuable testing capabilities are not lost due to evolving development practices. The modification will likely involve adding new conditional logic or entirely new parsing routines within the parse() method to differentiate between the old and new output formats, or perhaps to exclusively handle the new format if Criterion is no longer used at all. The key is to make the parser robust and adaptable to the current reality of the benchmark output.

For more information on benchmarking best practices, you can refer to Percona's Database Benchmarking Overview.