*ฅ^•ﻌ•^ฅ* ✨✨  HWisnu's blog  ✨✨ о ฅ^•ﻌ•^ฅ

Always check the code - Fasta benchmarks game

Introduction

I've encountered some slimy practises in not only one, but two of the biggest benchmarking websites. So in this post we are going to dig deep what happened and to remind everyone to always, ALWAYS check the code!

Update: added external comment section at the bottom page. Powered by komments.

What is Fasta benchmark?

It's a performance evaluation or test that assesses how well a particular system, algorithm, or implementation performs when executing FASTA searches or alignments. You can read more about it here.

Below are the pages for the Fasta benchmark:

  1. benchmarks-game. *Note there's no indication or remarks whether a program is single or multithreaded, they just lump in everything in one table.

  2. programming-language-benchmarks *Note in this one it explicitly mentioned about whether if a program is multithreaded or not. Multithreaded programs have -m postfix in their filenames. This website does a much better job on making it clear: those with -m are multithreaded, without are singlethreaded.

Let's check the code!

Since Rust (blazingly fast!) topped the #1 in both benchmarks, let's put the code under microscope, especially the one from benchmarksgame since it did not mention whether if the program is single or multithreaded.

Here's the Rust code, you can check it yourself by clicking the source column of the table. On this section I will only point out interesting parts of the code.

Rust program is multi-threaded and will use up to 2 threads.

system info: quad core.

From the screenshots above we understand the program will use 2 cores ~ a multithreaded program.

Comparison to C code

Here's the C code. Please review the code, or if you don't have time for a manual review, just copy paste the code to one of those AI/LLM and ask whether the program is single or multi threaded.

From here we can conclude the C program is single-threaded.

A more fair comparison

It is unfair to compare single threaded program to multi threaded one, so let's make the Rust code single threaded as well! It's very easy, just change num_threads to 1 (from previously 2): *Note the above code is from the same benchmarksgame source.

Re-run the benchmark in single-threaded

So what do you think of the single-thread result would be? I've decided I would throw in Zig as well into the comparison to see how it would fare vs C and Rust. Here it goes: *Note the top uses Hyperfine, the bottom uses Mitata -> both showing the same order of rank.

Voila, it's not that surprising to see now the whole narrative is turned upside down. This is why I tend to make my own benchmark coz numbers can be misleading if not intentionally deceiving if we are not carefully examining what's under the hood.

Conclusion

Always check and review the code! As I mentioned several times in my blog posts: my main is in finance as a hedgefund professional. Those equity reports you received from your broker, they can be twisted upside down, even numbers shown in statistics data are used if they "fits the narrative" ~ often to convince people that "it's backed by data".

Yes it's prudent to have something backed by data, however it's not unusual that people often mislead / misrepresent the data. For example you know the period where loads and loads of research article on "Bitcoin to be next Gold", did you notice how those "researchers" conveniently cutoff the data just before Bitcoin crash? So the time period they showed were all shiny and glazy to paint a rosy picture for Bitcoin.

It all depends on what's the narrative people want to sell you...same happened here. So please, do your homework, always review, do check and recheck.

Comments section here

Note on compiler flags used for each language

Source code: