Sun Mar 05 • 4 min read
Benchmarks
I almost don’t care about throughput benchmarks. For years, I’ve worked with Ruby On Rails which was just slow. Possibly the slowest. Didn’t win a single req/s competition since it was created. It didn’t matter though. Most of the request time in a typical web app I built was spent on IO anyway. Even if your bottleneck was not IO you could always throw more money at the CPU / RAM. It was still cheaper to buy hardware than to write code outside of Rails ecosystem. The tooling and community around was, and in many aspects still is, so superb, that it’s worth it.
There were though dev benchmarks that Ruby made me addicted to. How fast can you deploy a new version of app? Will it take 2 or 20 minutes? How fast can you run the test suite? 10 seconds or minutes? When I moved to TypeScript and NestJS I noticed how much those slowed down. It’s not a problem with any specific part of the ecosystem. The community just values and optimizes different things. And that’s fine. But I’m not going to lie, I was a bit disappointed.
Bun
Bun changes everything when it comes to TS stack, and below I’ll show you the results of my dev benchmark that will try to recreate my typical workflow when working on a backend project. I’m going to compare Bun and NodeJS.
Test Method
Run a CI pipelines that:
- Install runtime (node / bun)
- Installs dependencies (pnpm / bun)
- Type check (tsc —noEmit)
- Run tests (100 test files x 10 tests each – 1000 tests total, each test spawns server, makes few requests that interact with SQLite database, and then shuts down the server.) (vitest / bun test)
- Build a docker image able to run the app (buildx)
- Push image to GitHub Container Registry
Test Condition
Default GitHub Actions environment. I’ve figured that it’s the most common environment currently used by devs.
Each scenario runs 10 times.
Results
Benchmarks take into account 2 scenarios. With and without changing dependencies. Those 2 differ drastically from each other in speed usually. I’ve dropped the cold deploy completely as it happens once during the whole project lifetime.
1. Changing source Code
This scenario is just a single line change in a random source code file.
Time (s) | Bun | Node |
---|---|---|
Average | 35 | 107 |
Min | 19 | 86 |
Max | 48 | 134 |
When we look at the overall data we can see that Bun on average has an around 3x faster pipeline.
Time (s) | Bun | Node |
---|---|---|
Prepare | 8 | 20 |
Install | 1 | 1 |
TypeCheck | 2 | 2 |
Test | 3 | 60 |
Build and push Docker | 10 | 20 |
Other | 5 | 3 |
[Sum] | 29 | 106 |
Looking at the detailed data per step we can see that there are multiple differences between the runs.
Firstly preparation takes more than twice the time in Node because of the cache size. For some reason Setup Node GitHub action just tends to be very time-consuming. Install takes the same time as it is all cached. TypeChecking in the exact same.
When it comes to testing Bun just blows away vitest
in this scenario. No matter how many tests you throw at it, it’s blazing fast 😄. It’s as if there were 3 tests not 100. I even checked if there wasn’t a mistake in my code there. But no, it’s just that fast.
Docker building and pushing is also drastically faster in bun as it does not introduce 2 different images. There is no build step because the code is just ran as it is. Maybe if my app had an additional build step like compiling assets etc. than the timings would get closer.
Other is just a sum of all the other steps that are not listed above like saving cache. It’s not a big difference, but it’s still there.
2. Changing dependencies
This time we add 1 new dependency to app.
Time (s) | Bun | Node | Node WNL* |
---|---|---|---|
Average | 40 | 305 | 143 |
Min | 27 | 258 | 135 |
Max | 50 | 329 | 155 |
*Node WND – Node Without Native Libraries
On average Bun’s time didn’t change much. Around 5 sec increase on average. If your app does not have many dependencies it’s not going to change much in Bun.
In Node though the pipeline slowed down drastically. The main reason is the presence of 1 native library: better-sqlite3
. Installing it takes up to 60 seconds on slow GitHub Actions runner. Doing it 3 times (pipeline, docker build image, docker run image) takes a lot of time. If we dropped this library then the change would be smaller and the average would increase to 143 seconds rather than 305. Still the pipeline is slower by around 30 seconds to its source code change counterpart, and it will only grow bigger when the dependency tree grows. Other parts of the pipeline did not change.
What does this mean?
Finally! I’m so happy that there is a big movement making the dev experience in TS better and better. The future possibilities for new tools using Bun are endless. I’m excited to see what will come out of it. Even if you don’t care about benchmarks like I mostly do the work done by Bun maintainers is a huge step forward!
Discuss on
Leave A Comment