I get really excited about new tech I find interesting. When I first "got" the testing framework AVA, I was soooo excited. I was on the rooftops of twitter telling people to give AVA a shot. I created two screencasts about how to set up and use AVA, and a full blown workshop on how to test React with AVA:
I actually had used Jest long before, but had a less than awesome experience:
So, I convinced my team at PayPal and we started using AVA for all our new React tests. I was getting ready to start migrating our existing UI tests and our server tests when I started to notice something about AVA that really put a wrench in the whole thing.
It turns out that AVA was created to be fast for lots of async tests. Not for large test suites of async and sync tests. So for the use cases in my application, I found this to be true:
@tommy @ken_wheeler just testing things out. AVA just doesn't seem to be delivering on the promised perf-gains :-( But it's being worked on.
On a thread where I'm talking about making a AVA โก๏ธ Mocha codemod
It was just too much. As much as I loved all the other amazing and awesome features the AVA team had worked on (and there are many), performance was so poor I couldn't enjoy them as much. As the testbase grew to hundreds of tests, my MacBook Pro was brought to its knees. My fans would spin up like an airplane ready to take flight ๐ฉ and it wouldn't respond to me reliably until the tests were done.
These performance issues led
to my team (myself included) skipping the githooks with --no-verify so we
wouldn't have to wait to push our code changes. This led to the build breaking
for really simple things (like linting errors and broken tests) which slowed our
team down a lot.
Another major issue we found was with AVA's testing assertions. They use power-assert which is an amazingly helpful utility, but would sometimes lead to reeeeally weird things happening (like this). I realize that these are things that can be fixed, but as AVA is a community project built in the spare time of contributors (thank you and you so much!), those things getting fixed often took time. Along with that, engineers on my team sunk hours upon hours of work trying to figure out where these issues were and working around them. We don't have that kind of time.
Why migrate to Jest instead ofย Mocha?
So, at this point, we had all our React tests with AVA, previous UI tests (mostly backbone) with Mocha, and server tests with Mocha. The natural choice was to move all our stuff to Mocha. But then I got involved in this thread:
Which led to this comment from Trevor D. Miller which I responded to (in which I mention that I'd write this very blog post about me migrating from AVA to Mocha... I'd already started migrating my React Testing workshop to Mocha here. Hah). Then Christoph Pojer responded with this great comment. That pretty much won my curiosity back and I decided to give it another go on a project.
Unfortunately, I didn't have time to look into it right away, even though I wanted to:
But a few weeks later I gave it a go:
If you're curious about what it takes (tooling-wise) to migrate from AVA to Jest: test(jest): swap AVA for jest (#16) ยท kentcdodds/match-sorter@7982724 I'm reeeally loving Jest! ๐๐ป๐๐ป๐๐ป๐๐ป
I was impressed at how fast it was and even more how easy it was to set up. Especially since I'd tried Jest before and it was soooo slow. And I was really encouraged by this tweet from Mike Nikles:
So long, AVA. Inspired by @kentcdodds and what I heard in discussions at @ReactRally, I replaced AVA with Jest.
I was slowly discovering the truth of this statement:
So I eventually swapped AVA for Jest for my open source module generator and now all my open source stuff is Jest and I love it ๐ย .
As is the case for me, when I get excited about new tech, I really like to share! So I created a bunch of videos for egghead.io about it, had a tech chat with Christoph Pojer on YouTube about Jest, and rewrote my React Testing Workshop again in Jest.
The Migration
Client Side
My coworker Jamund Ferguson did most of the work for migrating our React tests from AVA. We were really helped by this codemod by Kenneth Skovhus. This saved us a ton of time manually updating our tests to work with Jest.
There were still some places we had to work through and we definitely had to
configure Jest in some special ways to handle our scenario. We have both server
and UI code in a single repo and each side gets its own specialย .babelrc which
complicated things slightly and we had some odd behaviors with some browser
globals, but for the most part it was pretty solid. We also had to migrate our
usage of proxyquire to Jest's (superior)
built-in
mocking capabilities.
After getting that merged, a week or two later I migrated the old mocha UI tests. Luckily Jest actually works with most of mocha's globals so that migration was relatively simple. Just some slight config changes and find/replace some stuff. Pretty easy...
Server Side
And then there was the ~300 server Mocha tests. Surprisingly, this was an absolute nightmare to migrate. Most of the problems stemmed from some of our dependencies doing some interesting and archaic things. But some of these were legitimate bugs in Jest that were reported and fixed. Finding those bugs took a really long time.
I ended up trying 5 different approaches to the migration. I really wanted to limit changing the existing tests to just find/replace for globals and stuff, but ended up manually changing every single test in unique ways. It took about 2 weeks of effort. In the process I learned a TON about how our dependencies work including Jest.
I'm not going to share everything that I learned (that would take a really long
time and a lot of background info), but
here's one really interesting one
that tripped me up for a very long time. It involved polyfilling JSON.parse.
And here's another that caused me
some grief. Oh, and
requireindex ended up not working at all
and we had to remove it from our project. Luckily we weren't using it heavily.
The reason it took so long for me to get the server tests migrated wasn't so much the amount of effort it took to do the code changes (though that did take quite a bit of time) but the amount of time it took to dig through our project's dependencies to figure out which one was the root cause of the issue. Most of the errors were thrown in bluebird promises which provided pretty much no useful stacktrace of any kind.
I should also mention that my experience was not at all typical. Most people are able to migrate to Jest fairly quickly and seamlessly.
Was it worth it?
For the client-side stuff, I can say for sure absolutely positively YES:
As for the server side. Well, my PR for the server changes was just merged
yesterday, so I can only say I think that it was worth it. Jest provides a ton
of great features, not the least of which is amazing performance as the number
of tests grow. It's exposes a surprisingly powerful API with great and
reasonable defaults. Jest ships with
easy code coverage instrumentation and reporting
right out of the box. It has
an interactive watch mode
that I love. It has the best mocking story of anything I've ever seen. It has
snapshot assertsions
which work really well for React components, server responses, and redux state.
And most importantly, I think my co-workers and I will stop running
git push --no-verify like we were before which will help our productivity, so
I think that it's definitely worth the migration!
Stats
So the stuff people always want to know is: "Is it really faster?" Well...
Here's some clipped output from running the test commands with time before and
after (note that the before tests are from over a month ago so there are a few
more tests in the after numbers):
If you're unfamiliar with the time command see this. "real" is the thing we're focusing onย here.
Note: This is with the cache in all cases. I ran the tests several times in a row with my machine otherwise idle to get the best numbers I could for all tests. I also ran these with code coverage because that's what our git hook runs. Also, you'll notice there are more total tests to run in the "after" case because "before" is one month ago.
So here are some important stats based on my runs:
UI Before: 144ms per test
UI After: 28ms per test
Server Before: 111ms per test
Server After: 116ms per test
One thing that I found with Jest is that the ms/test stat decreases as you add tests. Turns out it's really good at running tests in parallel. So I expect we'd see a wider gap with more tests.
Conclusion
Jest is awesome and I recommend you give it a solid look. Seriously. Oh, and don't forget to give it a star ๐!
See you onย twitter!
