Insites just got faster and better: Introducing “Speedy Mite”

“No matter how good you get you can always get better, and that’s the exciting part.” Tiger Woods

That’s a philosophy that we subscribe to here at Insites, and we are excited that our latest improvement to the platform is ready to go!

Rethinking our approach

We looked closely at the best way to analyse the tens of thousands of website pages passing through our platform to make the results more accurate and the analysis faster. The result is the new spidering approach we call “Speedy Mite.”

Previously, we downloaded the homepage in a real browser and then looked at the raw code on the other pages. We would then analyse the site sequentially, which extended the time needed to analyse the whole site, especially larger ones. 

While we were damn fast using this approach, we wanted to do better.

Through some clever thinking and testing by our tech team, we now download all the pages in a real browser to “see” what the site visitor would see and provide a more accurate view of the website performance.  

Performance Matters

But hang on… wouldn’t that be slower than just looking at the code?

Glad you asked!

Instead of analysing pages and tests sequentially, we now run everything in parallel, so the overall time it takes to analyse a website is much quicker. 

How much quicker?  About 20% faster according to our testing.

To add some icing on the Speedy Mite cake, we have also given our users the ability to set whether it is a mobile spider (to match Google’s mobile-first crawling) or a desktop spider. Of course, regardless of which one you choose, we will always check the homepage on both devices.

Because of the increase in performance, you can now decide if you want to analyse between 5 and 30 pages for each website, and we will still fall into the 30-90 secs average for the results depending on the tests you have enabled.

A new world on the horizon

This approach will allow us to deliver even more accurate results, particularly for heavily client-side sites (e.g., sites built with React or Wix) as we now “see” what the visitors see.

This new approach opens the door for us to explore new tests like the lazy loading of images. With Google’s Core Web Vitals growing in importance, we want to make sure we are accurately reflecting the users’ experience as much as possible and accurately reporting on those areas that can be improved.

Roll-out

For clients on our ecommerce plans, you will be automatically upgraded on Tuesday, September 21st, 2021.

Our enterprise clients will be migrated to the new spider by the end of 2021.