Case Study: How I Made a Widget That Sorts and Filters 12K+ Records Insanely Fast on Duda


Learn how some API work can improve your site's user experience


A customer came to me with a site that uses dynamic pages to display their blog posts–they have literally thousands and thousands of blog posts and anticipated having 12K+ eventually. At around 6K records, things were already crawling. In their region, internet speeds might top around 3 mbps. The pages needed to be optimized so they'd be far more usable. Check out the widget below, and scroll just below it to learn how we made it happen! For a more technical look, you can also read this Medium blog post meant for developer audiences.


****Please note that the images are NOT optimized quite yet so those load slowly if you are throttling or testing with a 3G network. Will be updated soon but right now the files are 10MB+ in size instead of mere kb.

Challenges


  • The built-in collection feature itself seemed to be part of the problem at the scale of 12K records – the widget content and design tabs loaded slowly, and the page took some time to load. Once the collection was loaded everything was fast, but that wasn't optimized enough IMO!
  • All of the data is managed in Airtable and needs to stay there. Airtable only lets you get 100 records per API request and limits you to 5 requests per second, so when you have 15K records, you'd have to wait around 30 seconds to get the entire table if querying directly from Airtable. So that had to be done BEFORE people came to the site. Direct querying would also result in API call limits being met if enough people were using the site at the same time, like at 8 AM when they sit down for a cup of coffee and to read the daily news.
  • The Airtable base is updated every day multiple times per day and needs to always be up to date as it's a news site, so stale data was NOT an option and manually uploading a new JSON file every day was also NOT an option. (If you do want to do it manually, you can actually export the Airtable base as a CSV and use any online CSV → JSON converter to get a quick JSON file. I like this one.


The Solution


  • Add some fields to Airtable to enable things like filtering the widget by category, featured data, year, etc. and be easier to maintain
  • Static hosted JSON file
  • Google cloud function to setup communication between the static file and Airtable
  • Google cloud scheduler to run the updates on a schedule
  • Airtable scripts to invoke updates on demand
  • Script to upload images to Duda using the Duda API to optimize images if needed and store them in the Duda CDN and send them back to the Airtable base to be used in the static JSON file


How Widget Pro made it happen


Right away, using a static-hosted JSON file seemed like the best option. I did a proof of concept by uploading a mock data file to Github and loading it in the editor, and there was either zero or minimal lag to load the file with a fetch request. (Minimal lag at my mountain cabin using 5 mbps wifi–hurry up, Starlink!)


As far as which platform to use, I wound up choosing an API-friendly storage platform called bunny.net, which a lot of people use to make their sites FAST as it caches your site and replicates it to servers globally. For storing a JSON file, it's as simple as either directly uploading it, or making a quick API call using an authentication token for authentication instead of needing an entire manual to compile authentication for a tool like S3. It was a real winner as soon as I found it. The pricing was super reasonable, and for this project, would be under $2 per month. $24/year is a drop in the bucket if it can bring in infinite more traffic to your site by making it load instantly instead of 30 seconds later IMO!


For the serverless function, I chose GCP because I'd already had some experience using Google Cloud for other tasks and I just like it 🤷‍♀️. I usually use Vercel, but the timeout (how long the serverless function can run) was too low, so I needed to be able to set a higher limit for maximum execution time. So GCP was a winner. If you're non-technical and wondering what the heck a serverless function is, here's a great summary from CloudFlare:


Serverless JavaScript is hosted in an edge network or by an HTTP caching service, which stores content to respond quickly to HTTP requests. Developers can write and deploy JavaScript functions that process HTTP requests before they travel all the way to the origin server.


Sorry that's still technical. All you need to know is that it runs code and if you're not calling it a lot, it's usually free! So no server costs to worry about.


The rest of the magic was really just writing a lot of code. For the Duda widget, I used my favorite library, list.js, which makes creating search + filter widgets soooooooo great. I just love it. You should ❤️ it too. In fact, I just set up a monthly donation to Jonny, who meticulously keeps the library up to date. Thanks, Jonny!


Now, I still made the widget optional to connect directly to a collection, but just added in some fields to let the client query the right static JSON file for each instance of the widget. Here are some of the considerations to keep in mind if your team develops something like this:


  • Logic to show and hide filters, search, page count, etc. The widget might get used on a page with multiple categories, like a Medium-style layout
  • JavaScript code to filter all of the unique checkbox options and render them on the screen–things change often and manually updating checkboxes is a pain in the ass
  • JavaScript to manage the filter criteria that are currently applied. Using a custom class or prototype works well for this so you can just rewrite a new one if you're clearing all of the filters and easily refer to the properties
  • Functions to handle adding or removing criteria from the selections object if you are selecting and deselecting filter criteria
  • Lots of design panel options to customize the widget and make it reusable for other websites
  • Flexible template options for cards -- make it possible to hide fields based on if the data is there or not or applicable to that page so the card can look nice, even if half of the fields are hidden


If you need to hire some technical for this type of project, send me an email or book a time on my calendar under About → Contact. Widget Pro is a certified Duda Expert in the Duda community. You can also hire other pros for help via the Duda Experts page.