Introducing Formulate: a programmable form-builder

banner
Tim Andrew
Dec 12, 2022

Formulate is a new type of form-builder. Unlike most other tools in this space, Formulate lets you build forms using code, which is a great way to express conditional logic and other forms of dynamic behavior.

You can use it to build the kinds of things you’d expect: forms, surveys, and quizzes. But it also works for basic apps, games, or even as an intro to coding.

Try it right away, look at the docs or some examples, or keep reading for more context about what this tool is and the problems it’s trying to solve.

No-code tools are more popular than ever before. You’ll hear about how they’re democratizing programming so everyone can write software, and this is true … to some extent. There are a ton of legitimate uses for tools that allow non-coders to build software.

A distribution with two peaks. Two categories of tools dominate, while others languish.

The unfortunate thing is, though: we’ve ended up with a largely bimodal distribution of available options: either you write software from scratch yourself, or you use a visual builder of some sort, typically a drag-and-drop GUI.

This should really look more like a spectrum: programmers could benefit from some of the advantages that no-code tools provide without having to jump all the way to a clicky GUI. This works the other way, too: non-coders are perfectly capable of programming their tools in small ways without having to write an entire full-stack webapp from scratch. Ask any Excel (or AppleScript) power user!

What’s the big deal with code, though? How is coding business logic any more powerful than defining logic in a visual way? Here’s a quick run-down:

Programmability

What does it even mean?

At its core, programmability is a means for end-users to have a system behave exactly they way they want it to, without having to make feature requests to the developers.

It’s literally impossible for a non-programmable system to anticipate everything a user may want to do with it, so the designers have to make assumptions. This allows for a wider audience (you don’t need to know how to code to use this thing) at the cost of power and expressiveness.

A programmable tool makes the opposite tradeoff. The audience is necessarily narrower, but you can use the tool in all sorts of interesting ways that the designers may not even have imagined.

While those are the extremes, it’s possible for tools combine the two, offering a programmable substrate that some users build plugins on, or simply as a secondary interface.

Programmable Systems

HyperCard

HyperCard was a tool to build applications interactively on early Macintosh OSes. You could build visually, and drop down into code when you needed to.

hypercard
HyperCard help screen describing how to add a button to a user interface.

HyperCard used the HyperTalk programming language, which attempted to use natural language to construct programs. HyperTalk eventually led to the creation of AppleScript.

hypercard req
HyperTalk documentation describing the request command

HyperCard was a programmable tool, and so was almost infinitely flexible. People used it to write video games, movie databases, landing pages, puzzles, service manuals, synthesizers, heart rate calculators, full-fledged apps, and even plugins.

Lisp

Heritage

Lisp is one of oldest programming languages, second only to Fortran.

Many ideas in CS that are common today are from Lisp: trees, dynamic types, conditionals, recursions, and REPLs.

Lisp (and its dialects) are programmable programming languages. That’s a bit of a mouthful, so let’s break it down before we move on. Feel free to skip this mini-tutorial if you already know what Lisp macros are.

What is a programmable programming language?

Take a "regular" programming language like Javascript. You define a function like so:

let add = (a, b) => a + b

You can then call the function with numbers as arguments:

add(5, 7) // Returns 12

You can also call the function with more complex arguments:

add(700 * 3 * 2, Math.pow(2, 16)) // Returns 69736

Importantly, the add function has no knowledge of the fact that you performed all those multiplications to compute the first argument, and it certainly doesn’t know that you called Math.pow to compute the second.

The Javascript runtime evaluates these two arguments before invoking add, so that by the time add is called, the call looks something like this:

add(4200, 65536)

In a language like Javascript, you can’t write this function in a way that allows add to inspect the operations that went into calculating its arguments.

In a Lisp dialect, though? It’s easy! You can write a special kind of function called a macro that receives all it’s arguments as-is. The macro can inspect the whole thing before deciding what to do, making some truly dynamic behavior possible.

Here’s the same add function in Clojure, a Lisp dialect:

(defn add [a b]
  (+ a b))

(add (* 700 3 2) (Math/pow 2 16)) ; Returns 69736

So far this is the same as the Javascript version. The add function has no idea that you multiplied a bunch of numbers together to create its first argument, ditto for Math/pow.

Let’s create an add macro instead. To keep things simple, let’s have it print its arguments and return nothing:

(defmacro add [a b]
  (println "arg a is" a "of type" (type a))
  (println "arg b is" b "of type" (type b)))

And here’s what you’d see if you invoked it:

formulate=> (add (* 700 3 2) (Math/pow 2 16))
arg a is (* 700 3 2) of type clojure.lang.PersistentList
arg b is (Math/pow 2 16) of type clojure.lang.PersistentList

The add function is able to "see" its arguments in their entirety! Note that clojure.lang.PersistentList is simply the type Clojure uses for its linked list data structure, so these arguments can be manipulated just like regular lists.

Crucially, you can now have the add macro generate an implementation based on these arguments. You could run Math/abs on every single integer that was used to construct the arguments to add, or attempt to detect if one of the arguments has overflowed.

This was a very contrived example, but it goes to show how a programmable programming language can let you do things that are impossible in other languages.

Programmable programming languages allow you to write functions that accept code as their input, as opposed to the values that that code may evaluate to. This lets you write programs that write programs, and do many things that are impossible in other programming languages.

  • Viaweb used macros to implement RTML, a programming language end-users could use to customize their online stores. [1]

  • Clojure used macros to implement core.async, bringing (the equivalent of) async/await to the language with no changes to the language necessary. Imagine implementing channels in Go or async/await in Javascript with just a library!

Spreadsheets

Spreadsheets are a ubiquitous no-code tool, but they’re also legitimate programming environments. Complexity is carefully hidden or abstracted away so using spreadsheets don’t really feel like programming, making them accessible to so many more people.

Excel in particular has turing-complete formulas, multi-threaded computation, and the ability to drop down to a more traditional programming language when necessary.

Figma

Figma is a collaborative design tool that ostensibly has nothing to do with programmability. You design things in a GUI with no code in sight. However, Figma has a robust plugin system that lets you write code that has full control over the artifact being designed.

Programmability in the Modern Web

The Spectrum of Available Tools

With that introduction to the concept of programmability, let’s segment web-based tools by how programmable they are. At a high level, we think there are four buckets:

  • From-scratch tools that you use to build software & services from scratch

  • Progammable tools let you use code as the primary means to build things without the complexity of doing everything from scratch

  • Hybrid tools are fully visual but allow dropping down into code when necessary

  • No-code tools are fully visual, with no programmability

To be clear, this is a good thing. The more people that have access to create content for the web, the better.

Once you start filling these buckets up[2], it quickly becomes clear that two buckets are very well-represented. If you’re using a fully visual tool with no programmability or writing something entirely from scratch, you have a ton of options to choose from!

The middle two buckets are important too, though, and they’re strangely sparse. This leads us to believe that there’s a whole segment of users – power users, developers, or simply anyone that isn’t put off by a learning curve – that today’s tools aren’t really focusing on.

No-code tools may democratize content creation on the web, but they certainly don’t democratize programming. We want more tools that nudge people towards programming, instead of leaving the true power walled off behind a GUI.

Here’s a quote from an essay that refers to this idea as end-user programming:

There is a moral imperative for end-user programming: to avoid a digital divide. A world where only a tiny elite of high priests (aka “programmers”) have control over what happens in our computing lives is concerning. This is the well from which programming literacy advocates such as Code.org draw their vigor.

More practically, the business case for end-user programming is substantial. In enterprise software, end-user programmers are sometimes referred to as “citizen developers”. Businesses of all sizes run their daily processes on tools built by non-programmer domain experts with enterprise-oriented tools like Filemaker, Microsoft Access, Force.com, and Quick Base.

If programmable tools can exist and be successful in the enterprise, why don’t we have more programmable tools that are consumer-focused? There’s this somewhat pervasive idea that consumer-grade tools have to be whittled down to the least common denominator, with the assumption that users want simplicity and minimalism above all else.

We think it’s misguided to apply this principle universally – there’s room for powerful tools that empower users by making programmability a first-class citizen.

Excel is probably the prototypical example of this: it is a reactive dataflow system, but you’ll never find any draggable widgets with little input ports and output ports you can connect.

It’s trivial to get started with Excel. It’s trivial to connect cells and get some computations going. It’s then a nice smooth transition all the way up to multidimensional analysis, and not once do you have to break out into a full programming language.

That’s what people want.

What you imagine they want is something childish and useless. To be honest, it’s simply arrogance. By assuming the user can’t code, it’s natural to assume that they think at a level of a young child and need only toys.

For fuck’s sake, Excel is 64-bit and fully parallel! Did you know that? I’ve seen 60 year old ladies from accounting produce spreadsheets that can bring a 16-core terminal server to its knees, and that spreadsheet was doing entirely legitimate, useful, productive computation. Show me the last system you built that can efficiently utilise a modern high-core CPU for a single user!

Your market is adults. They need power tools, not toys.

The word code is almost ridiculously overloaded, though. You’ve got UI code, backend code, business logic, configuration as code, and infrastructure as code. Code for persistence, routing, authentication, deployments, and monitoring.

Surely we don’t want to expose all this complexity to the end user?

Just Code

Well … no! One approach is to simply expose none of this complexity to the end-user, which is what no-code tools do.

Command-line tools

Here’s another way to think about this. Say you write a command-line solver for Wordle and want to put it up on the internet. You’ll have to write so much code that has nothing to do with the actual solving process!

What if you could port your solver script to the internet without really worrying about anything extraneous? Well, we did, and the code is reasonably close to what we’d have written for the command-line.

Here’s a different tack: expose just some of this complexity to the end-user. The system could handle all the incidental complexity, while giving the end-user programmable control only over the bits of code that are essential to the task at hand.

There isn’t really an established name for tools that work this way. The closest thing we’ve seen is the term Just Code:

We actually need "Just Code" tooling.

The No Code movements shines a light on how bad we let things become for programmers (and aspiring ones). Building a modern full-stack website these days involves a lot of knowledge, everything from docker to react, to how to use github, npm, vscode, the command line, etc.

No Code/Low Code recognizes that most of that is "accidental complexity", and removes it. In a sense, it really lets you focus on the actual program (the "essential complexity"), in a way that "real" coding does not.

The problem with No Code is that "code" is actually important, by which I mean functions and types and values. Things that are composable and decomposable - you can look under the covers and there’s more code, or you can combine two things together to make a bigger thing. That’s as opposed to "wizards", which many (not all) No Code tools are.

We can remove the accidental complexity without removing the code though.

This resonates with us: these tools let you get rid of all the baggage and effort involved with from-scratch development except the core business logic.

No servers, deployments, backups, containers, databases, APIs, load balancing, frontends, or security: just code. We think this is a valuable approach that not many tools take, and we’re exploring the idea by building a tool that has exactly this focus. Which brings us to…

Formulate

Let’s now zoom in on a specific type of web-based tool: form-builders. These tools typically allow you to specify "logic" to make decisions based on user input. You can skip, modify, or repeat questions based on the user’s answers, perform calculations, or even perform a preprocessing step by creating derived answers.

We think this is a great fit for a programmable tool. A form-builder that allows you to specify logic using a real programming language could be powerful.

But we’re getting ahead of ourselves. What is a form?

What is a form, anyway?

Ignoring all the bells and whistles, a form is simply a tool to gather user input on the web. HTML has had a form tag since 1995. From the original RFC:

A form is a template for a form data set and an associated method and action URI. A form data set is a sequence of name/value pair fields. The names are specified on the NAME attributes of form input elements, and the values are given initial values by various forms of markup and edited by the user. The resulting form data set is used to access an information service as a function of the action and method.

This had the monumental effect of changing HTML from a read-only medium to a read-write medium. You could now use it to accept arbitrary data from the end-user. You’d have to write some HTML code with a form tag, set up a server of some kind to handle the form’s action, and you were good to go!

The word form is now really a shorthand for any webpage (or section thereof) that accepts user input and sends it to a remote server, even if an actual form tag isn’t used.

Nowadays people use forms for all sorts of things: from market research, gauging employee engagement, receiving customer feedback, and handling signups & registrations, to formal surveys, quizzes, and even to accept payments.

Here’s our take on a programmable version…

Programming Environment

A Formulate form is simply a Javascript file that exports a default function. Here’s a tiny example:

export default async function() {
  let options = {banner: true};
  let name = await form.short("Hi, what's your name?", options);
  await form.statement(`Lovely to see you, ${name}!`, options);
}

Restrictions

You can do anything you like inside this file with a few caveats:

  • You have access to Javascript-the-language, but you can’t use any Web APIs, such as the DOM or the window object.

  • The default Math.random() implementation is replaced with a seeded version, so each instance of a form will see the same sequence of random values until the seed is reset, which only happens when a form is loaded for the first time, a form is completed, or when you Start Over.

  • new Date() and Date.now() are locked in place, and return the instant at which the form was loaded. As above, this only resets when you Start Over or complete a form.

That snippet of code produces this form:

Try it out!
This form is live. You can interact with it here or in a new tab.

The file has access to a form global variable that allows it to interact with the end-user. You can use it to present questions to the user, make HTTP requests, log to the browser console, or examine query parameters. It’s the only interface that the code in this file has to perform side effects of any kind.

The code executes in a sandbox inside the end-user’s browser.

The API

Here’s a quick rundown of everything you can do with the Formulate API.

Questions

This is really any form’s bread and butter. Present questions to the user and receive answers in return. The API currently allows for these six question types:

Short

Accept a single line of text, similar to an input field.

Long

Accept multiple lines of text, similar to a textarea.

Multi

User picks from options. All options are visible at once.

Select

User picks from options. Options are hidden behind a dropdown.

Rating

Accept a star rating.

Payment

Accept a payment via Stripe.

More "Questions"

There’s a Statement question type which just presents the user with a message

And a Hidden question type that allows you to specify both a question and an answer in code without any user interaction.

Each of these returns a Promise, which is then fulfilled when the end-user has responded in some way.

Utilities

The API also contains some peripheral tooling to round things out; here’s a quick list:

HTTP

You can use form.fetch to make HTTP requests at runtime, and have your form dynamically alter itself in response.

Logging

Use form.log or form.error to log data to the browser console.

Params

Use form.query to retrieve a single value from the URL query string.

Lodash

The Lodash utility library is always available for cases where the JS standard library just isn’t cutting it.

What can I build with this thing?

Those are the basic building blocks available so far. We haven’t explored the space of things that they enable you to build too much yet, but at least theoretically, the sky’s the limit!

Here’s a small list of ideas we’ve had so far that focus on use cases that programmability makes simpler (or even possible at all):

Randomization & A/B Testing
  • Each respondent sees a percentage of a large pool of questions

  • Each question is a random choice between multiple variants

  • Only 50% of respondents see a given question

  • Some form of feature flagging, where you instantly enable or disable certain questions

APIs – integrate with APIs at runtime using form.fetch
Misc
  • Custom validation and data cleaning: ask the respondent to redo a question until it matches your schema/format exactly

  • Maintain a user-visible running total (or score)

  • Respondents verify their email address with an OTP

Next Steps

And that’s everything we’ve got so far! As with any programmable tool, we’re sure Formulate can be used for a ton of things we aren’t even thinking about yet.

Here’s what we are thinking about, at least for the near future:

Features

We’ve only spent roughly a month’s worth of evenings and weekends working on Formulate so far, and feature-wise there are a ton of gaps. Customization options are limited, response data is only available as a CSV download, there are no real user accounts, the UI can be inconsistent (especially on small screens), and you can’t use it from the command-line.

Programmable Destinations

Here’s a quick example: an integration with Google Sheets could be useful in and of itself, but we want to allow controlling this behavior via the form interface.

This should let you do things like picking a target sheet/cell or applying custom formatting based on a user’s responses.

We want to fill in these gaps one by one, while also being thoughtful about making things programmable where possible.

Validate the idea

Formulate feels a lot more ergonomic to us than other form-builders we’ve used in the past, but it’s still very much in its infancy. We don’t know if anyone is going to be interested in a tool like this, all we really know is that we really like using it so far.

We want to have conversations with anyone who has opinions on this sort of thing to refine Formulate’s core premise until it’s perfect. If you’d like to help shape the direction this takes, please get in touch!

Smoother learning curve

Formulate’s only interface is code. This is a good starting point, but we’d really like to evolve towards something more like Excel, where you have a simpler interface presented as the default, with a fully programmable layer underneath.

We don’t have this simpler interface yet, and we aren’t quite sure what it should look like. We’re confident we don’t want a traditional drag-n-drop GUI – there are plenty of those already – but we haven’t gone much further here yet.

One initial idea is to use a plaintext interface, similar to Inform 7 or writing slide decks in Markdown. More ideas are welcome!

Conclusion

How much does it cost?

Nothing! Formulate is a side project that we have limited time to work on, so we’re focused on the core feature set, not pricing.

With that said, the infrastructure behind this thing is not free, so we’re going to want to implement some sort of pricing model in the future to cover costs.

Formulate started as an "is this even possible to build" hobby project, but we’ve since become convinced that that current state of web-based tooling leaves a lot of gaps for developers and power users. We hope you’ll give Formulate a shot and tell us what you think.

Squinting ever-so-slightly, you could even use Formulate for things that are decidedly not forms. Treating it more generally as an abstraction for user input in a browser, you could use it to build games, tiny web apps, or maybe even teach people to code!

We don’t have all the answers yet, but we like the shape this has taken so far, and we’re excited to keep going!

Finally, a couple of resources: we’re on Twitter, there’s a discussion forum if you’re stuck or have questions, you can find API documentation here, and here’s a tiny gallery of example forms to get you started.


1. For more about this read the Blub essay.
2. This is a list of tools we’re personally familiar with and isn’t intended to be comprehensive. That said, we think the point stands either way.

Receive updates via email
Share this post