FarmBot Software Developers

CeleryScript

What is CeleryScript?

CeleryScript is an Actively Maintained Standard

CeleryScript nodes are added, modified and removed often. The latest list of Celery Script nodes can be found here.

The Farm Bot system has many moving parts. Data must be exchanged between systems in a way that is predictable and asynchronous. Sometimes, this data is even used for telling the bot what to do in a similar fashion to traditional programming languages. To accomplish this, we use a special remote procedure call and data interchange format called "CeleryScript".

CeleryScript nodes are specially formatted JSON documents. FarmBot uses these documents for a variety of storage and communication use cases and also as an internal programming language for sequence scripting.

It is a programming language, serialization format and RPC protocol unified under a single schema known as a "corpus".

With adequate experience, it is possible for a developer to read and write Celeryscript manually, but this is an uncommon occurrence. Celeryscript is typically read and written by automated tools such as servers, FarmBots, compilers, and interpreters. Celeryscript's main goal is uniformity and ease of development when authoring said tools. It has a structure that is optimized for ease of consumption by developer tools which sometimes comes at the expense of human readability. Please keep this design tradeoff in mind as you read the documentation.

Intended Audience

This document is intended for advanced software development and debugging. Many software developers can avoid the low level details of Celery Script by using the FarmBotJS library. All of the low-level details of CeleryScript are abstracted away when using a wrapper library.

CeleryScript knowledge is required only if you prefer to not use the wrapper library, are developing new features for the FarmBot platform, or are trying to debug specific problems with the system. It's also a great way for an intrepid software developer to learn FarmBot system internals.

Javascript developers are encouraged to use FarmBot JS instead of raw CeleryScript for most use cases.

If you are writing CeleryScript for a new or unsupported language you are highly encouraged to write your own wrapper library, as writing CeleryScript by hand is tedious, error-prone and likely to have future compatibility issues. Conversely, migrating and managing auto-generated CeleryScript is often a trivial task that can be accomplished via scripting.

Where is CeleryScript Used?

Celery Script is used:

Where Can I See CeleryScript in Use?

Nodes are typically seen in the Message Broker in the form of RPC requests. They are also used in the /sequences endpoint of the REST API as stored sequences.

Example Sequence

The most prominent use case for CeleryScript is the Sequence Editor. A sequence is a collection of nested celery script nodes.

Creating this sequence in the web app...

...results in the creation of the following CeleryScript:

{
  "kind": "sequence",
  "name": "Example 1",
  "color": "gray",

  // These values are set by the server.
  "id": 1,
  "created_at": "2000-01-01T00:00:00.000Z",
  "updated_at": "2000-01-01T00:00:00.000Z",
  "in_use": false,

  "args": {
    // Set by the server; Tells us the Celery Script version of this sequence.
    "version": 20000101,
    "locals": {
      "kind": "scope_declaration",
      "args": { }
    }
  },
  "body": [
    // Each body item is what the user sees as a "step" in the editor.
    // "If statement" step
    {
      "kind": "_if",
      "args": {
        // "left hand side"
        "lhs": "pin1",
        // Operator
        "op": "is",
        // "right hand side"
        "rhs": 1,
        // Main branch
        "_then": {
          "kind": "execute",
          "args": {
            // The `id` of the "Water all plants" sequence
            "sequence_id": 2
          }
        },
        // Else branch
        "_else": {
          "kind": "nothing",
          "args": {
          }
        }
      }
    },
    // "Wait" step: Pause for 2.3 seconds.
    {
      "kind": "wait",
      "args": {
        "milliseconds": 2300
      }
    }
  ]
}

History of RPCs in FarmBot

In the early days of FarmBot, we chose JSON RPC as a means of passing messages between systems. It accomplished the job, but suffered from some issues:

  • JSON RPC 1.0 used positional arguments, which were easily placed in the wrong order and difficult to teach to new developers. It did not lend itself well to strongly typed languages such as Typescript.
  • JSON RPC 2.0 was built with a server/client architecture in mind, which did not represent the relation between users, devices, and APIs well.
  • FarmBot features a visual sequence builder, which had its own data format to represent custom built sequences. Having a separate data format for stored sequences essentially created two RPC formats.
  • Sequences were far more complex than traditional RPCs. Sequences built within FarmBot required recursion and other features that JSON RPC was not intended to address.
  • As the number of commands increased, it became increasingly difficult to spot spelling errors and out-of-date messages between systems.
  • As the platform feature count increased, so too did the number of communication protocols between software components. Disparate and unrelated protocols increased code duplication and decreased overall maintainability.

Nodes

With exception to the REST API and some other edge cases, all communication that happens between FarmBot users, devices and systems is wrapped in a Celery Script node.

A node is a specially formatted JSON document. It is a composable building block that can be nested and arranged to create trees of commands, similar to the way an abstract syntax tree is used to create programming languages.

Here is an example of an install_farmware Celery Script node, which tells a device to download and install a 3rd party add-on:

{
    // THE "kind" ATTRIBUTE:
    // Every CS node MUST have a "kind" attribute.
    // This is the nodes name.
    // Legal "kind" types are listed here:
    // https://github.com/FarmBot/farmbot-js/blob/7044292a6591753b3edad3ed7bf3a8a08a41fac4/dist/corpus.d.ts#L418
    "kind": "install_farmware",

    // THE "args" (arguments) FIELD:
    // Every node has an "args" object. It is REQUIRED.
    // The arguments for a node will vary based on the "kind".
    "args": { url: "http://foo.bar/manifest.json" },
    
    // THE "comment" FIELD:
    // You do not need to use this (or even include it), but you can if it is helpful.
    // We use comments for tracing and debugging.
    "comment": "Optional. Useful when debugging, but ignored by the system.",

    // THE "body" FIELD:
    // Not every node will have a body.
    // If a node does have a body, it will ONLY contain other celery script nodes.
    // Example: The body of a "sequence" node may contain a "move_relative" node.
    body: []
}

At a minimum, every node has a kind and args attribute, which will vary based on the use case. It may also have a body and comment in some circumstances. No other attributes are allowed.

The types of nodes that are available, as well as the format of arguments is available within the corpus.d.ts file.

Notable CeleryScript Nodes

As noted above, a CeleryScript node is comprised of:

  • A kind (string) that determines a node's purpose
  • An args object that contains key/value pairs (some of these values will be child nodes, others will be simple primitive values such as numbers). If a node requires a particular arg, it will always be present. There is no such thing as an optional arg in CeleryScript. This simplifies many aspects of the standard.
  • An optional body argument which, if defined, will contain a list of other CeleryScript nodes. A body will only ever contain CeleryScript nodes and will never contain primitive values such as strings or numbers.
  • An optional comment (string) field to aid developer readability.

Some common CeleryScript nodes include:

  • move_relative, move_absolute
  • execute - Runs a sequence via the sequence_id arg.
  • rpc_request, rpc_ok, rpc_error - discussed later in this document.

For a full listing, see this auto-generated list of nodes.

Not every CeleryScript node represents a device command

Some nodes, such as the "coordinate" node, are used to represent data.

The "rpc_request" Node

The CeleryScript specification defines three nodes used for real-time control of a device. These nodes are used extensively in the User Interface for one-off commands, such as device position adjustments.

  • rpc_request - Initiated by a user (or occasionally, the REST API) when requesting the device do something. The desired action (eg: move_relative, install_farmware, etc..) is held in the body of this node.
  • rpc_ok - Sent by a device to an end user. Indicates that the request operation has succeeded.
  • rpc_error - Indicates that the request operation has failed. The body of an rpc_error will often contain a number of explanation nodes describing the circumstances of the failure.

Problem: Identify Success and Failure

Imagine writing a software package that sends a single rpc_request node once every five minutes. You could easily track the status of your RPC request by simply waiting for the next rpc_ok, rpc_error node from the device over MQTT.

How would we handle this situation if the software requirements changed? What if, instead of one request every five minutes, we are now required to send 100 requests in a one-minute timeframe? Identifying success or failure is no longer a trivial task. This is especially true for operations that result in partial failure, where only a portion of the requests succeed.

When using the HTTP protocol, the solution is simple- open a connection, start a request and then close the connection when finished. The situation is more complicated over MQTT, which is a persistent, session-based protocol where many requests are sent over the same TCP socket.

CeleryScript authors can solve this problem by tagging each outbound request with a unique "label" tag. You can think of this as a sort of request ID. We will investigate this arg in the next section.

Solution: Multiplex Requests via the "label" arg

FarmBot has the ability to send many commands over a single MQTT channel. Messages do not follow a call/response cycle seen with HTTP servers. Unlike HTTP, MQTT connections are persistent, full duplex connections. If you send three commands to a FarmBot, they might not come back in the order that they were received.

     | BROWSER  Request "ABC"             DEVICE
     |        ------------------------->         
     |          Request "DEF"                   
Time |        ------------------------->         
     |          Request "GHI"                   
     |        ------------------------->         
     |          Response to "DEF"                   
     |        <-------------------------         
     v          Response to "ABC"                   
              <-------------------------         
                Response to "GHI"                   
              <-------------------------
              

To make sense of which messages have been acknowledged, the rpc_request, rpc_ok and rpc_error nodes all contain a label argument. The label serves as a unique identifier for a message. Assigning unique IDs to each RPC message allows re-assembly of message order on arrival.

In the case of an rpc_error, it allows us to specify not only that an error has occurred, but also which command caused the error to occur.

Here is an example of a move_relative command sent to a device over MQTT:

// Message published to MQTT channel:
// '/bot/device_123/from_clients'
{
    kind: "rpc_request",
    // Any unique identifier can be used as a label.
    // UUIDs are highly recommended.
    args: { label: "adslkjhfalskdha" },
    body: [
      { kind: "move_relative", args: { x: 0, y: 0, z: 0, speed: 100 } }
    ]
}
// ... after some time the bot will respond on MQTT channel `/bot/device_123/from_device`

{
    kind: "rpc_ok",
    // The "label" here matches the "label" of the move_relative request (shown above).
    args: { label: "adslkjhfalskdha" }
}

CeleryScript Terminology Glossary

Primary Node: Sometimes referred to simply as "node". This is a JSON object with a kind, args and optional body key. It is the basic building block of CeleryScript structures.

Edge Node: Any item found in the CeleryScript document that is not a fully formed Primary Node. This includes values like strings, numbers, and booleans.

Corpus: A JSON document that describes the allowed format of every possible CeleryScript node. The most recent version is available at https://my.farm.bot/api/corpus

Canonical Representation: The traditional format of CeleryScript discussed in this document. The root of the JSON document is a primary node that has more primary nodes and edge nodes nested within.

Flat Intermediate Representation: A special format of CeleryScript where nodes are not nested and all information is stored in a single flat array. This is essential for storage of nodes in relational (SQL) databases and for easy execution of nodes on a device. As of June 2018, the format is not fully documented and is mostly used for internal tools.

kind: A string identifier used to distinguish between the different types of CeleryScript primary nodes.

args: A set of key value pairs found on the "args" property of a primary node. The key is a string. The value is either a primary node or an edge node. CeleryScript args are never optional.

body: A property found on primary nodes. It is always optional. The length is always flexible and is never fixed in size. If populated, it only contains primary nodes and never contains raw edge nodes.

comment: An optional string field on the root level of a primary node. It is used similarly to a comment in a traditional programming language. It is often removed during runtime. It is considered poor practice to store data in the comment field of a node.

uuid: An optional string field that is always stripped from a node prior to storage or execution. It is an implementation detail due to technical limitations and is not considered part of the specification.