iLab Neuromorphic Robotics Toolkit  0.1
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
What is NRT?

the Neuromorphic Robotics Toolkit

At its heart, the Neuromorphic Robotics Toolkit is a C++ library and a set of tools to help you write high-performance modular software. NRT provides a framework in which developers can design lightweight, interchangeable processing modules which communicate by passing messages. NRT also provides libraries for performing various robotics related processing such as Image Processing, PointCloud Processing, etc.


NRT makes it easy for developers to create such modules which can be dynamically loaded and connected together at runtime. Modules can run in multiple threads of a same process. In this case, all message passing is done through shared pointers for maximum efficiency (no copy of message data as it is passed from one module to another). Modules can also run on different computers on a network. In this case, their messages are transparently serialized (a.k.a. marshalled) across network connections for maximum flexibility. For the user, or even the programmer of a module, it makes no difference whether the messages the module receives or sends to other modules are passed by local pointer or over the network.

NRT provides powerful facilities which make its modular message-based programming model intuitive and flexible, including the ability to split composite messages into parts, or the ability to write highly generic modules that can handle any message type.

Core features

Modular programming is a technique in which a software system is composed of self-contained, interchangeable components with defined interfaces. Because each module is self-contained, it can be tested independently from the rest of the system. Furthermore, having well-defined interfaces between modules allows researchers to effectively communicate their components' requirements and capabilities to the rest of the team.

After reviewing several existing modular programming frameworks, we determined the following set of requirements:

  1. Transmission of arbitrary C++ data types between modules. You are not limited to a small list of simple types like in many other message-passing frameworks, which often force you to convert your native complex data structures into their simple types.
  2. Zero-copy transmission of messages when modules are instantiated on the same machine.
  3. Transparent serialization (marshalling) of messages over the network when modules are instantiated on different machines, with no difference to a user or a programmer whether a message is transmitted locally or across a network (apart, of course, from transmission time).
  4. Statically (compile-time) defined module interfaces to avoid the sprawling of error-prone, string-based naming of module ports or publish/subscribe topics that plague most existing message passing frameworks.
  5. Enhanced ability to re-use modules by allowing (1) generic messages concepts and generic modules that can manipulate any message type (e.g., message queues) and (2) splittable messages such that complex composite messages can be broken down into elementary parts, which may then be processed independently before being re-assembled.

The Neuromorphic Robotics Toolkit is our solution to this list of requirements. Beyond these, it also provides a graphical tool for managing and creating networks of connected modules, and a suite of administration tools for starting, configuring, and stopping systems.

Programming model

The core inspiration for NRT is to consider the brain as an agency of concurrent specialized knowledge sources, whereby multiple such sources work together towards solving a complex problem. This vision is not new, and has been implemented in many early artificial intelligence systems, including VISIONS (Hanson & Riseman, 1978), HEARSAY-II (Erman et al., 1980), and Copycat (Hofstadter & Mitchell, 1995).

In this computing paradigm, a common knowledge base, the blackboard, is iteratively updated:

  • beginning with a problem definition and prior knowledge;
  • each knowledge source (i.e., software agent or module) is triggered as its pre-requisites for processing are met by the current contents of the blackboard (i.e., appropriate messages are posted);
  • working in parallel, each specialist knowledge source may update the blackboard with a partial solution, suggestions, or additional information, allowing other knowledge sources to trigger;
  • Over time, the specialists hence work together towards a solution.

Other groups have also explored schema-based and blackboard architectures in robotics (e.g., Arkin, 1989; Brooks, 1991; Xu & van Brussel, 1997) as well as in cognitive science (e.g., van der Velde & de Kamps, 2006). Here, with NRT we developed the necessary tools to bring these kinds of architectures to a new level of user friendliness through the use of highly modular, transparently distributed, graphical-oriented programming.

Novice users can assemble complex systems running on collections of computers by simply dragging and connecting software modules (the blackboard agents), where the modules interact via a distributed Blackboard Federation system behind the scenes (connections represent posting/subscribing of/to messages). Similar graphical programming metaphors have proven particularly effective in previous systems, including Khoros/Cantata, AVS (Advanced Visual Systems Inc.), Lego Mindstorms, Flowstone, and many others. While these previous frameworks have often been limited to single-CPU systems and simpler point-to-point information flow between modules, we have used our blackboard-based NRT framework on machines ranging from clusters with 288 CPU cores + 2048 GPU cores, to a single laptop, or even a 10-Watt netbook (Intel Atom).

A number of new features allow NRT to eliminate some of the major caveats of similar previous systems, described throughout this documentation.


Initial development of NRT started under the DARPA Neovision2 project. NRT was developed to allow us to build a neuro-inspired machine vision system that can attend to, track, and recognize up to 10 object categories (cars, trucks, boats, people, etc) in high-definition aerial video.

The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressly or implied, of the Defense Advanced Research Projects Agency or the U.S. Government.

Above: Flowchart of how different visual processes may give rise to visual scene understanding.
Above: Corresponding implementation as a large NRT system.
Above: Sample system output. Cars are marked in blue, people in yellow, cyclists in pink.