Why Upgrade from our earlier prototype?

Our robot ran reasonably well using the platform we hacked together in the summer of 2024... Or did it?

The demo event on July 15th did not go as planned—we had to cancel the robot’s appearance. This was due to a combination of Internet connectivity issues and unresolved software bugs. More concerningly, the system was difficult to analyze and debug afterward, leaving us without a clear understanding of what went wrong.

We have more demos ahead where the robot must perform reliably, under varying conditions and environments. Robustness, debuggability, and clear fault isolation are not optional—they’re essential.

Modularity was not Obvious

It was unclear how modularity worked. Parameters were somewhat global. Adding more nodes later would certainly have increased unclear modularity even further.

• Modularity now follows ROS2 packages and nodes more clearly and accurately.

Unclear Integration and Design

Due to time constraints, integration was mostly ad hoc. A better up-front design should allow people more time to focus and contribute.

• Now, using different repos provides clearer understanding and separation of concerns.
• mc_core only needs to change when the core hardware changes.
• mc_animate is the animation system. Once we have a better animation system, the entire package can be replaced without affecting the other packages.
• mc_ui and mc_monitor are Flutter projects, which are not part of the ROS2 infrastructure.
• (under construction: mc_speak development is currently waiting for Sergey's prototype)

Unfortunate Parameter Management

Parameters were described in one large yaml file without persistence. There were needless parameters like ROS topic names. Parameters did not adhere to ROS2 standards (i.e. were difficult to use in other ROS2 tools like RQt). Some parameters were not even ROS2 parameters. Also, an extra ROS1 dynamic reconfigure -style emulation layer was added to support a different parameter philosophy. There was a limited settings page to edit some parameters, but other parameters had to be edited in the yaml file. Editing this file proved to be error-prone and would sometimes cause the system not to boot.

• The parameters have been completely overhauled to be fully integrated in ROS2 parameter management.
• All declared parameters in the nodes can be edited from the Monitor, and are automatically persistent on a robot.

Unclear Logging

Some nodes would use ROS2 logging. Some nodes wrote to stdout. Some nodes did not produce any output. Also, it was never really investigated if ROS2 logging plays well with how we used Docker containers.

• All logging is now ROS2 logging.
• mc_core handles automatic logging of all active Docker containers.
• This log can be viewed from the Monitor in real-time.

Fragile Code and Unclear Testing

Some nodes would not crash, some nodes would only work in very specific environments. Some nodes had integration tests. Some nodes did not.

(- under construction: all nodes come with tests and are robustly designed for parameter changes, hotplugging of hardware, and graceful handling of exceptions)

400% CPU Usage

When software uses upwards of 90% CPU, it can start to display strange/unexplained behavior and delays. When this happens, it is wise to optimize the software and make sure it runs faster and uses less resources. This is especially important in production settings.

Unfortunately, running the prototype software completely filled up all cores of the Raspberry Pi5. Most of this was due to inefficient and unnecessary marshalling of big messages in Python and Dart, unnecessary use of extra nodes, missing upfront design, unfortunately placed containers and servers in the overall architecture, very heavy full-screen X11 + Firefox with Dart VM for extremely simple touchscreen UI and low-level servo command wrangling in Python.

• The core node has been rewritten in asynchronous Rust running on all CPU cores.
• Usage of the Feetech servo protocol is somewhat optimized in Rust.
• mc_core now combines what used to be the monitor server, servo manager, local sysinfo and eyes into one node.
• The UI Flutter project now builds into a bundle that can be run directly on the chest touchscreen, entirely without X11 and Firefox.
• Optimized messages sent between core and Monitor/UI, removing a lot of redundancy.

Non-deterministic Animation Problems

The animator had a few bugs where fingers would work properly sometimes, but not always. Also mixing the animations wasn't always clear, and the system would seem to forget animation keyframes here and there.

• The animator is simplified, runs all animations on separate channels, and allows for more flexible animation playing.

Fixed Servo Bus Assignment

The servo bus converters had to be connected to exactly addressed USB ports for the system to work.

• Servo management is now fully automatic, relying only on specifying the servo IDs in the jointsets.yaml file. The system automatically connects newly found FEE-URT-1 converters to the right bus.

Missing Documentation

Much time was lost figuring out how the system works, or people rerolling parts due to lack of clear design or documentation.

• You're reading the documentation right now.

Different Subsystems for TTS and ASR

under construction

We used the TTS server from HR SDK, which provided somewhat of a generalization over a few off-the-shelf TTS solutions, and the ASR solution(s) were made using a mostly different strategy. Also, the robot would sometimes block on ASR or TTS issues without any indication, which proved hard to analyze and solve. Viseme generation used a separate TTS service besides the HR SDK server, in hopes that the visemes would match somewhat the audio generated by HR SDK. Then, there was no solution for local offline TTS or ASR, making the robot completely reliant on cloud services. Regardless of which services to use, we had no way of reading metrics about the actual latency and performance of these services.

Also, heading into more elaborate demos, the audio+video pipeline will become a lot more complicated. After prototype experiments, this should be developed properly with upfront design, especially since this is one of our core strengths.

(due to continuing development on cancellation/interruption handling in the current prototype, a more grounded solution for the audio stack is postponed until the prototype is more stable)

• Both audio input and audio output is managed by the core.
• Audio input and output devices can be selected persistently from the Monitor.