Data Mining & Annotation Tooling: Modular, Front-end Architecture
When you combine three dimensions of fused spatial data, one dimension of time samples, and multiple sets of abstract objects and temporal events, you realize you're visualizing at least four dimensions of information simultaneously. This is the fundamental complexity our data annotation platform is designed to manage.
At Motional, our data annotation platform is where raw sensor data becomes structured intelligence. It’s a web application where users manipulate gigabytes of 3D point clouds, high-resolution video, and complex sensor data on a minute-to-minute basis around the clock. Building a front-end for this is a monumental challenge. It must be incredibly performant, highly scalable to accommodate dozens of new workflows, and robust enough to be the foundation of our entire data engine. It’s critical to scaling our ML pipelines.
Standard application patterns weren't enough. We had to innovate with a custom solution. There are four key pillars of our front-end architecture:
- The Scalable Shell: Data contract driven, modular architecture for supporting infinite workflows
- The High-Performance Engine: Custom "Action-driven Atomic State" model that delivers a fluid, non-blocking user experience
- The Maintainable Codebase: Each of our architectural decisions make our technical product more maintainable and reusable
- Task & Workforce Management: The technical components that support factors for an effective and well-managed labeling workforce
Why Autonomous Vehicle Annotation Tooling is a 4-Dimensional Data Science Problem
To the outside world, an autonomous vehicle operates smoothly through its environment. To our systems, every second of that journey is a torrent of high-dimensional data. Before we could build a single user interface (UI) component, we had to first model this complexity. Our analysis led us to a foundational truth: effective autonomous vehicle (AV) annotation is not a simple 2-dimensional labeling task, but a multi-layered, more than four-dimensional data science problem.
The Scalable Shell — A Foundation for Infinite Workflows
The Problem: How do you support dozens of unique annotation tasks—3D cuboid labeling, 2D semantic segmentation, event-based video tagging—without building and maintaining dozens of separate applications?
Our Solution: A universal application shell built on a scalable architecture
Instead of creating siloed applications, we built a single, modular shell that provides a consistent layout, a core set of services (like telemetry and common hotkey management), and a main content area. This shell is, in essence, a highly configurable host for a variety of data renderers and tools serving all of our 2D, 3D, temporal, data-frame, and audio-based workflows.
The modular magic that makes this work is that the entire application state is driven by composable data contracts shared by all sensor annotation task types.
A common URL contract integrates deeply with the application shell, enabling the same drive-log data extractions to be loaded into different types of annotation tools for ground truth data annotation and fusion. Our backend data service APIs seamlessly integrate with the UI across all types of labeling and data mining workflows – URL contract parameters drive API integrations and help to synchronize visual application state while allowing easily share-able data views
The High-Performance Engine — Atomic State for Flawless Interaction
The Problem: A user is rotating a 3D point cloud with 10 million points while editing a three-dimensional bounding box, with persistent updates that need to be saved to the server. How do you do this in a web browser without dropping frames or blocking the UI?
Our Solution: Rather than use standard state management patterns, we built our own: action-driven atomic state
The action-driven pattern in particular gives us qualities like recordable-replayable state transactions and ability to implement chain of custody for data mutations:
This is reminiscent of the canonical Flux/Redux state management pattern, but off-the-shelf solutions and document-based state management were not a good fit on account of developer experience and computational performance limitations.
Our model is inspired by libraries including Jotai or Recoil but with a crucial layer of control, because atomic state is endlessly flexible with minimum-possible cognitive overhead, but lacks any opinionated pattern for controlling state changes in the context of user actions and external effects – as in the diagram below, we see that out-of-the-box atomic state management directly mutates individual state atoms without any choreography:
In order to solve this, the application state is still broken down into (built up from) independent, "atomic" pieces. However, mutations are not performed directly. They are orchestrated through a central, non-blocking process that employs state transactions and effectful business layer integrations.
We call it the choreography of an action:
- Intent & Lock: A user action function is executed. The data editing system immediately enters a "locked" state, preventing conflicting user actions while allowing full application interactivity.
- Optimistic UI Update: We immediately update the relevant atomic state. The user sees their change instantly. This is the key to a responsive feel.
- Asynchronous API Call: In the background, a fetch request is sent to the backend. This happens completely asynchronously and never blocks the main thread.
- Record & Resolve: The action and its counter-action (the "undo" function) are pushed onto a history stack. Upon successful API response, the lock is released. If it fails, the state is gracefully rolled back.
This entire complex flow is encapsulated into a simple reusable higher-order-function pattern (as in a reusable React hook that we call usePerformAndRecordActionCallback), making it easy and intuitive for our feature developers to use this powerful pattern correctly and safely.
When we step back and look at our application architecture, what we see is strikingly similar to the reducer pattern used in libraries like Redux, with the added benefit of best-in-class performance, maintainability, developer experience, and code cleanliness.
Our solution to the 'four-dimensional' challenge of AV annotation combines a universal, modular data contract-driven shell with an action-driven atomic state management system. This allows us to scale across dozens of unique labeling tasks without code duplication, while delivering the high-performance, optimistic UI updates necessary for efficient minute-to-minute data manipulation.
Read the next part in this series: High performance web-based visualization.
