Implementing a Local 3-Model On-Device ASR Pipeline Using Claude Code

A technical walkthrough of developing a fully offline, on-device Automatic Speech Recognition (ASR) and translation pipeline, leveraging Claude Code to accelerate the deployment of a multi-model architecture on mobile hardware.

The Challenge: Eliminating Cloud Dependency in Real-Time Translation

Conventional translation applications typically rely on a client-server architecture, streaming audio data to a remote cloud provider for processing. This introduces latency, requires consistent internet connectivity, and raises privacy concerns. To solve these issues, the author developed a system that executes speech recognition, translation, and Optical Character Recognition (OCR) entirely on-device.

The primary motivation for this project was to facilitate seamless, real-time communication between speakers of different languages (specifically Cantonese and Tagalog) without the friction of WiFi dependency or the clunky user experience associated with traditional cloud-based translation tools.

Technical Architecture: The 3-Model Pipeline

The system is built around a sophisticated on-device pipeline consisting of three primary models working in tandem to handle the end-to-end translation process:

  • Automatic Speech Recognition (ASR): Converts spoken audio into text locally.
  • Neural Machine Translation (NMT): Translates the recognized text into the target language.
  • OCR Integration: Provides visual text recognition to complement the audio-based translation.

By shifting the compute load from the cloud to the mobile device's NPU/GPU, the application achieves offline functionality and enables advanced features such as real-time Bluetooth conversation capabilities.

Development Acceleration with Claude Code

The development cycle was significantly compressed, allowing the shipment of the pipeline within two months. This was achieved through the use of Claude Code, which assisted in the rapid prototyping and implementation of the complex integration required to run multiple ML models on a mobile operating system.

Note: The provided source material is a partial excerpt. Detailed specifics regarding the exact model architectures (e.g., Whisper, MarianMT) and the specific mobile framework used for deployment were not included in the provided text.
Original Source
On-Device AI ASR Edge Computing Claude Code Machine Learning Deployment