SpeakFlow V2 — Architecture & Business Plan

Audio Input (AVAudioEngine)
    │
    ├── Noise Gate (RMS threshold, hysteresis)
    ├── High-Pass EQ (80Hz, removes rumble)
    └── Input Gain (-20 to +20 dB)
    │
    ▼
Speech Recognition
    │
    ├── PRIMARY: SFSpeechRecognizer (iOS 26 on-device, free, private)
    │   └── requiresOnDeviceRecognition = true
    │   └── addsPunctuation = true (iOS 17+)
    │
    ├── ENHANCED (V2): CoreML Whisper (distilled, Apple Silicon optimized)
    │   └── whisper-large-v3-turbo via coremltools
    │   └── Handles: accents, code jargon, whisper-level input
    │   └── Runs parallel, merges with SF results for confidence boost
    │
    └── FALLBACK: MLX Whisper (Mac only, for maximum accuracy)
        └── whisper-large-v3 via mlx-whisper
    │
    ▼
Post-Processing Pipeline
    │
    ├── Smart Formatting (existing SmartFormattingService)
    │   └── NLP punctuation, capitalization, number formatting
    │
    ├── Voice Commands (existing VoiceCommandService)
    │   └── 20+ commands: delete, undo, copy, new line, etc.
    │
    ├── Command Mode (NEW — V2)
    │   └── "Hey Flow" trigger → on-device LLM processes edit instruction
    │   └── "Make this formal" / "Translate to French" / "Summarize"
    │   └── MLX Gemma 3 4B on Mac, CoreML distilled on iOS
    │
    ├── Context Awareness (NEW — V2)
    │   └── App-name detection via Accessibility API (NOT screen capture)
    │   └── Tone adaptation: formal (Mail), casual (Messages), code (Xcode)
    │   └── No screen capture. No cloud. Just app bundle ID.
    │
    └── N'Ko Transliteration (existing)
        └── Latin ↔ N'Ko via IPA intermediary
    │
    ▼
Text Injection
    │
    ├── AX API (Accessibility, preferred)
    └── CGEvent Paste (fallback for sandboxed apps)

┌─────────────────────────────────────────────────┐
│                 SpeakFlowCore                    │
│         (Shared Swift Package / Framework)       │
│                                                  │
│  SpeechService          AudioProcessingService   │
│  SmartFormattingService  VoiceCommandService      │
│  CommandModeService(NEW) ContextAwarenessService  │
│  NKoTransliteration     UserDictionaryService    │
│  VoiceSnippetService     PerformanceProfiler     │
│  WhisperCoreMLService(NEW)                       │
└──────────┬──────────────────┬────────────────────┘
           │                  │
    ┌──────┴──────┐    ┌──────┴──────┐
    │   macOS     │    │    iOS      │
    │  Menu Bar   │    │  Keyboard   │
    │   App       │    │  Extension  │
    │             │    │  + Host App │
    │ HotKeyMgr   │    │             │
    │ TextInject   │    │ KeyboardVC  │
    │ MLX Engine  │    │ CoreML Only │
    │ RecordPill  │    │ ListenOverlay│
    └─────────────┘    └─────────────┘

class CommandModeService {
    // Trigger: user holds hotkey a second time after dictating
    // Or says "Hey Flow" followed by an instruction

    func processCommand(_ instruction: String, selectedText: String) async -> String
    // On Mac: MLX Gemma 3 4B (fast, on-device)
    // On iOS: CoreML distilled model (3B or smaller)
    // Fallback: mesh route to Mac4/Mac5 via Tailscale

    // Built-in commands (no LLM needed):
    // - "make shorter" → extractive summary
    // - "fix grammar" → LanguageTool/NLP
    // - "translate to [lang]" → Apple Translation framework (iOS 18+)
    // - "make formal/casual" → template-based tone shift

    // LLM commands (needs model):
    // - "rewrite as bullet points"
    // - "explain this simply"
    // - Custom instructions
}

class WhisperCoreMLService {
    // CoreML-converted whisper-large-v3-turbo
    // Runs alongside SFSpeechRecognizer
    // Merges results: takes higher-confidence segments from each
    // ~300ms latency on M1+, ~500ms on A16+

    func transcribe(_ audioBuffer: AVAudioPCMBuffer) async -> TranscriptionResult
}

class ContextAwarenessService {
    // Detects active app via NSWorkspace (Mac) / UIApplication (iOS)
    // Maps bundle IDs to tone profiles:
    //   com.apple.mail → formal
    //   com.tinyspeck.slackmacgap → casual
    //   com.microsoft.VSCode → code (variable naming, no punctuation in identifiers)
    //   com.apple.dt.Xcode → code
    //   com.apple.MobileSMS → casual, short

    // NO screen capture. NO accessibility text reading.
    // Just bundle ID → tone mapping. Simple, private, effective.

    func currentContext() -> DictationContext
}

SpeakFlow App (on-device, primary)
    │
    ├── Normal dictation: 100% on-device, zero latency
    │
    └── Command Mode (complex instructions):
        ├── Try on-device MLX/CoreML first
        └── If device is phone/low-power:
            └── Route to mesh via Tailscale
                ├── Mac4: Ollama (large models)
                ├── Mac5: MLX Server (cognitive twin fine-tuned)
                └── exo cluster: distributed inference

On-Device Storage (Core Data / SwiftData)
    ├── Transcription history (searchable, local)
    ├── Custom vocabulary (synced via iCloud/CloudKit)
    ├── Voice snippets (synced via iCloud)
    ├── Usage analytics (local only, never uploaded)
    └── Performance metrics (local profiler data)

Sync (Optional, User-Controlled)
    ├── CloudKit: vocabulary, snippets, settings across devices
    ├── App Group: keyboard ↔ host app (existing)
    └── NO cloud transcription storage
    └── NO telemetry without explicit opt-in