🧠 AI Integration Layer

🎯 AI Philosophy

AI in FlowState should be optional, contextual, and friction-reducing. It's not about replacing the producer's creativity—it's about removing barriers and accelerating workflows.

🔀 Smart Inference Router

The router decides which model to use based on task complexity, latency requirements, and cost.

function routeInference(request: AIRequest): Provider {
  // TIER 1: Cloudflare Edge (realtime, simple)
  if (request.latency === 'realtime' && request.complexity === 'simple') {
    return 'cloudflare-workers-ai'; // FREE tier
  }

  // TIER 2: Free Models (background, creative)
  if (request.latency === 'background') {
    if (request.type === 'stem_separation') return 'htdemucs';
    if (request.type === 'music_gen') return 'stable-audio-open';
  }

  // TIER 3: Gemini (complex, audio understanding)
  if (request.audio_input || request.complexity === 'complex') {
    return 'gemini-3-flash';
  }

  return 'cloudflare-llama-4-scout'; // FREE, good default
}

📊 AI Capabilities Matrix (v0.1.99 - Implemented)

🚀 AI Studio Architecture (v0.1.99)

The AI Studio is built with a tiered architecture - FREE features run entirely in the browser, PRO features call cloud APIs:

FREE Tier (Magenta.js + Essentia.js)

• Pattern AI: Generate drums & melodies using MusicVAE, 100% client-side
• Audio Analysis: BPM, key detection, onset analysis using Essentia.js
• Morph Slider: Interpolate between patterns using VAE latent space
• Cost: $0/month - runs in user's browser

PRO Tier (Cloud APIs)

• ACE-Step: Full song generation with lyrics support (up to 4 min)
• Stable Audio Open: SFX generation (impacts, ambience, foley, up to 47s)
• OpenVoice v2: Voice cloning from 5-10s samples, 6 languages
• Chatterbox TTS: Expressive text-to-speech for scratch vocals
• Meta Demucs: Stem separation into vocals/drums/bass/other
• Cost: ~$0.02-0.10 per generation

⚡ Gemini 3 Flash Integration

Gemini 3 Flash (December 2025) is the core premium AI for complex tasks.

Key Capabilities

• Native Audio Input: Analyze beats, detect patterns without transcription
• 1M Token Context: Entire project history in one conversation
• Thinking Levels: MINIMAL → LOW → MEDIUM → HIGH (cost vs quality)
• Agentic Mode: Multi-step tasks with tool calling

When to Use Gemini

• ✅ "Analyze this beat and suggest improvements"
• ✅ "Help me write a hook for this track"
• ✅ Complex multi-turn conversations
• ✅ Audio understanding tasks
• ❌ Simple transport commands (use Llama instead)
• ❌ Basic text classification

Pricing

🎙️ Gemini 2.5 Flash Native Audio

For voice OUTPUT (avatar responses, assistant speech).

Features

• 30 HD voices in 24 languages
• Real-time streaming output
• Emotional expression
• Direct audio response (no TTS needed)

Cost Comparison

☁️ Cloudflare Workers AI Models

Available Models

Example: Whisper Integration

// workers/api/transcribe.ts
export async function transcribe(audio: ArrayBuffer, env: Env) {
  const result = await env.AI.run('@cf/openai/whisper', {
    audio: [...new Uint8Array(audio)]
  });

  return {
    text: result.text,
    language: result.detected_language,
    segments: result.segments
  };
}

🎤 Voice Command Pipeline

Supported Commands (MVP)

💬 AI Assistant Design

System Prompt Template

You are FlowState AI, a hip-hop production assistant.

Current project context:
- Tempo: ${project.tempo} BPM
- Key: ${project.key || 'not set'}
- Tracks: ${project.tracks.length}
- Current selection: ${selection}

You can help with:
- Production tips and techniques
- Sample recommendations
- Mixing advice
- Workflow optimization

Keep responses concise and actionable.
If the user asks you to do something in the DAW,
respond with a JSON action block.

Conversation Flow

1. User types or speaks query
2. Include current project context in system prompt
3. Route to appropriate model (Llama for simple, Gemini for complex)
4. Stream response to chat panel
5. If action needed, execute DAW command
6. Cache response in AI Gateway

💰 Cost Optimization Strategies

1. AI Gateway Caching

Cache common queries to reduce API calls by 40-70%.

// AI Gateway automatically caches based on:
// - Query similarity
// - TTL settings
// - Cache rules

// Configure in Cloudflare dashboard:
// AI Gateway > flowstate > Caching > Enable

2. Smart Model Routing

Use Llama 4 Scout (FREE) for 80% of queries, Gemini for 20%.

3. Context Caching

Gemini offers 75% discount on repeated system prompts. Cache project context.

4. Batch Embeddings

Embed samples in batches during off-peak hours.

Monthly Cost Estimate (10K users)