Apollo Icon

Apollo: AI Reasoning Engine

Apollo is a separate hardware system dedicated to AI and large language model processing. Running as an independent machine with dedicated GPU resources, Apollo provides natural language understanding, intelligent reasoning, and LLM orchestration capabilities to HALO. Rather than running as a container on Nexus, Apollo operates as a standalone server that exposes APIs accessible from Nexus services over the local network.

Apollo transforms HALO from a collection of automated systems into a truly intelligent home that understands, anticipates, and assists. Through tool calling to Home Assistant, Frigate, and n8n workflows, Apollo reasons about requests in context and generates responses that feel natural and helpful, acting as the bridge between human intention and technical execution.

Purpose and Scope

Apollo’s mission is to make interaction with HALO feel effortless and intelligent. It provides the reasoning layer that enables users to express their needs in natural language rather than learning specific commands or navigating through configuration screens. Apollo understands context from conversation history, current home state, and learned patterns to provide assistance that feels anticipatory rather than reactive.

Hardware Architecture: Apollo runs on dedicated hardware separate from Nexus, providing GPU acceleration for local LLM inference and sufficient resources to run multiple model types simultaneously. This separation ensures AI workloads don’t compete with critical home automation services for CPU and memory resources.

Network Integration: Apollo exposes REST APIs over the local network that Nexus services can call. Omnia widgets, n8n workflows, and other HALO components interact with Apollo as an external service rather than a co-located container.

Apollo’s capabilities extend beyond simple question-answering. It can orchestrate multi-step workflows, monitor system state for anomalies, suggest automation improvements based on observed patterns, and even explain why certain automations triggered or failed.

Architecture and Design

Apollo is built as a Python-based service with modular components that handle different aspects of AI reasoning:

Model Manager: Routes requests to appropriate language models based on task requirements, model availability, and configuration preferences. The manager handles model selection (local vs cloud), context window management, and fallback logic when preferred models are unavailable.

Context Tracker: Maintains conversation history and session state across multiple turns. The tracker can store messages in a local database, manages context window limits, and provides relevant historical context to the model for coherent multi-turn conversations.

Tool Orchestrator: Manages the catalog of available tools and handles tool calling workflows. When Apollo decides to call a tool (like querying Home Assistant or triggering an n8n workflow), the orchestrator validates parameters, executes the call over the network to Nexus services, and integrates results back into the conversation.

Memory System: Stores long-term context like user preferences, household patterns, and learned behaviors. Unlike conversation history (which focuses on recent exchanges), memory captures persistent knowledge that should inform all future interactions.

API Gateway: Exposes a REST API that other HALO components use to interact with Apollo. The gateway handles authentication, rate limiting, and request routing to appropriate internal components.

Model Support Strategy

Apollo supports multiple model backends to balance capability, cost, and privacy:

Local Models (Primary): Ollama running GGUF format models on the HALO server. Current focus is on models optimized for tool calling and reasoning (Llama 3, Mistral, Mixtral). These models provide good performance for most home automation tasks while maintaining complete privacy.

Cloud APIs (Optional): Integration with OpenAI, Anthropic Claude, and Google Gemini for tasks requiring cutting-edge reasoning capabilities. Cloud models are used only when explicitly configured and when the task justifies the privacy trade-off (like complex planning or creative generation).

Fallback Logic: If the preferred model is unavailable, Apollo automatically falls back to alternative models. For example: local Llama 3 → local Mistral → cloud GPT-4 → simplified rule-based responses.

This multi-tier approach ensures Apollo remains functional even when specific models are offline or overloaded.

Integration with HALO Systems

Home Assistant Integration

Apollo can query and control all Home Assistant entities through tool calling over the network:

Device State Queries: “What’s the temperature upstairs?” triggers a network call to Home Assistant’s REST API running on Nexus, retrieves sensor.upstairs_temperature, and returns the current reading with appropriate context.

Device Control: “Turn off all the lights” translates to a service call to Home Assistant’s API with entity group light.all_lights, then confirms the action was successful.

Scene Activation: “Set the house to bedtime mode” recognizes the intent, activates the scene.bedtime scene via API call, and can explain what changes were made.

Automation Queries: “Why did the porch light turn on?” can examine automation history through API queries, identify the trigger, and explain the logic in natural language.

Frigate Integration

Security-related queries leverage Frigate’s computer vision capabilities through network API calls:

Event Queries: “Any motion detected today?” makes HTTP requests to Frigate’s REST API on Nexus, searches the event database for detection events, groups them by camera and object type, and presents a summary.

Object Detection History: “Did any cars enter the driveway?” filters events by object type and zone through API queries, returning specific results with timestamps and confidence scores.

Recording Access: “Show me the front door camera” queries the Frigate API for the appropriate stream URL or latest snapshot for display in Omnia.

Pattern Analysis: “Is there unusual activity today?” compares today’s detection counts against historical averages retrieved from the Frigate API and flags anomalies.

n8n Workflow Integration

Apollo can trigger workflows via natural language by calling n8n webhook endpoints on Nexus:

Workflow Triggers: “Add milk to the grocery list” identifies the appropriate n8n workflow webhook URL, constructs the required payload, sends the HTTP request to Nexus, and confirms success.

Status Queries: “Is the morning routine running?” checks workflow execution state through n8n API calls on Nexus and provides current status.

Workflow Suggestions: Based on observed patterns, Apollo can suggest new workflows: “I notice you turn on the coffee maker every weekday at 6:30 AM. Would you like me to create an automation for this?”

Coordination: Multi-step requests like “Plan a birthday party” trigger coordinated workflow sequences through multiple API calls to n8n on Nexus.

Omnia Integration

Omnia widgets provide the primary user interface for Apollo by making API calls to Apollo’s REST endpoints:

Chat Widget: The main conversational interface where users can ask questions, issue commands, and receive assistance by sending requests to Apollo’s API.

Widget Recommendations: Apollo can suggest useful widgets based on usage patterns accessed through the Omnia API on Nexus.

Contextual Assistance: Widgets can send context-specific prompts to Apollo (like “Help me configure this automation”) that include relevant metadata for targeted assistance.

Dashboard Content Generation: Apollo can generate widget content like daily summaries and routine suggestions that are consumed by Omnia (API) and displayed in Omnia (Screen).

Core Capabilities

Natural Language Understanding

Apollo processes conversational input to extract intent and entities:

Intent Recognition: Identifies what the user wants to accomplish (control devices, get information, trigger automation, etc.)

Entity Extraction: Recognizes specific references to devices (“kitchen lights”), locations (“upstairs”), times (“at sunset”), and other relevant parameters

Context Awareness: Maintains conversation flow across multiple turns, understanding pronouns and implicit references (“turn them off” after discussing lights)

Clarification Handling: Asks follow-up questions when intent is ambiguous (“Did you mean the front yard lights or the backyard lights?”)

Tool Calling and Execution

When Apollo determines an action is needed, it selects and calls appropriate tools:

Tool Selection: Chooses the right tool from available options based on intent (Home Assistant for device control, Frigate for security queries, n8n for complex workflows)

Parameter Construction: Builds properly formatted tool calls with validated parameters extracted from natural language

Error Handling: Gracefully handles tool failures, attempting retries or alternative approaches when possible

Result Integration: Incorporates tool results back into the conversation naturally (“I’ve turned off the lights. Would you also like to lock the doors?”)

Content Generation

Beyond simple responses, Apollo generates useful content:

Summarization: Condenses lengthy information (automation logs, event histories) into digestible summaries

Suggestions: Recommends routines, scenes, or automation improvements based on learned patterns

Planning: Helps plan multi-step activities (trips, events, projects) by generating tasks, schedules, and resource lists

Explanations: Describes complex automation logic or system behavior in understandable terms

Proactive Assistance

Apollo can offer help without being explicitly asked:

Pattern Learning: Identifies recurring behaviors (“You usually turn on the porch light around sunset”) and suggests automations

Anomaly Detection: Alerts on unusual patterns (“The garage door has been open for 3 hours, which is unusual”)

Optimization Recommendations: Suggests efficiency improvements (“I notice your HVAC runs inefficiently on weekends. Would you like to adjust the schedule?”)

Contextual Reminders: Provides notifications based on location, time, and learned preferences (“You typically water the plants on Sunday mornings. Would you like a reminder?”)

Privacy and Security

Data Handling Philosophy

Apollo is designed with privacy as a core principle:

Local-First Processing: Primary operation uses local Ollama models where conversations and reasoning happen entirely on the HALO server

Sensitive Data Protection: Personally identifiable information (names, addresses, specific times) is filtered before sending requests to cloud APIs

User Control: Explicit configuration required to enable cloud API usage—never enabled by default

Encrypted Storage: Conversation history and user preferences stored in PostgreSQL with encryption at rest

Data Retention: Configurable limits on conversation history storage with automatic cleanup of old data

Access Control

Security measures ensure only authorized access to Apollo:

API Authentication: All API calls require authentication tokens issued by the Nexus infrastructure

Role-Based Permissions: Different user profiles have different capabilities (admin can modify automations, guests can only query status)

Audit Logging: All AI-assisted actions logged with timestamp, user, and action taken for accountability

Tool Restrictions: Certain high-impact tools (like door locks or security system disarming) require additional confirmation

Privacy Modes

Apollo supports different privacy configurations:

Maximum Privacy: Local models only, no cloud APIs, minimal logging

Balanced: Local models primary, cloud for complex tasks with PII filtering

Enhanced Capability: Cloud models allowed for all tasks (user explicitly accepts privacy trade-off)

Directory Structure

Apollo source code lives in the HALO monorepo but runs on separate hardware:

halo/
└── services/apollo/
    ├── src/
    │   ├── api/              # REST API endpoints
    │   ├── models/           # LLM interaction and routing
    │   ├── tools/            # Tool implementations
    │   │   ├── home_assistant.py  # Calls HA API on Nexus
    │   │   ├── frigate.py         # Calls Frigate API on Nexus
    │   │   ├── n8n.py             # Calls n8n webhooks on Nexus
    │   │   └── system.py
    │   ├── context/          # Memory and conversation tracking
    │   ├── config/           # Configuration management
    │   └── utils/            # Helper functions
    ├── tests/                # Unit and integration tests
    ├── requirements.txt      # Python dependencies
    └── README.md             # Apollo documentation

Deployment and Configuration

Apollo runs on dedicated hardware separate from Nexus:

Hardware Requirements:

Dedicated GPU (NVIDIA RTX 3060 or better recommended)
16GB+ RAM for local model inference
Ubuntu Server or similar Linux distribution
Network connectivity to Nexus services

Deployment:

# On Apollo hardware (not Nexus)
cd services/apollo
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python src/main.py

Environment Configuration

Apollo requires environment variables for:

Model Endpoints: Local Ollama URL, cloud API keys (if enabled)
Nexus Service URLs: Home Assistant URL/token, Frigate URL, n8n webhook URLs
Database Connection: Optional PostgreSQL on Nexus for conversation history
Feature Flags: Privacy mode, logging level, tool restrictions

Network Access

Apollo tools make HTTP requests to Nexus services:

# Example tool configuration
HOME_ASSISTANT_URL = "https://halo.local/ha"  # Through Traefik on Nexus
FRIGATE_URL = "https://halo.local/frigate"    # Through Traefik on Nexus
N8N_WEBHOOK_BASE = "https://halo.local/n8n/webhook"  # Through Traefik

Roadmap and Future Capabilities

Voice Integration

Wake Word Detection: Hands-free activation with local wake word processing

Speech-to-Text: Real-time transcription using Whisper or similar models

Text-to-Speech: Natural voice responses using Piper or cloud TTS services

Speaker Integration: Output through Home Assistant TTS or smart speakers

Advanced Reasoning

Multi-Step Planning: Autonomous execution of complex tasks requiring multiple tool calls and decision points

Autonomous Routine Creation: Observing patterns and creating automations without explicit instruction

Conflict Resolution: Identifying and resolving conflicts between automation rules or competing user preferences

Scenario Simulation: “What if” analysis of proposed automations before implementation

Learning and Adaptation

Fine-Tuning: Personalized model adaptation based on household-specific patterns and terminology

Preference Learning: Adapting responses and suggestions based on user feedback and acceptance rates

Success Metrics: Tracking which suggestions are accepted and using this to improve future recommendations

Cross-Household Knowledge: Privacy-preserved aggregation of insights from multiple HALO installations (opt-in)

Enhanced Tool Capabilities

Direct Database Queries: SQL generation for complex analytics across HALO data

External Service Integration: Weather, news, traffic, calendar services

Multi-Service Workflows: Coordinating complex operations (“Plan a dinner party” touches calendar, groceries, lighting, music)

Rollback Capabilities: Undoing AI-initiated actions when results aren’t as expected

Example Interactions

Simple Device Control:

User: "Make it darker in here"
Apollo: I'll dim the lights in the living room.
        [Calls Home Assistant: light.turn_on(entity_id="light.living_room", brightness_pct=30)]

Security Query:

User: "Did anyone come to the door today?"
Apollo: Yes, there were 3 motion events at the front door:
        • 9:23 AM - Mail delivery (person, 94% confidence)
        • 2:15 PM - Package delivery (person + truck, 98% confidence)
        • 5:47 PM - Neighbor walking by (person, 89% confidence)
        [Queries Frigate: GET /api/events?camera=front_door&label=person&after=today]

Multi-Step Planning:

User: "We're having guests over Friday night"
Apollo: Great! I can help prepare. Would you like me to:
        • Add a calendar event for Friday evening
        • Create a cleaning checklist for the guest areas
        • Set up a guest lighting scene
        • Temporarily adjust security camera zones
        Which of these would be helpful?
        [Prepared to call: n8n calendar workflow, task workflow, HA scene creation, Frigate config update]

Proactive Suggestion:

Apollo: I've noticed you manually turn on the porch light around sunset
        every day. Would you like me to create an automation for this?
        [Observed pattern: light.porch turned on within 15 minutes of sunset for 14 consecutive days]

For configuration details, see Apollo Configuration.

For integration with other systems, see Home Assistant, Frigate, and n8n Workflows.

For deployment infrastructure, see Nexus Hardware.