# 𓁟 Thoth — Architecture & Detailed Design > Full technical reference for every feature, module, and subsystem in Thoth. > For a concise overview, see the [README](../README.md). --- ## Table of Contents - [ReAct Agent Architecture](#react-agent-architecture) - [Long-Term Memory & Knowledge Graph](#long-term-memory--knowledge-graph) - [Wiki Vault](#wiki-vault) - [Dream Cycle](#dream-cycle) - [Document Knowledge Extraction](#document-knowledge-extraction) - [Brain Model & Providers](#brain-model--providers) - [Voice Input & Text-to-Speech](#voice-input--text-to-speech) - [Shell Access](#shell-access) - [Browser Automation](#browser-automation) - [Vision](#vision) - [Workflows & Scheduling](#workflows--scheduling) - [Designer Studio](#designer-studio) - [Thoth Status & Identity](#thoth-status--identity) - [Self-Knowledge & Insights](#self-knowledge--insights) - [Messaging Channels](#messaging-channels) - [Tunnel Manager](#tunnel-manager) - [X (Twitter) Tool](#x-twitter-tool) - [Tool Guides](#tool-guides) - [Image Generation](#image-generation) - [Video Generation](#video-generation) - [MCP Client & External Tools](#mcp-client--external-tools) - [Migration Wizard](#migration-wizard) - [Plugin System & Marketplace](#plugin-system--marketplace) - [Auto-Updates](#auto-updates) - [Habit & Health Tracker](#habit--health-tracker) - [Desktop App](#desktop-app) - [Chat & Conversations](#chat--conversations) - [Notifications](#notifications) - [Bundled Skills](#bundled-skills) - [Core Modules](#core-modules) - [Data Storage](#data-storage) - [Comparison with Other Tools](#comparison-with-other-tools) --- ## ReAct Agent Architecture - **Autonomous tool use** — the agent decides which tools to call, when, and how many times, based on your question - **30 core tools plus auto-generated channel tools** — web search, email, calendar, file management, shell access, browser automation, vision, image generation, video generation, X (Twitter), a personal knowledge graph, Designer Studio, scheduled workflows, habit tracking, Thoth Status self-inspection, external MCP tools, and more - **Streaming responses** — tokens stream in real-time with a typing indicator - **Thinking indicators** — shows when the model is reasoning before responding - **Smart context management** — automatic conversation summarization compresses older turns when token usage exceeds 80% of the context window, preserving the 5 most recent turns and a running summary; a hard trim at 85% drops oldest messages as a safety net; oversized tool outputs are proportionally shrunk so multi-tool chains fit within context; accurate token counting via tiktoken (cl100k_base) - **Dynamic tool budgets** — the agent automatically adjusts how many tools are exposed to the model based on available context headroom; when context usage is high, lower-priority tools are temporarily hidden to prevent the system prompt from crowding out conversation history - **Centralized prompts plus self-knowledge injection** — base prompt templates live in `prompts.py`, while `self_knowledge.py` injects a dynamic identity line, capability manifest, and live runtime state so Thoth can describe itself accurately without stale hard-coded copy - **Live token counter** — progress bar in the sidebar shows real-time context window usage based on trimmed (model-visible) history - **Graceful stop & error recovery** — stop button cleanly halts generation with drain timeout; agent tool loops are caught automatically (50-step limit for chat, 100 for workflows) with a wind-down warning at 75%; orphaned tool calls are repaired; API errors are surfaced as persistent red toasts and saved to the conversation checkpoint so they survive thread refresh - **Workflow cancellation** — running background workflows can be stopped from the chat header, activity panel, or workflow card; cancellation is checked between every LangGraph node for clean shutdown - **Displaced tool-call auto-repair** — if context trimming displaces tool-call/response pairs, the agent automatically detects and repairs the ordering before the next LLM call; orphaned tool calls trigger an automatic retry - **Date/time awareness** — current date and time is injected into every LLM call so the model always knows "today" - **Destructive action confirmation** — dangerous operations (file deletion, sending emails, deleting calendar events, deleting memories, deleting workflows, selected settings changes) require explicit user approval via an interrupt mechanism - **Workflow-scoped background permissions** — background workflows use a tiered system: safe operations always run, low-risk operations (move file, move calendar, send email) are allowed with optional runtime guards, and irreversible operations (delete file, delete memory) are always blocked; shell commands and email recipients can be allowlisted per-workflow via the editor UI --- ## Long-Term Memory & Knowledge Graph Thoth doesn't just store isolated facts — it builds a **personal knowledge graph**: a connected web of people, places, preferences, events, and their relationships. Every memory is an entity linked to others through typed relations, so the agent can reason about how things in your life connect. - **Entity-relation model** — memories are stored as entities with a type, subject, description, aliases, and tags; entities are connected by typed directional relations (e.g. `Dad --[father_of]--> User`, `User --[lives_in]--> London`) - **10 entity types** — `person`, `preference`, `fact`, `event`, `place`, `project`, `organisation`, `concept`, `skill`, `media` - **Memory tool** — 7 sub-tools let the agent save, search, list, update, delete, **link**, and **explore** memories through natural conversation - **Link memories** — the agent can create relationships between any two entities, building a richer graph over time - **Explore connections** — the agent can traverse the graph outward from any entity, discovering chains of relationships for broad questions like family, work, and projects - **Interactive memory visualization** — a dedicated **Knowledge** surface renders the entire knowledge graph as an interactive network diagram with search, filters, full-graph / ego-graph toggle, and detail cards - **Graph-enhanced auto-recall** — before every response, the agent retrieves semantically relevant entities via FAISS and then expands one hop in the graph to surface connected neighbors; recalled memories include their relationship context - **Automatic memory extraction** — a background process scans past conversations on startup and every 6 hours, extracting entities and relations the agent missed during live conversation; active threads and workflow threads are excluded; assistant messages are truncated to 200 chars to prevent extracting from AI-generated content; an 0.80 confidence floor rejects low-confidence entities - **Deterministic deduplication** — both live saves and background extraction check for existing entities by normalized subject before creating new entries; cross-category matching prevents fragmentation; alias resolution ensures related names merge; richer content is always kept - **Vague-type banning** — `related_to`, `associated_with`, `connected_to`, `linked_to`, `has_relation`, `involves`, and `correlates_with` are rejected before saving, preventing noisy low-value edges - **Relation pre-normalization** — alias forms are canonicalized before ban, confidence, and dedup checks - **67 valid relation types** — curated vocabulary with 60+ alias mappings plus document-specific relations like `extracted_from`, `uploaded`, `builds_on`, `cites`, `extends`, and `contradicts` - **Source tracking** — each entity is tagged with its origin (`live`, `extraction`, `dream_*`, or document-derived) for diagnostics - **Semantic recall** — FAISS vector index with Qwen3-Embedding-0.6B for similarity-based memory retrieval - **Memory IDs in context** — auto-recalled memories include their IDs so the agent can update or delete specific entries when the user corrects previously saved information - **Consolidation utilities** — built-in duplicate consolidation merges near-duplicate memories that may accumulate over time - **Local SQLite + NetworkX + FAISS storage** — entities and relations live in `~/.thoth/memory.db`, mirrored in a NetworkX graph for traversal, with FAISS vectors in `~/.thoth/memory_vectors/` - **Settings UI** — browse, search, visualize, and bulk-delete memories from the Knowledge tab in Settings, including graph statistics --- ## Wiki Vault The knowledge graph can be exported as a structured **Obsidian-compatible markdown vault** — one `.md` file per entity with YAML frontmatter, `[[wiki-links]]`, and auto-generated indexes. - **Vault structure** — entities grouped by type (`wiki/person/`, `wiki/project/`, `wiki/event/`, etc.) with one `.md` file per entity; sparse entities (<20 chars) roll up into `_index.md` per type; per-type indexes and a master `index.md` are auto-generated on rebuild - **YAML frontmatter** — each article includes `id`, `type`, `subject`, `aliases`, `tags`, `source`, `created`, and `updated` metadata - **Wiki-links** — related entities linked via `[[Entity Name]]` syntax, enabling Obsidian backlinks and graph view - **Connections section** — outgoing and incoming relations listed with arrow notation - **Live export** — entities are exported on save, deleted on entity removal, and rebuilt on batch operations - **Search** — full-text search across all `.md` files with title, snippet, and entity ID results - **Conversation export** — any thread can be exported as a vault-compatible markdown file - **Agent tools** — `wiki_read`, `wiki_rebuild`, `wiki_stats`, and `wiki_export_conversation` let the agent interact with the vault directly - **Settings UI** — enable/disable toggle, vault path configuration, stats display, rebuild, and open-folder actions in the Knowledge tab --- ## Dream Cycle A 5-phase background daemon refines the knowledge graph during idle hours and ends with an insight-generation pass over recent system activity. - **Phase 1: Duplicate merge** — entities with ≥0.93 semantic similarity and same type are merged; the LLM synthesizes the best description, aliases are unioned, and relations are re-pointed to the survivor - **Subject-name guard** — entities with different normalized subjects require ≥0.98 similarity to merge, preventing false merges of distinct people or concepts - **Phase 2: Description enrichment** — thin entities (<80 chars) appearing in multiple conversations get richer descriptions from conversation context and graph neighborhood - **Phase 3: Confidence decay** — stale `dream_infer` relations older than 90 days lose 10% confidence per cycle; very low-confidence edges are pruned automatically - **Phase 4: Relationship inference** — co-occurring entity pairs with no meaningful edge are evaluated for a specific typed relation; hub diversity caps, batch rotation, half-overlap reuse, multi-excerpt evidence, and a 7-day rejection cache improve quality and reduce repetition - **Phase 5: Insights analysis** — the system captures a snapshot of recent logs, provider/model/media configuration, channels, task state, memory stats, skills, and existing insights, feeds it to `DREAM_INSIGHTS_PROMPT`, and stores actionable results in `insights.py` - **Three-layer anti-contamination** — sentence-level excerpt filtering, deterministic post-enrichment validation, and strengthened prompting prevent cross-entity fact bleed - **Ollama busy check** — queries `/api/ps` before starting; defers if Ollama is actively serving a user request to avoid competing for GPU or CPU - **Configurable window** — default 1–5 AM local time; checks every 30 minutes if enabled, idle, in window, and not yet run that day - **Dream journal** — all operations logged to `~/.thoth/dream_journal.json` with cycle ID, summary, duration, merges, enrichments, inferences, insights, and errors - **Post-cycle rebuilds** — FAISS is rebuilt after the cycle, and the wiki vault is regenerated when enabled so downstream views stay in sync - **Manual trigger** — a dedicated Dream button in the Knowledge surface can start the cycle immediately - **Settings UI** — enable/disable toggle, quiet window controls, and last-run summary in the Knowledge tab --- ## Document Knowledge Extraction Uploaded documents are processed through a three-phase **map-reduce LLM pipeline** that extracts structured knowledge into the graph with full source provenance. - **Map phase** — each document is split into ~6K-char windows; each window is summarized to 3–5 sentences - **Reduce phase** — window summaries are combined into a coherent 300–600 word article - **Extract phase** — core entities and relations are pulled from the final article; extraction is capped at 12 entities per document to prevent over-extraction - **Curated relation vocabulary** — 67 valid relation types with 60+ alias mappings remove unknown-type churn and keep document-derived edges consistent with live memory extraction - **Hub entity** — the document itself is saved as a `media` entity; extracted entities link back via `extracted_from` for provenance tracking - **Quality gates** — minimum description length, self-loop rejection, and vague-relation bans keep output usable - **Cross-window dedup** — repeated entities across windows are merged before saving - **Cross-source merge protection** — document-derived content uses a stricter semantic merge threshold when it resembles a personal entity, reducing the chance of impersonal text overwriting personal knowledge - **Supported formats** — PDF, DOCX, TXT, Markdown, HTML, and EPUB - **Live progress** — the status bar shows an extraction pill with phase, progress, queue depth, and a stop action - **Background queue** — worker thread processes one document at a time - **Per-document cleanup** — removing a document deletes vector-store entries and extracted graph content associated with its source tag --- ## Brain Model & Providers The brain model is Thoth's default LLM — the model used for conversations, memory extraction, dream analysis, and any thread or workflow without a specific override. It can be a local Ollama model or an opt-in provider model. Thoth is built and tested for local models first. Every feature supports local models, and that remains the priority. Local models already handle tool calling, multi-step reasoning, memory extraction, and long conversations well with a 14B+ model. Provider models are supported for users without a dedicated GPU, for frontier reasoning on demand, or for trying many providers without downloading large local weights. Thoth supports opt-in provider models through **OpenAI** (direct API), **Anthropic** (Claude), **Google AI** (Gemini), **xAI** (Grok), **MiniMax** (M2 models through the Anthropic-compatible API), **OpenRouter** (many third-party models), **ChatGPT / Codex** (subscription-backed Codex models), and Custom/Self-hosted OpenAI-compatible endpoints such as LM Studio, vLLM, LocalAI, or private gateways. Provider connections, health, and credential sources are configured from Settings -> Providers; model catalog browsing, pinning, and defaults live in Settings -> Models. The `providers/` subsystem now owns provider config, auth metadata, model catalog normalization, runtime construction, display-safe status, and Quick Choices. Existing public functions in `models.py` remain as compatibility facades while provider-backed selection is rolled through the app. ChatGPT / Codex is deliberately modeled as a subscription provider, not as another OpenAI API-key route. Direct Codex runtime requires Thoth's in-app ChatGPT device-flow sign-in so Thoth stores its own runnable OAuth tokens in the local OS credential store. Existing Codex CLI auth files can be referenced only as display-safe metadata: Thoth records that the external login exists, path/fingerprint metadata, and broad auth-file shape, but it does not copy runnable tokens from `~/.codex/auth.json`. Codex runtime uses ChatGPT's subscription/internal Codex backend rather than the public OpenAI API. That means endpoint behavior, catalog shape, auth requirements, rate limits, and model availability may change upstream. When a ChatGPT / Codex model is selected, the current conversation plus model-visible tool context and tool results are sent to ChatGPT / Codex for that turn. Durable Thoth data such as memories, documents, files, and other conversations remain local unless explicitly included in the active conversation or surfaced by a tool result. - **Dynamic model switching** — change the brain model from Settings; choose from pinned local/provider Quick Choices managed in the Models catalog - **Per-thread & per-workflow model override** — conversations and workflows can each run on a different model, with overrides persisted locally - **Quick Choices** — models pinned from the consolidated Models catalog appear in chat, workflow, channel, Designer, and status-tool pickers - **Cost-efficient context management** — smart context trimming compresses older conversation turns and shrinks oversized tool outputs, reducing token usage and API costs for provider models - **Curated local models** — only tool-calling-capable local models are surfaced prominently - **Tool-support validation** — unsupported local models are warned about and can be auto-reverted if they fail a live tool-call check - **Download buttons** — local models not yet present show download actions with progress - **Configurable context window** — local and provider context caps can be set independently; actual model limits are still respected - **Local & provider indicators** — the UI clearly distinguishes downloaded local models, missing local models, and connected provider models - **Provider vision detection** — provider models with image capability are detected and reused by the Vision feature when available --- ## Voice Input & Text-to-Speech - **Toggle-based voice** — simple manual toggle to start and stop listening, no wake word required - **4-state pipeline** — stopped → listening → transcribing → muted, with explicit state transitions and mic gating during playback - **Local speech-to-text** — transcription via faster-whisper (tiny/base/small/medium), CPU-only int8 quantization, no cloud APIs required - **Voice-aware responses** — voice input is tagged so the agent knows you are speaking and can respond more conversationally - **Neural TTS** — high-quality text-to-speech via Kokoro, fully offline - **10 voice options** — US and British English, male and female variants - **Streaming TTS** — responses are spoken sentence-by-sentence as they stream in - **Mic gating** — microphone is automatically muted during TTS playback to prevent echo and feedback loops - **Hands-free mode** — voice input plus streaming TTS creates a fully conversational loop --- ## Shell Access - **Full shell access** — the agent can run shell commands on your machine through natural conversation - **Persistent sessions** — `cd`, environment variables, and other shell state persist across commands within a conversation; each thread gets its own isolated shell session - **3-tier safety classification** — commands are classified as safe, moderate, or blocked before execution - **Safe commands run instantly** — read-only operations like `ls`, `pwd`, `cat`, `git status`, or `pip list` execute without interruption - **Dangerous commands require approval** — destructive or system-modifying commands trigger an interrupt so you can accept or reject them - **Blocked by default** — high-risk commands like `shutdown`, `reboot`, or `mkfs` are rejected outright - **Background safety integration** — safe commands always execute; moderate commands are blocked by default in workflows but can be allowlisted per workflow; dangerous commands remain blocked - **Inline terminal panel** — command output appears in a collapsible terminal panel in the chat UI with clear and history controls - **History persistence** — shell history is saved per thread in `~/.thoth/shell_history.json` --- ## Browser Automation - **Full browser automation** — the agent can navigate websites, click elements, fill forms, scroll pages, and manage tabs in a real, visible Chromium window - **Shared visible browser** — runs with `headless=False` so you can see what the agent is doing and intervene when needed - **Persistent profile** — cookies, logins, and local storage survive across restarts in `~/.thoth/browser_profile/` - **Accessibility-tree snapshots** — after every action the tool captures the page's accessibility tree with numbered references so the model can click and type by number - **Smart snapshot filtering** — deduplicates links, drops hidden elements, and caps interactive elements to keep context under control - **Snapshot compression** — older browser snapshots are compressed to short stubs while the latest state remains detailed - **7 browser operations** — navigate, click, type, scroll, snapshot, back, and tab management - **Per-thread tab isolation** — each chat thread or background workflow gets its own browser tab; tabs are cleaned up on thread deletion or workflow completion - **Automatic browser detection** — prefers installed Chrome, then Edge on Windows, then Playwright's bundled Chromium - **Crash recovery** — if the browser closes externally, the next action relaunches it cleanly --- ## Vision - **Camera analysis** — capture and analyze images from your webcam in real-time - **Screen capture** — take screenshots and ask questions about what is on your screen - **Image file analysis** — analyze workspace image files by path without needing a camera or live capture - **Configurable vision model** — choose from local or provider-capable vision models - **Camera selection** — pick which camera to use when multiple devices are present - **Inline image display** — captured and workspace images are shown inline in chat - **Provider vision support** — provider models with image capability are auto-detected and work alongside local vision models --- ## Workflows & Scheduling Tasks have been renamed to **Workflows** throughout the application. The workflow engine adds a step-based pipeline runner, delivery routing, approvals, triggers, and safety gating on top of APScheduler. ### Core Engine - **Unified workflow engine** — named multi-step workflows run sequentially in a fresh or persistent thread and are scheduled through APScheduler - **7 schedule types** — `daily`, `weekly`, `weekdays`, `weekends`, `interval`, `cron`, and one-shot `delay_minutes` - **Template variables** — prompts can use `{{date}}`, `{{day}}`, `{{time}}`, `{{month}}`, `{{year}}`, `{{task_id}}`, and `{{step.X.output}}` - **Per-workflow model override** — each workflow can force a different model, then restore the default after completion - **Skills and tools overrides** — workflows can narrow the skill set globally and the tool set per step - **Channel delivery** — workflow output can be delivered to any registered channel via `delivery_channel` and `delivery_target` - **Persistent threads** — workflows can reuse the same thread across runs to preserve context - **Notify-only mode** — workflows can skip agent execution and just send notifications - **Webhook triggers** — workflows can be launched by HTTP webhook with per-workflow secrets - **Completion triggers** — one workflow can trigger another after finishing - **Concurrency groups** — related workflows can be serialized so only one runs at a time - **Safety mode** — `block_destructive`, `require_approval`, and `allow_all` modes control shell, workflow, and channel behavior inside background execution ### Step-Based Pipelines - **5 step types** — Prompt, Condition, Approval, Subtask, and Notify - **Conditional branching** — condition steps support `contains`, `not_contains`, `regex`, `json_path`, and `llm_evaluate`, each with `if_true` / `if_false` targets - **Approval gates** — approval steps pause execution, route requests through channels and desktop notifications, and resume on explicit user approval or denial - **Prompt chaining** — each step can see previous step output, enabling research → summarize → act patterns - **Agent-callable workflows** — the task tool can create, update, and run full step graphs programmatically ### Workflow Builder UI - **Simple/Advanced toggle** — simple mode preserves a single-prompt workflow editor; advanced mode exposes the full step builder - **Step builder** — reorder, delete, retarget, and retype steps visually - **Variable insertion menu** — context variables and prior-step outputs can be inserted without hand typing placeholders - **Flow preview** — Mermaid diagram generated from the step graph with manual refresh - **Validation** — required-field checks, reference validation, and operator-specific rules run before save ### Approval System - **Pending approvals panel** — approval cards show task name, request text, and Approve / Deny controls - **Sidebar badge** — pending approvals surface as a badge and quick actions above the thread list - **Multi-channel routing** — approvals can be routed through Telegram and desktop notifications with inline action controls - **Resume integration** — the agent and workflow runtime resume correctly on approve or deny and follow the appropriate branch ### Workflow Console - **Right-side console** — `ui/command_center.py` exposes running work, approvals, upcoming runs, quick launch actions, recent history, and insights in one drawer - **Live operational view** — running workflows, background states, and recent outcomes stay visible while you continue chatting elsewhere in the app - **Insight actions** — insight cards support pin, dismiss, and apply actions directly from the console - **Journal access** — extraction and dream journals are accessible from the same workflow-centric monitoring surfaces ### Existing Features - **Always-background execution** — workflows run without blocking the main chat UI - **Pre-built templates** — ships with starter workflows like daily briefings, research summaries, and reminders - **Home screen dashboard** — Workflows and Activity tabs show tiles, upcoming runs, run history, channel status, pending approvals, extraction journal, and dream journal - **Persistent run history** — execution history survives workflow deletion for auditability - **Monitoring / polling** — interval schedules plus condition steps support ongoing monitors like price checks or release watchers - **Stop / cancel support** — running workflows can be stopped from the chat header, activity panel, or workflow card --- ## Designer Studio Designer Studio is Thoth's dedicated visual-authoring subsystem. It spans five distinct **project modes**, a sandboxed interactive runtime, an authoring guardrail stack, and a mutation-reviewable tool surface for editing projects turn over turn. ### Project Modes Every project is created in one of five modes. Each mode carries its own canvas presets, template gallery, prompt budgets, critique rules, runtime behavior, and export targets. - **`deck`** — traditional slide decks; 16:9 canvas; ≤5 bullets per slide; PPTX export via `python-pptx` preserves editable text runs, images, and charts - **`document`** — long-form report / one-pager pages; A4 or letter canvas; 130–160 words per block; PDF export is the primary delivery format - **`landing`** — interactive marketing landing pages; vertical scroll canvas; CTAs and multi-section hero / feature / pricing layouts; published as interactive HTML - **`app_mockup`** — multi-screen app prototypes; route-aware navigator so the agent can define screens and declarative navigation between them; runtime bridge turns link / button clicks into in-preview route changes - **`storyboard`** — motion / ad storyboards; limited to 3–4 blocks per frame to avoid cropping; pairs naturally with the video generation tool for per-frame motion references ### Interactive Runtime Interactive modes (`landing`, `app_mockup`, `storyboard`) do **not** allow free-form `