feat: add comprehensive documentation for AI Workspace, including architecture, memory model, profiles, services, and security guidelines

docs/architecture.md

@@ -0,0 +1,68 @@
+# Architecture
+
+AI Workspace is organized around explicit boundaries: profile configuration, raw evidence, canonical memory, derived retrieval, local services, and AI client adapters.
+
+## System Flow
+
+```text
+Communication / screenshots / archives / manual notes
+        ↓
+Raw inbox evidence
+        ↓
+Agent or human curation
+        ↓
+Canonical Markdown project knowledge
+        ↓
+Derived local index
+        ↓
+Read-only MCP context server
+        ↓
+AI clients and agent workflows
+```
+
+## Responsibility Boundaries
+
+| Layer | Responsibility | Canonical? |
+|---|---|---|
+| `core/` | Reusable architecture and operating model | yes, for workspace design |
+| `profiles/<profile>/` | Project-specific configuration and assumptions | yes, for profile config |
+| `project-knowledge/` | Human-readable project memory for the active profile | yes, for project facts |
+| `ai/inbox/` | Raw evidence captured from connectors | no |
+| `.aiw/indexes/` | Rebuildable search indexes | no |
+| `.aiw/runtime/` | PID files, logs, local service state | no |
+| `scripts/aiw/` | Profile-aware service/index utilities | code source |
+| `scripts/mcp/` | MCP servers exposing local context | code source |
+| `apps/` | Local UI surfaces such as the macOS menu bar app | code source |
+
+## Current Repository Shape
+
+The current repo still keeps the first real profile's vault at root-level `project-knowledge/`. That is acceptable during migration, but reusable code should increasingly resolve paths from profile configuration rather than hardcoding Fidelity-specific locations.
+
+Target direction:
+
+```text
+profiles/<profile>/workspace.json   # where profile data lives
+workspaces/<profile>/project-knowledge/
+workspaces/<profile>/inbox/
+.aiw/indexes/<profile>/
+```
+
+## Design Principles
+
+- Keep the smallest useful context loaded by default.
+- Prefer just-in-time retrieval over dumping the entire workspace into prompts.
+- Keep human-readable Markdown as the project source of truth.
+- Keep raw evidence outside canonical memory until explicitly promoted.
+- Keep profile-specific facts out of `core/` and generic scripts.
+- Make local services observable through a single service manager.
+- Treat cloud memory systems as optional, not authoritative.
+
+## Why This Shape
+
+Current AI workflow guidance emphasizes context engineering: the model should receive the smallest high-signal context needed for the task. This workspace supports that by combining:
+
+- structured Markdown memory for durable facts;
+- raw evidence stores for auditability;
+- local indexes for retrieval;
+- MCP tools/resources for AI clients;
+- profile-specific boundaries for reuse across projects.

docs/getting-started.md

@@ -0,0 +1,56 @@
+# Getting Started
+
+AI Workspace is a local, profile-based context system for AI-assisted work. It keeps project memory, raw evidence, local services, and AI client integrations organized without making any single AI tool the source of truth.
+
+## Core Idea
+
+Use the workspace as a companion repo beside your real project or client work:
+
+```text
+implementation repo / corporate tools / chat evidence
+        ↓
+AI Workspace inbox and memory curation
+        ↓
+human-readable project knowledge
+        ↓
+local index and MCP context server
+        ↓
+OpenCode / Claude Code / Copilot / other AI clients
+```
+
+## First Run
+
+From the repository root:
+
+```bash
+python3 scripts/aiw/services.py doctor --profile fidelity
+python3 scripts/aiw/services.py status --profile fidelity
+python3 scripts/aiw/indexer.py build --profile fidelity
+```
+
+The `fidelity` profile is the first real project profile in this repo. New projects should follow the same shape but keep their own profile configuration and project memory isolated.
+
+## Daily Use
+
+1. Open the project knowledge vault in Obsidian or your Markdown editor.
+2. Start only the local services needed for the profile.
+3. Capture raw evidence into the profile inbox.
+4. Promote useful, verified facts into canonical Markdown.
+5. Let AI clients query context through MCP or direct file reads.
+
+## Key Rules
+
+- Canonical project memory is Markdown, not chat history or vector storage.
+- Inboxes contain evidence, not promoted memory.
+- Indexes are derived and rebuildable.
+- MCP is read-only by default.
+- Secrets belong in ignored local `.env` files.
+
+## Next Reading
+
+- [Architecture](architecture.md)
+- [Profiles](profiles.md)
+- [Memory Model](memory-model.md)
+- [MCP](mcp.md)
+- [Services](services.md)
+- [Security and Privacy](security-and-privacy.md)

docs/local-rag-index.md

@@ -0,0 +1,67 @@
+# Local RAG Index
+
+The local RAG index is the first retrieval layer over canonical Markdown memory.
+
+## Purpose
+
+It helps AI clients quickly find relevant snippets without loading the whole project knowledge vault into context.
+
+```text
+project-knowledge/ Markdown
+        ↓
+scripts/aiw/indexer.py
+        ↓
+.aiw/indexes/<profile>/project-knowledge.jsonl
+        ↓
+MCP tool: memory_hybrid_search
+```
+
+## Current Implementation
+
+The current indexer is dependency-free and lexical. It is intentionally simple so it can run on a new machine without a vector database.
+
+Build:
+
+```bash
+python3 scripts/aiw/indexer.py build --profile fidelity
+```
+
+Status:
+
+```bash
+python3 scripts/aiw/indexer.py status --profile fidelity
+```
+
+Search:
+
+```bash
+python3 scripts/aiw/indexer.py search "dismissal lifecycle" --profile fidelity
+```
+
+## What It Stores
+
+- source path;
+- heading;
+- text chunk;
+- mtime;
+- content hash;
+- chunk id.
+
+## What It Does Not Do
+
+- It does not replace Markdown.
+- It does not write project facts.
+- It does not index templates as real notes.
+- It does not send data to a cloud service.
+
+## Future Options
+
+Future phases may add:
+
+- better full-text ranking;
+- semantic embeddings;
+- Qdrant or Chroma as optional local vector stores;
+- hybrid lexical + semantic search;
+- index status in the menu bar app.
+
+Keep this as a derived layer. The project knowledge vault remains canonical.

docs/mcp.md

@@ -0,0 +1,73 @@
+# MCP Integration
+
+The Model Context Protocol (MCP) is the workspace's standard interface for exposing local context to AI clients.
+
+## Role In AI Workspace
+
+`aiw-context-mcp` is a read-only context server. It exposes bounded profile context through MCP tools and resources.
+
+It should not:
+
+- capture communication traffic;
+- send messages;
+- write canonical memory;
+- promote facts automatically;
+- expose secrets or raw credentials.
+
+## MCP Concepts
+
+MCP uses a host/client/server model:
+
+- **Host**: the AI app, such as OpenCode, Claude Code, VS Code, Copilot, or another client.
+- **Client**: the connection the host opens to a server.
+- **Server**: a local or remote program that exposes context.
+
+Servers expose primitives such as:
+
+- **Tools**: callable functions.
+- **Resources**: readable context objects.
+- **Prompts**: reusable prompt templates.
+
+AI Workspace currently focuses on tools and resources.
+
+## Current Tools
+
+Examples:
+
+- `context_profiles`
+- `project_current_context`
+- `project_search_memory`
+- `memory_hybrid_search`
+- `communication_latest`
+- `communication_date_context`
+- `communication_standup_context`
+- `photos_latest`
+
+## Current Resources
+
+Examples:
+
+```text
+aiw://profiles/fidelity/current-work
+aiw://profiles/fidelity/work-items
+aiw://profiles/fidelity/mattermost/latest
+aiw://profiles/fidelity/photos/latest
+```
+
+## Security Posture
+
+MCP tools can be model-controlled in many clients, so this workspace defaults to read-only context tools. If write tools are added later, they should require clear user intent, narrow scope, and audit-friendly outputs.
+
+## Start The MCP Server
+
+HTTP transport:
+
+```bash
+python3 scripts/aiw/services.py start aiw-context-mcp --profile fidelity
+```
+
+stdio transport:
+
+```bash
+python3 scripts/mcp/aiw-context-mcp/server.py --transport stdio
+```

docs/memory-model.md

@@ -0,0 +1,69 @@
+# Memory Model
+
+AI Workspace separates memory by purpose so that human-readable knowledge, raw evidence, agent behavior, and derived indexes do not collapse into one opaque store.
+
+## Memory Layers
+
+| Layer | Purpose | Examples | Canonical? |
+|---|---|---|---|
+| Canonical project memory | Durable project facts for humans and AI | current work, work items, people, decisions, daily notes | yes |
+| Raw evidence | Captured data before curation | Mattermost mirror, photos, archives | no |
+| Agent operating memory | Rules for agent behavior and workflows | promotion rules, communication style, verification rules | yes for agent behavior |
+| Derived index | Fast retrieval over canonical memory | `.aiw/indexes/<profile>/` | no |
+| External agent memory | Optional cross-agent recall | mem9, tool auto-memory | no for project truth |
+
+## Canonical Markdown
+
+The project knowledge vault should be readable without any AI tool:
+
+```text
+project-knowledge/
+  00-start/
+  01-current/
+  02-work-items/
+  03-context/
+  04-people/
+  05-decisions/
+  06-daily/
+  07-maps/
+  08-bases/
+  09-templates/
+```
+
+For future multi-profile setups, this same structure should live under a profile-specific workspace path.
+
+## Evidence Promotion
+
+Connectors write evidence. They do not decide what becomes memory.
+
+Promotion flow:
+
+```text
+inbox evidence → verified fact → smallest correct Markdown file
+```
+
+Do not promote:
+
+- secrets;
+- sync status;
+- generic chatter;
+- unverified guesses;
+- raw transcripts without curation.
+
+## Local Index
+
+The local RAG index is derived from canonical Markdown. It helps AI clients find relevant snippets quickly, but it is not the source of truth.
+
+If index output conflicts with Markdown, Markdown wins.
+
+## mem9 And Similar Memory Systems
+
+mem9 can be useful as an optional cross-agent recall layer for preferences, reusable workflow habits, and non-sensitive operational memory. It should not replace the project knowledge vault.
+
+Recommended stance:
+
+```text
+project-knowledge/ wins over mem9, vector stores, and chat memory.
+```
+
+For sensitive or corporate projects, avoid cloud memory ingestion unless the data policy is explicit and approved.

docs/profiles.md

@@ -0,0 +1,87 @@
+# Profiles
+
+Profiles make the workspace reusable across projects, clients, or personal workflows.
+
+Each profile should describe what the project is, where its memory lives, which communication sources matter, which local services are enabled, and how AI clients should access context.
+
+## Current Profile Layout
+
+```text
+profiles/
+  fidelity/
+    profile.md
+    services.json
+    context-sources.json
+  example/
+    profile.md
+```
+
+## Recommended Future Layout
+
+```text
+profiles/<profile>/
+  profile.md              # human-readable project profile
+  workspace.json          # profile paths and defaults
+  services.json           # local services for this profile
+  context-sources.json    # communication/source filters
+
+workspaces/<profile>/
+  project-knowledge/      # canonical Markdown vault
+  inbox/                  # raw evidence for this profile
+```
+
+## Profile Files
+
+### `profile.md`
+
+Human-readable summary for agents and developers:
+
+- project name;
+- workspace role;
+- communication sources;
+- work-item system;
+- stakeholders or roles;
+- important domain themes;
+- enabled workflows or skills.
+
+### `workspace.json`
+
+Planned path configuration. Initial versions can point to current paths:
+
+```json
+{
+  "profile": "example",
+  "display_name": "Example Project",
+  "knowledge_dir": "project-knowledge",
+  "inbox_dir": "ai/inbox",
+  "index_dir": ".aiw/indexes/example"
+}
+```
+
+### `services.json`
+
+Profile-specific local service manifest for `scripts/aiw/services.py`.
+
+Examples:
+
+- context MCP;
+- communication proxy/mirror;
+- photo inbox;
+- future indexer or dashboard services.
+
+### `context-sources.json`
+
+Source filters for profile-bounded reads. For example, a Mattermost profile can define which channels count as high-signal context.
+
+## Adding A New Project
+
+1. Copy `profiles/example/` to `profiles/<new-project>/`.
+2. Create or point to a project knowledge vault.
+3. Define services only for integrations the project actually uses.
+4. Put connector secrets in ignored `.env` files.
+5. Build the local index.
+6. Connect AI clients through the MCP server.
+
+## Migration Rule
+
+Reusable code should accept a `--profile` argument and resolve paths through profile configuration. Avoid adding new hardcoded references to Fidelity, channel names, ticket prefixes, or company-specific folders in generic scripts.

docs/security-and-privacy.md

@@ -0,0 +1,63 @@
+# Security And Privacy
+
+AI Workspace is designed for local-first, auditable context management. Treat it as a companion workspace that may contain sensitive project metadata and communication evidence.
+
+## Rules
+
+- Do not commit secrets, tokens, cookies, API keys, headers, or session IDs.
+- Keep connector credentials in ignored `.env` files.
+- Keep raw evidence outside canonical project memory until curated.
+- Keep MCP read-only unless a write tool has explicit safety rules.
+- Treat generated indexes as local artifacts because they may contain snippets from project notes.
+- Prefer local services for corporate or confidential projects.
+
+## Ignored Local State
+
+Examples of local-only data:
+
+```text
+.aiw/runtime/
+.aiw/indexes/
+ai/inbox/mattermost-mirror/
+scripts/*/.env
+```
+
+## Cloud Memory Systems
+
+Tools such as mem9 or managed vector stores can be useful, but they introduce a data boundary.
+
+Before enabling them for a project, decide:
+
+- what data may be stored;
+- whether cloud storage is allowed;
+- whether self-hosting is required;
+- who can inspect/delete memories;
+- what happens when cloud memory conflicts with Markdown.
+
+Default recommendation:
+
+```text
+Use cloud memory only for non-sensitive preferences unless a project policy approves broader use.
+```
+
+## MCP Safety
+
+MCP clients may let models invoke tools automatically. For that reason, workspace MCP tools should stay read-only by default and return bounded, source-attributed context.
+
+If future MCP write tools are added, require:
+
+- explicit user intent;
+- narrow target paths;
+- clear diffs or summaries;
+- no secret exposure;
+- easy audit through git.
+
+## Sharing The Repo
+
+Before sharing or open-sourcing a reusable version:
+
+1. Remove or isolate project-specific profile data.
+2. Confirm ignored inbox/runtime files are not tracked.
+3. Replace real profile examples with sanitized examples.
+4. Keep reusable architecture docs in `docs/` and `core/`.
+5. Keep confidential project knowledge in private profile/workspace data.

docs/services.md

@@ -0,0 +1,61 @@
+# Services
+
+The AI Workspace Service Manager provides one profile-aware lifecycle surface for local services.
+
+## Responsibility
+
+The service manager starts, stops, checks, and tails logs. It does not merge service responsibilities.
+
+```text
+Service Manager → process lifecycle/status/logs
+MCP Server      → read-only context access
+Connectors      → raw evidence capture
+Agent/Human     → memory promotion
+```
+
+## Common Commands
+
+```bash
+python3 scripts/aiw/services.py status --profile fidelity
+python3 scripts/aiw/services.py status --profile fidelity --json
+python3 scripts/aiw/services.py doctor --profile fidelity
+python3 scripts/aiw/services.py start --profile fidelity
+python3 scripts/aiw/services.py stop --profile fidelity
+python3 scripts/aiw/services.py logs aiw-context-mcp --profile fidelity
+```
+
+## Service Manifest
+
+Services are declared per profile:
+
+```text
+profiles/<profile>/services.json
+```
+
+A service can define:
+
+- command;
+- kind;
+- groups;
+- dependencies;
+- restart policy;
+- doctor checks;
+- health checks.
+
+## Current Service Types
+
+- `process`: long-running command with PID/log tracking.
+- `app-launcher`: one-shot command that opens an app/helper.
+- `mcp`: process service exposing an MCP-compatible context server.
+
+## Runtime Files
+
+Runtime artifacts are local and ignored:
+
+```text
+.aiw/runtime/pids/
+.aiw/runtime/logs/
+.aiw/runtime/state/
+```
+
+These files are operational state, not project memory.