1
The Hardware Stack — The "Black Box"
▼
The RTX 5090 is the first consumer GPU that makes local AI enterprise-viable. Its 32GB of GDDR7 VRAM lets you run 70B-parameter models at 4-bit quantization — something that previously required a $30,000 server rack.
RTX 5090 — 32GB GDDR7 VRAM
Runs 70B models (Llama-3, DeepSeek-R1) at 4-bit quantization with a 32k+ token context window. Or 30B models at full Q8 precision for maximum quality.
Runs 70B models (Llama-3, DeepSeek-R1) at 4-bit quantization with a 32k+ token context window. Or 30B models at full Q8 precision for maximum quality.
VRAM32 GB GDDR7
Max Model70B @ 4-bit
Context32k+ tokens
MSRP~$2,000
CPUThreadripper / i9
RAM128 GB DDR5
PSU1200W Platinum
StorageNVMe Gen5 SSD
Total Cost~$3,500
128GB RAM handles massive document caches — critical when you're embedding 5,000 PDFs for a law firm. NVMe Gen5 makes RAG indexing fast enough to overnight 10 years of legal filings.
No RGB. No gamer aesthetic.
Lawyers and accountants want an "appliance," not a gaming rig. Use a high-airflow server chassis or a quiet professional workstation. The machine should look like it belongs in a server room, not a LAN party.
Lawyers and accountants want an "appliance," not a gaming rig. Use a high-airflow server chassis or a quiet professional workstation. The machine should look like it belongs in a server room, not a LAN party.
Demo — What fits in different GPU VRAM tiers
RTX 5090 · 32 GB
70B model @ 4-bit
RTX 4090 · 24 GB
34B model @ 4-bit
RTX 4080 · 16 GB
13B model @ 4-bit
RTX 3060 · 8 GB
7B model only
Why does the RTX 5090's 32GB VRAM matter more than raw GPU speed for local AI?
2
The Software Flow — Your Private Knowledge Base
▼
You aren't selling "a chatbot." You're selling a Private Knowledge Base — an AI that knows only their documents and can be interrogated like an expert paralegal. Three components make this work.
Ollama
The engine. Manages model weights, provides a local REST API. Drop-in replacement for OpenAI's API — most tools connect to it out of the box.
Vector DB
LanceDB or Milvus stores document "embeddings" — mathematical representations of meaning. When you ask a question, the DB finds semantically similar passages in milliseconds.
Open WebUI
The interface. Looks and feels exactly like ChatGPT — zero learning curve for non-technical staff. Connects to Ollama and your vector DB automatically.
Ollama acts as the engine — it manages model weights and exposes a local REST API on port 11434. Models: Llama-3-70B (4-bit) or DeepSeek-R1 for legal/financial reasoning tasks that demand logic and precision.
RAG (Retrieval-Augmented Generation) is what transforms a generic chatbot into a firm-specific expert:
- Client drops PDFs into a shared folder
- System embeds every document into a local vector database
- At query time, semantically relevant passages are retrieved
- The LLM answers only based on retrieved context
Result: "Does the Smith contract have a force majeure clause?" → AI searches only their local files to answer.
Open WebUI — the zero-friction interface. Partners with AnythingLLM for enterprise document management. Supports "personas" (e.g., "Contract Review", "Tax Research") so users see a tailored assistant for their role.
Demo — A RAG Query in Motion
Click Play, then click any step to inspect it.
What makes RAG fundamentally different from just asking the LLM a question directly?
3
The Business Models — How You Get Paid
▼
Two viable paths to market. Both work. The right one depends on your risk tolerance and how much you want to be on-site vs. remote.
Demo — 12-Month Revenue Calculator
3
Option A pitch: "This box has no Wi-Fi card. It is physically impossible for your data to leak." — Upfront cash flow funds hardware.
Option B pitch: "Your data stays on my private, encrypted hardware, never touching OpenAI or Google." — Recurring MRR scales without requiring site visits.
Option B pitch: "Your data stays on my private, encrypted hardware, never touching OpenAI or Google." — Recurring MRR scales without requiring site visits.
For Option B ("Private Cloud"), which tool lets clients securely access your machine without opening public internet ports?
4
The Workflow Example — A Law Firm in Practice
▼
Here's what the first week with a law firm client looks like. Four phases from onboarding to daily use — every step happens on their hardware or yours, never a third-party server.
Demo — Law Firm Onboarding Timeline
Click Play, then click any step to inspect it.
The key privacy guarantee: step 4 ("Daily Use") logs stay on the machine. No one at OpenAI, Google, or Microsoft sees that this firm is working on a high-profile merger or acquisition.
What happens during the overnight "Indexing" phase that makes the system firm-specific?
5
The Reality Check — Three "Adult" Problems
▼
The tech works. But to charge $1,000/month — and keep clients — you must solve three non-technical problems that will make or break your business.
Demo — Business Risk Navigator (click a risk, then click its resolution)
Select a risk to understand its impact and resolution path.
SOC2 / HIPAA: You become a Data Processor
Even if the hardware is on-prem, if you have remote access for maintenance, you are legally a "Data Processor." You need a signed Data Processing Agreement (DPA) before touching client data. Draft one with an attorney — yes, your first client may literally be a lawyer writing their own DPA.
Even if the hardware is on-prem, if you have remote access for maintenance, you are legally a "Data Processor." You need a signed Data Processing Agreement (DPA) before touching client data. Draft one with an attorney — yes, your first client may literally be a lawyer writing their own DPA.
Wrong citations = malpractice liability
Configure your RAG system to always surface citations: "According to Document_A.pdf, Page 4…" Never let the model answer without attribution. Consider adding a confidence threshold — if the model can't find a relevant document, it should say "I don't know" rather than fabricating a case citation.
Configure your RAG system to always surface citations: "According to Document_A.pdf, Page 4…" Never let the model answer without attribution. Consider adding a confidence threshold — if the model can't find a relevant document, it should say "I don't know" rather than fabricating a case citation.
Hardware failure = client workflow stops
For a $1k/mo retainer, clients expect an SLA. You need: (1) a spare GPU on the shelf, (2) a documented 24-hour replacement procedure, and (3) a temporary fallback (even cloud-hosted, with client consent) while the machine is down. Price the spare hardware into your contract terms.
For a $1k/mo retainer, clients expect an SLA. You need: (1) a spare GPU on the shelf, (2) a documented 24-hour replacement procedure, and (3) a temporary fallback (even cloud-hosted, with client consent) while the machine is down. Price the spare hardware into your contract terms.
The Verdict: Sell certainty (the AI will never leak your data), not just utility (the AI can write emails). $1,000/month is a steal for a mid-sized firm's legal liability protection alone.
What is the most important reason to force the AI to provide document citations in legal use cases?
6
Document Lifecycle — Keeping the Knowledge Base Fresh
▼
The vector database doesn't auto-update when source files change. Stale embeddings are the silent killer — the model will confidently cite outdated clauses or reference deleted documents until you explicitly re-sync.
Contract v2 replaces v1
Delete the old document's chunks from the vector DB by doc ID / filename, then re-embed the new version. Without this, both versions exist in the DB and the model may cite contradictory clauses from v1 and v2 simultaneously.
Delete the old document's chunks from the vector DB by doc ID / filename, then re-embed the new version. Without this, both versions exist in the DB and the model may cite contradictory clauses from v1 and v2 simultaneously.
Step 1Detect mtime change via folder diff
Step 2Delete old embeddings by doc ID
Step 3Re-embed updated file
Closed case files removed from the drive
The model won't "forget" a document just because you deleted the file. You must explicitly delete its embeddings from the vector store by document ID. AnythingLLM and Open WebUI both have per-document delete buttons — or automate it with a folder-watch script.
The model won't "forget" a document just because you deleted the file. You must explicitly delete its embeddings from the vector store by document ID. AnythingLLM and Open WebUI both have per-document delete buttons — or automate it with a folder-watch script.
Step 1Diff source folder vs DB metadata
Step 2Identify orphaned doc IDs
Step 3Purge embeddings by ID
Nightly cron sync — the $1k/mo differentiator
A script watches the source folder and diffs filenames + last-modified timestamps against the vector DB's metadata. Only changed/new files are re-embedded; deleted files are purged. LanceDB and Milvus both support metadata filtering so this is targeted, not a full re-index. Include this in your maintenance retainer — manual "re-sync" buttons don't cut it for paying clients.
A script watches the source folder and diffs filenames + last-modified timestamps against the vector DB's metadata. Only changed/new files are re-embedded; deleted files are purged. LanceDB and Milvus both support metadata filtering so this is targeted, not a full re-index. Include this in your maintenance retainer — manual "re-sync" buttons don't cut it for paying clients.
Demo — Document State Machine (trace the update and delete paths)
Click "Indexed" to start, then follow a document through its lifecycle.
A law firm updates a contract PDF. What happens if you don't re-sync the vector database?