Systems I've built and evaluated. Each project includes methodology,
metrics, and honest limitations.
Multi-Agent Google ADK MCP MongoDB Atlas Human-in-the-Loop
Event Commerce Ops Agent
Nine-Capability Agent with Human Approval
Google ADK agent that takes a batch of post-match photos through to Shopify listings and a queued social post. Nine capabilities run in sequence, from ingest and Gemini Vision scoring to human approval and execution. Every step's state lives in MongoDB Atlas, reached through the MongoDB MCP server. Built for the Google Cloud Rapid Agent Hackathon (MongoDB track).
Key findings
- → ADK's docs suggest a long-running tool re-runs its body when it resumes. The ADK source shows it doesn't. The operator's reply goes to the coordinator LLM instead, so approval became two tools, one that only pauses the run and one that records each decision.
- → Starting the MongoDB MCP server with npx @latest on a restricted network froze the event loop while npm resolved the version, and the timeout couldn't interrupt it. A pinned, vendored binary fixed the hang. Reusing one session cut each call from about 2.9s to under 1s.
- → Images that resemble past high performers are routed by a similarity-weighted vote over history, and the LLM can't override it. Images with no precedent get the model's judgment and a written rationale. That half is uncalibrated. Validating it needs a human-scored reference set.
Results
Eval ship threshold
≥19/20 runs
Multi-Agent RAG FastAPI ChromaDB Safety
AI-Powered Jira Assistant
Five-Agent RAG System for Issue Intelligence
Five-agent RAG system over a 4,020-issue Jira corpus. Retrieves relevant issues, surfaces hygiene suggestions, and supports sprint planning with simulation-first write safety that previews all mutations before execution.
Key findings
- → On one project only 4% of open issues had story points, so the greedy sprint planner reached 6% capacity utilization. Injecting the suggestion agent's story-point estimator as a callable took it to 99%.
- → Upgrading from gpt-4o-mini to gpt-4o cured the Medium-bias distribution but left priority agreement at 38%. Corpus label noise was the ceiling, not LLM capability.
- → Chain-of-thought on priority labeling regressed by 4 percentage points. More structured reasoning made the model more cautious, pulling confident Critical/Blocker predictions toward safer categories.
Results
Simulated sprint utilization (1 of 2 projects)
99%
Agents AWS Bedrock MCP Deterministic Pipeline Property-Based Testing
Cost Assassin
DynamoDB N+1 Diagnosis Agent
Five-step agent that reads an X-Ray trace, finds a DynamoDB N+1 read, points at the source line and writes a BatchGetItem fix. Detection, source location and cost math are deterministic. The LLM writes only the fix. Built for Código Facilito's AWS hackathon (Specialized Agents challenge).
Key findings
- → The cost headline was wrong. AWS bills each BatchGetItem item as a GetItem, so batching saves no read cost. The 50% came from pricing the N+1 reads at the strongly consistent rate.
- → Given 500 keys, Nova Pro wrote one BatchGetItem call with all 500, past AWS's 100-key limit, while citing the passage that states it. The output was schema-valid, so the test suite couldn't see it. gpt-oss-120b chunked the keys and retried the leftovers, at $0.0007 a run against Nova Pro's $0.0033.
- → X-Ray ingests a trace incrementally. The first recording at N=100 captured 74 of 100 reads, and the volume math silently fell back to a default. The recorder now waits for the trace to settle.
Results
Read-cost saving (per AWS)
$0
Correctness properties
22
Multi-Agent CrewAI DSPy Embeddings
Customer Sentiment & PM Intelligence
Multi-Agent Review Analysis & Roadmap Alignment
Four-agent pipeline that ingests cross-platform customer reviews, scores sentiment and pain intensity, discovers themes via LLM map-reduce, aligns them to a product roadmap with semantic embeddings, and surfaces priority-ranked gaps.
Key findings
- → Reframing the sentiment output (predict 5-star, collapse to 3-class) beat two rounds of prompt refinement
- → Empty high-priority bucket survived two independent formula re-tunes. A corpus property, not a tuning failure
- → Spec's 0.75 cosine threshold was wrong for text-embedding-3-small; calibrated to 0.45 against actual distribution
Multi-Agent LangGraph Communication Style FAISS
Digital Clone
Five-Agent Communication Style Pipeline
Five-agent LangGraph pipeline that ingests 2,156 Enron emails, extracts communication patterns, and generates query responses in the target employee's writing style. Each agent owns a discrete stage: ingestion, profile extraction, retrieval, generation, and scoring.
Key findings
- → Post-hoc rewriting degraded style scores from 0.678 to 0.565. Allen's vocabulary richness is 0.128; he repeats words constantly. Raw output accidentally matched his repetitive fingerprint better than deliberately polished content.
- → The 0.90 style target proved unrealistic. Allen never wrote about machine learning. Applying his email fingerprint to computer science content is extrapolation into territory he never occupied.
- → Upgrading to Sonnet 4.6 improved scores but broke the fallback rate. Higher quality pushed borderline off-topic queries above the delivery threshold. The fallback is quality-driven, not category-driven.
RAG Hybrid Retrieval Cross-Encoder Evaluation
PaperSearch
Academic Paper Research Assistant
RAG system that retrieves relevant passages from 1,000 academic papers and generates cited answers. Validated against the Open RAG Benchmark with 3,045 human-authored queries.
Key findings
- → Hybrid retrieval (dense + BM25) dominated all top configurations
- → MiniLM matched mpnet quality at 5× the speed
- → Reranking improved MRR by 7.6% (unlike the financial system)
RAG Chunking Evaluation Financial
Financial Report Search
RAG Pipeline for PDF Documents
Retrieval pipeline for a 160-page annual report. Systematic evaluation across 66 configurations covering chunking strategies, embedding models, and retrieval methods, to find what actually works for financial documents.
Key findings
- → Sentence-based chunking outperformed fixed-size and semantic methods
- → Overlap caused 12× chunk explosion and consistently destroyed accuracy
- → Reranking hurt. Cohere rerank-v3.5 cut MRR by 0.070 and added 4,000x latency
Synthetic Data LLM Structured Output FastAPI Evaluation
Synthetic Data Pipeline
Resume-Job Match Dataset Generator
Production-style pipeline generating and validating synthetic job/resume datasets. Combines rules-based evaluation with LLM-as-judge analysis, runtime benchmarking, and failure-mode analysis across 250 resume-job pairs.
Key findings
- → Rules-based filtering caught 40% of mismatches without LLM calls
- → Sub-second latency for rules-only mode (p95 0.016s vs 2.0s target)
- → Template-specific quality variance identified through failure correlation analysis
Synthetic Data LLM Structured Output
Synthetic Data Generator
DIY Repair Q&A Dataset
Pipeline to generate realistic Q&A pairs for DIY home repair. Instructor library for structured outputs, LLM-as-judge validation, quality metrics.
Key findings
- → Structured output constraints eliminated formatting failures
- → Diversity gap exists at dataset level, not individual item level
- → LLM-as-judge enabled automated quality filtering
Fine-Tuning Embeddings Contrastive Learning SentenceTransformers
Dating Compatibility
Fine-Tuned Embedding System
Sentence transformer fine-tuned with contrastive loss to predict relationship compatibility from profile text. Tests whether semantic similarity, what embedding models are trained for, is a useful proxy for actual compatibility.
Key findings
- → Pre-trained model scored AUC 0.40, worse than random. It rated incompatible pairs higher than compatible ones, confusing shared vocabulary with attraction.
- → Contrastive fine-tuning lifted AUC from 0.40 to 0.91 and Cohen's d from -0.36 to 2.17.
- → Seven hyperparameter iterations across all tunable parameters plateaued at 83-84%. The ceiling is data quality. Training profiles substitute vague sentiment for concrete preferences, leaving the model without sufficient signal.