Systems I've built and evaluated. Each project includes methodology,
metrics, and honest limitations.
Multi-Agent CrewAI DSPy Embeddings
Customer Sentiment & PM Intelligence
Multi-Agent Review Analysis & Roadmap Alignment
Four-agent pipeline that ingests cross-platform customer reviews, scores sentiment and pain intensity, discovers themes via LLM map-reduce, aligns them to a product roadmap with semantic embeddings, and surfaces priority-ranked gaps.
Key findings
- → Reframing the sentiment output (predict 5-star, collapse to 3-class) beat two rounds of prompt refinement
- → Empty high-priority bucket survived two independent formula re-tunes. A corpus property, not a tuning failure
- → Spec's 0.75 cosine threshold was wrong for text-embedding-3-small; calibrated to 0.45 against actual distribution
RAG Hybrid Retrieval Cross-Encoder Evaluation
PaperSearch
Academic Paper Research Assistant
RAG system that retrieves relevant passages from 1,000 academic papers and generates cited answers. Validated against the Open RAG Benchmark with 3,045 human-authored queries.
Key findings
- → Hybrid retrieval (dense + BM25) dominated all top configurations
- → MiniLM matched mpnet quality at 5× the speed
- → Reranking improved MRR by 7.6% (unlike the financial system)
RAG Chunking Evaluation Financial
Financial Report Search
RAG Pipeline for PDF Documents
Retrieval pipeline for a 160-page annual report. Systematic evaluation across 66 configurations covering chunking strategies, embedding models, and retrieval methods, to find what actually works for financial documents.
Key findings
- → Sentence-based chunking outperformed fixed-size and semantic methods
- → Overlap caused 12× chunk explosion and consistently destroyed accuracy
- → Reranking hurt. Cohere rerank-v3.5 cut MRR by 0.070 and added 4,000x latency
Synthetic Data LLM Structured Output FastAPI Evaluation
Synthetic Data Pipeline
Resume-Job Match Dataset Generator
Production-style pipeline generating and validating synthetic job/resume datasets. Combines rules-based evaluation with LLM-as-judge analysis, runtime benchmarking, and failure-mode analysis across 250 resume-job pairs.
Key findings
- → Rules-based filtering caught 40% of mismatches without LLM calls
- → Sub-second latency for rules-only mode (p95 0.016s vs 2.0s target)
- → Template-specific quality variance identified through failure correlation analysis
Synthetic Data LLM Structured Output
Synthetic Data Generator
DIY Repair Q&A Dataset
Pipeline to generate realistic Q&A pairs for DIY home repair. Instructor library for structured outputs, LLM-as-judge validation, quality metrics.
Key findings
- → Structured output constraints eliminated formatting failures
- → Diversity gap exists at dataset level, not individual item level
- → LLM-as-judge enabled automated quality filtering
Fine-Tuning Embeddings Contrastive Learning SentenceTransformers
Dating Compatibility
Fine-Tuned Embedding System
Sentence transformer fine-tuned with contrastive loss to predict relationship compatibility from profile text. Tests whether semantic similarity, what embedding models are trained for, is a useful proxy for actual compatibility.
Key findings
- → Pre-trained model scored AUC 0.40, worse than random. It rated incompatible pairs higher than compatible ones, confusing shared vocabulary with attraction.
- → Contrastive fine-tuning lifted AUC from 0.40 to 0.91 and Cohen's d from -0.36 to 2.17.
- → Seven hyperparameter iterations across all tunable parameters plateaued at 83-84%. The ceiling is data quality. Training profiles substitute vague sentiment for concrete preferences, leaving the model without sufficient signal.
Multi-Agent LangGraph Communication Style FAISS
Digital Clone
Five-Agent Communication Style Pipeline
Five-agent LangGraph pipeline that ingests 2,156 Enron emails, extracts communication patterns, and generates query responses in the target employee's writing style. Each agent owns a discrete stage: ingestion, profile extraction, retrieval, generation, and scoring.
Key findings
- → Post-hoc rewriting degraded style scores from 0.678 to 0.565. Allen's vocabulary richness is 0.128; he repeats words constantly. Raw output accidentally matched his repetitive fingerprint better than deliberately polished content.
- → The 0.90 style target proved unrealistic. Allen never wrote about machine learning. Applying his email fingerprint to computer science content is extrapolation into territory he never occupied.
- → Upgrading to Sonnet 4.6 improved scores but broke the fallback rate. Higher quality pushed borderline off-topic queries above the delivery threshold. The fallback is quality-driven, not category-driven.
Multi-Agent RAG FastAPI ChromaDB Safety
AI-Powered Jira Assistant
Five-Agent RAG System for Issue Intelligence
Five-agent RAG system over a 4,020-issue Jira corpus. Retrieves relevant issues, surfaces hygiene suggestions, and supports sprint planning with simulation-first write safety that previews all mutations before execution.
Key findings
- → Greedy sprint planner reached 4% capacity utilization until the suggestion agent's SP estimator was injected as a callable. Utilization jumped to 99%.
- → Upgrading from gpt-4o-mini to gpt-4o cured the Medium-bias distribution but left priority agreement at 38%. Corpus label noise was the ceiling, not LLM capability.
- → Chain-of-thought on priority labeling regressed by 4 percentage points. More structured reasoning made the model more cautious, pulling confident Critical/Blocker predictions toward safer categories.