AI Agent Review Paper Fastrack

Full-stack AI Review Engine for automatically evaluating essays, research papers, and business plans

Executive Summary

FastTrack AI Review Engine is an asynchronous multi-agent evaluation platform designed to perform structured, objective, and domain-specialized reviews of complex documents. Inspired by Stanford ML Group's [PaperReview.ai](https://paperreview.ai), FastTrack expands the evaluation paradigm across three distinct document ontologies: Academic Research Papers, Scientific & Academic Essays, and Startup Business Plans / Pitch Decks.

Evaluating dense, multi-page PDFs using standard single-prompt LLMs typically results in superficial feedback, hallucinated literature citations, and context degradation. FastTrack resolves these challenges through a deterministic Directed Acyclic Graph (DAG) state machine powered by LangGraph, multi-source grounding (arXiv, Semantic Scholar, Tavily), a two-stage hybrid re-ranking pipeline, and persona-driven evaluation rubrics.

Key Technical Highlights & Metrics

1. 18-Node LangGraph State Machine: Designed a type-safe, modular state graph (`ReviewEngineState`) with dynamic branching based on document ontology and extraction validation gates.
2. Multi-Source External Grounding: Automated verification of external prior work and market data querying arXiv, Semantic Scholar, and Tavily with strict URL/title deduplication.
3. Two-Stage Hybrid Ranking: Implemented heuristic relevance filtering (keyword overlap, recency scoring, source weighting) paired with Groq LLM reranking to curate the top-k evidence context.
4. Decoupled Distributed Architecture: Separated compute-heavy AI tasks (FastAPI) from business logic and persistence (Laravel 11 + Redis Queue), providing sub-second job acknowledgments and real-time step-by-step WebSocket telemetry.
5. Temporal-Aware Academic Scoring: Specialized evaluation prompts that calibrate novelty and methodology against a paper's actual publication year rather than modern baselines.

System Architecture & Distributed Flow

The system is built as three-tier distributed architecture, ensuring that long-running AI evaluation pipelines never block client interactions or overload application databases.

Component Responsibilities & Boundaries

1. Frontend Client (React 18 + Vite + TypeScript + Tailwind CSS)
   • Handles multi-format PDF uploads, live step-by-step progress tracking, and renders an interactive multi-axis radar chart for evaluation dimension breakdown
2. Backend Orchestrator (Laravel 11 + MySQL + Redis Queue)
   • Serves as the single source of truth for user authentication, file access validation, and evaluation records.
   • Dispatches long-running jobs (`TriggerAIReviewJob`) to a Redis worker with exponential backoff retries (60s, 120s, 240s) and a 10-minute worker timeout ceiling.
   • Exposes authenticated internal callback endpoints protected by HMAC/`X-Internal-Key` headers for status and result synchronization.
3. AI Agent Service (FastAPI + LangGraph + LangChain)
   • A stateless Python microservice executing the LangGraph evaluation pipeline asynchronously (`await graph.ainvoke()`).- Dispatches step progress logs (`extracting`, `classifying`, `searching`, `scoring`, `generating`) back to Laravel after each graph node completes.
4. Real-time Synchronization (Laravel Reverb / WebSockets)
   • Pushes live status updates and the final completion event directly to private user channels (`private-analysis.{userId}`), with an automatic 5-second polling fallback if WebSocket connections drop.

The LangGraph Agentic Pipeline & State Machine

Because an academic research paper requires citation novelty verification, an essay requires thematic coherence analysis, and a business plan requires unit economics validation, a uniform linear pipeline is insufficient. We structured the evaluation as a deterministic StateGraph.

Pipeline Execution Lifecycle

1. Document Ingestion & Validation (`extract` → `metadata_extract`):
   • Converts raw PDF documents into clean, structural Markdown via `pymupdf4llm` to preserve tables, formulas, and structural boundaries.
   • Evaluates defensive validation rules (minimum page threshold, text density, character limits). Invalid documents route directly to `end_with_error`.
   • Extracts structured front-matter: title, abstract, author list, keywords, and publication year using targeted context slices (`HEAD_SIZE = 3500`, `TAIL_SIZE = 1500`) to minimize token consumption.
2. Domain-Specific Profiling & Deep Extraction:
   • Research Paper Path (`research_document_profile`): Identifies domain classification, methodology type, benchmark claims, and generates academic search queries.
   • Business Plan Path (`bizplan_metadata_extract` → `bizplan_financials` → `bizplan_document_profile` → `bizplan_search_prep`): Parses revenue models, pricing signals, burn rate, runway months, break-even timelines, and red-flag financial signals.
   • Essay Path (`essay_document_profile`): Assesses thesis structure and applies heuristic rules to decide if external web search is needed.
3. Multi-Source Retrieval & Two-Stage Re-Ranking (`retrieval_prep` → `search_execute` → `search_rank`):
   • Queries arXiv (preprints), Semantic Scholar (citation graphs), and Tavily (market intelligence/web search) in parallel.
   • Eliminates duplicate URLs and scores candidates using a 2-stage hybrid ranker:
     • -Layer 1 (Heuristic Scoring): Evaluates title/keyword overlap, source credibility weighting (`Scholar > arXiv > Tavily`), publication recency bonuses, and snippet quality.
     • Layer 2 (LLM Reranker): Re-ranks the top candidates to curate the final 3–5 high-relevance references.
4. Specialized Persona Agents (`essay_agent` | `research_agent` | `bizplan_agent`): Synthesizes document evidence chunks, profile metadata, and retrieved references into domain-specific evaluation contexts.
5. Weighted Scoring & Final Payload Generation (`score` → `generate`):
   • Applies structured, weighted rubrics across 5–6 dimensions per document type.
   • Calculates the overall weighted score (0.0 - 10.0) and compiles strengths, improvement areas, reference citations, and executive summaries into a validated JSON schema.

State Machine Schema (`ReviewEngineState`)

The state acts as a centralized data contract across all pipeline nodes:

```python
from typing import Annotated, Literal, TypedDict
import operator
class ReviewEngineState(TypedDict):
    # Input & Ingestion
    analysis_id: str
    file_url: str
    doc_type: Literal["essay", "research", "bizplan"]
    raw_markdown: str
    page_count: int
    title: str | None
    is_valid: bool
    error: str | None
    # Academic & Paper Profiling
    abstract: str
    authors: list[str]
    keywords: list[str]
    year: int | None
    domain: str | None
    paper_type: str | None
    
    # Financial & Business Plan Specifics
    financial_metrics: dict[str, object]
    burn_rate: str | None
    runway_months: float | None
    financial_red_flags: list[str]
    market_validation: dict[str, object]
    # Retrieval & RAG Evidence Context
    search_queries: dict[str, list[str]]
    ranked_results: list[dict]
    top_references: list[dict]
    evidence_chunks: list[dict]
    review_context: str
    
    # Multi-Agent Evaluation & Scoring
    dimension_scores: dict[str, float]
    score_overall: float | None
    dimensions_feedback: list[dict]
    overall_feedback: str
    summary: str
    final_result: dict | None

Evaluation UI & Multi-Dimensional Scoring Report

The output is rendered on an interactive dashboard structured for rapid assessment by researchers, students, and founders.

Key Report Features

1. 6-Axis Radar Visualization: Displays visual symmetry across key criteria (e.g., Thesis Clarity, Coherence, Evidence Quality, Writing Clarity, Citation Rigor, Originality) to immediately highlight structural imbalances.
2. Dimension-Level Auditing: Details explicit weights, individual scores (1.0−10.01.0−10.0), and qualitative feedback tied to specific document sections.
3. Actionable Improvement Matrix: Separates high-priority conceptual revisions from secondary editorial improvements.
4. Curated References: Lists relevant real-world papers and market sources fetched during the pipeline run to support review recommendations.

Future Enhancements

1. Automated Benchmark Suite (RAGAS): Integrating a continuous offline evaluation harness tracking Faithfulness, Context Precision, and Aspect Critique across curated reference datasets.
2. Persistent Vector Caching: Caching computed embeddings for common academic literature to accelerate repeated domain lookups.
3. Multi-Format Ingestion: Extending parser capabilities to natively support .docx and LaTeX source bundles.