I evaluate AI models.
Then I built the system
that picks between them.
Prashant Dimri — founder of Agent Forge. I came to AI from civil engineering and a decade of enterprise sales and delivery, not from a research lab. That turns out to be the useful part: most AI failures I have watched up close were not model failures, they were workflow failures — and you only see those if you have sat with the person whose job the model just broke.
I design and grade frontier-model evaluations. I built an OCR-and-LLM document pipeline that took commercial insurance underwriting from about forty minutes a document to roughly two. And I build Agent Forge — a platform that spreads work across many models and learns which one to trust from what actually happened last time.
"Building begins before the product. It begins with understanding the problem well enough that people recognize themselves in the solution."
Frontier-model evaluation.
Designing and grading tasks that test whether a model can reason correctly across a real document set — W-2s, 1099s, brokerage statements, loan records, draft returns — and reach a defensible number a person could act on.
Each task ships as a complete package: the consumer prompt in their own voice, the critical elements a correct answer must reach, a golden trajectory tracing every figure to its source document, a weighted scoring rubric, and a scored justification for how each candidate model performed. Every one of those documents is published in full in the archive — not summarised.
The consumer asks about their tax bill and a 9% student loan. The real answer is a $3,383.83 credit card at 24.99% that is negatively amortizing — monthly interest of $65.59 against an autopay of $63.08, so the balance grows while being paid. A model can compute the tax correctly and still fail, because it answered the question asked instead of the one that mattered.
Design principle: the headline finding is never the question the consumer asks.
Corrected federal balance due of $3,494.17 against the consumer's draft of $4,703.00 — a $1,208.83 adjustment, reached by disallowing a $2,500 student-loan deduction above the phase-out ceiling, adding $55.50 of Additional Medicare Tax, and crediting $1,864.13 of excess Social Security withheld across two employers. Every constant is pinned to the tax year before a single response is scored.
A rubric is only as good as the ground truth underneath it.
35 weighted criteria, 795 positive points, −430 in penalties for violating a stated constraint. Weight concentrates where responses actually diverge: the hidden headline, the exact corrected balance, debt priority order. Of three model responses scored against it, two failed — one by 8 points — and the failures were itemised criterion by criterion rather than called weak.
If two graders can reach different scores, the rubric is not finished.
Bringing AI into commercial insurance.
My next major chapter was working as a Business Analyst focused on AI infrastructure and automation inside a commercial insurance organization — large document volumes, inconsistent formats, scanned files, handwritten information, complex terminology, regulatory requirements, and costly manual verification. I worked directly with executives, business teams, developers, data specialists, and insurance stakeholders to translate operational problems into AI-enabled workflows.
A central initiative was an OCR and LLM-powered document-processing pipeline supporting motor vehicle records, IFTA reports, loss runs, insurance applications, driver records, and scanned or handwritten underwriting files. The workflow combined document extraction, OCR, language models, schema mapping, confidence scoring, validation rules, and human review — covering roughly 95% of the targeted document types, including difficult scanned and handwritten forms. A process that could take about 40 minutes of manual work was reduced to around two minutes, including verification.
I also helped design feedback mechanisms letting reviewers correct extracted information and capture those corrections as structured data — a repeatable learning loop that directly influenced how Agent Forge approaches telemetry, failure tracking, and outcome-based learning. My role often functioned as the translation layer between business requirements and technical implementation: requirements discovery, process mapping, API schema alignment, confidence thresholds, user acceptance testing, and executive reporting. Redesigning project structures and operational discipline raised process compliance from roughly 20% to 95%.
The larger lesson was clear: successful AI adoption requires more than a capable model. It requires architecture, accountability, validation, and a deep understanding of the workflow surrounding the model.
One model is not the answer.
Working across different language models, I repeatedly hit the same problem: no single model is best at everything. One may reason well but cost too much for routine work. Another may be fast but inconsistent with structured output. A self-hosted model may offer privacy and control but need stronger validation. A smaller model may outperform a larger one when the task is narrowly defined. Most AI applications still treat model selection as a fixed configuration. Agent Forge treats it as a decision — answering questions like which model should handle this specific task, whether it should run hosted or self-hosted, whether a smaller model can do it reliably, when multiple models should work concurrently, how failures get detected and classified, and how performance improves from real outcomes rather than guesswork.
It's an evolving control plane for intelligent model orchestration — built to know how to use each model's strengths, recognize its weaknesses, and keep improving from what actually happens when it runs.
Where it started.
I began with a Bachelor of Technology in Civil Engineering from Lovely Professional University. Engineering taught me to think in systems: understand the foundation, identify dependencies, test assumptions, and recognize that a structure is only as reliable as the components supporting it.
My early career then took me into real estate, SaaS, EdTech, sales, business development, and customer experience — including roles at BYJU'S, Lido Learning, and PlanetSpark. I learned how products are positioned, how customer decisions are made, and why even strong products fail when they don't solve the right problem. I managed customer relationships, closed hundreds of transactions, exceeded revenue targets, and led a business development team. I also co-founded an early service startup — it didn't become my final destination, but it gave me a founder's understanding of discovery, positioning, execution, and adaptation.
The MBA, and the start of the AI journey.
In 2022 I moved to the United States for an MBA in Marketing Analytics at Pace University's Lubin School of Business in New York. The program connected customer intuition with structured analysis — data, statistics, segmentation, visualization, and analytical decision-making.
Around the same time, generative AI entered the mainstream. When ChatGPT launched, I didn't see it only as a chatbot — I saw a new interface between humans and computing, one capable of understanding instructions, generating content, analyzing information, and eventually performing complete workflows. I began experimenting with prompts, outputs, limitations, and model behavior. What started as curiosity quickly became a serious professional direction.
Learning how language models think.
Over 1,000 prompts and model responses written, graded, and ranked — RLHF work, comparative evaluation, hallucination and instruction-following analysis, and checking structured output that has to parse as well as read well. Business tasks, analytical tasks, multimodal tasks, and files that had to come back as valid JSON or CSV.
The work required more than deciding whether an answer sounded good. I had to determine whether the model understood the real intent, followed every material instruction, reasoned soundly, avoided unsupported claims, handled ambiguity, and produced an output that was genuinely usable.
"A model can produce an impressive answer and still fail the task."
Selected projects.
Agent Forge itself is a private production system, but a curated engineering dossier of it is public and readable — architecture notes, examples, and tests, with CI and CodeQL running against it. Read the code on GitHub →
Multi-model AI orchestration and model-intelligence platform. Intelligent routing, concurrent multi-model processing, specialized agent orchestration, outcome-based Karma scoring, tenant isolation, and cost/quality/latency optimization. Python, FastAPI, MongoDB Atlas, BGE-M3 embeddings, Cloudflare Workers, Docker.
Explore the platformOCR and LLM-powered document-processing system converting scanned, handwritten, structured, and unstructured insurance documents into validated business data. ~95% coverage of targeted document types; workflows reduced from ~40 minutes to ~2. OCR, PyMuPDF, PaddleOCR, JSON schemas, confidence scoring, human review.
Shipped · ~95% coverage of targeted documentsTraining and evaluating frontier model behavior — creating prompts, testing edge cases, comparing responses, classifying failures, and evaluating outputs across text, analytical, multimodal, and structured-file tasks. Prompt engineering, RLHF, comparative evaluation, rubric design, output validation.
Read the full archivePython and Google-based automation collecting information from emails and forms, standardizing it, and organizing it in spreadsheets — reducing repetitive manual work and improving consistency. Python, Google Apps Script, Gmail, Google Sheets.
A planning and project-visibility application transforming project information into structured timelines and executive-level views — natural language to planning artifacts. Python, FastAPI, structured planning workflows.
What I'm exploring now.
Three things, specifically. Training a classifier of our own — every time the system disputes its own classification, that case is kept, and those cases are becoming the training set for a local model that replaces a general-purpose one. Letting unproven models earn a track record — bounded exploration, so a model with no history gets a real chance at work instead of being excluded forever by a cold start. And retrieval that does not flatten the detail — late-interaction scoring, for the cases where compressing a document to a single vector throws away the thing that actually mattered.
The through-line is the same each time: take something the system already sees happen, and turn it into something it can act on the next time round.
"How do we build AI systems that don't merely generate outputs, but become more reliable through every task they complete?"
How I build.
Before choosing a model or architecture, understand the actual problem, the workflow, and the person expected to use the result.
A good prompt cannot compensate for weak architecture. Models need boundaries, validation, observability, and recovery mechanisms.
A successful API call is not the same as a successful task. Measure whether the result was accurate, useful, timely, and worth its cost.
Failures shouldn't disappear inside logs. They should become structured evidence that improves routing, prompts, validation, and future behavior.
A note from Prashant.
Agent Forge represents the convergence of everything I've worked on so far: engineering thinking, customer understanding, analytics, model evaluation, enterprise automation, and system design. I didn't begin this journey knowing I would build an AI platform — I arrived here by repeatedly hitting problems existing tools didn't fully solve.
I saw impressive models fail basic workflows. I saw businesses buy AI without the infrastructure to evaluate it. I saw teams manually correct the same failures without turning those corrections into learning. I saw smaller models get ignored, expensive models get overused, and critical decisions made without enough visibility into why a model was selected. Agent Forge is my attempt to build the layer that connects those pieces — not to replace human judgment or declare one model the winner, but to create infrastructure that knows how to use each model's strengths, recognize its weaknesses, and keep improving from real outcomes.
This is still the beginning. Every experiment, failed workflow, successful deployment, model comparison, and corrected output is bringing the platform closer to that vision.
Build with the right model.
Learn from every outcome.
Forge intelligence into a system.