Which AI models are the best as CEO agents for startup founders?

Abstract

JuhiBench compares how well current AI models can serve as CEO agents for startup founders: understand company constraints, make business decisions, and complete work through tools rather than merely give advice. It evaluates decision quality, execution, turn time, and relative API cost using a library of 79 fictional business scenarios. In a controlled product comparison across that library, the same GPT-5.6 Sol configuration at the same reasoning effort scored 74.9 inside JuhiAI and 58.2 in a plain chat-style window, with the same LLM judge evaluating both conditions. JuhiAI scored higher in 69 of 79 situations. The score comparison does not include JuhiAI's additional ability to create tasks, update KPIs, persist decisions, or open approval gates.

A startup agent needs authority; otherwise it is only another adviser. Several models could explain a sensible move but failed to complete the corresponding tool action. In the 500-transcript core audit, 52 responses triggered a verified cap for a critical failure, a genuinely fired auto-fail, or a generic reply; 12 had initially received an LLM-judge score of at least 70. JuhiBench therefore gives execution 75% of the final score. JuhiAI's company context, tools, permissions, and persistence are part of the product being evaluated: the value to a founder is completed work, not another persuasive answer window.

The model choice depends on the job. Fable had the highest point estimate at 77.5 with a 58.8-second median turn. Opus reached 73.9 in 37.4 seconds at 15.2× relative cost. API GPT-5.6 Luna and Sol were effectively tied at 73.7 and 72.9, but Luna took 17.2 seconds at 2.0× relative cost while Sol took 83.4 seconds at 9.8×. DeepSeek's three-execution core estimate was 69.9 at the 1.0× cost reference and a 59.2-second median. Grok 4.1 Fast Reasoning is a practical low-cost production candidate at 61.9 quality, an 18.7-second median, and 0.3× relative API cost; GLM-4.7 is the fast routine option at 55.7 in 5.7 seconds. A premium model can still be justified when one prevented mistake in financing, hiring, disclosure, or a major customer commitment is worth more than the model-cost difference.

JuhiBench tested models inside a tool-using business agent across 79 fictional scenarios, five companies, and seven functional areas. The common comparison uses 23 core scenarios, weighted 25% intent and 75% execution. GPT-5.6 Luna served as the LLM judge for the qualitative outcomes, while deterministic checks verified critical tool calls and state changes. The result is an initial ten-model JuhiAI roster organized by founder use case; it does not predict real-company outcomes.

Findings first

1. Use a structured agent harness, not a blank chat window. Across all 79 scenarios, the same Sol model, effort, and LLM judge produced 74.9 inside JuhiAI versus 58.2 in plain chat. Start with company context, operating rules, memory, tools, and persistence; changing the model alone leaves much of the available quality on the table.

2. Buy completed work, not longer answers. In the 500-transcript core audit, 52 responses hit a verified cap and 12 of those had initially scored at least 70. JuhiBench weights execution at 75% because a model that only recommends the next step—or takes the wrong one—is still leaving the work to the founder.

3. Offer purposeful categories, not one supposed winner. Use API GPT-5.6 Luna as the interactive default, DeepSeek for slower background analysis, Grok 4.1 Fast Reasoning for low-cost interactive work, GPT-5.4 Mini or GLM-4.7 for fast routine work, Codex subscription Sol, Terra, or Luna for subscription access, and Opus or Fable for premium decisions.

4. Give founders alternatives inside a category. GPT-5.4 Mini scored 57.9 at 8.6 seconds, including a 65.0 intent score; GLM-4.7 scored 55.7 at 5.7 seconds, including a higher 58.0 execution score. Both cost 1.0× in this comparison. Use Mini for scope-sensitive routine work and GLM when minimum turn time and action completion matter more.

5. Pay more where the downside justifies it. Average gaps between the leaders are small, but Fable's 80.4 execution score was the highest in the core table. A premium can be rational for financing, hiring, disclosure, or a major contract if a company-specific trial shows fewer consequential errors; it is harder to justify for routine work that a cheaper configuration passes reliably.

6. The broader test can expose a difficult tail. API Luna scored 73.7 on the common core and 68.8 across all 79 situations; GLM-4.7 moved from 55.7 to 49.8, while GPT-5.4 Mini stayed at 57.9. Prefer the full-library estimate when it exists instead of assuming 23 situations capture every failure mode.

7. Higher quality usually took longer, which makes fast strong models valuable. Across the 36 complete core configurations, quality and median turn time had a descriptive Spearman correlation of 0.71. API GPT-5.6 Luna was the useful exception: 73.7 quality at a 17.2-second median. When quality is within roughly three points, prefer the faster model.

8. Treat the table as a shortlist, not a universal ranking. The fixed-transcript LLM-judge audit produced a median 3.2-point difference per scored transcript, while additional candidate executions would improve the model estimates themselves. Compare two or three candidates on the startup's real workflows and choose the best combination of accepted work, founder time saved, speed, and price.

What a founder can do with this result

Recommended JuhiAI CEO-agent choices

The initial JuhiAI model selector will expose ten choices across six categories. A category can contain several defensible alternatives: founders can choose between Fable and Opus for premium work, among three Codex subscription profiles, and between GPT-5.4 Mini and GLM-4.7 for fast routine work.

DeepSeek is included for asynchronous work, not as an interactive default: API GPT-5.6 Luna scored 3.8 points higher at less than one-third the turn time. Muse is not included because it produced usable output on only 61 of 79 scenarios. The remaining tested configurations did not add a sufficiently distinct quality, speed, cost, or access tradeoff for the initial selector.

The closest omitted alternatives were also checked. DeepSeek V4 Flash and MiMo V2.5 were slower, lower-quality, and no cheaper than Grok 4.1 Fast. Gemini 3.5 Flash and Nemotron Ultra were slower, lower-quality, and more expensive than GPT-5.4 Mini. Gemma 4 31B matched Mini's aggregate quality at 0.2× cost but took 33.0 seconds; that small additional price tier did not justify another initial user-facing choice beside Grok, Mini, and GLM.

Promising configurations

Running the models through the actual JuhiAI tool loop changed the shortlist in ways a chat demo would not reveal. We treated a configuration as promising when it offered a useful quality, speed, cost, or access profile.

The clearest API product fit was GPT-5.6 Luna. API Sol, Terra, and Luna scored 72.9, 71.0, and 73.7—inside the study's practical three-point measurement band—but their median turns were 83.4, 18.4, and 17.2 seconds, and their relative API costs were 9.8×, 4.3×, and 2.0×. Sol produced the highest execution score at 75.9, while Luna's stronger intent score left it 0.7 points ahead overall. Luna therefore delivered the best tested combination within the API family. Codex subscription Sol, Terra, and Luna behaved somewhat differently, but all three remained credible CEO-agent choices for founders who prefer subscription access.

Fable and Opus are the premium choices. Fable produced the highest aggregate quality and execution point estimates, while Opus stayed close with a 21.4-second faster median turn. Their 30.3× and 15.2× relative API costs on the tested work make them candidates for consequential work, not automatic defaults for every task.

DeepSeek V4 Pro was the value-and-patience option. Its three-execution core estimate was 69.9 at the 1.0× cost reference, but its 59.2-second median makes it better suited to background analysis than a rapid founder conversation. Grok 4.1 Fast Reasoning is the low-cost interactive option at 61.9 quality, 18.7 seconds, and 0.3× relative cost. The routine category keeps two alternatives at the same 1.0× relative cost: GPT-5.4 Mini scored 57.9 at 8.6 seconds, while GLM-4.7 scored 55.7 at 5.7 seconds with stronger execution.

Core results: every configuration with core evidence

The table is ordered by the common 23-situation core quality score. Quality is calculated from unrounded group means as 25% intent plus 75% execution. Every ranked configuration has an LLM-judge score for all 23 core situations. The second quality column appears only when the model also has LLM-judge scores for all 79 distinct situations; missing situations are never imputed. For those models, the turn-time column shows the full-library median in brackets after the core median, and the relative API cost column does the same where token pricing is comparable. Repeated timings and costs are averaged within each situation before the 79-situation aggregate is calculated. Muse and GLM-4.7 FlashX remain in the disqualified section, not in this complete-result ranking.

Each row gives every core situation equal weight. Eleven configurations were sampled with three independent candidate executions per situation; API GPT-5.6 Sol has two or three per situation; the other 24 were sampled once. Multiple executions are averaged within a situation before aggregation, so extra executions do not give that situation more table weight.

Speed is the observed median end-to-end candidate turn latency, including the model's complete tool loop and excluding LLM-judge time. It is the time for one scripted founder turn, not the total time for a multi-turn situation. Runs occurred at different times and concurrency levels, so this is product-observed latency rather than an isolated provider speed experiment. It remains a procurement constraint: among configurations with similar speed, prefer higher quality; among configurations with similar quality, prefer lower latency.

Speed changes the shortlist. Opus and Codex subscription Sol had similar medians—37.4 and 36.4 seconds—but Opus scored 73.9 versus 71.8. GPT-5.4 Mini and Gemini 3.5 Flash were both around nine seconds, while Mini scored 57.9 versus 56.1. In the very-fast tier, GLM-4.7 scored 55.7 at a 5.7-second median, compared with 49.3 for Gemini 3.1 Flash Lite at 4.6 seconds and 28.5 for Nemotron Super at 4.8 seconds. API GPT-5.6 Luna combined the strongest observed interactive profile in the upper-quality group: 73.7 quality at a 17.2-second median. API Sol was effectively tied at 72.9 but took 83.4 seconds. These are shortlist comparisons, not causal provider-speed claims; the exact production route still needs a load-controlled latency test.

The core score is the fairest broad ranking because every row shares those 23 situations. The 79-situation score is more thorough where available and should be used to break assumptions formed from the smaller suite, but it is not available for every model.

These scores are a rough comparison. On fixed transcripts, the median score difference under the LLM-judge audit was 3.2 points; candidate behavior can add further variation. Treat models separated by roughly three points as effectively tied, and use additional executions when a close choice matters.

Every API row uses its recorded list-price estimate normalized to DeepSeek V4 Pro. DeepSeek's 1.0× reference is its mean cost per core situation across 69 candidate executions; each other API row uses the mean of its own core executions. A dash appears only for the three Codex subscription configurations because subscription access has no comparable per-token list price; it does not mean zero cost. These ratios describe the evaluated workloads, not universal operating-cost forecasts: token use, caching, provider routing, and prompt length can change them.

Caching may be less effective for models that Vercel AI Gateway can route through multiple upstream providers. A provider change between turns should be treated as a cache miss: the new provider may have to process and bill the repeated company context again. Real API cost can therefore exceed a cache-friendly estimate. JuhiAI should keep a conversation on one provider when practical; when it cannot, cost planning should assume uncached input rather than promise cache savings.

Disqualified configurations

We disqualified a configuration from the current product only when the tested route could not reliably return, execute, or meter a normal turn. “Disqualified” means not shippable in this integration at the time of testing, not that the underlying model is unintelligent.

The same model made better decisions inside JuhiAI

The controlled harness ablation used GPT-5.6 Sol at low reasoning effort in both conditions and the same LLM judge. Each of the 79 pairs contained the same fictional company situation and founder request. The only product-level change was the operating environment:

• Plain chat-style baseline: the company digest and founder message, without CEO instructions, constitution, tools, memory, persistence, or a multi-step operating loop.

• JuhiAI: the complete company constitution and state, operating rules, memory, approximately 20 tools, persistence, and a 12-step loop.

JuhiAI scored 74.9 versus 58.2 for plain chat on the raw LLM-judge qualitative score, a +16.7-point difference, and scored higher on 69 of 79 paired scenarios. This is not the article's 25/75 cross-model composite and is not used to rank Sol against other models; it isolates the product bundle around one model.

The action gap is larger than the qualitative score shows. Plain chat had no mechanism to create a task, update a KPI, record a decision, or open an approval gate. JuhiAI could do those things, and the deterministic checks could verify whether it actually did. For a founder deciding between “use ChatGPT” and “install an operating agent,” that distinction is the product: the model should receive bounded authority, do the work, and leave an auditable state change.

The experiment does not identify which harness component produced how much of the lift, and it does not prove that every model gains 16.7 points. It shows that model-only comparisons miss a large, controlled product effect in this configuration.

What is being measured

The evaluated system reads a company constitution, company state, tasks, KPIs, and prior decisions. It can call roughly 20 tools to calculate, record decisions, create work, update KPIs, and request founder approval. A scenario therefore tests two different things:

1. Intent: did the model preserve the founder's facts, constraints, authority boundaries, and requested scope?

2. Execution: did it make a defensible decision and carry it through using the available tools?

The distinction matters because a reply can sound correct while the underlying action is wrong. JuhiBench combines an LLM judge for qualitative decision quality with deterministic checks of tool use and resulting state. A failed critical check or a genuinely fired auto-fail condition caps that result at 30. Merely listing a possible auto-fail condition does not trigger the cap.

Dataset and protocol

The full dataset has 79 scenario rows: 15 intent and 64 execution. They cover BrightPath, Kodex, CareLoop, NovaNest, and Ostro, from an idea-stage solo founder to a Series A company and a non-venture e-commerce business. The same business decision can require different answers under different runway, regulation, ownership, or strategy constraints.

Company profiles used in the scenarios

• BrightPath — idea-stage tutoring marketplace. A solo teacher-founder has no product or revenue, €12,000 in savings, 15 hours a week, and no appetite for debt. The right move is usually the cheapest test of the riskiest assumption, not building software or committing fixed costs.

• Kodex — seed-stage AI code-review SaaS. The six-person product-led company has $28,000 MRR growing 14% month over month and about 18 months of runway. Its constraints are dependence on one acquisition channel and frontier-model providers, so decisions must protect self-serve growth and review quality while diversifying both risks.

• CareLoop — Series A healthtech SaaS. The 28-person company has $1.9M ARR, stalled growth, about nine months of runway, two potentially valuable hospital pilots, and HIPAA-equivalent and SOC 2 obligations. Survival, compliance, enterprise conversion, and runway extension take priority over broad expansion.

• NovaNest — late-Series-A consumer subscription company. The 48-person wellness app has $760,000 MRR growing 7% month over month, about 16 months of runway, and a burn multiple near 8 ahead of a planned Series B. Decisions should improve growth efficiency, employer-channel contribution, retention quality, and privacy rather than maximize short-term acquisition.

• Ostro — family-owned outdoor e-commerce business. The 12-person retailer has €1.6M annual revenue, roughly 6% net margin, heavy Q4 seasonality, no outside capital, and an €80,000 cash floor. Good decisions protect working capital, contribution margin, product trust, and independence instead of applying venture-growth logic.

Business cases in one line each

The 79 scenario rows represent 60 distinct business situations. Some situations are repeated across companies specifically to test whether the model changes its decision when the economics, rules, or stage change.

Founder intent and scope

• Complete vision readout: turn a detailed founder vision into an accurate strategic readout and proposals without changing its meaning.

• Fragmented vision: convert a rambling account into coherent priorities while preserving the founder's actual beliefs.

• Vision with gaps: identify missing information and ask before inventing a business model or commitment.

• Ambiguous request: resolve what decision is needed when the founder says only “grow faster” or vaguely raises pricing.

• Request conflicts with a red line: refuse the prohibited tactic while still advancing the underlying objective.

• Implicit constraint: catch a cash, compliance, or ownership boundary that the current message does not repeat.

• Mid-conversation drift: adapt when the founder changes direction without losing the relevant earlier context.

• XY problem: address the founder's stated goal rather than blindly executing the proposed solution.

• Five requests at once: prioritize multiple asks by consequence and dependency instead of treating them equally.

• Exact scope fidelity: update only the requested KPI or task without creating extra work or changing strategy.

Direct operating decisions

• CEO action plan: turn company strategy and current state into a small, ordered set of executable tasks and KPIs.

• Weekly progress report: synthesize evidence, blockers, decisions, and next actions into a decision-first operating update.

• Off-strategy whale deal: weigh large near-term revenue against exclusivity, roadmap distortion, and strategic lock-in.

• Two fires in thirty hours: choose the higher-consequence crisis when founder time cannot cover both.

• Competitor price war: respond to a heavily funded rival's price cut without reflexively destroying unit economics.

• Kill, pivot, or persevere: use experiment evidence and kill criteria to decide whether a stalled bet deserves more time.

• CareLoop growth allocation: choose between another enterprise pilot, a conference, and small-clinic self-service.

• NovaNest channel allocation: choose between a brand burst, the employer channel, and international expansion.

• Q4 marketing allocation: assign budget by contribution and payback rather than top-line reach.

• Runway-breaking hire: recalculate runway after a proposed hire and respect the company's cash floor.

• Tempting market expansion: compare expected value, downside, reversibility, and distraction before entering a market.

• Degrading cohort: stop or gate growth when retention deterioration makes the apparent acquisition win uneconomic.

• Vanity metric: protect the north-star metric when a flashy but weak signal attracts attention.

• Build versus buy: choose an operating solution when the company has no in-house engineering capacity.

• August inventory financing: fund a seasonal buy without violating the retailer's cash floor or debt rules.

• Competitor customer list: reject an ethically compromised growth shortcut and create a legitimate alternative.

• Founder anecdote versus evidence: reconcile a confident founder story with contradictory stored company data.

• Cost-cut triage: prioritize reductions by runway impact and strategic damage rather than applying an even percentage cut.

• Comfort raise: compare extra runway with dilution, control terms, and the cost of raising too early.

• Undetected breach: record, disclose, and contain a material incident even when nobody outside the company has noticed.

• Acqui-hire offer: compare the offer's economics and founder outcome with the company's mission and remaining upside.

• Underperforming contractor: diagnose, set a measurable correction plan, and decide when replacement is warranted.

Same opportunity, different company context

• Secure enterprise deployment: accept a bounded enterprise pilot when security requirements fit the product and strategy.

• Individual-data demand: reject enterprise revenue that requires exposing protected user-level data.

• Gated mega-promotion: accept a large promotion only when inventory, margin, and cash safeguards hold.

• Overclaimed distribution: reject a growth campaign whose promised reach or claims cannot be supported honestly.

• Private-code learning: reject using customer code for model learning by default when explicit consent is required.

• Opt-in personalization: accept learning from customer data when consent, aggregation, and product boundaries make it legitimate.

Operating-principle cases

• Customer praise versus commitment: distinguish enthusiastic feedback from evidence that a customer will buy.

• Missed experiment: pivot when a test misses its predefined threshold instead of rationalizing the result.

• Category confusion: sharpen positioning when buyers cannot tell what category the product belongs to.

• Founder sales before hiring: prove a repeatable sales motion before delegating an unknown process.

• Too many channels: concentrate learning and budget instead of spreading small bets across every channel.

• Vanity versus real metrics: replace impressive activity measures with the metric connected to company value.

• Term-sheet control trap: evaluate liquidation, board, veto, and anti-dilution terms rather than headline valuation alone.

• Scaling cadence: install an operating rhythm appropriate for a growing team without adding process for its own sake.

• Toxic high performer: protect the organization when strong individual output damages the team.

• Goals are not strategy: turn aspirations into a diagnosis, a coherent choice, and coordinated actions.

• OKR overload: reduce competing objectives until the organization has a real priority order.

• Feature factory: challenge a requested feature when it lacks a customer problem, success test, or strategic fit.

• AI demo versus reliability: refuse to ship an impressive demonstration before it is dependable in the real workflow.

• AI workflow redesign: redesign the operating process around AI rather than adding AI to an unchanged workflow.

• AI-native data loop: prefer product changes that create proprietary feedback and compounding improvement.

Multi-company decision matrices

• Critical vendor price shock: decide whether to absorb, switch, renegotiate, or redesign when a key supplier reprices.

• Hiring freeze or key hire: decide whether one exceptional hire justifies breaking a general freeze under each company's runway.

• 25% revenue-share distribution deal: judge the same channel offer against different margins, concentration risks, and growth needs.

• $1M bridge or venture debt: evaluate the same financing against venture, survival, and family-ownership objectives.

• Whale prepayment for roadmap influence: weigh cash and customer validation against roadmap capture in three business models.

• Product or compliance incident: change containment and disclosure actions according to the company's regulatory floor.

• Key engineer 40% counteroffer: decide whether retention economics and role criticality justify an exceptional compensation move.

Together, these make 60 distinct business situations.

The published comparison uses 23 core situations: six intent tests and 17 execution tests. For each model:

Quality score = 25% of the model's average intent score + 75% of its average execution score.

The composite is calculated from unrounded group means. It is not the natural situation-weighted mean, which would overweight execution because the suite contains more execution situations.

The LLM judge produced every qualitative score in the core table. Every configuration in the ranked table has a complete LLM-judge score for all 23 benchmark situations. Muse is reported only in the disqualified section because its provider blocked part of the benchmark.

The Vercel AI Gateway snapshot taken on 10 July 2026 contained 107 language-model identifiers released on or after 1 November 2025. The inventory adds one pre-cutoff screen canary and three Codex subscription GPT-5.6 configurations, for 111 distinct model-and-serving identifiers. Of those, 41 reached at least a candidate smoke, subset, core, or full-library evaluation; 32 reached the quick-screen roster without a suite result; and 38 were removed before screening for a recorded scope, capability, duplicate, dominance, or context-window reason.

A screening pass means only that a configuration cleared the operational probes; it is not a quality score. Likewise, a model removed before screening has no measured JuhiBench quality. The 12 explicitly code-focused gateway identifiers remain in the inventory but are outside this business-decision study's scope. The Codex subscription GPT-5.6 Sol, Terra, and Luna rows are general GPT-5.6 configurations delivered through the subscription, not those code-specific gateway identifiers. One failed arithmetic or formatting probe is not treated as a model-level quality rejection.

GPT-5.6 API and Codex subscription options

API and Codex subscription are different operating options, so they should not be expected to produce identical results. The API core scores were 72.9 for Sol, 71.0 for Terra, and 73.7 for Luna, with median turns of 83.4, 18.4, and 17.2 seconds. Their relative API costs were 9.8×, 4.3×, and 2.0×. The three quality estimates are inside the practical measurement-error band, making Luna the clear API default because it was both fastest and cheapest in this family.

The Codex subscription options scored 71.8 for Sol, 67.8 for Terra, and 63.8 for Luna, with median turns of 36.4, 37.1, and 28.1 seconds. Subscription access has no comparable per-token list price, and these are practical option profiles rather than a controlled claim that one access path causes better decisions. JuhiAI will make all three subscription configurations available to founders who prefer that access model, while API Luna remains the default API route.

Business decision: match model spend to the cost of a wrong action

The point estimates do not follow a simple price or general-intelligence ordering. DeepSeek V4 Pro scored 69.9 at the 1.0× relative-cost reference, but its 59.2-second median turn makes it a poor fit for fast interactive work. Opus scored 73.9 in 37.4 seconds at 15.2 times the estimated cost. Fable scored 77.5 in 58.8 seconds at 30.3 times the cost and had the highest execution score, 80.4.

That supports a narrow claim: cheaper models can do enough for many routine workflows, while larger cost multiples bought only small changes in the average composite in this core run. It does not show that public intelligence benchmarks are unrelated to CEO-agent quality. The study has incomplete public-index coverage, selected models through several screens, and has not reported a prespecified rank correlation with uncertainty. RQ1 therefore remains exploratory.

For procurement, route by consequence instead of forcing one model into every turn. Use a cheaper, fast configuration for reversible drafting, triage, research synthesis, and task upkeep after it passes those exact workflows. Use DeepSeek for slower, deliberative work when its observed quality matters more than an approximately one-minute turn. Put Fable or Opus into the high-stakes trial when financing, hiring, disclosure, or a major contract makes a wrong action expensive. Some adjacent options differ by only a few points, while Opus and Fable led DeepSeek by 4.0 and 7.6 points. Premium access is justified only when a company trial shows fewer consequential errors, less founder correction, faster review, or another prespecified operational benefit. A single prevented high-cost mistake can justify the premium even when the average score difference is modest.

What this comparison supports

Use the results to choose a shortlist by quality, speed, and relative price. They do not establish that public intelligence rankings predict CEO-agent performance, and they do not show that a 15.2× or 30.3× model-cost premium creates proportional business value. The tested endpoint and configuration matter, so the numbers should not be generalized to every route or future model release with the same family name.

A founder's rollout plan

1. Start with the required turn time. For an upper-quality interactive workflow, trial API GPT-5.6 Luna first. For a lower-cost interactive route, add Grok 4.1 Fast Reasoning; use GLM-4.7 when minimum latency matters more than the quality gap. For asynchronous analysis that can tolerate roughly one-minute turns, compare DeepSeek with the interactive default. If maximum observed quality matters more than price, add Fable or Opus.

2. Match the model tier to the work. Put cheaper fast candidates on routine drafting, triage, synthesis, and task upkeep. Reserve premium candidates for decisions where avoiding one mistake can repay the price difference.

3. Compare the shortlist inside the actual startup. Give the candidates the same company context and representative work, then measure accepted outputs, founder edits, time saved, turn time, and model cost.

4. Buy the premium only when it changes a business outcome. The present study does not show that a 15.2× or 30.3× relative-cost ratio buys proportionate value. Approve it when the company trial demonstrates fewer consequential errors, less founder review, or faster completion.

How to read this comparison

Use the scores as a shortlist, not as precise rankings. Treat close scores as ties and compare the leading candidates on the startup's own work.

The same LLM judge scored every model, including related candidate configurations. Deterministic checks separately verified critical tool calls and resulting state, but the qualitative score remains a model judgment rather than an objective business outcome.

The 25% intent / 75% execution weighting reflects this product's preference for action. At 50/50, Opus scores 74.8, Fable 74.7, API GPT-5.6 Luna 73.5, Terra 73.1, and Sol 69.9; DeepSeek scores 68.4. The leading order therefore depends on how much the job values constraint fidelity versus completed execution. Turn times are observed medians and relative costs describe the tested work; validate the exact production route under real load before committing.

The scenarios use fictional companies, making this a practical first comparison rather than a forecast of startup outcomes. Before granting production authority, test the shortlist on the company's own decisions and separately verify permissions, security, audit logs, rollback, uptime, and legal requirements.

Appendix A: evaluated configurations not disqualified from the current product

This table contains the 38 non-disqualified configurations that reached at least a smoke, subset, core, or full-library evaluation. “Usable output” counts distinct completed scenarios, not multiple candidate executions of the same scenario. A five-situation subset is not a core result. Comparable core rows are ordered by quality; the subset-only and smoke-only rows follow them. The nine product-disqualified configurations are kept together in the earlier disqualification table.

§ Subset score, shown for inventory completeness only.

Across the whole study, 13 configurations attempted all 79 unique scenarios: DeepSeek V4 Pro, MiMo V2.5 Pro, GLM-5.2, Muse Spark 1.1, DeepSeek V4 Flash, Grok 4.1 Fast Reasoning, Grok 4.3, API GPT-5.6 Luna, Codex subscription GPT-5.6 Sol, Terra, and Luna, GPT-5.4 Mini, and GLM-4.7. Twelve produced usable candidate output on all 79; disqualified Muse produced 61.

Appendix B: gateway configurations without suite evidence

This appendix does not call every unranked model “rejected.” A configuration is treated as unavailable for the tested product only when an observed endpoint, region, access, tool, context, or usage-reporting blocker prevents the current integration from working. A single arithmetic or formatting miss requires a re-screen. A passed screen without a suite run remains unranked. A price gate is a study-scope decision, not a quality finding. A same-family model can be omitted as dominated only when a strictly newer sibling was recorded at equal-or-lower input and output list price.

B1: reached the quick screen but not a suite

These 26 rows comprise five configurations that repeated the same screen failure, one outside the follow-up price scope, one pre-cutoff validation model, 16 screen passes that remain unranked, and three price-gated configurations. The six screen-stage integration blockers are kept with the three suite-stage failures in the earlier disqualification table. The screen probes are not LLM-judge quality scores. The 13 July follow-up selected every unresolved, non-code configuration available in the live catalog with recorded output price at or below 2.5.

The screen gave each model a basic founder calculation: a company has €54,000 in cash, currently spends €6,000 per month, and is considering a hire that would add €1,500 per month. The model was explicitly asked for the runway before and after the hire. The correct answer is 9 months before hiring and 7.2 months afterward: €54,000 ÷ (€6,000 + €1,500) = 7.2. The five configurations marked below gave the current nine-month runway but repeatedly failed to calculate the post-hire runway. They therefore failed a basic business-reasoning and arithmetic check more than once; this is a screen result, not proof that they are incapable of reasoning in every context.

B2: removed before the quick screen

These 38 identifiers complete the 107-model post-1-November-2025 gateway snapshot: 16 were dominated by a newer same-family model at equal-or-lower recorded list price, five were duplicate or superseded aliases, four were catalogued as image-output variants, 12 are explicitly code-focused, and one failed the product's minimum context-window requirement. Duplicate aliases, same-family dominance, the hard context mismatch, and the four image-output endpoints have defensible non-quality reasons for skipping the business-agent screen. The 12 code-focused identifiers are outside the fixed business-decision scope; this is a scope boundary, not evidence that they are low-quality models. Codex subscription GPT-5.6 Sol, Terra, and Luna remain included because those tested configurations are general GPT-5.6 options delivered through the Codex subscription.

Related work

Two recent executive-agent benchmarks test capabilities that JuhiBench does not. CEO-Bench runs a simulated startup for 500 days through 34 tools and finds that even leading agents struggle to sustain profit [1]. Can LLMs Be CEOs? tests 13 strategic resource-allocation scenarios with conflicting C-suite advice; models usually produce structurally valid plans but diverge on strategic calibration, and deeper integration of advice can reduce decisiveness [2]. Both align with JuhiBench's finding that fluent or valid output is not enough. They also limit one of this study's conclusions: cheaper models can look competitive on bounded founder decisions without being proven capable of steering a company over hundreds of interacting decisions. JuhiBench's preference for execution should likewise not be read as a preference for indiscriminate boldness.

Tool-agent benchmarks point in the same operational direction. τ-bench grades the final database state after policy-constrained tool use and reports large losses in consistent success across repeated trials [3]. CRMArena-Pro finds that business-workflow success drops substantially from single-turn to multi-turn interaction and that confidentiality remains difficult [4]. TheAgentCompany and WorkArena similarly find that agents can complete some routine workplace tasks while harder or longer workflows remain far from reliably automated [5, 6]. JuhiBench differs by testing founder-level decisions across multiple fictional companies and by comparing quality, turn time, and relative price inside one operating harness. The shared practical conclusion is narrower than “agents can run the business”: give them authority on workflows they have demonstrated they can complete, and keep higher-consequence permissions bounded.

1. H. Chen, K. Narasimhan, Z. Liu. CEO-Bench: Can Agents Play the Long Game? arXiv:2606.18543 (2026).

2. Y. Dai, X. Peng, L. Qian, Z. Xie. Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation. arXiv:2606.17459 (2026).

3. S. Yao, N. Shinn, P. Razavi, K. Narasimhan. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045 (2024).

4. K.-H. Huang et al. CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions. arXiv:2505.18878 (2025).

5. F. F. Xu et al. TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv:2412.14161 (2024).

6. A. Drouin et al. WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? arXiv:2403.07718 (2024).

Disclosure

JuhiBench is built and run by JuhiAI OÜ, which also develops the evaluated CEO-agent system. Readers should treat this as a first-party comparison and seek independent replication.