Benchmark · 1,613 questions · 30 projects · 10 multimodal LLMs

MMEnterprise-Bench: Evaluating Enterprise Agents over Evolving Multimodal Records

Seohyun Lee^1,∗ Jongha Kim^2,∗ Dohwan Ko² Ji Soo Lee¹ Seoung Choi¹ Jayun Ku¹ Hyunwoo J. Kim^1,†

¹ KAIST ² Korea University

∗ Equal contribution † Corresponding author

Paper Code Dataset coming soon Leaderboard coming soon

News

Oct 21, 2026 Preprint released on arXiv (2610.27911).
Coming soon Dataset and evaluation toolkit, full generation pipeline code, public leaderboard.

Overview

Abstract

Teaser: a cross-source conflict where Notion, Slack and a PDF table give three different fund sizes (27.8bn, 28.7bn, 27.3), and a temporal conflict where a briefing deadline is revised from Sep 1 18:00 to Sep 1 20:00 and later to Sep 2 09:00; an agent must decide which value applies. — **Conflicts across evolving multimodal records.** Enterprise agents struggle with **cross-source conflicts** (top), in which different sources contain inconsistent versions of the same information, and **temporal conflicts** (bottom), in which records undergo multiple revisions over time.

Enterprise agents answer questions over workplace records that evolve through team collaboration. As information is shared, corrected, and revised across documents, chats, and figures, multiple versions of the same information accumulate in the workspace. Reliable question answering therefore requires agents to reason over sufficient evidence, resolve cross-source and temporal conflicts, and recognize when the available evidence is insufficient to establish an answer. To systematically evaluate these capabilities, we introduce MMEnterprise-Bench, a benchmark comprising 1,613 questions grounded in evolving multimodal records from simulated collaboration projects, covering three categories: Evidence Reasoning, Conflict Resolution, and Unanswerability. We evaluate 10 proprietary and open-source multimodal LLMs and find that even the strongest models struggle when evidence contains cross-source or temporal conflicts or spans multiple images. Moreover, increasing the retrieval budget or employing a larger retriever improves evidence recall but does not consistently translate into higher answer accuracy. These findings highlight the persistent challenges of reasoning over evolving workplace records and establish MMEnterprise-Bench as a testbed for developing more reliable enterprise agents.

At a glance

What’s inside

1,613

questions

1,526 answerable · 87 unanswerable

projects

seeded from public papers & reports

7,140

retrieval records

one shared workspace corpus

1,415

visual records

PDF pages, slides, images

task types

in 3 categories

conflict types

cross-source & temporal

Evidence Reasoning

Answer from sufficient, non-conflicting evidence spread across chats, emails, meeting notes, trackers, reports, and slides: extraction, attribution, computation, comparison, and interpretation.

Conflict Resolution

Earlier records are retained rather than overwritten, so superseded values stay in the corpus. Agents must pick the value that applies across sources, or recover the state at a specific point in time.

UNS

Unanswerability

Questions built by perturbing one entity or condition of an answerable question, so that no record in the project supports an answer. The agent has to say so instead of guessing.

Task taxonomy

Three categories, eight task types

Evidence Reasoning (ER) derives answers from sufficient evidence, Conflict Resolution (CR) resolves conflicting evidence, and Unanswerability (UNS) identifies questions that cannot be answered from the records.

Category	Task	Description	Example question	#
Evidence Reasoningsufficient evidence1,102 questions	Extraction	Extract entities or values explicitly stated in text, tables, or figures.	Q: Which model does BLiM use?	268
	Attribution	Identify the source, requester, author, or communicator of information.	Q: Who sent Nora Bell’s welcome email?	80
	Computation	Apply question-specified calculations to recorded values.	Q: How much does the deficit-to-GDP share increase?	310
	Comparison	Compare results or properties to identify differences, rankings, or choices.	Q: Which frame setting fits the latency budget?	212
	Interpretation	Draw conclusions or assess claims using recorded evidence.	Q: Do the counts support a 12-risk increase?	232
Conflict Resolutionconflicting evidence424 questions	Cross-source	Identify conflicting information across records or determine which value applies.	Q: Which action-log deadline applies after the correction?	212
Conflict Resolutionconflicting evidence424 questions	Temporal	Identify information at a specified time or trace its revisions.	Q: What was the deadline at 14:00?	212
Unanswerabilityinsufficient evidence87 questions	Unanswerability	Identify missing information or unsupported premises.	Q: What is the measured serving cost?	87

Scroll horizontally to see all columns

Example questions are taken from Table 1 of the paper. The last column gives the number of questions per task (1,613 in total).

Benchmark construction

How the benchmark is built

Construction pipeline: seed documents provide key facts and visual assets for a project; organizational context and event planning generate chats and artifacts into a shared workspace of team chat, email, database, documents and slides; a rule-based and LLM-based QA generator with task-specific instructions and quality control produces questions with gold evidence and answers. — **Construction pipeline.** Each seed document serves as the information source for a project. Based on the extracted information and the project’s context, collaborative events and their corresponding records are generated over time. The projects collectively form a shared workspace of evolving multimodal records that contain cross-source and temporal conflicts, over which questions are generated and validated.

Workspace construction

Text, tables, and figures are extracted from 30 public papers and reports. For each project, a team with roles, channels, access relationships, and deadlines is specified, and an LLM plans collaborative events (result sharing, error detection, corrections, document revisions) in chronological order. Events are realized as messages, emails, meeting notes, trackers, reports, and slides; earlier versions are kept.

QA generation

Rule-based generation covers questions derivable from structured project information; an LLM with task-specific instructions generates the rest, jointly producing the question, reference answer, step-by-step solution, and supporting records. Unanswerable questions perturb one entity or condition of an answerable question so that no project record supports an answer.

Quality control & human validation

Independent LLM-based checks verify records against the source and planned events, and verify that the cited evidence derives the reference answer, flagging ambiguity, wrong answers, and leakage. In a blind audit of 100 annotations by three annotators, 71% were rejected by no one and 29% by at least one; none of the 50 LLM-accepted items was rejected by any annotator, and all 29 human-rejected items had also been rejected by the LLM.

Evaluation protocol. For each question, a Qwen3-VL-Embedding retriever (2B by default; 8B also released) ranks the 7,140 workspace records; the top-k records are presented to the model ordered by timestamp and ID, each as its text followed by its image when it has one. An LLM judge (GPT-5.6 Luna) compares the response with the reference answer, and accuracy is reported over all 1,613 questions. Both retriever rankings and the judge prompt are released, so results are reproducible without re-running retrieval.

Experiments

Main results

Accuracy (%) of 10 multimodal LLMs on all 1,613 questions with the Qwen3-VL-Embedding-2B retriever. Best result per column in bold. Attr. = Attribution, Comput. = Computation, Compar. = Comparison, Interp. = Interpretation, UNS = Unanswerability.

Model	Evidence Reasoning (ER)					Conflict Resolution (CR)		UNS	Overall
Model	Extraction	Attr.	Comput.	Compar.	Interp.	Cross-source	Temporal	UNS	Overall
# questions	268	80	310	212	232	212	212	87	1,613
Proprietary models
GPT-5.6 Sol	67.5	61.3	58.1	47.2	58.6	36.3	32.5	65.5	52.6
GPT-5.6 Luna	66.4	57.5	55.5	44.8	58.2	35.4	33.5	63.2	51.3
Gemini 3.8 Flash	67.9	52.5	54.2	44.3	34.5	28.8	30.2	87.4	47.6
Gemini 3.1 Pro	66.8	58.8	54.8	42.5	30.6	27.4	28.8	74.7	45.9
Open-source models
Qwen3.5-27B	63.1	52.5	37.7	33.5	28.9	22.6	20.3	75.9	38.6
Qwen3.5-9B	60.1	40.0	36.5	25.9	27.2	14.2	21.2	73.6	34.9
Qwen3.5-4B	62.7	40.0	37.4	31.1	28.4	19.8	19.8	55.2	36.0
Qwen3.5-2B	58.6	20.0	21.9	17.9	12.9	12.7	13.2	19.5	23.6
InternVL3.5-8B	63.4	51.3	31.9	23.1	27.2	15.6	19.3	36.8	32.7
Kimi-VL-16B-A3B-Instruct	56.3	40.0	14.5	16.0	16.4	11.8	14.2	10.3	22.6

Model	Evidence Reasoning (ER)					Conflict Resolution (CR)		UNS	Overall
Model	Extraction	Attr.	Comput.	Compar.	Interp.	Cross-source	Temporal	UNS	Overall
# questions	268	80	310	212	232	212	212	87	1,613
Proprietary models
GPT-5.6 Sol	77.6	68.8	71.6	67.5	73.3	55.2	56.1	72.4	68.0
GPT-5.6 Luna	76.5	65.0	69.4	63.7	72.4	52.8	54.2	67.8	65.8
Gemini 3.8 Flash	78.0	71.3	71.6	65.6	53.4	46.7	54.2	74.7	63.9
Gemini 3.1 Pro	76.9	72.5	69.0	62.7	49.6	49.1	50.9	80.5	62.5
Open-source models
Qwen3.5-27B	73.9	56.3	50.3	43.9	42.7	35.8	38.7	71.3	50.3
Qwen3.5-9B	71.3	38.8	43.9	37.7	32.8	26.4	33.0	63.2	43.1
Qwen3.5-4B	69.8	43.8	48.7	42.9	35.3	26.4	30.2	47.1	43.8
Qwen3.5-2B	60.8	25.0	16.5	18.4	18.1	19.3	16.0	14.9	25.0
InternVL3.5-8B	66.4	53.8	35.5	28.8	31.0	24.1	23.1	29.9	36.6
Kimi-VL-16B-A3B-Instruct	57.5	35.0	12.6	19.3	19.4	15.6	9.9	13.8	23.1

Scroll horizontally to see all columns

The strongest model, GPT-5.6 Sol, reaches only 52.6% (k = 10) and 68.0% (k = 50). Errors concentrate in Conflict Resolution: at k = 10 it scores 67.5% on Extraction but 36.3% on cross-source and 32.5% on temporal conflicts. The best open-source model, Qwen3.5-27B, reaches 50.3% at k = 50.

Analysis

Where enterprise agents fail

Conflicts are the hard partAcross models, cross-source accuracy ranges from 11.8% to 36.3% and temporal accuracy from 13.2% to 33.5% at k = 10, far below extraction from a single record.
Multiple images hurt mostGPT-5.6 Sol drops from 83.3% on single-text-record questions to 52.9% with one image and 27.5% with two or more images.
Recall is not accuracyRaising k from 50 to 100 lifts recall from 57.0% to 64.8%, yet Qwen3.5-2B falls from 25.0% to 22.2%; the 8B retriever adds recall but not consistently accuracy.

Accuracy by number and modality of gold evidence records

Model	Text		Image		Text + Image
Model	1	2+	1	2+	2+
# questions	324	369	425	138	270
Proprietary models
GPT-5.6 Sol	83.3	51.5	52.9	27.5	25.6
GPT-5.6 Luna	81.5	49.1	52.2	25.4	25.9
Gemini 3.8 Flash	78.7	40.9	49.4	18.8	18.1
Gemini 3.1 Pro	78.4	41.5	47.3	15.2	17.4
Open-source models
Qwen3.5-27B	67.0	34.7	36.7	13.8	13.7
Qwen3.5-9B	65.4	28.7	32.5	10.1	10.7
Qwen3.5-4B	69.8	30.4	33.9	10.9	13.0
Qwen3.5-2B	53.1	19.5	22.1	6.5	6.3
InternVL3.5-8B	63.6	30.4	28.0	13.8	14.8
Kimi-VL-16B-A3B-Instruct	55.2	18.4	16.2	11.6	8.5

Scroll horizontally to see all columns

2B retriever, k = 10. Columns group answerable questions by the modality of their gold evidence (text only, image only, or mixed) and by how many records are required (1 vs. 2+). Every model performs worse when multiple records are needed, and worse again when those records are visual.

Scaling the retrieval budget and the retriever

Model	k = 10		k = 50		k = 100
Model	2B	8B	2B	8B	2B	8B
Evidence Recall@k	41.3	43.0	57.0	59.6	64.8	67.3
Proprietary models
GPT-5.6 Sol	52.6	53.8	68.0	70.0	75.9	77.2
GPT-5.6 Luna	51.3	52.2	65.8	66.9	69.9	71.2
Gemini 3.8 Flash	47.6	47.7	63.9	66.0	72.5	72.5
Gemini 3.1 Pro	45.9	47.7	62.5	63.1	68.7	70.3
Open-source models
Qwen3.5-27B	38.6	38.3	50.3	51.8	56.2	56.5
Qwen3.5-9B	34.9	36.0	43.1	43.8	47.1	47.6
Qwen3.5-4B	36.0	36.8	43.8	45.7	46.9	46.6
Qwen3.5-2B	23.6	23.5	25.0	23.7	22.2	22.4
InternVL3.5-8B	32.7	34.7	36.6	38.0	36.2	37.6
Kimi-VL-16B-A3B-Instruct	22.6	23.3	23.1	23.1	23.6	22.9

Scroll horizontally to see all columns

Overall accuracy (%) for k ∈ {10, 50, 100} with the Qwen3-VL-Embedding-2B and -8B retrievers. The first row reports Evidence Recall@k (macro average over the 1,526 answerable questions). More retrieved records consistently raise recall, but for smaller models higher recall alone does not ensure better answers.

Qualitative examples

What the questions look like

Two failure cases. Left: an interview count is corrected from 43 to 34 in a revised memo that also adds a new row with value 23; asked for the value in the new row, the model answers 34 instead of 23. Right: a deadline is revised from Sep 1 20:00 to Sep 2 18:00; asked for the deadline before the revision, the model returns the revised Sep 2 18:00. — **Representative failure cases of GPT-5.6 Sol.** **Left, cross-source conflict:** corrections flow through Slack and a revised memo while a Notion table lags behind; the question targets the newly added row (23), but the model returns the corrected interview count (34). **Right, temporal conflict:** a deadline moves from Sep 1 20:00 to Sep 2 18:00 with both values preserved; the model answers the current deadline correctly but returns the revised value when asked for the deadline before the revision.

One example panel per task type from the paper appendix, plus a sample of the simulated Slack workspace. Click to enlarge.

CRCross-source conflict

Qualitative example panel: Temporal conflict

CRTemporal conflict

Qualitative example panel: Unanswerability

UNSUnanswerability

Qualitative example panel: Simulated Slack workspace

RecordsSimulated Slack workspace