Paper review · November 10, 2026
LongBanana: An Expert-Verified Benchmark for Long-Context Multi-Reference Image Composition
LongBanana: An Expert-Verified Benchmark for Long-Context Multi-Reference Image Synthesis
The authors built 90 tasks that require gathering clues scattered across about 32 reference images to produce a single image, along with graph-based checklist evaluation, and used them to diagnose agentic image generation systems.
Problem
Real image generation requests require selecting what is needed from many reference images and excluding similar distractor images. Attributes and relations must also be bound to the correct subjects and composed into one image. Existing benchmarks focus on editing, personalization and a small number of references, so they do not measure this ability to use evidence. Provenance errors, where a plausible-looking but wrong reference is used, also do not show up in a single overall score.
Method
An LLM structures expert-designed scenarios into entity-relation graphs. The graph's node, edge and whole-graph constraints are converted into atomic checklists, each scored on three dimensions: Content Fidelity, Relational Binding and Global Constraint Satisfaction. The 90 tasks follow a 3×3 taxonomy (scene composition / appearance / reference reasoning). At least 30 tasks include hard negative references. The grader, LongBananaEval, gathers evidence for each item and sends it to a dimension-specific critic. The critic calls tools for OCR, counting, layout checks and identity verification as needed, and records a score of 0/1/2 with a rationale. On the generation side, LongBanana-Gen compares direct generation, minimal agent harnesses (Hermes-Agent, OpenCode, OpenClaw), and the RPR (ReAct–Plan–Exec–Reflect) workflow.
Results
- The benchmark contains 90 tasks and 4,670 checklist items. Each task has on average about 32 reference images, about 20 constraints and about 50 evaluation items.
- Of 46,700 human labels on 900 generated outputs, 15% (7,005) were labeled independently by two annotators. Exact agreement was 89.4%, and quadratic weighted kappa was 0.83.
- The authors report that with Nano Banana 2 fixed, minimal agent harnesses had a better overall score profile than direct generation.
- The effect of RPR depended on the harness. It helped only with harnesses that reliably used retrieved evidence and revision feedback; otherwise, results drifted from the original intent. The top scores per subcategory were also spread across several approaches, so no approach dominated.
Limitations
The number of tasks is small at 90 (10 per subcategory). Only one agent model, Claude Sonnet 4.6, and three image backends were compared. An LLM is involved in graph construction, and scenarios focus on storyboard, product and scene composition cases. The provided text is cut off partway through the experiments, so agreement between the automatic grader and human labels and per-system detailed scores could not be checked.
Why it matters
In generation tasks that must accurately reflect many references, such as advertising and product images, "was the right reference used in the right place" matters more than plausibility. This benchmark provides an auditable evaluation framework that pinpoints which item an error occurred in. It can serve as a shared target when developing agentic image generation systems.
Key terms
- Multi-reference image composition
- A task that takes the appearance and attributes of subjects from multiple reference images and composes them into a single new image.
- Entity-relation graph-based checklist
- An evaluation sheet that breaks down the subjects that must appear in an image (nodes), the relations between them (edges), and whole-scene constraints into small, checkable items. It makes it possible to pinpoint exactly where a failure occurred.
- Hard negative references
- Distractor images that look relevant on the surface but should not be used. They are used to evaluate whether a model filters them out.
- Agent harness
- An agent execution framework that wraps an image generator and handles reference state management, evidence selection, prompt construction and revision.
Original
Authors: Haoxiang Cao, Yuxuan Zhang, Penghui Du, Bo Li, Jiajiong Cao, Yingying Fan, Zimeng Wu, Yifu Luo, et al. · Posted 2026-11-04 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.