Knowledge Graphs as Files:
An Agent Harness Where Tool Creation and Tool Use Co-Evolve

Yuyang Bai¹, Yu Wang², Yu Zhang¹

¹Texas A&M University, ²University of Oregon

Figure 1: a predefined neighbors tool versus DKGI on a STaRK-MAG request

Figure 1: An example from the STaRK-MAG dataset: “Papers on data quality enhancement algorithms affiliated with the University of the Balearic Islands”. (a) A neighbors call can only return 20 (out of 66) researchers affiliated with the university. The agent never encounters the gold paper in this run. (b) In our DKGI, tool creation and tool use co-evolve: execution feedback guides how the agent revises and extends its code, and the revised code gathers further evidence, ultimately isolating the gold paper. Bracketed numbers denote call indices.

Abstract

Searching text-rich knowledge graphs requires following relations while examining textual evidence. Predefined graph interfaces constrain what evidence agents can access, even when they write code to combine tool calls or process retrieved subgraphs. We propose direct knowledge graph interaction (DKGI), a harness in which language model agents create graph access tools directly over complete node and edge files. The agent writes Python code to follow relations, combine node sets, and select textual evidence, interleaving execution with shell search and file inspection. This lets agents process large node sets in bulk while returning selected text for the model to assess. Execution feedback guides tool creation and revision, while the evolving tools determine what evidence the agent gathers next. On benchmark datasets spanning three domains, training-free DKGI exceeds the strongest published baseline by at least 6.7 Hit@1 points on every graph, even with a minimal harness. Controlled ablations show the benefits of multi-turn programming over shell-only access and predefined graph tools, while trajectories show agents revising code to extend searches or address errors. These findings highlight the value of allowing tool creation and tool use to co-evolve during graph search.

Three Settings of Graph Access

Existing methods typically access the graph through specialized interfaces: database query languages, retrievers that return reasoning paths or subgraphs, and tools for entity search, node lookup, and neighbor expansion. Some agents write code that combines predefined graph tool calls, while others generate code to traverse graph objects provided by a dedicated subgraph retriever. Across these designs, access to the underlying graph remains governed by a predefined interface. DKGI exposes textual node records and typed graph relations as files, together with the graph schema, and lets the agent implement graph access directly over the complete node and edge files.

	Predefined	Written by the agent
Setting	graph API / retriever	code	revisions	direct file access
① Direct tool use ToG, Graph-CoT, ARK, GRASP; Graph tools	✓	–	–	–
② Code over a graph interface semantic parsing, KG-Agent, Code-on-Graph, AvaTaR; Graph tools + code	✓	✓	(✓)	–
③ Direct knowledge graph interaction Full DKGI	–	✓	✓	✓

Table 1: Graph access and agent-written code across three settings. Direct file access lets agent-written code access all node and edge files without a predefined graph API or subgraph retriever. “(✓)” indicates support in some systems. Italicized entries denote our controlled variants.

On the same request, DKGI writes and revises graph access code (tool creation) and executes it to gather evidence (tool use). Execution feedback guides which access operations to create or refine. The resulting code, in turn, changes how the agent explores the graph and what evidence it obtains. Tool creation and tool use thus co-evolve: the agent develops its tools as it searches, and its evolving tools guide the subsequent search.

Main Results on STaRK

We evaluate DKGI as a training-free method on the three knowledge graphs of the STaRK benchmark (Amazon, MAG and Prime), in the minimal Pi harness and in the more feature-rich Codex CLI. Even in Pi, our best DKGI configuration exceeds the strongest published baseline (GRASP) by at least 6.7 Hit@1 points on every dataset.

	Amazon			MAG			Prime
Method	H@1	R@20	MRR	H@1	R@20	MRR	H@1	R@20	MRR
Trained or tuned on STaRK
AvaTaR	49.9	60.6	58.7	44.4	50.6	51.2	18.4	39.3	26.7
PASemiQA	45.9	–	55.7	43.2	–	50.2	29.7	–	31.0
mFAR (All+2)	53.0	66.3	64.3	55.9	74.1	64.3	40.0	72.6	52.0
MoR	52.2	59.9	62.2	58.2	75.0	67.1	36.4	63.5	46.9
GraphRAFT	–	–	–	69.6	89.1	76.2	63.7	76.4	69.0
GRASP	71.2	70.6	78.7	82.8	91.5	88.3	67.8	88.1	76.8
Training-free
BM25	44.9	53.8	55.3	25.9	45.7	34.9	12.8	31.3	19.8
ColBERTv2	46.1	53.4	55.5	31.2	43.9	38.4	11.8	25.0	17.4
GritLM-7B	42.1	56.5	53.5	37.9	46.4	47.3	15.6	39.1	24.1
KAR	54.2	57.2	61.3	50.5	60.3	57.5	30.4	50.8	39.2
HybGRAG	–	–	–	65.4	65.7	69.8	28.6	43.6	34.5
AF-Retriever^†	61.2	35.5	67.3	78.6	61.4	84.0	46.2	51.2	54.0
ARK	56.1	58.6	65.7	73.1	86.2	81.0	50.6	70.0	59.6
KG reasoning methods adapted to STaRK by Yang et al. (2025), with GPT-4^‡
ToG	16.7	–	21.2	5.4	–	6.1	13.2	–	15.8
RoG	31.2	–	36.0	32.9	–	38.7	22.7	–	25.2
GoG	21.0	–	26.3	41.3	–	47.8	18.9	–	21.1
Ours: direct knowledge graph interaction (training-free)
DKGI (Pi, GPT-6-Luna)	57.8	58.1	64.7	88.0	88.4	89.8	63.8	78.9	70.4
DKGI (Codex, GPT-6-Luna)	59.8	55.2	66.2	88.0	88.1	89.6	66.6	80.9	72.4
DKGI (Pi, GPT-6-Astra)	79.4	71.1	84.0	91.8	93.2	93.2	74.5	86.1	79.1

Table 2: H@1, R@20 and MRR on the synthetic test sets of STaRK. Bold marks the best value in each column. Baseline numbers are published values (backbones in the paper's appendix), and a dash marks a value absent from the source paper. ^†R@20 from the complete-pipeline row of its ablation table. ^‡From the H@1 and MRR tables of Yang et al. (2025). The DKGI rows cover the full test sets (7,108 queries) and share the prompt and the time limit.

Ablation Study

We fix the model (GPT-6-Luna), the harness (Pi), the budgets and the prompt apart from its tool section, and vary which actions the agent can take. A 2×2 design crosses graph access (the graph files or predefined graph tools based on ARK) with Python. All variants run on STaRK's test subset (710 queries).

			H@1					Overall
Variant	Graph access	Python	Amazon	MAG	Prime	Overall	Δ	R@20	MRR
Full DKGI	node and edge files	✓	59.1	91.0	69.6	75.2	–	78.0	78.9
Shell only	node and edge files	–	48.8	86.1	62.9	68.3	−6.9	69.5	72.0
Graph tools + code	predefined graph tools	✓	60.4	90.2	67.1	74.2	−1.0	79.4	78.9
Graph tools	predefined graph tools	–	59.8	87.6	66.4	72.8	−2.4	79.5	77.6
One program	node and edge files	once	48.2	87.2	63.9	69.0	−6.2	70.8	72.5
Retrieval only	`search`, `get_node`, `get_schema`	–	58.5	48.5	40.7	47.7	−27.5	55.1	53.9
No edges	node files	✓	49.4	45.5	40.0	44.2	−31.0	52.6	50.6
Full DKGI (Codex)	node and edge files	✓	58.5	87.6	67.5	73.0	−2.3	75.5	76.9

Table 3: Ablation on the test subset (710 queries) with GPT-6-Luna in Pi, unless marked Codex. Python: ✓ over several turns, “once” for a single program per query, and – for no Python access. The first four rows form the 2×2 design, the fifth row limits Python to one program, and the last three rows are controls. Δ: overall H@1 minus that of Full DKGI.

With Python, the agent outperforms predefined graph tools in overall H@1. With the shell but no Python, the agent scores 68.3 overall H@1, below Graph tools (72.8). With Python, Full DKGI reaches 75.2, 2.4 points above Graph tools.

Python improves the agent far more than it improves predefined graph tools. On the graph files, adding Python improves overall H@1 by 6.9 points. With the graph tools, adding Python improves overall H@1 by 1.4 points.

The agent gains most from Python when it writes and executes programs over several turns. One program reaches 69.0 overall H@1 and recovers about a tenth of the 6.9-point gain from Python on the graph files.

The agent relies on the relations in the graph. No edges reaches 44.2 overall H@1, 31.0 points below Full DKGI. The closed-book ID probe, in which the model answers from its own knowledge, shows little recall of benchmark IDs.

How Tool Creation and Tool Use Co-Evolve

Figure 2: behavior of Full DKGI on the full test sets

Figure 2: Behavior of Full DKGI (Pi, GPT-6-Luna) on the full test sets. (a) Python commands, rg searches and other calls (mostly read) by position in the trajectory. (b) Programs by operation type (types overlap). The blue types have counterparts among predefined graph tools. (c) Error types of incorrectly answered queries.

The agent searches first and then writes programs. A trajectory starts with file reads and rg searches, and Python makes up 83.3% of the calls in its last 20%. The agent writes 4.0 programs per query on average.

Programs reuse IDs from earlier outputs and code from earlier programs. Of all programs, 48.4% contain a node ID shown in the output of an earlier turn. After a program fails, the next program revises it in 88.6% of cases, and after a program runs successfully, the next program revises it in 49.7% of cases.

The agent's lead over predefined graph tools is largest on queries where both tool-based agents make many calls. On the 100 test-subset queries for which both Graph tools and Graph tools + code call neighbors at least ten times, Full DKGI leads them by 9.0 and 6.0 H@1 points.

With a stronger LLM, the agent reuses earlier outputs more often and revises more of its failed programs. With GPT-6-Astra, 59.9% of its programs contain an ID from an earlier output, and the next program revises a failed one in 92.2% of cases.

Code and Data

Resource	Content	License
leopoldwhite/DKGI	code that exports the STaRK graphs to files, runs every configuration and computes the paper's numbers; exact prompt texts	MIT
yuyangbai/DKGI-STaRK-Graphs	the exported node and edge files, questions and labels of the three STaRK graphs	CC BY 4.0
yuyangbai/DKGI-Trajectories	for every test configuration and question: the agent's trajectory, its answer and its scores	CC BY 4.0

BibTeX

@misc{bai2026dkgi,
  title  = {Knowledge Graphs as Files: An Agent Harness Where Tool Creation and Tool Use Co-Evolve},
  author = {Bai, Yuyang and Wang, Yu and Zhang, Yu},
  year   = {2026}
}

Knowledge Graphs as Files:An Agent Harness Where Tool Creation and Tool Use Co-Evolve