Benchmarking Large Language Models for Geolocating Colonial Virginia Land Grants
Understanding the historical landscape of Virginia during the seventeenth and eighteenth centuries often hinges on navigating a myriad of land patents that exist primarily as narrative descriptions. These patents contain metes-and-bounds data that, while rich in historical context, pose significant challenges for spatial analysis. Enter the world of large language models (LLMs), which offer promising pathways for translating these intricate prose abstracts into actionable geographic coordinates.
The Need for Accurate Geolocation
Virginia’s colonial land records—comprising 5,471 patents from 1695 to 1732—are crucial for numerous fields, including history, geography, and archiving. However, the prose nature of these descriptions often leaves scholars, researchers, and genealogists at a crossroads. Traditional geospatial techniques may fall short, hampered by the lack of standardized coordinate data. This study reveals how LLMs can step in to bridge this gap, presenting a scalable solution to historical georeferencing.
Research Overview and Methodology
Submitted by Ryan Mioduski, this innovative study systematically evaluates the effectiveness of current-generation LLMs in converting historical texts into geographic coordinates. The research focuses on six OpenAI models, spanning three architectures: o-series, GPT-4-class, and GPT-3.5. Each model underwent rigorous testing using two paradigms:
- Direct-to-Coordinate: Models generated coordinates directly from the narrative text.
- Tool-Augmented Chain-of-Thought: Models invoked external geocoding APIs to refine the outputs.
Among its key offerings, a digitized corpus of Virginia patent abstracts was released, including 43 rigorously verified test cases that serve as a controlled benchmark for evaluation.
Results: Performance Metrics
The results garnered from this extensive evaluation shed light on the capabilities of LLMs. The standout performer, the o3-2025-04-16 model, achieved a mean error of just 23 km, with a median error of 14 km. This performance not only surpasses the median LLM’s error rate of 37.4 km by an impressive 37.5%, but it also left behind the weakest LLM’s result of 50.3 km by 53.5%. Additionally, when compared to established external benchmarks such as GIS analysts and the Stanford NER geoparser, the LLMs demonstrated a significant advantage—reducing errors by 67% and 70%, respectively.
More Read
To further refine accuracy, a five-call ensemble was employed, further lowering errors to 19.2 km on average, with a median error of 12.2 km. Remarkably, this improvement came at a minimal cost of approximately USD 0.20 per grant.
Insights on Model Efficacy
An interesting facet of the research was the ablation experiment concerning patentee-name redaction, which revealed a slight increase in error (~7%). This suggests that LLMs are not merely memorizing data; rather, they rely significantly on textual landmarks and the contextual relationships within the narratives to produce accurate geographic coordinates.
The cost-effective model gpt-4o-2024-08-06 demonstrated a mean error of 28 km, costing about USD 1.09 per 1,000 grants. This establishes a new benchmark for researchers looking for both accuracy and cost-efficiency in historical georeferencing.
Geospatial Tools and LLMs
One prominent finding from this study was that external geocoding tools did not provide measurable benefits in this evaluation. The direct application of LLMs surfaced as a powerful alternative for translating historical texts, challenging the traditional reliance on established geospatial methodologies.
Implications for Future Research
Mioduski’s study opens the door to further implications for historical research, emphasizing how LLMs can play a pivotal role in academia and related fields. The ability to produce scalable and accurate geolocation for historical land grants paves the way for deeper explorations into genealogy, land use, urban planning, and heritage conservation.
In summary, the advancements in LLM technology signal a transformative shift in the way we approach historical data. With continued exploration and refinement, large language models are set to redefine our understanding of Virginia’s colonial past, unlocking countless opportunities for new discoveries.
Inspired by: Source
