Geoparsing from Text — Bachelor's Thesis Project

Estimate the geographic location a piece of text describes or implies — without relying on explicit place names. The input is a short text (a social-media post or a news article); the output is a predicted location (Wikidata QID, coordinates, country, continent). This is the segment of geoparsing where the standard approach (detect a toponym → look it up in a gazetteer) fails: either no toponym appears at all, or the one that does is ambiguous and needs contextual disambiguation.

Nine pipeline configurations are implemented and evaluated against each other and against the external GeoCorporA benchmark.

Repository layout

src/                  pipeline code + all datasets
  main.py             the 6 top-level geolocate_* entry points (mode dispatch)
  ner.py              Flair NER (ner-large)
  refined_linker.py   ReFinED entity linker (mention -> Wikidata QID)
  wikidata.py         single client for all Wikidata access (cache, rate-limit, property-chain resolution)
  ranking.py          E5 multilingual cross-candidate reranker
  llm_geo.py          shared LLM-fallback pipeline (name -> lookup, never QID directly)
  llm_gemma.py        Gemma (Google) backend for llm_geo
  llm_groq.py         gpt-oss-120b and llama-4-scout backends (Groq API) for llm_geo
  evaluate.py         evaluation harness — CLI entry point for running a mode over a dataset
  *.csv               datasets (see DATASETS.md)

scripts/              dataset construction, validation, and analysis utilities
                       (standalone, run individually — see scripts/README.md)

results/
  raw/                one CSV per (mode x dataset) evaluation run
  summary/            aggregated tables (headline, per-signal, continent, failure modes)
  plots/              PNG visualisations
  logs/               text logs of individual runs

prompt                 the LLM specification used to generate the synthetic datasets
DATASETS.md             dataset provenance, generation metadata, validation results
pipeline_architecture.md  design doc: generalising the experiment into a production architecture (Czech)
geolocation_partner_overview.md  research overview for partner institutions
geolocation_overview_june.md     project-status summary
repository_analysis.md / analyza_repozitare.md   critical analysis of the current repo state (EN / CZ)
dataset_plan.md          plan for a redesigned, manifest-first dataset generation process

Pipeline modes

All modes are exposed as geolocate_* functions in src/main.py and are selectable via --mode in src/evaluate.py.

Mode	Approach
two_stage	Flair NER → Wikidata top-10 candidates per entity → E5 rerank
refined	Flair NER (LOC/PER/ORG) → ReFinED links each mention to a QID → Wikidata property chain → E5 rerank
refined_search	refined, plus a wbsearchentities fallback for mentions ReFinED can't link
gemma	Pure LLM (Gemma 3 12B via Google API)
groq_oss	Pure LLM (gpt-oss-120b via Groq API)
groq_llama	Pure LLM (llama-4-scout via Groq API)
refined_gemma	Routing gate: use refined if a LOC mention resolves to a real geographic QID, else fall back to gemma
refined_groq_oss	Same gate, LLM fallback is groq_oss
refined_groq_llama	Same gate, LLM fallback is groq_llama

See pipeline_architecture.md for the rationale behind each design choice (why routing, why the LLM never returns a QID directly, etc.), with pointers back to the specific experimental evidence.

Setup

Requires Python 3.12.

pip install -r requirements.txt

requirements.txt covers the Wikidata/Flair/ReFinED-adjacent stack (flair, sentence-transformers, SPARQLWrapper, pandas, numpy, scikit-learn, rdflib) but is currently missing packages the code actually imports for some modes. Install these as needed:

pip install refined python-dotenv groq google-generativeai

• refined — needed for any refined* mode (refined_linker.py)
• python-dotenv — needed by evaluate.py to load src/.env
• groq — needed for groq_oss / groq_llama / refined_groq_*
• google-generativeai — needed for gemma / refined_gemma

API keys

The pure-LLM and hybrid modes need API keys, read from environment variables loaded from src/.env (git-ignored, never commit it):

GOOGLE_API_KEY=...   # for gemma / refined_gemma
GROQ_API_KEY=...     # for groq_oss / groq_llama / refined_groq_oss / refined_groq_llama

two_stage, refined, and refined_search need no API key — they only call the public Wikidata API.

Running

Quick smoke test — runs all six original modes on one sentence:

cd src
python main.py

Full evaluation over a dataset:

cd src
python evaluate.py --dataset social_media_dataset.csv --mode refined --output eval_results.csv
python evaluate.py --dataset journalistic_dataset.csv --mode refined_gemma --output eval_results.csv
python evaluate.py --dataset geocorpora_eval.csv --mode groq_oss --limit 20

--mode accepts any of the nine values from the table above. evaluate.py prints a per-run summary (exact / country / continent / within-161km / no-result rates, overall and per signal type) and writes a detailed per-row CSV.

Wikidata and LLM responses are cached on disk (src/.wikidata_cache/, src/.llm_cache/, both git-ignored) and flushed every 10 rows, so an interrupted run doesn't lose already-fetched lookups — but evaluate.py itself does not resume from a partial output file; a rerun starts from row 0 and replays from cache.

Results

Headline numbers (results/summary/table_FINAL_headline.csv), exact-match / country / continent / acc@161km, on the clean 144-row social-media set and the GeoCorporA benchmark:

Mode	clean_sm exact	clean_sm country	geocorpora exact	geocorpora country
two_stage	34.0%	49.3%	18.1%	70.8%
refined	46.5%	53.5%	52.1%	76.4%
refined_search	48.6%	56.9%	52.1%	77.8%
gemma	42.4%	72.9%	37.5%	81.2%
groq_oss	48.6%	72.9%	31.2%	79.9%
groq_llama	33.3%	66.7%	37.5%	82.6%
refined_gemma	53.5%	75.7%	50.0%	86.8%

refined_gemma (structured linking + LLM fallback) is the best-performing mode overall. refined alone (no LLM, no external API, no per-row cost) is the strongest fully self-contained option. Full breakdowns (per signal type, noisy-text robustness, failure modes, continent confusion) are in results/summary/ and plotted in results/plots/.

Datasets

Three synthetic CSVs (generated interactively via an LLM from the spec in prompt) plus the external GeoCorporA benchmark. Full provenance, signal-type definitions, and validation methodology are in DATASETS.md.

File	Rows
src/social_media_dataset.csv	144	synthetic, clean
src/social_media_noisy.csv	144	same locations, distractor text
src/journalistic_dataset.csv	72	synthetic, journalistic register
src/Geocorpora.csv / src/geocorpora_eval.csv	~144	external benchmark

Known caveats with the current dataset (see repository_analysis.md for the full analysis): the journalistic set is a 50% subset of the SM locations rather than an independent parallel set, the noisy set has no clean control group, and per-row generation metadata isn't recorded. dataset_plan.md sketches a manifest-first redesign addressing these.

Scripts

scripts/ holds standalone dataset construction, validation, and analysis utilities (QID/coordinate enrichment, Wikidata validation, failure-mode analysis, NER-recall diagnostics). See scripts/README.md for the full list and usage.