Go to NeSy 2025 Conference Phase 2 homepageTalking to GDELT Through Knowledge Graphs
Keywords: Retrieval Augmented Generation, Knowledge Graphs, Large Language Models, Question Answering, Ontology-based Knowledge Graphs
Abstract: In this work we study various Retrieval Augmented Regeneration (RAG) approaches to gain an understanding of the strengths and weaknesses of each approach in a question-answering analysis.
To gain this understanding we use a case-study subset of the Global Database of Events, Language, and Tone (GDELT) dataset as well as a corpus of raw text scraped from the online news articles.
To retrieve information from the text corpus we implement a traditional vector store RAG as well as state-of-the-art large language model (LLM) based approaches for automatically constructing KGs and retrieving the relevant subgraphs. In addition to these corpus approaches, we develop a novel ontology-based framework for constructing knowledge graphs (KGs) from GDELT directly which leverages the underlying schema of GDELT to create structured representations of global events.
For retrieving relevant information from the ontology-based KGs we implement both direct graph queries and state-of-the-art graph retrieval approaches.
We compare the performance of each method in a question-answering task. We find that while our ontology-based KGs are valuable for question-answering, automated extraction of the relevant subgraphs is challenging. Conversely, LLM-generated KGs, while capturing event summaries, often lack consistency and interpretability. Our findings suggest benefits of a synergistic approach between ontology and LLM-based KG construction, with proposed avenues toward that end.
Track: Knowledge Graphs, Ontologies and Neurosymbolic AI
Paper Type: Long Paper
Resubmission: Yes
Changes List: # **Response to reviews of “Talking to GDELT Through Knowledge Graphs” for NeSy25**
We greatly thank the reviews for their thoughtful and insightful observations and suggestions. In addition to the submission of the revised manuscript, we are responding specifically to them here.
---
## **Meta-review**
> **“The paper has added value but seems to be too weak on the evaluation. We would like to encourage the authors to improve this aspect and resubmit the paper for the late deadline.”**
Thank you, we agree that the paper is in scope, and have provided an increased quantitative evaluation as follows. In Section 3.3 and specifically highlighted in Table 2 and Figure 5, we have added ground truth answers for the questions used in our analysis. We then used the cosine similarity between the embeddings of the generated answers and the ground truth to quantitatively assess the semantic accuracy of each pipeline. This provides a more robust and objective measure of each method’s performance.
---
## **Reviewer ZwiN**
> **“I think this work is out of the scope of the conference.”**
We agree with the Program Chair that the work is in scope.
---
## **Reviewer zey5**
> **“There is a lot going on and it's hard to describe it all clearly. I found myself drawing a pen-and-paper diagram to keep track of what relates to what with respect to the various datasets, KGs, and LLM-with-RAG pipelines. Eventually I got to Figure 4 in Section 3.2 on page 6 which provides the kind of diagram I had been drawing for myself. It might be helpful to place the Figure 4 diagram earlier in the paper to help anchor and drive all of the lengthy description of pipeline scenarios, rather than leave it until page 6 as a way of summarising the scenarios.”**
Excellent observation, we have advanced Fig 4 to be Fig. 1 in the intro, and provided an initial précis of our neurosymbolic architecture there.
> **“Comparing the performance (in an LLM-with-RAG pipeline) of a KG built from pre-distilled GDELT data with that of KGs built from a text corpus of the source news articles may well be novel, but I'm not sure that the GDELT ontology itself is as novel as the author's appear to suppose. There is a known, 1-to-1 scheme for mapping cells of tabular data to triples (and vice versa): 1) the entity of the table row becomes the 'subject' of a triple, 2) the name of the table column becomes the 'predicate' of a triple, and 3) the cell value becomes the 'object' of a triple. It looks to me like the GDELT ontology proposed by the authors is an expression of this standard conversion scheme. That said, other researchers may well find provision of this GDELT ontology helpful.”**
Another excellent observation. We fully appreciate the standard “star graph” method for representing an RDB as a KG. While it is true that this method broadly informs our “ontology” for GDELT, there are notably some additional constraints present in the event table that warrant departure from this mechanical approach, yielding it not actually just a union of stars. We have now commented on this fully, including noting the equivocation about the use of “ontology” in various parts of the community.
> **“More work is needed to automate this LLM-with-RAG pipeline to make it viable in practice.”**
We do not disagree with this observation. Our position is that in the wide-open world of various neurosymbolic architectures, our experience with this one is very valuable to advance community knowledge and shared experience.
---
## **Reviewer zfU3**
> **“Unfortunately, the experiments are reduced to a very small set of queries and it is only qualitatively evaluated over them. By using recent evaluation systems like RAGAS, it is possible to evaluate quantitatively larger datasets providing a series of interesting measures like faithfulness and factual correctness. Quantitative evaluation is necessary in this kind of papers.”**
We agree with this point regarding the necessity of quantitative evaluation. We have addressed this concern in our results Section 3.3 by now including a cosine similarity based quantitative evaluation framework. This includes curating ground truth answers (Table 2) for our existing set of questions and then employing cosine similarity (Figure 5) to measure the similarity of the answers generated by each pipeline compared to the ground truth.
Publication Agreement: pdf
Submission Number: 24
Loading