How to implement a knowledge graph completeness assessment with the guidance of user requirements

doi:10.23919/JSEE.2024.000046

Journal of Systems Engineering and Electronics ›› 2024, Vol. 35 ›› Issue (3): 679-688.doi: 10.23919/JSEE.2024.000046

• SYSTEMS ENGINEERING • Previous Articles

How to implement a knowledge graph completeness assessment with the guidance of user requirements

Ying ZHANG(), Gang XIAO()

Institute of Systems Engineering, Academy of Military Sciences, Beijing 100107, China

Received:2023-05-23 Online:2024-06-18 Published:2024-06-19
Contact: Ying ZHANG E-mail:yingzhang199608@foxmail.com;searchware@qq.com
About author:
ZHANG Ying was born in 1996. She received her M.S. degree in data science from King’s College of London, British, in 2020. She is working toward her PH.D. degree in the Institute of Systems Engineering, Academy of Military Sciences. Her research interests include knowledge graph refinement and knowledge graph evaluation. E-mail: yingzhang199608@foxmail.com

XIAO Gang was born in 1973. He received his B.S. and Ph.D. degrees in computer science and technology from the National University of Defense Technology. He is currently a professor at the National Key Laboratory for Complex Systems Simulation. His research interests include complex system theory, system design, knowledge graphs, and deep learning. E-mail: searchware@qq.com
Supported by:
This work was supported by the National Key Laboratory for Complex Systems Simulation Foundation (6142006190301).

Abstract

Abstract:

In the context of big data, many large-scale knowledge graphs have emerged to effectively organize the explosive growth of web data on the Internet. To select suitable knowledge graphs for use from many knowledge graphs, quality assessment is particularly important. As an important thing of quality assessment, completeness assessment generally refers to the ratio of the current data volume to the total data volume. When evaluating the completeness of a knowledge graph, it is often necessary to refine the completeness dimension by setting different completeness metrics to produce more complete and understandable evaluation results for the knowledge graph. However, lack of awareness of requirements is the most problematic quality issue. In the actual evaluation process, the existing completeness metrics need to consider the actual application. Therefore, to accurately recommend suitable knowledge graphs to many users, it is particularly important to develop relevant measurement metrics and formulate measurement schemes for completeness. In this paper, we will first clarify the concept of completeness, establish each metric of completeness, and finally design a measurement proposal for the completeness of knowledge graphs.

Key words: knowledge graph completeness assessment, relative completeness, user requirement, quality management

Ying ZHANG, Gang XIAO. How to implement a knowledge graph completeness assessment with the guidance of user requirements[J]. Journal of Systems Engineering and Electronics, 2024, 35(3): 679-688.

Figures/Tables 9

Fig 1

Table 1

Fig 2

Fig 3

Fig 4

Fig 5

Fig 6

Table 2

Table 3

References 33

1	SHEN T, ZHANG F, CHENG J W. A comprehensive overview of knowledge graph completion. Knowledge-Based Systems, 2022, 255: 109597.
2	XUE B C, ZOU L Knowledge graph quality management: a comprehensive survey. IEEE Trans. on Knowledge and Data Engineering, 2023, 35 (5): 4969- 4988.
3	PRASOJO R E, DARARI F, RAZNIEWSKI S, et al. Managing and consuming completeness information for wikidata using COOL-WD. Proc. of the 7th International Workshop on Consuming Linked Data, Co-located with the 15th International Semantic Web Conference, 2016. https://ceur-ws.org/Vol1666/paper-02.pdf.
4	FARBER M, BARTSCHERER F, MENNE C, et al Linked data quality of dbpedia, freebase, opencyc, wikidata, and yago. Semantic Web, 2018, 9 (1): 77- 129.
5	HOGAN A, BLOMQVIST E, COCHEA M, et al Knowledge graphs. ACM Computing Surveys, 2021, 54 (4): 1- 37.
6	BRACK A, HOPPE A, STOCKER M, et al Analysing the requirements for an open research knowledge graph: use cases, quality requirements, and construction strategies. International Journal on Digital Libraries, 2022, 23 (1): 33- 55. doi: 10.1007/s00799-021-00306-x
7	WANG X Y, CHEN L Z, BAN T Y, et al Knowledge graph quality control: a survey. Fundamental Research, 2021, 1 (5): 607- 626. doi: 10.1016/j.fmre.2021.09.003
8	ISSA S, ADEKUNLE O, HAMDI F, et al Knowledge graph completeness: a systematic literature review. IEEE Access, 2021, 9, 31322- 31339. doi: 10.1109/ACCESS.2021.3056622
9	LAJUS J, SUCHANEK F M. Are all people married? Determining obligatory attributes in knowledge bases. Proc. of the World Wide Web Conference, 2018: 1115–1124.
10	BALARAMAN V, RAZNIEWSKI S, NUTT W. Recoin: relative completeness in wikidata. Proc. of the Web Conference, 2018: 1787–1792.
11	CAPPIELLO C, NOIA T D, MARCU B A, et al. A quality model for linked data exploration. Proc. of the International Conference on Web Engineering, 2016: 397–404.
12	WISESA A, DARARI F, KRISNADHI A, et al. Wikidata completeness profiling using proWD. Proc. of the 10th International Conference on Knowledge Capture, 2019: 123–130.
13	BARONCINI S, SARTINI B, VAN ERP M, et al Is dc: subject enough? A landscape on iconography and iconology statements of knowledge graphs in the semantic web. Journal of Documentation, 2023, 79 (7): 115- 136.
14	RAZNIEWSKI S, ARNAOUT H, GHOSH S, et al. Completeness, recall, and negation in open-world knowledge bases: a survey. ACM Computing Surveys, 2023, 56(6): 150.
15	PELLISSIER T T, STEPANOVA D, RAZNIEWSKI S, et al Completeness-aware rule learning from knowledge graphs. Proc. of the 16th International Semantic Web Conference, 2017, 507- 525.
16	GALARRAGA L, RAZNIEWSKI S, AMARILLI A, et al. Predicting completeness in knowledge bases. Proc. of the 10th ACM International Conference on Web Search and Data Mining, 2017: 375–383.
17	DARARI F, RAZNIEWSKI S, PRASOJO R, et al. Enabling fine-grained RDF data completeness assessment. Proc. of the International Conference on Web Engineering, 2016: 170–187.
18	LUGGEN M, DIFALLAH D, SARASUA C, et al. Non-parametric class completeness estimators for collaborative knowledge graphs — the case of wikidata. Proc. of the International Semantic Web Conference, 2019: 453–469.
19	CHERIX D, USBECK R, BOTH A, et al. CROCUS: clusterbased ontology data cleansing. Proc. of the CEUR Workshop, 2014: 7−14.
20	SOULET A, GIACOMETTI A, MARKHOFF B, et al. Representativeness of knowledge bases with the generalized Benford’s law. Proc. of the International Semantic Web Conference, 2018: 374–390.
21	SIMSEK U, KARLE E, ANGELE K, et al A knowledge graph perspective on knowledge engineering. SN Computer Science, 2022, 4 (1): 16. doi: 10.1007/s42979-022-01429-x
22	ABIAN D, MERONO-PENUELA A, SIMPERL E. An analysis of content gaps versus user needs in the wikidata knowledge graph. Proc. of the International Semantic Web Conference, 2022: 354–374.
23	SYCHEV O Combining neural networks and symbolic inference in a hybrid cognitive architecture. Procedia Computer Science, 2021, 190, 728- 734. doi: 10.1016/j.procs.2021.06.085
24	JI G L, LIU K, HE S Z, et al. Knowledge graph completion with adaptive sparse transfer matrix. Proc. of the AAAI Conference on Artificial Intelligence, 2016. DOI: https://doi.org/10.1609/aaai.030i1.10089.
25	AKINNUBI A, AJIBOYE J Knowledge graph: a survey. Journal of Robotics and Automation Research, 2023, 4 (2): 366- 377.
26	CHEN X J, JIA S B, XIANG Y A review: knowledge reasoning over knowledge graph. Expert Systems with Applications, 2020, 141 (112): 948. doi: 10.1016/j.eswa.2019.112948
27	PAN S R, LUO L H, WANG Y F, et al. Unifying large language models and knowledge graphs: a roadmap. IEEE Trans. on Knowledge and Data Engineering, 2024. DOI: 10.1109/TKDE.2024.3352100.
28	TIWARI S, Al-ASWADI F N, GAURAV D Recent trends in knowledge graphs: theory and practice. Soft Computing, 2021, 25 (31): 8337- 8355. doi: 10.1007/s00500-021-05756-8
29	CUI Y N, WANG Y X, SUN Z Q, et al Lifelong embedding learning and transfer for growing knowledge graphs. Proc. of the AAAI Conference on Artificial Intelligence, 2023, 37 (4): 4217- 4224. doi: 10.1609/aaai.v37i4.25539
30	TAMASAUSKAITE G, GROTH P Defining a knowledge graph development process through a systematic review. ACM Transactions on Software Engineering and Methodology, 2023, 32 (1): 1- 40.
31	CHEN Z, WANG Y H, ZHAO B, et al Knowledge graph completion: a review. IEEE Access, 2020, 8, 192435- 192456. doi: 10.1109/ACCESS.2020.3030076
32	WANG S, HUANG X, CHEN C, et al. Reform: error-aware few-shot knowledge graph completion. Proc. of the 30th ACM International Conference on Information & Knowledge Management, 2021: 1979−1988.
33	YOSHIDA M, ARASE Y, TSUNODA T, et al. Wikipedia page view reflects web search trend. Proc. of the ACM Web Science Conference, 2015: 65.

Searched entity	Number of searches	Hits(Yes/No)	Search weight
h₁	10	Yes	0.1
h₂	5	No	0.05
h₃	5	Yes	0.05
h₄	15	No	0.15
h₅	15	Yes	0.15
h₆	10	No	0.1
h₇	10	No	0.1
h₈	15	No	0.15
h₉	15	No	0.15

Knowledge graph	Metrics
Knowledge graph	Entity completeness	Schema completeness	Property completeness
Douban	0.16082	0.15978	0.15978
Top1000	0.08578	0.02930	0.02930

Knowledge graph	Merics
Knowledge graph	Entity completeness	Schema completeness	Property completeness
Douban	0.11902	0.12199	0.12199
Top1000	0.07922	0.02718	0.02718

How to implement a knowledge graph completeness assessment with the guidance of user requirements

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 9

References 33

Related Articles 0

Recommended Articles

Metrics

Comments