Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Spotify | Amazon Music | Android | Youtube Music | RSS
Any knowledge graph or other semantic artifact must be modeled before it’s built.
Panos Alexopoulos has been building semantic models since 2006. In 2020, O’Reilly published his book on the subject, “Semantic Modeling for Data.”
The book covers the craft of semantic data modeling, the pitfalls practitioners are likely to encounter, and the dilemmas they’ll need to overcome.
We talked about:
- his work as Head of Ontology at Textkernel and his 18-year history working with symbolic AI and semantic modeling
- his definition and description of the practice of semantic modeling and its three main characteristics: accuracy, explicitness, and agreement
- the variety of artifacts that can result from semantic modeling: database schemas, taxonomies, hierarchies, glossaries, thesauri, ontologies, etc.
- the difference between identifying entities with human understandable descriptions in symbolic AI and numerical encodings in sub-symbolic AI
- the role of semantic modeling in RAG and other hybrid AI architectures
- a brief overview of data modeling as a practice
- how LLMs fit into semantic modeling: as sources of information to populate a knowledge graph, as coding assistants, and in entity and relation extraction
- other techniques besides NLP and LLMs that he uses in his modeling practice: syntactic patterns, heuristics, regular expressions, etc.
- the role of semantic modeling and symbolic AI in emerging hybrid AI architectures
- the importance of defining the notion of “autonomy” as AI agents emerge
Panos’ bio
Panos Alexopoulos has been working since 2006 at the intersection of data, semantics and software, contributing in building intelligent systems that deliver value to business and society. Born and raised in Athens, Greece, Panos currently works as a principal educator at OWLTECH, developing and delivering training workshops that provide actionable knowledge and insights for data and AI practitioners. He also works as Head of Ontology at Textkernel BV, in Amsterdam, Netherlands, leading a team of data professionals in developing and delivering a large cross-lingual Knowledge Graph in the HR and Recruitment domain. Panos has published several papers at international conferences, journals and books, and he is a regular speaker in both academic and industry venues. He is also the author of the O’Reilly book “Semantic Modeling for Data – Avoiding Pitfalls and Dilemmas”, a practical and pragmatic field guide for data practitioners that want to learn how semantic data modeling is applied in the real world.
Connect with Panos online
Video
Here’s the video version of our conversation:
Podcast intro transcript
This is the Knowledge Graph Insights podcast, episode number 23. In order to build a knowledge graph or any other semantic artifact, you first need to model the concepts you’re working with, and that model needs to be accurate, to explicitly represent all of the ideas you’re working with, and to capture human agreements about them. Panos Alexopoulos literally wrote the book on semantic modeling for data, covering both the principles of modeling as well as the pragmatic concerns of real-world modelers.
Interview transcript
Larry:
Hi everyone. Welcome to episode number 23 of the Knowledge Graph Insights podcast. I am really excited today to welcome to the show Panos Alexopoulos. Panos is the head of ontology at Textkernel, a company in Amsterdam that works on knowledge graphs for the HR and recruitment world. Welcome, Panos. Tell the folks a little bit more about what you’re doing these days.
Panos:
Hi Larry. Thank you very much for inviting me to your podcast. I’m really happy to be here. Yeah, so as you said, I’m head of ontology at Textkernel. Actually, I’ve been working in the field of data semantics, knowledge graph ontologies for almost now 18 years, even before the era of machine learning, back when it was mostly about symbolic AI. Yeah, I’ve been working a lot on this field. I’ve seen its ups and downs, I’ve seen it’s good and bad things, and I think our discussion is going to focus on these things. What I’ve been doing lately now with the field of AI, and I think… No, let me say this differently. I think that the field of data semantics, even in the era of AI and large language models, et cetera, is even more important and this is something that I’m actively looking now. I’m actually looking a lot on the synergy and in the interrelation between large language models with data, with knowledge graphs and ontologies.
Larry:
Yeah, I’d love to talk more about that because that just seems to be in the air. One thing that I want to talk about, and I realized I totally left out of my intro, that you wrote this brilliant book called Semantic Modeling for Data.
Larry:
That’s right. I kind of buried my lead there, as we say in journalism, but one of the first things that, and we talked about this a little bit before we went on the air. Can you describe to folks what semantic modeling is? What are we doing there when we’re modeling?
Panos:
Yes. So the definition I give to the term of semantic modeling is the practice of building descriptions of data that have three important characteristics. The first thing, the first characteristic is that this description should be accurate, that this we should describe data and domains in a correct way, right? We don’t want to have statements and assertions that are wrong. The second characteristic is that these descriptions should be explicit both for people and machines. What does that mean? If I have a data, if I have a data set, a set of data and I give it to you, and when you read it, you cannot understand what it is about. That’s not good semantics, right? The meaning is lost, and the same applies for systems, for machines. If I call an API, I take some data back and my machine, enterprise system is not able to interpret the meaning of this data, then I have an issue.
Panos:
The third characteristic is agreement. It’s not enough to have explicit meaning on data. It’s also very important that we both agree on the validity of that meaning and that we serve the same meaning, right? And it starts with, I can give as example very simple things like what is a knowledge graph? If you go and you try to find a definition of what a knowledge graph, you will see many definitions that are not necessarily consistent to each other, right? So there’s already disagreement there, and actually the word of a good ontology, semantic modeler would be to try to start with defining that. So that’s what semantic modelers do and the artifacts that we build, this is practically an umbrella type, semantic modeling that covers a lot of artifacts that we build. These artifacts can range from database schemas, taxonomies, hierarchies, glossaries, thesauri, ontologies as many of our audience already heard, knowledge graph, et cetera.
Panos:
So when you do semantic modeling, you’re not building necessarily one type of artifact. Then the key to remember is that you are describing data, you’re describing domains by means of formal symbolic representation. That’s also another important thing that semantic data modeling is about creating explicit human understanding, not only system understandable, but also human understandable descriptions of data. That means that embeddings, large language model, et cetera, do not fall into this category because their representations, their underlying representation are subsymbolic, there are numbers, so they maybe contain some meaning. They may be encoding some meaning, but according to my definition, they do not fall into the practice of semantic data modeling.
Larry:
Got it. The way you just said that sub-symbolic, because I’ve heard a lot of people talk about the symbolic AI and neural networks or machine learning and other, the sub-symbolic stuff. Is there a distinct dividing line there or is this a continuum going up and down then the AI stuff?
Panos:
For me, I don’t think it’s a continuum. Sub-symbolic is when you have numbers practically, right? So when you have a word, and this is encoded by a vector of 1600 or so numbers, this representation is subsymbolic. Symbolic means words, symbolic means human language, right? Terminology that doesn’t happen in machine learning.
Larry:
Got it. Yeah, no, as soon as you said that, I was like, “It’s more of a Boolean.” Like, “Yep, it’s symbolic or it’s not.”
Panos:
Of course of you can have, let’s say hybrid models. For instance, you can take a knowledge graph of entities and the attributes of each entity. You can have the name of the entity, some other characteristics, and then you may have also an embedding of that entity. So you can have at the same time, two representations, a symbolic one and a subsymbolic one for the same entity, and you use that for different purposes. For example, an embedding is very good to find similarities between entities, which is more difficult to do it in the symbolic space.
Larry:
Interesting, yeah. And this gets into RAG architectures and other hybrid architectures that are emerging, or does it? Is that how those come together that if you’re looking for similarities, you revert to like a subsymbolic system, but if you’re looking for, I don’t know, knowledge-based stuff, you shift into the symbolic.
Panos:
Yeah, so RAG is a nice example, right? So what’s the idea of RAG? RAG stands for Retrieval Augmented Generation. It’s the idea that you have your LLM, but you want it to give you answers based on your own data, based on your own knowledge. So when you give a prompt, when you give a query to the LLM before the LLM answers, you have a step where you retrieve relevant information to your query from some database, from some other set of documents, or as it’s lately more into vogue structured data and knowledge graphs. That provides you more context to your query and that increases the likelihood that the LLM will give you a correct answer and not a hallucinated answer. Yeah, that’s the main rationale there, and indeed, it’s a nice example of actionable hybrid architecture with symbolic and subsymbolic machine learning AI they call it’s called.
Larry:
Yeah, and I mean that’s going to be such a rich vein to mine for years to come, the exact mix of that. I just dropped, we’re recording this, I think it’s two or it’ll be two or three episodes back. But I just dropped an episode with Juan Sequeda and he and Dean Allemang did that interesting research on just how using knowledge graphs to improve querying of enterprise relational databases and how three X-ed, the accuracy of the answers. How does modeling help that? Like you mentioned that you can just consult documents or something, but you can also consult in those RAG architectures structured data or even a knowledge graph. There’s probably a case for semantic modeling in there that the more modeled and the better modeled it is, the better results. Is that provable or is that true?
Panos:
Right, so let’s take the traditional RAG, right? So in traditional RAG you have, I said a set of documents. The documents can be a lot, and you have to retrieve based on your question or your query the most relevant ones. So maybe you have 10,000 documents, but probably you will need to retrieve five or 10. There is a high likelihood that this retrieval will not be completely accurate or will not be, sorry, complete, right? There is also a common problem that has been reported called missing, lost in the middle, you have relevant things that they are not retrieved because they do not make the high score.
Panos:
What you do with semantic modeling, what you do with structured data is that you distill out of your domain knowledge of your documents, the knowledge that is more important for your situation, right? And a knowledge graph is much smaller in terms of size and in terms of expressivity than a large set of documents, right? It’s less verbose, which means that you only have hopefully the information that you need and that makes it more appealing. Of course, the flip side is that you need effort to create that knowledge graph, whereas just feeding documents is easier because you have the documents, they’re there.
Larry:
And that’s again, back to the benefit of modeling is that a document can have some kind of like HTML markup know or semantic kind of markup about what things are in that document, but a knowledge graph provides more granularity, right? More specificity that can filter, that you can retrieve by-
Panos:
Knowledge graph does that, but also modeling, you can model the domain in many ways, and it always starts with the intention, why you want to model it? What do you want to achieve? And based on that you model the information. So usually when we do semantic modeling, we start with, okay, what questions do we need to answer for our domain using an ontology or a knowledge graph? And based on those questions, which are usually called competency questions, in the parlance of semantic modeling, we decide what entity types we have, what classes, what relations, et cetera. If you don’t have those requirements, then you can create any model, any semantic model, knowledge graph, which is not necessarily what you need for your case. So semantic modeling is a way to guide your applications into the right direction on how to interpret the information and how to interpret your information needs and your questions and your tasks.
Larry:
Yeah. And that goes back to those things you talked about, the three things you talked about, the need for accuracy, the explicitness, and I think maybe most key, the agreement on what those things mean. I mean, I know mean all of those sound important, but in the practice of modeling to get to that point of view, you just said you’re getting the right direction or answering the right… having the ability to answer the right questions. Can you walk me through, I know this is the whole book, we can’t do it obviously, but just the basics of modeling as a practice, how you get to where you’re confident of your ability to answer those questions.
Panos:
Right. So to answer a question through a knowledge graph, it means two things. It means that you have the right structure that will allow you to answer the question and then to have the right content. For example, let’s say that if we take the HR domain, the recruitment domain, someone wants to know what skills are necessary or are important in order to enter a particular profession in order to practice a particular profession. For example, what skills does an ontologist need to have in order to do their work, right? Okay. If we want these questions to be answered through a knowledge graph, the first thing we need to think is what kind of elements we need. Obviously we need an element of skills.
Panos:
We need to have somehow modeled the notion of a skill and the concept of a skill and then also to have the notion of a profession, and then to have a relations between them. So the first step is to define these elements, and you see that these elements come from the question that we’re trying to answer. And after we do that, okay, then it’s technically answerable, but we need the actual content. We need also then in the knowledge graph to have concrete skills with concrete professions being related to each other. So it’s not enough to just define it theoretically to define the structure. You also need to provide the content either by humans or from other data sources, or even sometimes from pure text from mining.
Larry:
Yeah. As you talk about that, I’m reminded, I’ve heard a number of ontologists talk about there’s both the bottom up and the top down part of that. It sounds like that surveying the content instances is the bottom up part of that, having a lot of those. And then what you described at first seems like the top down part like, “Well, what are we talking about here?” We got skills, we got professions. Is that an accurate way to think about it?
Panos:
The bottom up and the top down is it has to do with where you start with your analysis. Sometimes you are modeling a domain, but you don’t have requirements. You just say you are just given data. So you try to discover relations to discover class, discover elements out of data. That’s the bottom up. The top down starts with, “Okay, this is what I want to do and I’m adding content.” In most of the cases you have nowadays, you have a hybrid approach, at least that’s my favorite one. So you start with top down and defining a minimum schema of a minimum set of elements that you want to have in your knowledge graph, your entity, your many entity types.
Panos:
And then when it comes to populating those, to put instances of those, of course you can use domain experts or people, but this is not scalable and it’s actually very expensive. That used to happen actually 15 to 20 years ago, a lot before the machine learning, but nowadays, that you have big data, you have a lot of data, and you have much more advanced mining methods and machine learning, you can take data and try to extract knowledge from there. The good thing with the hybrid, the reason why you need a hybrid approach is that the manual work, the conceptual manual work that you’re going to do as an ontologist to define the small schema, can help you design the the automatic population process.
Panos:
Otherwise, you go blindly. You’re going blindly like, “Okay, these are the data. Tell me what you see.” And actually I’ve seen this a lot lately with some approaches saying, “You know what, no worries. Give me your documents or your data. I’m going to feed them into a large language model and we’re going to derive your knowledge graph without any direct, without any guidance.” That will probably give you a knowledge graph, not necessarily the knowledge graph that is important to you, and very likely it will suffer from severe quality problems. So there’s no magic way to just derive the knowledge graph that you need out of data yet. That’s why semantic models and ontologies are still needed.
Larry:
Yeah. As you say that it’s because… not to disparage my engineering friends, but a lot of engineers, “Oh, we can just discern that from the data.” But you just said you’ve talked a lot about the importance of that top-down, the input of the people. But you can also still use the LLMs to assist in this kind of thing. How do you balance the benefits of being able to automatically extract stuff but still keep the human wisdom in the process?
Panos:
Right. So you try to investigate and see what the humans can give you that the LLM doesn’t give you and vice versa, right? So for example, as you said, LLMs can help with knowledge graph development, with semantic modeling, and actually there are three. I’ve been doing some work and I’ve let’s say identified three main ways that an LLM can help a knowledge graph developer in knowledge graph development. The first way is to use it as a direct source knowledge. Yeah, knowledge source, sorry. So for example, you have in your knowledge graph the class European Union country. You go and you ask the LLM, “Tell me the European Union countries and then populate it.” The thing with using the LLM as a knowledge source, the problem is that for many facts it will hallucinate. It doesn’t know everything, and many times it just makes up things. So it can be used to some extent for that aspect, but only in collaboration with other data sources.
Panos:
It’s not the single data source to use it, so as a knowledge source is not that strong yet. Another way to use an LLM to help you with semantic modeling and building knowledge graphs and ontologies is to use it in the similar way as you use it for software engineering, for helping you with coding. So when you have the task of taking a set of competency questions and you have to transform them into RDF or a label property graph and you don’t know exactly how or you’re too bored, you can ask an LLM, “Okay, give me the formal representation.” So far in my team and myself, I’ve been doing a lot of experiments on that aspect, and I’ve seen that LMS are quite mediocre in doing semantic modeling, in doing transformations into formal representations.
Panos:
Which means that it can potentially help a senior person, a knowledgeable knowledge engineer, a semantic modeler who can easily discern the problematic issues. But it would be detrimental if it was to be used by junior engineers, junior modelers, who will omit it, which is the same principle as in coding, right? If you’re inexperienced coder, you will get code that it’s not as good and you won’t be able to understand that it’s not good. That’s the problem. The third usage, and I think this is where LLMs are really valuable, especially when at some point hopefully they become faster and less expensive, is extraction, NLP. Imagine that before the LLMs, if you wanted to build an entity extractor or relation extractor, you had to follow or supervise the approach. You have to give it a lot of examples, create big data sets, train your models and you hope that you get a good quality extractor of entities and relation.
Panos:
With LLMs, there is a lot of transferable knowledge already, so sometimes all you need to do for some entity relations, all you need to do in order to extract them properly is give them a prompt. “This is the document, just tell me what skills and occupations are in there.” It might work, it might not, but it can already be done like that. Then there is the option of fuel shot where you just give it also some examples, and if that doesn’t work, then of course you go into more advanced approach, like fine-tuning, et cetera, but it already lowers the bar a lot and we’ve seen both in my team and also I feel discussing it in my courses and with other people that there is a lot of potential there for LLMs. Of course, all this needs to be done carefully, right? It’s not about blindly trusting things. You always have to evaluate, always you have to check if the quality is good.
Larry:
Yeah. As you say that, it does seem like you use the analogy of it’s very similar to the way LLMs can help software engineers and they’re… again, you said the more experienced you’re, the easier it is to recognize potential issues and stuff and to deal with them. Has this accelerated or just sort of modified your modeling practice, incorporating these three things you just mentioned?
Panos:
So in my experience, I haven’t applied. I only have applied LLMs for extraction. And in that case, indeed they managed to accelerate things and also with quality, so we needed much less need for human-based quality check ins. We really increased precision and recall a lot.
Larry:
Yeah. It feels like we’re at a stage right now where everybody’s kind of figuring out. You’ve got these three things you’ve been looking at, that one’s working pretty well. Are we still sorting out how best to use LLMs and these kind of workflow? And are you revisiting things like old-fashioned NLP stuff or machine learning techniques in your modeling practice as well?
Panos:
Of course, not only revisiting, we are still using old-fashioned techniques. Believe it or not, we still have tasks that we do with syntactic patterns, with heuristics, with regular expressions. I mean, when you have a task where the simplest approach works, you don’t need the fancy approach immediately, it’s much cheaper. You don’t go directly to the LLM, that will be an overkill. You go to the LLM at least now when you have something that we can all easily… you always need to see the tradeoff.
Larry:
Yeah, I guess that’s why I asked, because I think I’ve met a lot of people who just jump right to the LLM in their problem solving, but I don’t know how many of those are data modelers now that I think about it.
Panos:
I think it applies for every task, right? Of course, people that start now, indeed, it’s very tempting, will say, “Yeah, I can do this with an LLM. I don’t need to revisit the office.” But when you have been working in any field in software engineering, in data modeling, in NLP already with previous methods, and especially when you’re not so much doing research, but rather you’re in the industry, you know very well that despite the hype for much more the short term, you will still be having… your standard approach is there. It is not easy to know that transition suddenly to say, “I just ditch everything that I’ve been doing and moved to an LLM based approach.”
Larry:
Yeah, some stretches of LinkedIn, you might think that wasn’t the case. Like LLMs are going to turn everything for you. Yeah.
Panos:
Yeah, but LinkedIn is a bubble.
Larry:
Yeah, exactly. Yeah. Well, and that’s why I wanted to talk to you because you’ve been in the trenches doing this work hands-on, wrote a whole book about it. That’s why I’m really excited to share this with folks. But hey, Panos, I can’t believe it, we’re coming up close to time already, but before we wrap up, is there any last thing you want to share or revisit from the conversation?
Panos:
I think the message is based on the discussion that we have, is that semantic modeling, symbolic AI in general still has a place in what we call the AI and data world. Actually, I think probably your previous guests like Juan Sequeda and et cetera, already said that it’s even more important now. Of course, we’re a bit biased because we come from there, but it’s even more important and I think the future is about synergy. The future is about hybrid models. At least, I don’t know, perhaps scientifically wise, I don’t see that long term. Maybe at some point we don’t need all this work, but from a business perspective, from the industry and pragmatic perspective, we’ll need to use hybrid ways in order to get business value, et cetera. We’re nowhere around having an agent, let’s say, doing everything for us without any guidance, without anything else. And when you guide something you need to understand it comes from symbols. Humans cannot process numbers.
Larry:
I love that notion that to the extent that there is guidance and that it sounds like, I’m inferring from what you said that for agents to really take off that there’s going to need to be as much if not more symbolic AI in there as there is…
Panos:
There’s also an issue with agents that I’m still, have wrote the agents back in my masters or so. The main thing that we go outside of semantic modeling, but the main thing that we need to consider when we talk about agents and when we talk about is not as glorified at say methods, but rather as autonomous things, is we need to define what is the notion of autonomy. An agency. Agency means that you have accountability. Agency means that you have intentions, and this is when you have an agent that just has the intentions of whatever you program it with, right? They’re not really autonomous agent. It’s actually more like a way to justify your lack of accountability.
Larry:
Yeah, you said that. Yeah.
Panos:
Yeah. It’s a way to hide things. I don’t know.
Larry:
Yeah, because that notion, I haven’t seen a lot of talk about accountability. I’ve seen a lot of imagineering about task, and a lot of it just seems like automated workflows, not anything super fancy, but that notion of what do you mean by agent? Can you sue an agent if it does something wrong? That kind of thing. Yeah.
Panos:
Exactly.
Larry:
Exactly. Cool. Well, thanks so much, Panos. This has been a fantastic conversation. One very last thing. If folks want to connect with you online, what’s the best place to find you?
Panos:
LinkedIn, definitely, and email secondly, I think these are the main two channels.
Larry:
Okay, great. I’ll put that in the show notes as well. Well, thank you so much, Panos. I really enjoyed this conversation.
Panos:
Me too.
