Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Spotify | Amazon Music | Android | Youtube Music | RSS
The complementary nature of knowledge graphs and LLMs has become clear, and long-time knowledge engineering professionals like Quentin Reul now routinely combine them in hybrid neuro-symbolic AI systems.
While it’s tempting to get caught up in the details of rapidly advancing AI technology, Quentin emphasizes the importance of always staying focused on the business problems your systems are solving.
We talked about:
- his extensive background in semantic technologies, dating back to the early 2000s
- his contribution to the SKOS standard
- an overview of the strengths and weaknesses of LLMs
- the importance of entity resolution, especially when working with the general information that LLMs are trained on
- how LLMs accelerate knowledge graph creation and population
- his take on the scope of symbolic AI, in which he includes expert systems and rule-based systems
- his approach to architecting neuro-symbolic systems, which always starts with, and stays focused on, the business problem he’s trying to solve
- his advice to avoid the temptation to start projects with technology, and instead always focus on the problems you’re solving
- the importance of staying abreast of technology developments so that you’re always able to craft the most efficient solutions
Quentin’s bio
Dr. Quentin Reul is an AI Strategy & Innovation Executive who bridges the gap between high-level business goals and deep technical implementation. As a Director of AI Strategy & Solutions at expert.ai, he specializes in the convergence of Generative AI, Knowledge Graphs, and Agentic Workflows. His focus is moving companies beyond “PoC Purgatory” into production-grade systems that deliver measurable ROI.
Unlike traditional strategists, he remains deeply hands-on, continuously prototyping with emerging AI research to stress-test its real-world impact. He doesn’t just advocate for AI; he builds the technical roadmaps that translate the latest lab breakthroughs into safe, scalable, and high-value enterprise solutions.
Connect with Quentin online
Video
Here’s the video version of our conversation:
Podcast intro transcript
This is the Knowledge Graph Insights podcast, episode number 44. We’re far enough along now in the development of both generative AI learning models and symbolic AI technology like knowledge graphs to see the strengths and weaknesses of each. Quentin Reul has worked with both technologies, and the technologies that preceded them, for many years. He now builds systems that combine the best of both types of AI to deliver solutions that make it easier for people to discover and explore the knowledge and information that they need.
Interview transcript
Larry:
Hi, everyone. Welcome to episode number 44 of the Knowledge Graph Insights podcast. I am really delighted today to welcome to the show Quentin Reul. Quentin is the director of AI Strategy and solutions at expert.ai in the US in Chicago. So welcome, Quentin. Tell the folks a little bit more about what you’re up to these days.
Quentin:
Hi, thank you, Larry, for accepting me and getting me on your podcast. So my name is Quentin Reul. I actually have been around the RDF and the knowledge graph since before it was cool in the early 2000. And today, what I’m helping people in news, media, and entertainment is to see how they can leverage all of the unstructured data that they have and make it in a way that can be structured and they can make their content more findable and discoverable as part of what they are offering to their customers.
Larry:
Nice. And I love that you’ve been doing this forever. And one of the things we talked about before we went on the air was your early involvement in the SKOS standard. Can you talk a little bit about your little contribution to that project?
Quentin:
Yeah. So for this, we do know what SKOS stands for Simple Knowledge Organization System. It’s a standard that has been created by the W3C standard around 2005. And being at the University of Aberdeen in Scotland, we had a lot of involvement with the W3C voicing the web ontology language and SKOS.
Quentin:
For SKOS, I was actually working on my PhD, and the idea of my PhD was to look at two ontologies and trying to map entities from one ontology to the entities in the other one. And a lot of the approach that were taken at the time were either leveraging philosophical kind of representation. And there was not really a lot of things that were looking at linguistics. So the approach that we were taking was looking at WordNet and using the structure of WordNet and maps that to the linguistic information, so the labels that were associated with nodes in the taxonomy.
Quentin:
But to do that, we needed to have a structure that was transitive. And at the time, SKOS only had broader and narrower, and they didn’t have the transitive property. So my contribution was to push for the W3C standard and SKOS to include the SKOS broaderTransitive and SKOS narrowerTransitive, so that I could now have that if A broader B and B broader C, that A broader C was also correct, and having that description logic structure that would enable that.
Larry:
Well, that’s so cool. I love that you have your ideas are ensconced in this 20-year-old standard now. But hey, what I wanted to talk about today and really focus on, I know I was excited to get you on the show because you’re doing a lot of work in the area of neuro-symbolic AI, the idea of integrating LLMs and other machine learning technologies with knowledge graphs and other symbolic AI stuff.
Larry:
It’s one of those things that everybody’s talking about, but I haven’t had the chance to talk on the podcast with many people who are actually doing it. So I’m hoping that you can help the listeners take the leap from this conceptual understanding of the natural complimentary nature of them to actually putting them together in an enterprise architecture. I guess maybe start with the strengths and weaknesses of each of the kinds of AI that we’re talking about here.
Quentin:
Yeah. So if we look at the history of AI, symbolic AI was a thing that came up in the ’70s and led to the first AI winter and the second AI winter for that matter. But where they were very good was in the structure and the explainability. So if you aren’t very well set set of rules or predictive kind of aspect, it would do it consistently, repeatably, and all of that type of things.
Quentin:
Now, when you were trying to adopt a rule-based system for new data, it would die off because you had never seen that or a new set of rules or a new set of business requirements, it would just not handle that. And that’s where machine learning really helped in making that transition to where we are today.
Quentin:
And the LLM, contributing further to that, in as much as the machine learning was pretty good at dealing with new patterns, as long as it was similar to the data that you were training with. I think one thing that the LLMs have really shine is in the way that it’s able to surface things that you were not predicting from the data.
Quentin:
One thing that I think that we could have predicted or seen from the data if we had LLMs back in 2020 is we could probably have seen the topic of COVID emerging a bit earlier than what it did. And the reason is, it’s because it’s very good at surfacing things that it’s never seen before. It’s able at interpreting the language and analyzing the language in its structure. And by the sentence structure, understanding that things are very similar, and you may use different words for them, but now you’re able to interpret them.
Quentin:
So if we think about information retrieval in the ’90s, 2000s, and even in the 2010s, the way that we were doing a lot of these things was using control vocabulary, CISORI, or other dictionaries, and they were used to do query expansion. So you add a keyword, you were looking in the dictionaries, the dictionary were doing an expansion, and then you add something else.
Quentin:
Well, now with the LLM, that kind of expansion is intuitive to the actual LLM because you had seen so many different aspect and so many occurrence of text that it can actually predict and see what these different terms are associated with a holistic concept.
Quentin:
Now, that’s a good thing. On the bad thing, the LLMs don’t have … Well, they have a cutoff point or knowledge cutoff point, which means that when they are trained, they are trained of information that is in the past. So they’re not always that great at predicting, especially current event or information about things that are happening today, they’re not very good at that.
Quentin:
I think if I look at the data, generally between the release of a new model and the nature of the data or the cutoff point, it’s about six months to a year. This is like going a bit slower now or shorter in terms, but you have to remember that the time that it takes to train these models, we’re speaking about days, weeks, and sometime months as opposed to hours with machine learning models. So they’re expensive as well from that perspective.
Quentin:
Another aspect that they don’t have, it’s a knowledge base to just take a higher level from a knowledge graph, like the knowledge base. So it’s not able to disambiguate information in a large corpus. It’s very good to do entity linking within the context of one document.
Quentin:
So if you pass it one document, let’s say a financial document, and it refers to Acme as an enterprise, if Acme is mentioned several times during the document, it will infer that there is only one entity and that entity is Acme.
Quentin:
But now, imagine that you have a group of financial reports, and these financial reports refer to Acme, a bakery in Illinois, and Acme, a construction company in Maryland. You put all of that together. Well, now it’s not able to differentiate because what do you have in financial reports? You have things like what your income was, what your loss were, and these are not really information about baking or about the construction industry.So you’re not going to be able to disambiguate them. And what is likely going to happen is that you’re going to now put together the information, and that’s where hallucination come from, or come in, because you have Acme as a string. The context around the financial information is very similar. So the LLM think that it’s the same entity.
Quentin:
Now, if you bring symbolic AI and particularly a knowledge graph, where in the knowledge graphs you have uncoded that the bakery in Illinois or Acme, the bakery in Illinois was … it’s a bakery type of an organization, it’s located in Illinois, and the other Acme is a type of a construction company and based in Maryland. Now you have that information. The financial report tells you where it was filed. It was filed in Illinois or it was filed in Maryland, but now you produced two things together and you are clearly going to be able to disambiguate that the first one was referring to Acme and the other one to Acme as a construction company. So when you ask the question about what was the revenue by Acme, the bakery, well, you know because of the context around it that you’re referring to one and not the other.
Larry:
I love that example because so many of the … As generally the observation there is that the LLMs have read the whole internet, their training data is so broad, but there is that linguistic commonality that everybody uses the same accounting language, and that’s a really good way to help people, I think, understand how the importance of entity resolution in that kind of thing, business, and it must apply in many other realms as well.
Quentin:
Yeah. And as you pointed out, the LLMs are trained on the data that is on the web, on the open web.
Quentin:
So another area where we see a limitation of the LLM is when you are bringing it to the enterprise. So if you think about what Anthropic, OpenAI, and all of these companies are looking for is artificial general intelligence. So a way to speak about the world in general. Enterprise tend to be very narrow AI. They’ve got a very specific problem that they have, but not only that, they have a jargon that is very specific to them. And in a lot of cases, all of the data that they’ve curated over the years is proprietary. So it’s not going like to be on the open web.
Quentin:
Well, the consequence of that is that all of that jargon, all of that information that is relevant to the industry is not on the open web, and therefore, not in the LLM. So that’s another limitation as well of LLM is that it’s not industry-specific, it’s as much a good thing because it means that you can apply it to anything, but it’s also a bad thing when you’re trying to bring it to the enterprise because it doesn’t really have that knowledge that is specific to that particular industry or that particular sector.
Larry:
Yeah. And that’s interesting, there’s the ongoing sort of, not tragedy, but the observation that the semantic web never took off, but these technologies are alive and well, mostly in enterprises. So this is super relevant, that distinction between enterprise knowledge and general knowledge out on the web.
Larry:
But hey, so I feel like we’ve got a pretty good feel for the pros and cons of each of these kinds of technology, but given all that, what are some of the obvious complimentary natures of them that you’re putting together in the projects you’re working on?
Quentin:
Yeah. So there are two ways that I can think about it. The one way where LLMs have really revolutionized or accelerated the semantic web technology and the knowledge graph technology as we are seeing over the past couple of years compared to its adoption in the enterprise in the early 2010s is on the knowledge-based population or the knowledge graph creation and population side of things, so how do you translate unstructured information into structured information?
Quentin:
I used to work at a publisher, and I recall being part of a … Well, I was responsible for the content enrichment platform that we were offering to our internal customers. So we were applying NLP, machine learning to do categorization, entity extraction, and so forth. One of the things that is needed to do knowledge graph population is relationship extraction.
Quentin:
Now, if you go back 5, 10 years, the way to do that was to train a machine learning model, which meant having large amount of data labeled on the type of relationship and having been very clear about the type of relationship that you were interested in.
Quentin:
Now, in a business, you may see the value of it, but then you always have to look at it from the perspective of what revenue or what return on investment am I going to get. And back then, the time that it would’ve taken a team, especially a team where the annotation and the labeling was secondary to their main job, would really have taken too long, and therefore, was never something that we could push within the business.
Quentin:
Now, enter the LLM, you have extracted your entities using traditional named entity recognition. So you have people, you have locations, you have organizations, and so forth, and control vocabularies. And you use an LLM and you say, “Well, this type of entity and that type of entities, they would be related by this type of relationship.” And because of its understanding of the language, it was able to do that much faster.
Quentin:
So to take a concrete example, imagine a court case. In my case, I was looking at the Supreme Court of the US, and what I did with an LLM was take the text and say, “Well, go and extract the parties, be it an organization or a person, the attorneys that represented those parties, what was the role of the parties in the court case, were there defendant, whether the one bringing the court case forward, and so forth.” And within a couple of days, I was able to create the relationship between the party, what was the role of the party in the court case, the party and the attorney, and the attorney and the law firm that they worked for. And it was 90%, 95% correct, obviously depending on which model I like to use, but this was a eye-opener and definitely an area where we could not do anything in the past, it was just too expensive, but the technology enabled that to be done much faster.
Quentin:
Now, so that’s the population side of things. And we covered with the Acme example, the other side of it, where you are able to use that information to do the downstream disambiguation to make sure that you’re speaking about the right thing.
Quentin:
Another example where I can see the relationship between the two, again, in the legal sector, what we have seen, especially in early 2023, although I still get notification every so often that a lawyer filed a brief to the court with wrong citations. I really don’t understand why it happens, but I think some people still take the LLM as a search engine. But what happened in that case is that people were asking the LLM, “Here is my brief or the case that I’m working on, find me any type of citation that would help me build my case for or against the argument that I put forward.”
Quentin:
Well, due to their nature, the LLM is a pleasing algorithm. So it’s always going to try to give you an answer, and therefore, it was creating citations that were not real, that didn’t exist.
Quentin:
Well, one way that you can alleviate that problem is say that now you have the list of all the legislation that exists within a country. In the US, you have the USC or the United States Codified system for law. You store that in a knowledge graph, and now you use that as a way to validate or guardrail or essentially as a guardrail to determine that if the law is not known into the knowledge graph, well, then it’s likely that it doesn’t exist. So now you have put some ways of leveraging that information as a way to filter out what could have been hallucinated or what could not be relevant to your user.
Larry:
Nice. Those are two really good examples. And the thing that comes out of both of those is the broader issue of trust and trustworthiness. And another thing that’s come up is, and you kind of alluded to that in both of those examples, I think, but the notion of the fact that knowledge graphs provide facts, whereas LLMs are not just, but primarily next-token predictors, and therefore, unable to … They’re a black box, you can’t go back in and trace back to the data. Does that kind of consideration come into the architectures that you’re crafting?
Quentin:
Yeah, and it comes in multiple ways. We mentioned symbolic AI, and you were nicely a bit loose in your definition of symbolic AI, which I very much appreciate, because when we speak about symbolic AI, we don’t really limit ourselves to knowledge graph.
Quentin:
In a lot of cases, we include expert system or rule-based system that was very predominant in the early days of AI. And they are very useful when you have these very strict rule that you want to implement. So having the rule-based approach as well in combination with the LLM enables you to look at, well, in my business problems, I have these things that are very strict. I know that it’s always going to happen if this, that, then this. But then I have my exceptions, and my exceptions can be very well handled by the LLM or I have a level of complexity, go back to citations.
Quentin:
When you have a codify citation, to take the US as an example, the Codify Federal Register is a volume CFR and then the name or the number for paragraph. That’s an easy pattern recognition. It’s a regular expression, it’s easy to do. Why would you use an LLM to do that? The cost of inference is still very high to use something that can be, with very little computational overhead, done by regular expression.
Quentin:
But now when you have the Dodd-Frank Act of 1995, then that’s a lot more complicated to do. And that’s where the LLM can play because they understand the nature of the language to be able to extract that better than the others.
Larry:
Interesting, yeah. And then again, it’s that ability to just read everything and tie things together that way. Hey, one thing I want to make sure we get to is all of these insights, the implementation of them. I know you guys probably have your secret sauce and you probably can’t divulge all your trade secrets and stuff, but in terms of a generic approach to architecting these neuro-symbolic systems, how do you mediate and manage this interaction between these two technical paradigms?
Quentin:
Well, you start with the problem. At the end of the day, what we are trying to solve, especially when you are a B2B enterprise or consulting firm, or even when you are working internally within an enterprise, as I said earlier, you are trying to solve that very narrow problem. So it’s really trying to look at what is that problem that I’m trying to solve?
Quentin:
I’ve seen so many projects over the years fail for two reason. One, because people didn’t really frame the problem correctly, and they were just looking too widely, and they didn’t have KPIs or they were looking at a full end-to-end flow, but really the problem that they needed to solve was a very narrow problem, or they came with technology first. And we see a lot of companies that are coming today and say, “Well, I want a AI, I want a LLM,” and they have already pre-sold the solution, but they didn’t really think about the problem. So they’re trying to retrofit the technology to a problem that they may have.
Quentin:
Problem in both cases is that it’s not because you can solve a problem that someone is willing to pay for that problem or for the solution, which is what leads to a lot of these project failing ultimately. So really leveraging frameworks in the user experience world, like the Jobs to Be Done framework, which is really helping to frame what the problem is. And then once you have that problem, you design the solution, and you look at what you can do.
Quentin:
So when I mentioned citation extraction, well, as part of citation, you have different type of citation. And by their nature, there are different ways that you can really do it. So to go back to my example, why using an LLM when you can do something with a pattern extraction, it makes no sense from a financial and from a performance perspective.
Quentin:
So it’s really going back to that problem, how do I understand what the problem is, and how do I leveraging all of these different technology and understanding that each type of technology has got its pros and cons?
Quentin:
So we spoke about the brittleness and the explainability of rule-based system. You look at machine learning, they need a lot of data, but they are going to be able to predict patterns better. And then you have LLM, they’re very costly from an inference perspective, but they’re able to surface these things that you have never seen before, which is all great. So if you put them together, now you’re able to create a solution that is going to be robust and enable things better.
Quentin:
Just to give you an example, I remember this project back in the days where we took it from the technology perspective, and this was a machine learning project where we were doing classification, and we had a set of folders where we were putting documents according to different categories. We were using a statistical model at the time. And what we found when the subject matter experts were reviewing the information is that every time they were seeing something that was not code, so a false positive or a false negative, they would move it to one of the folder as being a new example.
Quentin:
Well, the problem is a lot of them were exceptions. So what happened to the model? The model ended up not learning the right patterns because now you are the exception with the traditional thing that were together.
Quentin:
If you were to take the exceptions out and just learning these patterns that are what happens naturally in the world, well, you’re going to have very robust machine learning, and on top of that, you put a rule-based system to deal with the exception. And now instead of having 80% accuracy or F1 measure, you can go to 90, 95% because you are dealing with the problem from a holistic perspective and using technologies that fits best.
Larry:
Nice. I’m loving these examples, and I’m sure we could list many more, but I can’t believe it, Quentin, we’re already coming up close to time. But I want to make sure before we wrap up, is there anything last, anything you want to revisit from the conversation or you just want to make sure we share before we close?
Quentin:
Well, I think if anything, what to remember is really start from the problem, and then look at the technology and see how you can, more efficiently, achieve the outcome that you want to achieve. Do not come from technology first. That’s what generally leads to failure in terms of projects.
Quentin:
The other thing is be aware of technology and the ways that technology evolves. I’ve been around for 20 years in the world of AI. I’ve never seen things going as fast as it goes now, but what you are able to know by having the right understanding of the different technology is you can see how they fit together and how you can create the solution that are becoming efficient.
Larry:
Nice. And I think I’m really loving this because probably everybody who’s doing this is on a similar learning journey as yours, but with slightly different details. So I’m going to try to get a bunch more people on to talk about this stuff. Maybe we can piece together some … Because you’re focused in publishing and media, that’s a whole different use case than finance or pharma research or something. It’s really exciting to watch all this come together.
Larry:
Hey, one very last thing, Quentin, if folks want to follow you or connect online, what’s the best way to connect?
Quentin:
Yeah, I am on LinkedIn. It’s Unlock Data with Quentin is my handle. It’s the same on YouTube where I’ve got a YouTube channel. It’s the same on Medium. Everything is Unlock Data with Quentin, and I’m also on Bluesky. So if you are interested in connecting with me, please do so. And I love to hear your thoughts on this episode.
Larry:
Excellent. Yeah, and we didn’t even get a chance to talk about your awesome YouTube channel and stuff, but I’ll link to all that so folks can explore on their own. Well, thank you so much, Quentin. I really enjoyed the conversation.
