Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Spotify | Amazon Music | Android | Youtube Music | RSS
Knowledge graphs provide the digital foundation for some of most visible companies on the web.
Mike Dillinger built LinkedIn’s Economic Graph, the knowledge graph that powers the social media giant’s recommendation systems.
Mike now helps people understand knowledge graph technology and how it can complement and improve generative AI, whether by acting as “jet fuel” to better train LLMs or by providing “adult supervision” for their unruly, adolescent behavior.
We talked about:
- how he describes knowledge graphs
- how the richness of information in a knowledge graph helps computers better understand the things in a system
- the differences between knowledge graphs and LLMs
- how LinkedIn’s Economic Graph, which Mike’s team built, works
- how LLMs can help build knowledge graphs, and how knowledge graphs can act as “jet fuel” to train LLMs
- the RDF “triples” that are at the foundation of knowledge graphs
- the importance of distinguishing between unique concepts in a knowledge graph and how practitioners do this
- the two main crafts needed to build knowledge graphs: linguistic expertise and software engineering
- the job opportunities for language professionals in the LLM and knowledge graph worlds
- the propensity of tech companies to staff knowledge graph efforts with engineers while there is actually a need for a variety of talent, as well as better collaboration skills
- his assertion that “language professionals aren’t janitors,” put on teams only to clean up data for software engineers
- how knowledge graphs provide “adult supervision” for unruly, adolescent LLMs
- his hypothesis that using KGs as a separate modality of data rather than as training data for LLMs will advance AI
Mike’s bio
Mike Dillinger, PhD is a technical advisor, consultant, and thought leader who champions the importance of capturing and leveraging reusable, explicit human knowledge to enable more reliable machine intelligence. He was Technical Lead for Knowledge Graphs in the AI Division at LinkedIn and for LinkedIn’s and eBay’s first machine translation systems. He was also an independent consultant specialized in deploying translation technologies for Fortune 500 companies, and Director of Linguistics at two machine translation software companies where he led development of the first commercial MT-TM integration. He was President of the Association for Machine Translation in the Americas and has two MT-related patents. Dr. Dillinger has also taught at more than a dozen universities in several countries, has been a visiting researcher on four continents, and has a weekly blog on Knowledge Architecture.
Connect with Mike online
Video
Here’s the video version of our conversation:
Podcast intro transcript
This is the Knowledge Graph Insights podcast, episode number 2. If you’ve ever looked for a job or recruited talent on LinkedIn, you’ve seen Mike Dillinger’s work. His team built LinkedIn’s Economic Graph, the knowledge graph that powers the social media platform’s recommendation system. These days, Mike thinks a lot about how knowledge graph technology can work with generative AI, seeing opportunities for the technologies to help the other, like the ability of knowledge graphs to act as “jet fuel” to train large language models.
Interview transcript
Larry:
Hi everyone. Welcome to Episode Number 2 of the Knowledge Graph Insights podcast. I am really delighted today to welcome to the program Mike Dillinger. Mike is a cage-free consultant based in San Jose. He’s been doing knowledge graph and other technical things for many years. So welcome, Mike. Tell the folks a little bit more about what you’re up to these days.
Mike:
Thanks a lot, Larry. What am I up to? I’m trying to help people wrap their heads around knowledge graphs and how we can make AI, transform GenAI into Next-GenAI by leveraging more explicit content and human knowledge.
Larry:
Ooh, I love that. Next-GenAI. I hope you’ve copyrighted that.
Mike:
No.
Larry:
Yeah, no, but that’s … because right now, it seems like most of the oxygen in the room has been sucked up by Open AI and LLMs and chatbots and GPTs, but knowledge graphs have something to offer as well. I guess the first thing I’d like to ask is, can you describe for folks, because I think a lot of folks listening to this podcast might not be as familiar as some of us with what a knowledge graph is and what it does. Can you sort of set out for folks what a knowledge graph is?
Mike:
Sure. The usual way I describe knowledge graphs are as collections of densely interconnected facts about individual things and categories of things, based on a range of different relations. So one thing that people get caught on is, oh, so there is a taxonomy? Not really, but taxonomies are a part of the knowledge graph. Oh, so it’s an ontology? No, but ontologies are a part of the knowledge graph. So there are a range of different kinds of facts in a knowledge graph, so it’s broader than an taxonomy or an ontology.
Larry:
And I think a lot of people come … like, I work in the content world mostly, and information architecture. Most people in that world, I think the first time they go to organize stuff, they start thinking taxonomically, which I guess makes sense. But tell me the benefits of going beyond a simple taxonomy or just an ontology. How does it come together? How does it help people do more interesting and better stuff?
Mike:
Well, the question that we’re talking about here is, you might call it the richness or the depth of the knowledge representation. With a taxonomy, you only have relations like this is a subcategory of that, or this is an instance of that, and you don’t have information about what is this for or what are its attributes or what are its components? So when you move from a taxonomy to a knowledge graph, we’re talking about giving algorithms more information in more detail about the things that we want them to think about.
Larry:
Interesting. And I think a lot of people right now, a lot of the curiosity and interest around these kinds of technologies is around LLMs and OpenAI and ChatGPT and all those things. Can you contrast a knowledge graph and what it can do with what those kinds of systems are doing?
Mike:
Oh, sure. So language models focus very much on strings, and sequences of strings, and knowledge graphs focus more on facts or concepts. So concepts built into facts, as it were. So they’re really focusing on very different things: sequences of words or strings, or graphs of concepts. So the notion of meaning is really different. They’re both in terms of similarity, but an LLM computes similar meaning in terms of context.
Mike:
So if two words have similar words around them, then those two words are considered in an LLM to have similar meanings. But in a knowledge graph, you compare to concepts by saying, oh, do they have similar components and similar characteristics? If so, then they’re related in meaning.
Mike:
So they’re very different ways of getting at a similar problem.
Larry:
Interesting.
Mike:
You might phrase it, for linguistic people, you might say LLMs focus on syntax and knowledge graphs focus on semantics, or LLMs focus on data and knowledge graphs focus on knowledge. There are a lot of different ways of describing it. So they’re very, very much complementary technologies for getting at some of the same problems.
Larry:
You know what I’d love to do now is I’d love to ground what you just said in some examples. Like, what are your … the first accomplishment of yours that I learned about was your work on the Economic Graph at LinkedIn. Can you talk a little bit about what the aim of that is and how a knowledge graph helped LinkedIn do better stuff with their data?
Mike:
Oh, sure. Okay. So the Economic Graph at LinkedIn is a model of the entities in the economy, focusing on schools that produce talent, people with talent, and companies that absorb that talent, okay, and then companies produce products. So we have things like companies, products, workers, schools, these are main entities, and there are a wide range of relationships between them. So when we want to find a worker who fits in a company in a particular position, we need to have a detailed and reliable description of both the position and the worker.
Mike:
This is what LinkedIn’s technology is all about, is matching workers to openings, or now increasingly doing other things, like matching, in their ads, business matching products to people, kind of thing. So knowledge graphs are all about making matching work more systematically and in a more understandable way.
Mike:
So this is what we did at LinkedIn. We built up a kind of vocabulary for describing people or workers and for describing jobs, but we used the same vocabulary for both, so that we could translate, as it were, your worker profile and this company’s job profile into a same meta-language. And it made it much easier and much more accurate to compare one with the other. And that meta-language is what we call a knowledge graph.
Larry:
And some of the mechanisms that permit that … I work mostly in the content world, and we are famous for being bad at our core competency, which is naming and labeling things. And so there’s a lot of people doing content jobs that they’re doing the same thing, but they have a different job title. I know there’s techniques in the knowledge graph world for resolving that kind of discrepancy. Can you talk a little bit about … and I assume that must have happened at scale in the economic graph.
Mike:
Yes. Oh yeah. Yeah. So when I built a team there, we faced a little problem of having 150 million distinct job titles to navigate. So this is way bigger than anything that normal taxonomists usually deal with. So we had to cut that problem down to size, and then, with all the linguistic diversity, we would find literally hundreds or thousands of ways of saying software engineer, for example. So we had to separate out the concept of software engineer from all of its different manifestations in English, in French, in Portuguese, in all the languages that we had available, there were 23 at the time. So this distinction really made a big difference.
Mike:
So we would translate, as it were, software in Wichle, in German, to our concept ID 69137, which is software engineer, language independent, and then the algorithms could work on the IDs and not have to deal with all the variants and variability of the strings. And this is what made search and analytics and recommendations so much better.
Larry:
Those are, in particular search, but also analytics, are kind of classic use cases that I think many … almost every business area struggles with. And it’s that ability … Tell me from a perspective of discoverability, of searching for something, how does that … so again, I work a lot in the design world, so how does that manifest at the end of the process? What are the kinds of end user applications that can draw on the power of a knowledge graph to … like I’m thinking maybe a LinkedIn is a good example.
Mike:
Yeah. So my team was a horizontal team, which means that we produced resources for all the product teams to use. So it was applicable across the board, like you said, again, for search or for recommendations.
Mike:
So discoverability I think is the key concept here. You’re talking about discoverability in the face of extreme variability. Some people would say, we want system analysts II, or level II, and others would say, SWEE, S-W-E-E. They sometimes even spelled it. Okay, so how are we going to match those? Unless we had some sort of internal translation mechanism, which was the knowledge graph, map everything.
Mike:
I don’t know if you know, but your profile on LinkedIn is not what they process. There’s a shadow profile, which is an interpreted and normalized version of your profile. Your profile is yours. That’s your data that they believe very strongly that it belongs to you, but they create a shadow profile, which is that profile translated into the meta-language that they use for processing. And that meta-language was a knowledge graph, so that we could, based on that much smaller vocabulary and more reliable vocabulary, do all the kinds of processing that needed to be done.
Larry:
So I’m curious now, you said, what was the figure, 150 million or something like, different job titles.
Mike:
Yep.
Larry:
By what kind of a factor were you able to reduce that into … how many smaller buckets did you come up with? It was like a factor of 10 or…?
Mike:
Oh, way more than that. So since it’s a standard longtail distribution, it was really straightforward to eliminate most of the titles because they only have a frequency of one or two or five. And the ones that were really relevant in terms of frequency or occurrence were about 50,000. So most of our work focused on those 50,000 and then compressing them into a smaller number of concepts, and saying, oh, well these are pretty much synonyms, these are synonyms, or they have components in common, and we reduced it to this much smaller meta-language to talk about.
Larry:
Fascinating that you could, and then once you’ve done that, then you can draw connections that you might never have made before, because you would’ve just been lost in this jumble of language.
Mike:
In the weeds. Exactly.
Larry:
Hey, one thing about this, I wonder, again, we were talking earlier about the connection, the comparison to the capabilities of LLMs and knowledge graphs, but it sounds like, one you said is mostly, I guess, math, and one is more like, there’s sort of a mathematical similarity versus a conceptual similarity. It sounds like, can they help each other or are there architectures that pull them together?
Mike:
It’s a very active area of research to see how LLMs and knowledge graphs can be leveraged to help each other. So there’s a lot of work on using LLMs to generate the triples that make up a knowledge graph, which is really interesting because the methods to do so are getting better and better. For some triple relations, LLMs work quite well. For others they are more unreliable, which is fine. So there’s a range of ways we can use LLMs to build knowledge graphs.
Mike:
But the other side is, I think, more interesting. So there are a couple of actually very good reviews of literature about how to use knowledge graphs to improve training of LLMs. Okay? So you use the knowledge graph as a loss function to guide how the model, the LLM, relates words to each other. People have used knowledge graphs as simple training data on the input. I don’t think they do it well yet, because in my view, it’d be better to treat knowledge graphs as a separate modality, the way we treat images and videos. So if we could keep those two distinct, then I think it would work a lot better.
Mike:
But since the knowledge graph, think of a knowledge graph, a tiny one with, I don’t know, half a dozen triples, that will correspond to 20,000 sentences. So the knowledge graph is like jet fuel for training LLMs. So those two parts of the step of knowledge graph development are affected very much by knowledge graphs. Knowledge graphs are used for evaluation, they’re used for guardrails, filtering the output. They’re used to make RAG systems more efficient. So it turns out that at every step along the way of building an LLM, if you add knowledge graphs that helps things along, and improves accuracy, for example. I haven’t seen yet though, someone, any group, use knowledge graphs at every step of the way. So that seems to me like a huge opportunity that’s not being exploited.
Larry:
Right, because you hear about RAG and it’s often used to train the model or to fine tune the model, and then it goes, but the end result is still the LLM, which works differently. So there’s still opportunities for, if not hallucination, at least misunderstandings.
Mike:
That’s right.
Larry:
Hey, actually, before I explore that, I want to back up just a little bit because you used a couple of words that are familiar to me, but I want to make sure that folks new to knowledge graphs get them. The notion of a triple, in the knowledge graph world, I guess if you could talk a little bit like the RDF and the foundation of how you build a knowledge graph.
Mike:
Sure. So a knowledge graph, the way I think it’s easiest to look at it is a knowledge graph is a collection of facts. A dog is a subcategory of mammal, so that’s a really small fact, and the canonical way of describing those with this three-piece statement, dog subcategory of mammal, okay? So those are called triples because they’re made of three parts. Sure, there are other kinds of statements with more pieces, and we can go into hyper graphs to talk about more complex ones. But the standard one that people talk about and think about today are the really small three-piece assertions that we call triples.
Mike:
And how do we build them? Well, we build them in a range of ways. Sometimes you can use LLMs, standard natural language understanding systems can generate them for drafts. Sometimes people create them manually because a lot of times it’s human expertise that we’re just trying to record. You can import taxonomies, ontologies. So we have existing structured knowledge that we can include in the knowledge graphs.
Mike:
There are a range of ways of doing this. It doesn’t have to be manual. This is something that I think is important to point out. A lot of people assume that the knowledge graphs have to have 16 billion triples to be useful, and that’s not the case. We had a really interesting story at LinkedIn when we first started and we just took two concepts. Manager was giving us a lot of problems because it was a very, very common title, and we split it into resource manager and team manager. And just making that distinction yielded visible improvements in search and recommendations and a range of things. So even small scale knowledge graphs, because we’re usually working in with the most abstract and most impactful concepts, can have a really significant impact.
Larry:
And that distinction you just articulated, that’s so interesting that like … tell me a little bit about the work process, the practices in knowledge graph development that led to that discovery that yeah, we’re just butting our head up against something. Oh, I know. If we distinguish between resource managers and team managers, boom, a bunch of problems are solved. How do you get to making a good decision like that?
Mike:
So the key thing that we want to do with knowledge graphs is identify distinct concepts. If we can see that, well, this manager has one set of skills and this other manager has a different set of skills, then we have to call them distinct objects, give them different names. Even if we’re in the text, for example, that we’re using as input, they’re called the same way, we want to say, wait a minute. There are actually two really different things here.
Mike:
And inversely, like we said earlier, you have things like a software engineer, but it can be called a SWEE, an analyst, an architect. There are lots and lots and lots of different names. So what we’re looking at identifying is distinct concepts that we can … a really fundamental step there is defining them explicitly so we can double check that they’re distinct, and with the definition, which is a collection of triples itself, we can give that information to the algorithms so that they can unpack these concepts and process them piece by piece and compare them piece by piece in a much more systematic and reliable way.
Larry:
Interesting. And along with that, you have both that notion of what that concept is, and then you have this, I’m picturing a sigh of relief is you’re like, oh, and this big pile of stuff over here that we didn’t know what to do with fits nicely in that new bucket. Is that kind of how it feels when you’re practicing?
Mike:
It does. It does. And it’s like, oh, wait a minute. If we put these together, then all of a sudden, hey, this makes sense. And if we separate out those, it’s like, ah, like you said, sigh of relief. Now it kind of makes sense. Things are clearer.
Larry:
Nice. That kind of gets into the practice itself, like “Daddy, where do knowledge graphs come from?” Or where do knowledge graph practitioners come from? I guess, who are the people putting these together? Is it mostly engineers or other folks?
Mike:
No, so it’s interesting. If you do a job search with the term knowledge graph, then you get back on LinkedIn, you still get back engineers, because there’s a lot of work on the infrastructure for the knowledge graphs, and they can’t quite distinguish yet, not even on LinkedIn, between the people who build the infrastructure and the people who build the content to put into that infrastructure.
Mike:
So yeah, it turns out that engineers are particularly poorly suited to building the content part of knowledge graphs, even though they do great with the infrastructure, because they simply have no training at all with qualitative or linguistic data. And so my joke is, they call it unstructured data, like randomly related stuff. That tells you right there that they don’t have a very nuanced view of linguistic constructs.
Mike:
So my team was built of almost … the team I built at LinkedIn was built almost entirely of people trained in analytic linguistics, and they were really perfect for this. So they had lots of good training in taking apart sentences and noun phrases, in syntax, and identifying similarities and differences in meanings, and they were, they really rocked. This is really great. And sometimes the engineers would just stand in awe and say, “How did you do that?”
Mike:
But so this is a really difficult problem because the engineers kind of suffer, I think, from the situation where they’re in charge, they’re the alpha males of the organization in tech, and I think they suffer from managers who say, “Oh, we pay you enough. You can figure it out.” Even though they’ve had no training at all.
Larry:
Interesting.
Mike:
For engineers, these are really difficult issues to grapple with.
Larry:
That’s so interesting. I started this podcast because I have plenty of other content podcasts going, but I’m like, wow, this is so great that it comes right back to that. And it sounds like, and we were talking before we went on the air about the hope that you offer to language professionals in this new world of … and probably not … is it just knowledge graphs or are there opportunities in the LLM world as well?
Mike:
Oh, the opportunity, I saw someplace, since they’re running out of data from the web that the new wave is going to be domain-specific expertise kinds of data. So there’s an opportunity there for people to identify, curate, and furnish the data that’s missing. For that, again, you’re going to need people with linguistic talents and skills, but I think a lot of it is going to be, well, let me think. How are we going to improve on what we have now? Do LLMs produce engaging and culturally relevant text or do they just overuse words like “delve”? So we’re seeing a lot of this now if we want to move on and make LLMs create more useful, more reliable texts, then we’re going to start to have to add more nuance, so they don’t say the wrong thing or something offensive or these sorts of things. And we need language professionals for that.
Mike:
Something similar is going on with translation. So sure, LLMs can … generative AIs can translate, but they can only cover a handful of languages for which there’s enough data, and then the quality decreases really precipitously when there’s not enough data. So we need language professionals to help not only provide the data, but to take up the slack for the things that the AIs can’t do.
Larry:
Interesting. So we thought they were helping us, but now we need to help them, it sounds like.
Mike:
That’s one way of looking at it.
Larry:
Yeah. But the other thing that reminds me of is there’s already talk in the LLM world of the kind of, I’ve heard it described as smoothness, that over time as LLMs start learning from machine-generated content, they get less and less kind of human over time. And it seems like this re-interjection of human, like you said, like in the translation example, there’s nobody translating Tagalog or … Well, that’s probably a bad example because there’s a lot of speakers, but anyhow, some language that doesn’t have that-
Mike:
But it’s not a top tier language, so you’re right, Tagalog doesn’t have enough, in most cases, doesn’t have enough data for a first-rate generative AI translation system.
Larry:
So you have to involve-
Mike:
So yeah, you’re starting to talk about the … I love the term that’s starting to run around, the enshittification of the language on the internet. Okay? So the more you add machine-generated content to the training of the next generation machine-generated content, the worse it gets. So people are starting to be really seriously worried about that, with good reason.
Larry:
Yeah. And you just coined my favorite, Cory Doctorow’s notion of in enshittification is, it’s tragic how many places-
Mike:
Oh yeah, that’s right, Cory.
Larry:
Yeah, how many places it applies. But one thing about this, just in terms of the practical part of getting people ready for this, it sounds like, hopefully this year knowledge graphs are the center of Gartner’s Impact Radar thingy, and so I guess there’s some anticipation of growth and interest in the field. How can people prepare, or what do you see as a manager of people who create knowledge graphs, what kind of talent is needed and is the infrastructure there to create the next wave of talent that we’re going to need to build all these knowledge graphs?
Mike:
So the people who are most vested in building knowledge graphs are usually in tech. They build teams starting from engineers, and a lot of times it takes them a while to see that the engineers don’t have all the skills that they need. So we went through this at LinkedIn, and had to bring in different talent, not only engineering talent. So I think this is a key to moving ahead, but that also means that we have to not only bring in new talent, but train the engineers to work with people who have different skills and not say, oh, that’s the cleanup crew, they will make our dirty data clean. Okay, so language professionals aren’t janitors. They have a lot of very sophisticated skills, but it takes a while for the engineers to wrap their heads around this, that even though they can’t program or work with sophisticated mathematical models, they do have sophisticated and important skills.
Mike:
So there are the two parts. One is to bring in this new kind of talent, and then the other one is to make sure that they can talk to each other, which is not always an easy thing to do.
Larry:
Yeah, I’ve done over 200 interviews the last several years, and it always comes up that like, yeah, processes, technology, blah, blah, blah, it’s always people stuff. It’s the limiting factor in any of this stuff. But one thing, I just love what you just said, that language professionals aren’t janitors. I’m going to have a T-shirt made with that on it.
Mike:
Yeah, we don’t do data cleanup, so the way I started phrasing it was, no, we don’t do cleanup. Sorry, that’s not what we do. We do enrichment. We enrich the data in a wide range of other ways. If you start throwing out bad data, then the system’s robustness suffers really amazingly. If you try to kind of normalize the data, then you lose the ability to deal with the variation. So it’s definitely not cleanup.
Larry:
Interesting. Yeah. Wow. That’s a whole other conversation. But hey, Mike, I can’t believe it. We’re coming up close to time already, but before we wrap up, is there anything last, anything you’d like to revisit from the conversation or just make sure that you share with the folks before we wrap up?
Mike:
I’m really looking forward to more knowledge graphs. They provide an extra layer of what I call adult supervision to LLMs, which are a lot like teenagers, in one of my blog posts, kind of unruly, full of energy and unruly. So it’s really clear to many people at this point that putting them together is going to help us jump to the next level of functionality in AI, and that’s why a key area of research is how can we put them together? Okay? My favorite hypothesis is that if we stop dumping the knowledge graphs in as training data and use them as a separate modality of data, then we’ll make a lot more progress, but I haven’t seen a lot of work on that particular hypothesis.
Larry:
I’m intrigued. I’m going to be keeping my eyes peeled for instances where people are using knowledge graphs more across the board, like you said earlier.
Mike:
Yeah, there’s that too. I haven’t seen any projects where, okay, we have a knowledge graph, we want our LLM, our AI to work particularly well in this given domain. Let’s use the knowledge graphs at every step of the way of building this knowledge, this domain-specific LLM, and see how well it works. I haven’t seen that either.
Larry:
Oh, well, hopefully in a few years that’ll be a best practice and we can test your hypothesis.
Mike:
Fingers crossed.
Larry:
Exactly. Yeah.
Mike:
Good to see you.
Larry:
Hey, one very last thing, Mike. If folks want to connect or follow you online, what’s the best way to find you?
Mike:
I have a profile on LinkedIn where I blog every week about knowledge graphs and sometimes about LLMs and about the people who build and use them.
Larry:
It’s a fantastic-
Mike:
Called Mike Dillinger on LinkedIn.
Larry:
I’ll link to that in the show notes as well, and I’ll observe that your feed, that’s why I invited you on the show. I’ve been reading you for a year now, and it’s like, this guy knows what he’s talking about. So thanks so much for your generosity in sharing there.
Mike:
Thanks, Larry. Appreciate it. It’s been a pleasure talking to you.
