Jim Hendler: Scaling AI and Knowledge with the Semantic Web – Episode 43

Podcast: Play in new window | Download

Subscribe: Apple Podcasts | Spotify | Amazon Music | Android | Youtube Music | RSS

As the World Wide Web emerged in the late 1990s, AI experts like Jim Hendler spotted an opportunity to imbue in the new medium, in a scale-able way, knowledge about the information on the web along with its simple representation as content.

With his colleagues Tim Berners-Lee, the inventor of the web, and Ora Lasilla, an early expert on AI agents, Jim set out their vision in the famous “Semantic Web” article for the May 2001 issue of Scientific American magazine.

Since then, semantic web implementations have blossomed, deployed in virtually every large enterprise on the planet and adding meaning to the web by appearing in the majority of pages on the internet.

We talked about:

• his academic and administrative history at the University of Maryland, Rensselaer Polytechnic Institute, and DARPA
• the origins of his assertion that “a little semantics goes a long way”
• his early thinking on the role of memory in AI and its connections to knowledge representation and to SHOE, the first semantic web language
• his goal to scale up knowledge representation in his work as a grant administrator at DARPA
• how different departments in the US Air Force used different language to describe airplanes
• the origins and development of his relationship with Tim Berners-Lee and how his use of URLs in SHOE caused it to click
• how he and Berners-Lee brought Ora Lassila into the semantic web article
• how his and Berners-Lee’s shared interest in scale contributed to the “a little semantics goes a long way” idea
• why he lives in awe of Tim Berners-Lee
• Berners-Lee’s insight that a scaleable web needed the 404 error code
• how including an inverse functionality property like in a relational database would have ruined the semantic web
• how they came to open the Scientific American paper with an anecdote about agents
• his early involvement in the AI agent community along with Ora Lassila
• their shared conviction of the foundational importance of interoperability in their conception of the semantic web
• how the lack of interoperability between big internet players now is part of the reason for the inability to fully execute on the agent version they set out in the SciAm article
• the impact of LLMs on the semantic web
• early examples of semantic web linked data interoperability
• Google’s reclamation of the term “knowledge graph”
• the reason that the shape of the semantic web was always in their mind a graph
• how the growth of enterprise data led to their adoption of semantic web technology
• how the answer to so many modern AI questions is, “knowledge”

Jim’s bio

James Hendler is the Tetherless World Professor of Computer, Web and Cognitive Sciences at RPI where he also serves as a special academic advisor to the Provost and the Head of the Cognitive Science Department. He also serves as a member of the Board, and former chair of the UK’s charitable Web Science Trust. Hendler is a long-time researcher in the widespread use of experimental AI techniques including semantics on the Web, scientific data integration, and data policy in government. One of the originators of the Semantic Web, he has authored over 500 books, technical papers, and articles in the areas of Open Data, the Semantic Web, AI, and data policy and governance. He is the former Chief Scientist of the Information Systems Office at the US Defense Advanced Research Projects Agency (DARPA) and was awarded a US Air Force Exceptional Civilian Service Medal in 2002. In 2010, Hendler was selected as an “Internet Web Expert” by the US government, helping in the development and launch of the US data.gov open data website and from 2015 to 2024 served as an advisor to DHS and DoE board. From 2021-2024 he served as chair of the ACM’s global Technology Policy Council. Hendler is a Fellow of the AAAI, AAIA, AAAS, ACM, BCS, IEEE and the US National Academy of Public Administration. In 2025, Hendler was awarded the Feigenbaum Prize by the Association for the Advancement of Artificial Intelligence, recognizing a “sustained record of high-impact seminal contributions to experimental AI research.”

Connect with Jim online

• RPI faculty page

People and resources mentioned in this interview

• Tim Berners-Lee
• Ora Lassila
• Deb McGuinness
• The Semantic Web, Scientific American, May 2001
• Introducing the Knowledge Graph: things, not strings
• Massively Parallel Artificial Intelligence paper
• Attention Is all You Need paper
• Vision conference
• Is There An Agent in Your Future? article
• “And then a miracle occurs” cartoon

Video

Here’s the video version of our conversation:

Podcast intro transcript

This is the Knowledge Graph Insights podcast, episode number 43. Twenty-five years ago, as AI experts like Jim Hendler navigated the new World Wide Web, they saw an opportunity to imbue in the medium, in a scale-able way, more knowledge than was included in the text on web pages. Jim combined forces with the web’s inventor, Tim Berners-Lee, and their mutual friend Ora Lasilla, an expert on AI agents, to set out their vision in the now-famous “Semantic Web” article for Scientific American magazine. The rest, as they say, is history.

Interview transcript

Larry:
Hi everyone. Welcome to episode number 43 of the Knowledge Graph Insights Podcast. I am super extra delighted today to welcome to the show, Jim Hendler. Jim, I think it’s fair to say he literally needs no introduction. He was one of the co-authors of the original Semantic Web article in Scientific American. He’s been a longtime well-known professor at Rensselaer Polytechnic Institute. So welcome, Jim. Tell the folks a little bit more about what you’re up to these days.

Jim:
Sure. Just to go back a little further in history, I’ve been doing AI a long time and my first paper was about ’77, but a lot of the work we’re going to be talking today happened when I was a professor at the University of Maryland, which was from ’86 to 2007. And then from 2007 on, I’ve been at RPI where I was really hired to create a lab that really would be a visionary lab on semantic web and related technologies. I think the president of the university saw the data science revolution coming and saw that that was a key part of it.

Jim:
So who am I? What am I? Really, what happened was very early in the days of AI, I was working in a lot of different things. I started under Roger Schank at Yale, took a few years off to work professionally at Texas Instruments, which had the first industrial AI lab outside of the well-known ones at Xerox Park and stuff. Then decided no, I really was an academic at heart. So I came back, went to grad school with Gene Charniak at Brown and went from there to the University of Maryland. So you know my job life history. I’ve bumped around during that time. Living in Maryland, you tend to bump into the Defense Department and things like that and funding and things like that. I was on a few committees and things like that. Eventually asked to come to DARPA for a few years, which is really where a lot of our conversation today probably starts.

Jim:
And then again, just because it was successful and we had a visionary president here at RPI, she asked me to come and said, “Not only do I want to hire you, but I want you to hire a couple other people you’ll work with who’ll help put us on the map and this stuff.” And I hired Deb McGuinness and I’m sure that’ll come up later. And then past 15 years have been a combination of research and administration. So I’ve done both, doing my own work, working with my students, and also trying to really set up some significant presence of AI on our campus, AI and beyond.

Larry:
Nice. Yeah, and we’ll talk definitely more about your research work and everything. But hey, I want to set a little bit of context about how we met, because I know Dean Allemang from the Knowledge Graph Conference community, and we’ll talk a little bit more about the book that you wrote with him later on. But one of the things that he famously says, and always attributes it to you, is that phrase “A little semantics goes a long way.” I’d love to open up by talking a little bit about that.

Jim:
So early on in AI, it was becoming very, very clear to me, and now I’m talking 70s, early 80s, so a long time before we were where scaling means what it does today. But it’s very clear to me that a lot of the problem with AI is it didn’t scale. And meanwhile, I was seeing these other technologies coming along, the ones that really led to the web, that were looking at a much, much broader thing than the typical AI system. So one of the things I started asking is, how do we scale up AI? And we were looking at traditional knowledge representation languages. I actually have a paper from the 80s. I actually did a book with Hiroki Katano, who’s now the… I believe he’s still the vice president for research at Sony, if not something higher. And Katanosan and I actually had a book called Massively Parallel Artificial Intelligence in the 80s, but it became clear to me that the machines were part of the story, but the lots and lots of people doing lots and lots of different things was the much more interesting part of the story.

Jim:
And then also, I’ve always been intrigued by human memory. You asked me a question and I not only answered that question, but I’m doing right now. It’s associating a million things in my mind. And what I’m really doing is winnowing rather than trying to come up with the precise answer. And so I started thinking about how does AI memory start to look like human memory more? In those days, a thousand and then 10,000 and then a million “axioms” were very, very large things, and that’s what I wanted to do. And then the web was coming along and I saw that, well, if I’m going to get a million facts about something, where can I go get it? I have two choices. I can ask Doug Lenat, who in the early days of Cyc was talking about how that was scaling, or I could figure out how to mine it from the web.

Jim:
So a couple students and I started figuring out how to mine it from the web. That led us to realize that if we just had a little bit of markup on the web, a little semantics, we could do a lot. So for example, I said, every single computer science department in the country is creating a website and every one of them has a faculty page with a list of all their faculty and their faculty interests. Why can’t we just go get that? And at the time when a lot of people were writing web scrapers, a couple of my students came to me and said, “What if we just had people put in little tags that said this is where we talked about our faculty?” And that led to us saying, “Well, that looks a lot like knowledge representation. What if we had these little knowledge representation tags?” And out of that came something called the Simple HTML Ontology Extension, SHOE, which arguably was the first semantic web language, certainly the first in the US.

Jim:
And so that’s really when I started getting much more interested in these things. And we had a T-shirt made for our lab, which had the word SHOE on it. And then underneath it in little letters with a little pointer coming up was a slogan, “A little semantics goes a long way.”

Larry:
Nice. I love that. Do you still have one of those T-shirts, I hope?

Jim:
Believe it or not, I still have one of those T-shirts. I’ve either got some moth balls around it now because-

Larry:
That’s great. If you go to KGC, I hope you’ll bring it.

Jim:
I have the entire history of semantic web in T-shirts.

Larry:
Hey, I would love, so the shoot, now we had a good conversation last week to prepare for this and I’ve done some other research, and I’m not sure I have the exact story of how SHOE relates to your research at DARPA and then how that fed into your growing conception of the semantic web.

Jim:
Sure. So basically the idea behind SHOE started with just what if we had these shared, what we now call ontology, still back then we called ontologies, but we were thinking a much simpler level. So we knew they couldn’t just be taxonomies and a tree structure because we’ve been KR researchers and we knew we really wanted networks and graphs. So really we were talking about something that didn’t look that different from knowledge graph except that we wanted to be able to link peoples. And the minute you start talking about linking peoples, you have to either do one of two things, either you have to say, these are the 10 links you’re allowed to use, or you have to come up with an infinite way that people can create their own links and then tell if they were linking to someone else.

Jim:
And about, oh, I don’t know, it must have been 91, 92, one of my students who had been paying more attention to the web than I was at that time said to me, “Look at these URL things. What if we use those URLs as a way to designate a page with an ontology on it? And then I would know if you and I were using the same ontology if we were pointing at the same page.” We started working on that, and after a while we started realizing that we didn’t actually even want it at the page level. We wanted it at the term level. So I wanted to say, when I say professor and you say professor, are we talking about the same thing in the same context or not?

Jim:
And that led us to develop a small language based around that. That language had URLs in it. So when I went to DARPA, I really went there with the idea that I wanted to scale up what people were doing in AI knowledge representation. And it was arguably we were nearing the end of the AI winter. I just don’t think anyone realized that yet. So I started there in 99. So we had had the 86 expert system days, early 80s, the early neural net stuff, and all of that stuff, funding was down, things like that. In fact, at one point I was told I was the largest funder of artificial intelligence in the world with about $65 million a year in funding, research funding. Obviously the world has changed quite a lot.

Jim:
So I was looking at that and I was looking for some examples that might help convince DARPA to do this. And at the time, Ed Feigenbaum had been the chief scientist of the Air Force a little while before. Ed and I had met in a lot of context. Ed, I owe much of my career to Ed, and that would be another whole story. And by the time this airs, he’ll have just celebrated his 90th birthday, so let me wish him happy birthday. But he said to me, one of the things that drives me crazy is you look at an airplane and on the tail of that airplane is typically two letters and then depending Air Force or Navy three or four numbers. That’s called the tail number. He said, “But I go look at the database and there’s at least a dozen different ways people represent tail numbers. How do I fix that?”

Jim:
And I started looking at that and started asking people questions. And again, this could become a very long anecdote, but the fast answer is I started realizing that different parts of the Air Force, different based on their tasks, based on their jobs, had different ways of looking at the same thing. And that you could sort of say, “Here’s a hundred words I would recommend you use of terms, but use the ones you can and then add your own.” And I found some fairly compelling examples of that, some of which were in the classified community, a lot of which were in unclassified, but just where one person would say the most important part of a plane is the weapons load, because to them that’s what an airplane is. It’s something that goes and drops a bomb somewhere. And to someone else, they’d say, “No, no, no, that’s not part of an airplane. That’s something you take off an airplane before you take it to a logistics space.”

Jim:
So I suddenly realized that you had two different people with different conceptualizations when no difference in arguing what an airplane was, but what its usage was, what’s for, et cetera. And so that led me to some really compelling examples. I was doing that in the US. I met my counterpart in Europe who was also interested in some of the same things, but coming from much more of a logic perspective, sort of description logic. I got some funding from DARPA, he got some funding from Europe. We created something called the US/EU Ad Hoc Task Force on Agent Markup Languages or something like that, the word international somewhere. Tim Berners-Lee was one of the honorary leads from the US side because he was in my program. Dieter Fensel was the lead for the European side.

Jim:
We started bringing those languages together. That started leading to some commonality. As I was leaving DARPA, that led to the starting of the Web Ontology Working Group at the W3C, and that’s where the language OWL came from.

Larry:
Got it.

Jim:
That’s a lot of history in the short term.

Larry:
That’s a lot there. And when we were talking to prepare for this, one of the fascinating things about that era was the evolution of your relationship with Tim Berners-Lee. Can you talk a little… I just love the way you described the unfolding of that relationship.

Jim:
Yeah, so it was… We get into some stereotypes here, and I apologize in advance, but since I’m making fun of myself as much as anyone else, it’s probably fair. So I had this idea about doing this program, and usually when you start a program at DARPA, you ping a few people to see what they think. And I had sent some email to Tim and some other people about some things, and I hadn’t heard back – in many years till I realized that Tim just never answered email right away. But I was sort of talking to a former DARPA person, and I said, “I want to get this thing started, but I really need these people at Stanford and MIT and things like that to sort of… How do I talk to them?” He said, “What do you mean how do you talk to them? You’re DARPA. You call them up and say, Do you want money? Then you got to do this.” I’m like, “I can’t do that.”

Jim:
So I sent an email to Tim at MIT that said, “Hey, I’m starting this program at DARPA. I’d like you involved. Would you like some funding?” And 10 minutes later, my phone rings and it’s Tim. Now, Tim, of course, is thinking that I am the typical DARPA guy in a suit, military, doesn’t really understand any of this technology and is really just trying to hook into some of the big schools and what they’re doing. I’m thinking of Tim as you’ll forgive me for saying so, but MIT professors are not typically known for humility and wanting to have a lot of people who disagree with them around them. They tend to build very, very smart groups, but groups that share assumptions. So I was thinking of Tim as an MIT professor and he was thinking me as a DARPA guy in our first couple conversations, I think we were talking past each other.

Jim:
So I went up to MIT to visit, and I had made a little diagram of what I had in my head with some layers of languages and different kinds of maybe logics high up. And I went into Tim’s office, and first of all, I think he was surprised because I was in generic shirt and probably jeans at the time, so I didn’t look like a DARPA guy. And I had also figured out by then that a lot of the people who Tim had surrounded himself within his group were people who would argue with him. So I knew he wasn’t just the traditional… So I think we later learned that both of us said, this is going to be our last attempt.

Jim:
But what happened is I got to his office, I put my drawing down and said, “Let me tell you what I’m thinking of.” He looked at it, picked up his phone, called one of the members of this group, said, “Ralph,” this is Ralph Swick, who at the time was one of the leads at the Worldwide Web Consortium said, “Ralph, he has one too, and it looks a lot like ours. Come bring the group.” So they all came in and we started a conversation, and then they looked at SHOE and said, “Wait a minute, you built a language using URLs?” I said, “Of course, what else would you use on the web?” I think that was the moment we clicked.

Jim:
And the way I describe it is Tim had this view of the semantic web. I describe it as a big circle, and I had this view of adding semantics to the web that I think was a much smaller circle. The good news is it put a pimple on the side of his circle. It made his circle bigger, and I had the funding to make the circles real. And so we started working together, and then I got a call at some point, periodically Scientific American trolls DARPA and says, “You doing anything that’s worth us doing a little column on or something?” I said, “Well, I’m actually working with Timbers Lee and some others on… ” And they said, “Stop right there. You get us a real article with Tim as one of the authors and we’ll publish it.” Because I had previously sent one or two articles to Scientific American which sat in their drawers forever and never saw the light of day, including one on agents, by the way, because now that agentic AI is really like, we could have had the first article of agent.

Jim:
So we brought agents into the semantic web article, but that’s really what happened. And then adding Ora was just both Tim and I knew Ora. We felt like Ora spoke both of our languages, and the article came out of a combination of us doing some brainstorming. We kinda knew where we were now, and we kinda knew what it looked like at scale. And Tim walked up to the whiteboard where we were wrote that and writing down our very, very brief outline. In the middle he wrote, “and then a miracle occurs” based on an old Far Side cartoon where you see two scientists, and I think it was Far Side, but at the top of the board is a dense equation. The bottom of the board is dense equations. In the middle, it says, then a miracle occurs, and one scientist is saying to the other, “I think you need to be more precise there.”

Jim:
And Tim really said to me, “You know, actually we don’t need to be more precise there. If we get the bottom part right, the miracle is that people use it all over the place, and that takes us to the top.” So you build it right and it’ll scale, and that became one of my mantras. And in fact, that fits so nicely with the little semantics goes a long way view of the world that that’s kind of what we wrote up in that article. And it had interoperability, it had disambiguation, had a lot of stuff we talked about now. So really what it had in it was a view of knowledge graphs plus a little bit of inference and just how powerful that would be primarily for bringing data to the web. And once you had data on the web, you could talk about agents on the web.

Larry:
Yeah, a couple of things about that. One, I have to ask, back to the Air Force tail number example, was that in any way involved in Ora’s getting involved because he’s such an aviation fan?

Jim:
Actually, interestingly, no.

Larry:
Okay.

Jim:
I didn’t even know until… I guess I was relating that story sometime with Ora. It must’ve been five or six years later, and he just started laughing and said, “You may not know this, but I collect aircraft. I build aircraft models and things like that.” No, it had really total coincidence.

Larry:
Oh, that’s great. Well, yeah, sorry for interjecting. I just had to ask that. But I think more germane, more germane to this is that the ostensible failure of the semantic web has been rebutted by any number of people. And one of the things they talk about is, well, more than half the webpages on the internet have semantic markup and every single enterprise on the planet has a knowledge graph somewhere in there. So that bubbling up-

Jim:
That’s that miracle, yeah.

Larry:
Yeah, that was the miracle occurring. That’s awesome.

Jim:
In fact, when I went to DARPA, when you presented a program, you had to have a schedule and I had what I called my notional schedule. And on one side, it showed a little kid with a snowball, and then there was an arrow, and then there was an avalanche running down a mountain, and I had a little arrow that pointed to the middle of that big arrow that said, “You are here.” And the DARPA director looked at that, looked at me, looked at that, looked at me and said, “That’s your schedule?” And I said, “Uh-huh.” And amazingly enough, he gave me the money anyway. And frankly-

Larry:
I love that because-

Jim:
…always in my view of the semantic web, we threw the snowball and let other people…

Larry:
There’s that famous chart that everybody shows what we think progress looks like a line up to the left. And what it actually looks like is a bunch of squiggly stuff. And now I’m going to picture that squiggly stuff is like a mountain with… It’s about to have an avalanche that come playing-

Jim:
Yeah, you have a lot of different chunks of snow and something up top causes them to… I have no idea if that’s how avalanche really happened.

Larry:
Exactly. Yeah.

Jim:
I’ve never studied them and luckily I’ve never seen one, but the snowball getting bigger and bigger… In fact, at one point we had the beginning of Rocky and Bullwinkle in one of our things because there’s one… The original Rocky and Bullwinkle cartoons had, but I forget whether it’s Rocky or Bullwinkle, but one of them throws a snowball and you see it rolling down the hill and it captured… Anyway.

Larry:
Yeah. No, interesting.

Jim:
But yeah, we decided that that was going a little too far for DARPA.

Larry:
Yeah. No, well, in addition to that, so this organic evolution was built in from the start, but some of the key insights that underlay that success of that snowball turning into an avalanche was some of… You’ve talked about the brilliance of Tim and a couple of the insights he had early on that really that you were just in awe of, it sounds like. Can you talk a little bit, start with maybe with the 404 error?

Jim:
So I live in awe of Tim. Tim is one of the few true geniuses I’ve ever really met. And I’ve met a lot of smart people. I’ve met 20 Turing award winners and lots and lots of people out in the political world. I’ve been lucky enough to meet three presidents, that kind of thing. But Tim is different. He thinks different. And so he taught me two really important things. One is he said, when you’re looking at something that a lot of people are doing, question the assumptions. And his example of that is when he looked at what was going on in the hypertext world, everybody was making the assumption that when you clicked on a link, the thing you were pointing at would be there. And Tim said it dawned on him that you couldn’t possibly have a scalable system really scalable distributed where that was true because you either needed one of two things.

Jim:
You either needed their central link repository. So anytime a link was changed, it would know to change that, or you would need a human to human thing. So I’d say, “Hey, Larry, I want to point at your webpage, so you should know that I’m pointing at it or you should let me know anytime you change it.” And you could theoretically see a model where if anything I was pointing at changed, I’d be informed of the change. But as the thing grew, then almost all the messages would be change messages. So at one point, Tim was asked, “What really was the insight behind the web?” And he said, “It was the 404 error. It was the fact that you could point to a page and it couldn’t be there.” And that’s what led it grew because now I could point to your page without your permission.

Jim:
By publishing it, you were giving everybody in the world permission to point at it. So that’s lesson number one I learned from Tim. I saw him do similar things a few times. One time while we were in the working group, must’ve been five, 6,000 emails had flown by. And my phone rings as Tim saying, “Jim, there’s these five or six emails talking about this particular feature. You can’t let that feature be in the language.” I’m sitting there going, to me, it was just one term in a big chunk of terms we were looking at. I looked at it carefully and I realized, yes, it has this horrible name, inverse functional property. What it really means is database key. So I was able to say in traditional description logics, every person has a unique ID number, but I couldn’t say every ID number denotes an individual and having both directions is obviously crucial in data and that’s a database key is internal.

Jim:
So if we want an external database key, and I want my database to point at something in your database, we have to have something that can designate as where that cell and that cell can be unique. And in fact, part of solving the tail number problem was that in traditional relational database technology, adding a new field or two, adding columns requires a lot of work, and especially if anyone else is using your database, then they all have to re-engineer and things like that. And in the semantic web, essentially, the idea was that columns could almost essentially be added dynamically just by declaring it. And so my little slogan for the semantic web, at one point I was asked for the elevator speech is, “My webpage can point at your webpage. My database can’t point at your database.” So Tim and I were really thinking data on the web.

Jim:
I had also cut my teeth on agents and Ora was much more into the sort of agent side of things. And that’s, I think, what made the article really work because we went all the way from web technology to web plus data terminologies, and then eventually that you wouldn’t be able to get to agents without that.

Larry:
Yeah, I hadn’t thought about that link between, because that’s what agents are going to have to do is access things like databases to do that kind of stuff. So that early insight about doing things in… Questioning that assumption that probably all the data people involved had about, well, no, that’s just how you do it and Tim’s brilliance to ignore that.

Jim:
And he had one other aphorism that plays into the same thing. So this is the other one, he got me several of them and I could go into long digression, I won’t. But another one he taught me somewhere along the line when we were doing some of the design was build small but viral. And I thought really hard about what he could mean by that because he had said it to lots of people who all sort of, yeah, that’s Tim. But I realized he actually meant something very important. Make it simple, but make it so that when I show it to my friend, they say, “Ooh, I want that too.” And when I show that to my competitor, they say, “Oh, heck, I better have that too.” And now the thing will grow. That’s viral. So it precedes viral memes and all that-

Larry:
No, I was going to say that point – That’s years before the arrival of social media and YouTube and all the current conception of viral-

Jim:
It’s what made it possible.

Larry:
Yeah. Yeah, just his genius just keeps popping out more and more. Well, and I want to talk a little bit more about the article because it’s such a… I read it probably once a year and every time in the last couple of years as I’ve read it, I’m like, wait a minute, it starts with agents. I totally forgot about that. So can you talk a little bit, you mentioned just a minute ago that Ora’s involvement in the project was part of that, but the three of you worked on the article together. How did you decide to open with that? Was that the most compelling story or what was the…

Jim:
So to us, The most important thing was… The one thing that all three of us wrote as the first most important word in our thinking about the article was interoperability. Things had to be able to work with other things, regardless of whether they had been designed explicitly to or not in some heavy duty sense. If you look at Worldwide Web Consortium recommendations, they’re almost never as heavy duty as the ones you get out of traditional standards bodies, largely because they were always built along that notion. Some of them are heavier than some people might like and they have to work. I’m sorry. So I had met Ora through agent conferences. So we were both very early in the AI agent community. In fact, probably my second most cited article after the Scientific American one is one that occurred and Nature had for a while, an online adjunct called Nature Web Matters.
And I had one called, Is There An Agent in Your Future?

Larry:
What year was that?

Jim:
I think it got published in ’99 or 2000, but it was an idea I’d had for a long time. And we had an AI agents community. The planning community as is now called, had a lot of agent people in it. And there were things called BDI architectures. If you look at the early days “cognitive” or smart robotics, it was all there. So I mean, we actually had a fairly thriving distributed AI, AI agents community. So that was where Ora and I had met on the AI side. And Ora had met Tim more on the web side. And so I was the latecomer to the web and Tim was the latecomer to agents, but we saw where these things all fit together. So we wanted an opening scenario which would require multiple people working with multiple data sources and different preferences.

Jim:
And so we came up with the two people trying to schedule the parents hospital visit in such a way it was useful for both of them and where all of the information, remember we’re talking the year 2000 now when we’re writing this, was already on the web. In other words, we wanted a problem that we could solve by hand, but which you couldn’t solve by just asking the web to do it. When we actually sent that to the Scientific American editor, he kind of liked it, but he said, “But we need to open with something that’s more of a human, compelling story.” So he came up with, John was listening to music on his stereo and the phone rang and the stereo got quiet and it was his sister. And the funny part is if you look today at that article, that first paragraph, the one he saw as science fiction worked just fine.

Jim:
Alexa, all those things, but the other stuff is still not there. And some of that is interoperability because big entities came and took over things. And there’s a question of what is their motivation to be interoperable. Right now I go to a lot of doctors because I’m just about 70 and I pay more in copays, I think, than many people pay in medical bills, but I have four different medical portals, each of which some subset of my doctors can use. And I have to physically download stuff as a PDF document, email it to my doctor at a different portal who then uploads the piece of information from that that he finds useful. I’m like, that’s crazy in this day and age. So we still aren’t there. And when people talk about Agentic Web, I feel like they added that “-ic” so that they could ignore the early literature just as deep learning took over from the term neural nets so that people wouldn’t have to remind people that people were doing it in the ’70s and the ’80s, but that may be my old man cynicism coming through.

Jim:
But anyway, so there was this vision that’s still there and is now even more important than ever because there’s more stuff on the web, more things you want to do that are personalized, but require interacting with multiple websites and you do all of the work.

Larry:
Yep. Yeah. Well, at least the scenario you described, at least it’s not a fax machine somewhere in that workflow, so there’s that. No, but seriously, I’m thinking a couple of things. Well, I guess the main question there is what would it take to get to that vision now at this point? The web is, god, we’re 25 years in, there’s way more stuff there. How close do you think we are? How far do you think we are from actually being able to implement the full scenario you set out?

Jim:
So the way I would describe it is there currently are several systems in which if you built the whole thing in that system, you could probably do it. Problem is they’re no one to interact with the other systems. So again, it’s the interoperability thing, but I tell a story when I’m trying to simplify things. I taught a course on the history of AI. So the question was, how do I take 16 weeks of two-hour lectures, actually four hours a week of lectures and turn that into an elevator pitch. And really, there was kind of this symbolic AI thing. A numeric thing started happening with search and stuff like that simply because you had enough data that people could write an article called The Unreasonable Power of Data. And that started people going in that math direction and the semantic knowledgey direction started to fall off a little bit.

Jim:
And then I draw a picture of both of those hitting a brick wall when large language models come along, which came out of an article called Context is All You Need or something like that. I forget the exact title, but it was I think-

Larry:
I think it was Attention Is all You Need.

Jim:
Attention, sorry.

Larry:
Yeah, that was the transformer group.

Jim:
But anyway, it was basically saying that if you had a bunch of words and just could predict what the next word is, that didn’t require a lot of pre-training. A lot of the early vision stuff needed annotation, the language stuff needed semantic annotation. So you went to semi-supervised, self-supervised, and then “unsupervised”. We could get into a long discussion of that. So what happened is both the people who were into numbers as Bayesian type stuff, and the people who were into semantics says, “Look, you’re never going to be able to do this within your own enterprise or a bigger system without some semantics. Start being listened to again as we’ve started realizing the problems with LLMs.” So my view, and the Knowledge Graph Conference started partly, it started before LLMs got big, but you could see them coming, and you could see that a lot more people were saying, “What I need for my company is a way of managing my internal data, but now I’m the size of an Amazon or I’m the size of a Google.” And also those big companies could make metadata agreements with each other.

Jim:
So Google didn’t have to scrape the Amazon site, didn’t have to search it. Amazon would just hand Google all the metadata. I don’t know the exact details and things like that and some of the other sites, whereas the little guys were getting shut out of that. So the little guys started saying, “Well, if we had interoperability, we can compete.” So you went through these different phases of first, the big companies trying to lock out everyone else, then the little companies being able to compete by a lot of little fish chasing the big fish in the old cartoon, and then the big fish saying, “Hey, let’s invite the little fish in.” And that’s when you really saw the super growth of the Google, the Amazon, the Microsoft world.

Larry:
Interesting, that kind of begs the question… Oh, I’m sorry, go ahead.

Jim:
Yeah. Yeah, no, and I was going to say, and that meant you had the computer power to start taking ideas like these transformer models and doing them at scale, but it meant leaving out all of the semantic stuff that made it all worthwhile.

Larry:
It makes me wonder, when did schema.org start and how does that fit in the little guys competing with the Amazons and the mass brokerages of data? Yeah.

Jim:
So schema.org has its history actually a little further down the road. Yahoo was doing some things. They had something called SearchMonkey and there were some other things. But around 2012, you had a company called Freebase that Google acquired. Microsoft had bought, I forget the name of the company that became Bing, but several of these acquisitions happened where they were buying companies that were actually ingesting RDF and/or able to share data in RDF. So they were starting to use the interoperability formats and also to realize that connectedness of these things was important. So Google really reclaimed the term “knowledge graph”. So you can go all way back to the 1970s, find filing papers that used the exact term knowledge graph. Over time, that idea started becoming clearly needed if you were going to manage very large scale assets. So some of that was just in cataloging and things like that, internal to a website.

Jim:
Google started saying, “Hey, one of the examples we used in the Semantic Web article in 2001 actually grew out of a real example, which is one of my students had been at a conference, had met someone he wanted to connect back with. He remembered their last name was Cook and that they worked somewhere and we couldn’t find them on the web, even though he knew they had a webpage because it kept wanting Cook to be cooking somebody who cooks. And later we used Gates as example and things like that, but you needed some precision in there. So Google said, well, there was an article that came out of Google that said, there’s this thing that people have been calling Semantic Web. We prefer to call it knowledge graph. That came down about four days later and was replaced by a blog called the Google Knowledge Graph.

Jim:
And then where Google really reclaimed the thing and said, a lot of links of relations triples were done. The other thing mathematically I haven’t gotten into, but at the heart of some of the development of the semantic web model was it always was in graphs because if I have two trees and I join them, I’m not guaranteed to get a tree. But if I have two graphs and I join them, I’m guaranteed to have a graph. It’s not necessarily guaranteed to be consistent, but it’s a graph. So mathematically, the graph is the simplest model, I believe, that when you unify it, you’re guaranteed something other than a set and sets are a whole different entity. So Tim from the very beginning said the semantic web’s going to have to be based on graphs. And so where the normal web had a webpage and then a link and then a webpage, and both of those webpages could be described as URL, we said, why doesn’t that link also get described by a URL?

Jim:
And then it becomes de-referenceable. And now you can go to the document, which can tell you what that is. And there was some thought about making those just human-readable right at the beginning. And then when my program and DAML and the Semantic Web came along, we said to make those human-readable. And Google was able to just take advantage of that and then say, hey, now how did we turn that into money? We turned that into money by search engine optimization. We tell people, you put this stuff on your page, your stuff’s going to show up higher. And for a while they had their own language and Microsoft had their own language and there were these different embedded languages. It turned out people would call me up and say, which one do I use? And I said, it doesn’t matter because they each read the others.

Jim:
But more and more people started saying, well, we don’t want to be bought in if those companies make changes. And look, there’s this standard out there, which is RDF and its many descendants, RDFS and things like that. That was what people started using. So things like Neo4j and things like that grew out of that. And so the bottom of most knowledge graphs today are things that were semantic web technologies or grew out of semantic web technologies. The things at the very large scale companies, we don’t know what’s down there and some of it is made to be directly compatible with their great big AI learning systems. So most of us don’t have a server farm to play around with and four million GPUs.

Larry:
Yeah, what a different world because I remember 30 years ago we were picturing like, great, you connect your computers to the internet and you can talk to each other. But now so much of the fundamentals are, again, there’s a different view of scale. Yeah, you have to have a few billion dollars laying around to run enough data centers to build these foundation models. So what’s the hope or the promise going forward, what does the technology look like that might finally stitch things together?

Jim:
Yeah, I think, and I can tell the story partly historically and partly modern. Historically, one of the things is I pointed out somewhere around, I think it was 2012 to 2015 in a talk I gave, that I could show you a lot of companies whose internal web was the same size as the worldwide web was in its fourth, fifth, sixth year. So some of the external web technologies were now becoming useful at that scale internally. And therefore, of course, the external web had to be able to link those things and therefore be looking at a much, much larger scale. So you really needed to keep the exponent going. So that’s, I think, a piece of the story that led to higher and higher need for computation. I think people started having supercomputer access would let them play with some of these ideas on supercomputers. The traditional supercomputer was still primarily filled with people doing mathematical stuff, Limpach, things like that, but people started developing some of these AI tools such that they would move.

Jim:
That led to some scaling. People started realizing some of the algorithms we were doing worked really nice on graphics processors, GPU got big on and on. But I think what’s happening now is people are beginning to realize that things are happening at scale that look really great when you’re at a surface level, but now when you want to get to that deeper level, you need a different kind of scaling. And that’s where the knowledge graph stuff is coming in. That’s where Deb has recently been using the term much more often than she used to, “just enough ontology”. So she and I feel like a little semantics has gotten to be a little more semantics and a little ontology or enough ontology has become a little less to become just enough ontology. The two of us are starting to put together talk and hopefully it’ll become a paper if the two of us ever take enough time off of running around, giving talks to actually sit together and think.

Jim:
But that’s what I think the future looks like just in that narrow space. I’m my own company, I want my own data, I want my data sources, I want them to be interoperable, not now in the sense of somebody coming in and grabbing my data, but with a language model. And I don’t want to build that language model because I don’t want to have to build a server form and run it for six months and da, da, da, da, da. So I want to make a business arrangement or some other arrangements that says, if I have my data in this format, you’ll be able to integrate with it. And I started as documents, but now more and more it’s structured documents, it’s triple stores, it’s knowledge graphs. And again, once you have those knowledge graphs, now I have to just be able to tell, when you say Berlin, do you mean the city or do you mean the person, Irving Berlin, or do you mean the donut or the Berliner? And it’s a very-

Larry:
Same the Cook guy, your student is turning. Yeah.

Jim:
So again, so I see the future next five to 10 years is a lot of that stuff starting to grow together. And I don’t know if it’ll grow from industry out or from large AI in. I suspect it’ll be emerging of the two, that people are starting to build their own stuff within their enterprise that’s starting to get big enough that they’re saying, “Hey, I’d really like to be just using a web resource to do this, a web service.” And those web services are scaling. And at the same time, you have the bigger thing starting to say, “We need some places where we have more precision. If we’re going to help lawyers, we can’t be making up the name of cases, and we got to know how to tell the difference between a real one and a fake one.” And all these systems now have built in rules, guardrails…

Jim:
I’ve just been reviewing a set of papers for the Vision conference, which is all about how to break the visual. All the papers they sent me were on breaking visual guardrails, which is weird because the paper I sent them had nothing to do with that. But when they searched my history, they saw I published some of that in language literature. So we’re in this running battle and part of the answer is knowledge and has always been knowledge. Question is how much, what form, what scale, where? And that’s a question I’ve been asking for 50 years, and I hope I’m around long enough to be asking it. I doubt I’ll be around long enough to be asking it for another 50 years, but I’d settle for being around long enough to ask it for another 10 or 20 because that’s where I see things going.

Jim:
And then I’m going to throw one last thing in, I hope it’s last thing, unless you have more questions, is I also think that the GPU running on a server farm, beginning to become running on a supercomputer, beginning to talk to a quantum computer, we’re in a new curve in computing where the traditional drivers of Moore’s Law, I don’t think we’re seeing the end of Moore’s Law. What I think is we’re seeing the end of Moore’s Law as an individual machine, and more and more Moore’s Law as a network, and if we’re going to have those networks, why aren’t they heterogeneous and interoperable?

Jim:
So I’ve just started recently a couple of meetings and workshops trying to bring together people who are working on quantum meets machine learning and AI. I think that’s going to be an important part of the future. We’re not there yet. I think that’s 10 years off from when it will really be practicable.

Jim:
I think you need super computers talking to quantum computers to clean up some of the noise for the foreseeable future. So I see a heterogeneous world. I’ve always seen a heterogeneous world, but the problem is a heterogeneous world where things can’t communicate is a problematic world. And so that’s what I think we did with the semantic web for the web, for the technologies that now sit on top of the web, apps, social network, things like that. And I think in the future, we’re going to see a growing number of new kind of apps that need that as well.

Larry:
Yeah. That’s a fantastic place, I think, to wrap, but I don’t want to just stop because I want to make sure if there’s anything last that you want to make sure you share before we leave or if there’s anything you want to revisit from the conversation, anything last?

Jim:
I think if anyone wants to come away with this with two slogans, “a little semantics goes a long way” is still true. There are places where you need deep reasoning. I’m not against deep reasoning, but those need to be carefully elucidated to be those areas where you need deep reasoning and they need to integrate with the places where you don’t need deep reasoning, where a little semantics goes a long way. The other one is I still use Tim’s two aphorisms, which in my mind have become the same one, which is question assumptions and build small but viral. Look at where those assumptions are not letting you do that and try to find ways around that.

Larry:
That’s great. That’s such pragmatic advice too. If you could just adopt those as your mindset as you go into things. Well, fantastic, Jim. I really have enjoyed this conversation. Oh, but one very last thing, if folks want to connect with you or follow you online, what’s the best way to connect?

Jim:
Generally the best way to find me is just find me on my webpage and there’s a link to my email there. It’s a fairly standard email address, but I like to keep it there because unfortunately spam became so much of a problem that now my traditional spam crawlers starting to take real messages out. So that’s the other thing. If somebody wants to send me mail to follow up on this, say, I heard you on Larry Swanson’s podcast, can I ask you this? That’ll cue me that you’re actually a real person asking a real question, and it’s worth my time to try to give you an answer.

Larry:
Perfect. Well, I’ll just link to your webpage then and let people do the human work that’ll actually connect you.

Jim:
Yeah, and if that won’t work, Hendler and RPI, just remember those two and you’ll find me.

Larry:
Okay. That’s a good mnemonic. Well, thank you so much, Jim. This was an awesome conversation. I really appreciate you taking the time.

Jim:
My pleasure, Larry. And it’s a great podcast. I enjoyed listening, so I’m really happy to now be part of it. Just mad that I didn’t get number 42, but that’s another story.

Larry:
That’s funny. Okay. Quick aside on that, Brad and I were disappointed. We only realized after the episode that it was 42, we had the answer, but I’m glad I got you on, Jim.

Jim:
Take it easy.