Podcast: Play in new window | Download
Subscribe: Apple Podcasts | Spotify | Amazon Music | Android | Youtube Music | RSS
When Fran Alexander looks at the current AI landscape she sees some interesting parallels between the Alien vs Predator science fiction franchise and the way RAG and other architectures are combining LLMs and knowledge graphs.
We talked about:
- the analogy she draws between the Alien and Predator science fiction franchise with LLMs and knowledge graphs
- how the human-esque (if malevolent) cognitive and behavioral nature of Predators aligns more with knowledge graphs and how the unpredictable and stochastic nature of Aliens aligns more with LLMs
- how the eloquence of LLM outputs can deceive humans
- the lack of explainability and transparency in both Alien and LLM behavior, and the opposite in knowledge graphs
- the difficulty of dealing with baked-in biases in LLMs
- the lack of repeatability in LLMs and the opposite in KGs
- the current trend of architectures and practices like RAG that draw on the strengths of KGs and LLMs to get better results, just as the Alien and Predator media franchises combined forces
- how over the past year or so investment in LLMs has overshadowed all other investments, just as Aliens are out to wipe out anything that’s not an Alien
- her approach to AI architectures that combine LLMs and knowledge graphs
- how different kinds of people consume LLM output
- how she helps enterprise decision makers choose whether to address a use case with a knowledge graph or an LLM
- how taxonomists and ontologists can use LLMs in their work
- the Alien Loves Predator UK Facebook group and Alien and Predator on a seesaw
Fran’s bio
Fran started her career as a writer and editor of dictionaries and thesauruses in the UK, and, as technology evolved, she specialised in information architecture, search systems, and digital archives, and more recently, the use of semantics in knowledge graphs and LLM applications. Having worked on reference publications including the Collins English Dictionary, and as Taxonomy Manager for the BBC Archive, she now lives in Montreal, Canada, and is the Senior Taxonomist for Expedia Group. She was Taxonomy Bootcamp London’s Taxonomy Practitioner of the Year 2023.
Connect with Fran online
Video
Here’s the video version of our conversation:
Podcast intro transcript
This is the Knowledge Graph Insights podcast, episode number 15. When two impressive domains converge, amazing things can happen. When the Alien and Predator science fiction franchises joined forces, both enjoyed new commercial success. Similarly, in the AI world right now, Fran Alexander sees knowledge graphs and large language models combining forces to create retrieval augmented generation and similar architectures that work together to create systems more useful and valuable than the sum of their individual capabilities.
Interview transcript
Larry:
Hi everyone. Welcome to episode number 15 of the Knowledge Graph Insights podcast. I am really delighted today to welcome to the show, Fran Alexander. Fran is an independent taxonomist and ontologist based in Montreal. And so, welcome Fran. Tell the folks a little bit more about what you’re up to these days.
Fran:
Hi Larry. Well, it’s nice to talk to you again. I really enjoyed talking to you on the previous podcast that we did a little while ago. And that one was kind of a bit of a general introduction to taxonomies, ontologies, thesauruses, knowledge modeling and semantics. But this time, I thought we could talk about knowledge graphs and LLMs. They’re a big hot topic and I did a presentation earlier on in the year for Taxonomy Boot Camp London, Bite-sized Taxonomy Boot Camp London. That was a lot of fun and has been really popular. A lot of people have been asking me about it. I’ve revisited it a couple of times and that’s LLMs versus knowledge graphs, Alien versus Predator.
Larry:
Okay. So why Alien versus Predator? Why not King Kong and Godzilla or…?
Fran:
I did think maybe King Kong versus Godzilla. Godzilla as LLMs and King Kong as knowledge graphs. Certainly, the idea is that it’s a fun analogy. It’s a fun way to start thinking about the differences between knowledge graphs and LLMs. It’s not supposed to be a serious study of science fiction characters, but you certainly could pick your own pair of monsters and do the analogy there. I did consider, as I say, I did consider Godzilla versus King Kong. That would work, but personally, I happen to really like the Alien franchise. I’m a bit more familiar with the Alien franchise. I thought it was really, really fun to have the Alien versus Predator crossover and that’s actually become quite a successful franchise in its own right. So yeah, so you could use many monsters, pick your own monsters and run your own analogy. But as a starting point for talking about LLMs, they can seem very Alien. They can seem very scary. So that was my starting point.
Larry:
Interesting. Yeah, and so I’m just trying to, I’m getting it as you talk about it. Well, tell me a little bit more about, because in ontology work, we figure out what the entities in the domain are and ascribe properties to them. What are the properties of an Alien that make you think they’re like an LLM?
Fran:
So Aliens are very different from humans and one of the reasons why I like the Alien versus Predator analogy is in these characters, they’re societies and the way they operate and what we know of them are very different. So Aliens, we don’t really know much about Alien society. They’re not at all like humans. They have acid for blood and what an Alien does is basically, just goes around killing everything in its path and making more Aliens. We can’t really communicate with them, we don’t know much about them. They’re very, very different. Their approach to the universe is very, very different to ours.
Fran:
Whereas Predators, are still big, scary monsters, still very powerful, but they’re much more humanoid. They kind of look humanoid. They move in a more humanoid way and Predators actually have a much more human-like society. So they have some kind of moral principles, maybe not that many. They’re basically, they’re mercenaries for hire, but they do have social structures, social hierarchies, complex societies. You can’t go and hire an Alien to work for you in the way that you can hire a Predator.
Fran:
So Predators are starting to get into those kind of complexities, hierarchies. You talk about the way that we build knowledge graphs, they’re usually very specific. So Predators will have a specific target. They’re out to assassinate their designated target that they’re doing for money. Knowledge graphs are very specific and targeted and precise. Whereas Aliens, they’re not really involved in that at all. They’re just going off to do their Alien thing in their own Alien way. So that was my starting point. You’ve got these two contrasting characters, one that’s very, very strange and different. The acid for blood of Aliens, I compared to LLMs having maths for blood.
Fran:
The way that LLMs work in a kind of, and I’m not an expert and machine learning engineers and experts, I don’t know whether they’d completely agree with my overview, but the way I look at how LLMs work is that, you take a big corpus of something – texts, documents, images – chop it up into lots of little pieces, and then you layer on lots and lots of algorithms and calculations and machine learning to figure out what the probability of one piece appearing next to another piece is. And that essentially is what LLMs are doing. They’re very, very complicated probability engines. Whereas knowledge graphs are built up using structures and hierarchies and conceptual models that come from humans and come from people.
Fran:
Now I don’t know anyone who learns by chopping things up and calculating probabilities of bits of text. Humans don’t read books and learn from them like that and humans don’t discuss and describe concepts and approach the world like that. But a human way of looking at the world is in thinking of things with structures and hierarchies that we’re used to. Our taxonomies that are kind of like the backbone with knowledge graphs, that kind of parent-child broader narrower concept relationship is something very, very familiar to us. We talked last time about supermarkets being organized with the dairy section and milk within the dairy section and types of milk. That’s very natural to us and we approach the world with a mental map or a mind map.
Fran:
It’s very much like in ontology. So when you build a knowledge graph, you tend to start with subject matter specifics like the Predator having a specific target and a specific reason for going out after its targets, a specific motivation and you’ll build up, you start to put your lists, your labels, your taxonomies, your ontologies. You’re building them up from a very human perspective with your subject matter experts or your business drivers to come up with a knowledge base that answers specific questions in a focused and targeted way. So that’s kind of where I started with the analogy.
Larry:
Yeah and especially the way you punctuated at the end there. But when you were talking earlier about the LLMs and instead of acid, they have math for blood, maths for blood. And that’s so Alien and I think what’s interesting to me, there’s weird contrast there between, I agree with you that that’s like a metaphor and an analogy that really works for me in feeling LLMs, but I think because of their conversational interface that they typically use, I think a lot of people perceive them as more, ascribe more humanity to them than they deserve. Does that make sense?
Fran:
Yeah, I think that makes sense. I think it’s really interesting. I think it’s one of the dangers actually and there are probably other sci-fi monsters. My mind goes to some of the GELFs in Red Dwarf that could kind of fool you by presenting themselves in the image that you want to see. And that’s something where LLMs are tricky and deceptive because they can be so eloquent and we, as humans, we are fooled by eloquence and confidence and we’re much more likely to believe something that’s a load of old nonsense. But if it’s presented to us with beautiful language and with confident expression, we’ll fall for it and we’ll go for it and we’ll trust it more than facts that may be true and accurate, but are not presented in such a pretty package for us. So we’re not so eloquently expressed and that’s really, really dangerous. That’s one of the tricky things about LLMs.
Fran:
And another thing that I think is difficult with LLMs like that is what goes under the surface. So another kind of point that I make in the analogy, another one of the differences between Predators and Aliens is you don’t really know much about what’s going on in the mind of an Alien. You don’t really know what’s going on under the surface of this nice output that you’re getting from your LLM, but with your Predator you kind of do. It’s clear what they’re trying to achieve. So if they get it wrong, if your knowledge graph is wrong and it’s got bad data in it, you can go and find that bad data. You can isolate that bad data. You can remove it from your knowledge graph and then it’s not going to come up in the results you get from that knowledge graph anymore. So trace ability and that ability to go and improve the results of a knowledge graph, is really important.
Fran:
It’s much more difficult with an LLM to do that. It’s very hard. If you detect bias in your LLM, it’s probably bias that’s coming in from your training data, but you may not… Once you’ve trained it on a massive corpus that can take a long time, you probably don’t know which bits of training data has led to that bias and then it’s just in there, in the model, it’s just been absorbed into the LLM. It’s very hard to go and unpick and pull out and go, “Well, that’s the bias bit. I’m going to take that out,” and have the LLM carry on. It’s much more of a problem. With knowledge graphs, it’s much more straightforward. There’s a problem there, trust, traceability, explainability, accuracy of results. Maybe the knowledge graph is a bit less glitzy and eloquent, but it’s a bit more straightforward when it comes to those questions of bias and validity and validation. And that ultimately goes to trust.
Larry:
Mm-hmm. Because in a knowledge graph, if you discovered some source of bias, you could find it. Whereas the LLMs, as I understand it, even the engineers who built them can’t see under the hood. And more to the point, maybe it’s not like a database. You can’t just pluck out things.
Fran:
Exactly. Yes.
Larry:
That it’s a set of learned patterns that is just math at that point. Yeah.
Fran:
Yeah, exactly. Somewhere in those weightings and algorithms and models, something skewing it one way or another. But the other thing that’s hard with LLMs compared with knowledge graphs is the repeatability. And this is why prompt engineering has become a hot topic and such interest, there’s a kind of fluidity around the output that you’ll get from an LLM because your input can vary. So there’s a kind of lack of stability. You’re not necessarily always going to get the same answer out of your LLM. Whereas with your knowledge graph, it’s very clear you can go and you can examine your ontology classes and any inferences that are being drawn in the datasets that have gone in there and you can go, “Oh, yes. It’s predictable. We put this query in, we get these results out, and those results are going to be comparable over time.” They’re not going to fluctuate if we express the input in a slightly different way. So that’s another way that knowledge graph data can be easier to handle.
Larry:
As you’re talking, you’re reminding me of the Alien and Predator franchise, like two venerable science fiction franchises then joined forces and became something even bigger themselves. And this feels like there’s something similar happening now in the knowledge graph and LLM world. How do you see that unfolding?
Fran:
Well, I think the analogy that’s a great way of describing it. The analogy is that actually, you can see these differences. They’re two very powerful tools and what people are looking at now is how do you use them together? And I think that’s really interesting. People talk about retrieval augmented generation, which is using things like knowledge graphs to work with LLMs. So you get the best of both worlds approach by saying, “Okay, the LLM…” It’s really good at particular things. It’s really good at passing out natural language and coming up with nice eloquently written paragraphs in its results which people love. People do love interacting with that.
Fran:
Your knowledge graph though, has got your steady stable source of facts. So retrieval augmented generation is about using the two together and there’s different architectures that people are experimenting with and working with. So do you sit an LLM on top of a knowledge graph? Do you say, do you train the LLM only on the data that’s in that knowledge graph or on validated sources of data so that then you still get the nice output but it’s restrained, it’s not just data that’s come in from anywhere out on the web, it’s data that you’ve controlled?
Fran:
Do you use a knowledge graph to fact check the output that you’re getting from the LLM and can you use the two together in a sort of feedback loop from one to another to use the knowledge graph to provide a starting point, then have the LLM run over more content, produce you answers, compare that with the knowledge that’s in the knowledge graph, feed back into the knowledge graph and improve the knowledge graph? Another thing, another issue with LLMs, a tricky thing about LLMs is that you do tend to train them up and get them going, but then the data goes stale over time and it can be harder to top up an LLM with fresh data. Whereas it’s very easy to keep, once you’ve got your knowledge graph set up, it’s very easy to keep adding more data into that and then that data is available immediately.
Fran:
So some people are looking at using the knowledge graph to kind of keep the LLM up to date. And a lot of people are talking about putting guardrails around LLM responses. This is a lot of things that content strategists are looking at. What can we allow the LLM to say? What kind of constraints? What kind of questions can we allow it to answer and what questions should we not allow it to answer? And a loss of that can come from a knowledge graph and can be set up with taxonomies and ontologies that work alongside the LLM. So that’s, I think, where the future is heading. It’s a bit of a finding the right use cases and finding good ways of getting the best of both.
Larry:
I’m trying to figure out now a way, I don’t want to overdraw your analogy, but one of the things that seems to be in these architectures is that the LLM, because of its natural language interface and conversational nature, seems to appeal to people more, but you don’t want to rely on it for facts, but it might be good at writing the query that would get those facts out of a authoritative source by a knowledge graph. What do you see in terms of that last quarter inch, that last little bit of interaction to ensure that you get the accurate, because everything you said about knowledge graphs, it’s like they’re kind of authoritative, up to date, which might not always be the case with an LLM?
Fran:
Yeah.
Larry:
How do you see that little querying and answering loop going?
Fran:
Yeah, I think using the two together, I like the idea of kind of an LLM layer that sits on top of knowledge graphs and so, you let human beings chat via the LLM, but you have the knowledge graph sitting underneath it to make sure it stays accurate and valid and so on. The other thing to think about is cost because LLMs and in my talk, I talked about LLMs consuming all investment. I’d say like Aliens are out to wipe out anything that’s not an Alien. It has felt like over the past year or so, that the LLMs have been wiping out investment in anything else because everybody just wants to throw lots of money at LLMs and not invest in anything else. But actually, they are pretty expensive. The time to answer query can be quite slow depending on the size and the scope of the LLM. And they tend to work better. They tend to be more eloquent when they’re trained on more data.
Fran:
So that can be expensive. You can’t cut costs by having a little LLM. The smaller it is, the less eloquent it is and knowledge graphs, retrieving data from a knowledge graph via a sparkle query for example, can be really, really fast. So a business needs to think really carefully about what jobs it’s getting the LLM to do. And I think this is something that’s going to sort itself out. People are going to figure out, well yeah, when I’m just doing factual retrieval, it’s much, much quicker, less expensive, more reliable, more environmentally friendly to have a knowledge graph. But that top layer where I want to put a virtual assistant, a little chatbot in front of my customer so that my customer can type in natural language and get a nice natural language answer back out, we’ll just use the LLM for that little kind of icing on the cake layer and not make it do all the heavy lifting that a knowledge graph can do more effectively.
Larry:
Yeah, you got me thinking… I wonder how soon we will see cost affecting these architectures because it seems like right now, my feeling, it feels like the cost of the LLMs is kind of hidden somewhere that if you were really comparing, if you were really paying the bill to answer that question… I don’t know. I haven’t done this at enterprise scale and been in that P&L ownership to actually see that. Do you have any feel for that? Are there ways in your ontology practice that you can expose those costs or make people aware of them?
Fran:
I think a lot of it, well then there’s a lot of variation. So it really depends where you’re starting from. It’s like if you want to go there, I wouldn’t start from here. So a lot of organizations have already invested in big knowledge graphs. So for them it’s kind of a no-brainer to keep using the knowledge graph technology that’s already been built and developed with subject matter expertise, domain expertise and so on. And for some organizations actually getting up and running with a knowledge graph is going to be quicker and easier than getting set up and training an LLM to be a specific in-house LLM. So there’s different ways to look at the costs. What have you already got?
Fran:
So a lot of organizations already have a lot of structured data that’s very close. Even if they haven’t actually formalized that into a complete knowledge graph, it’s not much of a step to take that into a knowledge graph and have that there as a knowledge base for an LLM. Whereas it would be a lot more complex process for them to build their own in-house LLM from scratch and go through some of the hidden costs, easing the hallucinations, the answers that are not right, the answers that are right 80% of the time, but 20% are completely wrong. The business criticality of that plays a huge… There’s a huge variety of use cases. And I think what people are doing a lot at the moment with LLMs and when they’re experimenting with LLMs is they’re actually looking for those use cases where it doesn’t matter if it’s completely wrong occasionally. And that-
Larry:
Yeah, that’s… Go ahead.
Fran:
Yeah. Those are quite hard to, well they’re quite distinct because you start to think about it with your business. Generally anything to do with money and the law, you want to be accurate or things that are going to be to do with people’s health or criticality of people actually getting what they expect. Those are where you want really good, really good reliable facts and 80% of the time being right and 20% of the time being crazy wrong is not good enough. But there are other use cases where maybe it doesn’t matter so much if it’s wrong sometimes, is that the human beings going to interact. So that interactiveness is an interesting one to look at.
Fran:
Can the human being, whether it’s an internal analytics person for example or a taxonomist using an LLM to build a taxonomy or an ontology or using an LLM to write some code, if you’ve got a human being who’s taking those results and then what they’re doing is looking at those results and going, “Okay, I’ll use that bit, I’ll use that bit, and now I’m going to throw that bit of nonsense away and have a laugh about it,” then that kind of interaction is very different from a kind of interaction of say, a customer with a problem trying to contact customer service and get their problem solved or someone with some kind of critical situation that they need an accurate answer really quickly.
Fran:
I need to put this insurance claim in and I need to know how to put this insurance claim in. You don’t want to be told something, when you are claiming on your insurance and you need the money quickly, you don’t want a result that’s right for 80% of the customers, but completely wrong for you. That’s not good enough. So I think there’s kind of situations like that where the way that you’re dealing with the responses is different and what those responses and answers, the purpose they’re serving is different. And so there’s different levels of criticality and different needs for accuracy.
Larry:
As you say that, all those different use cases, it sort of speaks to different priorities at every… I can just picture a whole range of architectures serving this because some of those, if you’re in, I don’t know, financial services, you better be 101% correct every time or you’re going to face some legal liability. And what was the other thing about the…? Oh, the trade-off, it’s not just financial cost, it’s sort of optimally using your enterprise resources. Do you have any feel for that? If you’re coming into a project like an ontologist does with a use case that you’re trying to get, and when you get to the point of should I do this with a knowledge graph or an LLM, do you have any rubrics or frameworks for evaluating that kind of decision?
Fran:
Well, I think it goes back to those understanding the purpose, understanding the need for accuracy, understanding what you already have formalized already, what the other elements are in the project that you’re putting together. Are you looking to put together different aspects of technology for a kind end-to-end overall ecosystem or you are looking to provide specific answers in one particular part of the business? I think those are different viewpoints, different perspectives on solving the problems. I think there’s a lot of experimentation going on. One of the questions that I get asked a lot is, can you use an LLM to make building a taxonomy quicker and easier? And I think that’s quite interesting because in those kind of specific use cases, what the LLMs are doing is very similar to what we used to with what used to be called text mining or concept extraction and that a lot of the semantic systems do with pre-LLM different techniques and technologies. And it’s not clear to me that what the LLMs do is actually better.
Larry:
I want to elaborate on that because is that that thing of them being kind of glib and eloquent, but maybe not actually informed? Is that what’s going on there, do you think?
Fran:
Yeah. Yeah, exactly. So you can say to a traditional concept extraction, give me the key concepts in this document and it will give you lots and lots of key concepts that are good candidates for a taxonomy. And it might even suggest patterns and hierarchies that you can use. You do that with an LLM and it’s the 80/20% thing again. It will come up with a kind of… And I think you see this actually in images. I don’t know if you’ve played with the image generators. It’s more or less right, but then there’s this just crazy bit that doesn’t work.
Fran:
So it’s got a person with three legs suddenly or… You try and get image generators to draw people playing the violin, they can’t do the bow. They’ll often have a bow sticking out of the side of their head. These things that when you look at it as a human being, you’re like, “Well, it’s nearly right, but that bit’s crazy.” And that seems to me to be what we’re getting with LLMs, when you ask them to say, “Build me a taxonomy.” A lot of it’ll be really good, but then there’ll be this chunk that’s just completely out there and you just have to be able to pick that bit out and go, “Yeah, sorry.”
Larry:
You’re reminding me now, in terms of professional taxonomy practice, because I’m thinking of a bunch of things here that, I have a number of friends who, content modeling friends who really appreciate some of just the grunt work that an LLM can do for you. And you got to do some tidying up, but it’s generally worth it to do that. But I also have a lot of engineer friends who a year and a half ago, they were super excited about just given coding tasks, the ChatGPT and going, “Wow, that’s pretty good code.” But increasingly, they’re kind of backing away a little bit and going, “You know, it’s a lot harder to troubleshoot that code that I didn’t write.” Is there an analogous thing or are there practices you can do… I guess the more germane thing might be, what can you do as a taxonomist to get something usable with the LLM?
Fran:
Yeah. Yeah. Yeah, well, I think it’s fine as a source of research. So I think good practice is always to not to rely on one source, it’s back to that old chestnut that we all know. Have multiple sources, cross-check things against different sources, look at a variety. If you’re looking to build a domain taxonomy, look at other similar taxonomies in that domain. Look at what other people have done and compare and pick the bits that you need. And I think adding in, and we have different kind of concept extraction techniques that are kind of similar. They’re not always perfect. You use them as a starting point.
Fran:
And that kind of taxonomy building process. So to add that in as another source into the mix, I think that’s fine and that’s useful. And that’s like the code is saying, “Well, you know, I might get that as a starting point.” But you do have that little bit of a danger of maybe the eloquence will kind of make you gloss over or not notice a bit of craziness that’s crept in there or it’ll sort of influence you in an unexpected way. And I can see how some people would not like that and would rather just say, “Okay, I’m going to start with my own domain expertise and it’s easier for me if I’ve written all the code, I understand why I did it in this way.”
Fran:
But that’s back again to what we were saying about the Alienness. Maybe you don’t know what’s gone in, what’s behind the scenes with the result you’re getting from an LLM. So you can’t be sure why. One example that I can give you playing around, getting LLMs to categorize things. They can throw in things that you go, “Well, that really doesn’t belong in this category.” It’s like you’d say, “Will you give me a taxonomy of items of clothing?” And most of them are items of clothing, but somewhere in there is motor car and car seats and something like that.
Fran:
And you go, “Well, why would a car and car seats be items of clothing?” And you try and think what could have come up there? Maybe there’s some leather car seats somewhere. And maybe it’s trying to draw a connection between, there’s been some descriptions of car interiors that sound a bit like descriptions of clothing. So now it’s pulling car interiors into and cars into the set of things that it considers to be like clothing. And that’s the sort of sudden bit of craziness that if you let that slip through your quality control, you’re going to have a real problem. But if you’ve got a human there, you’ll probably catch it.
Larry:
Yeah, you’re reminding me of another thing that a friend said that it’s like you’ve just hired 10,000 interns who are maybe precocious and bright, but not as grounded in the work and the domain as you are. Hey, I can’t believe it, Fran. We’re coming up on time already, but before we wrap up, is there anything last, anything we haven’t got to yet or that you just want to share before we close?
Fran:
Well, I think we’ve talked about most things, all the things that we’ve gone through. I think that the message for the future I would say to people is that you want to have the best of both worlds. So an image that I picked up in my talk for Bite-sized Taxonomy Bootcamp London that I found that really amused me was Alien Loves Predator UK Facebook group. And it had two like cosplay, people cosplaying an Alien and a Predator, and they were in a children’s playground and they were on a seesaw together. And I think that’s what I like the analogy to bring to people is that this is a way of thinking. We can think about these big monster technologies, but we can make them play nicely together. And we should remember, we are in control of the technology. We shouldn’t let the technology be in control of us so we don’t have to be afraid of it and it can be fun. So I’d like to end on that note.
Larry:
Thank you. No, that’s a perfect note and I’m going to go out and try to find, if you can send me a link, I want to put the link to that in the show notes, the cosplay Alien and Predator.
Fran:
Yeah, yeah.
Larry:
One very last thing, Fran, if folks want to connect with you, what’s the best place to find you online?
Fran:
So I’m on LinkedIn. So there are a few Fran Alexanders out there, but Fran Alexander on LinkedIn, taxonomist and Montreal will find me, and that’s a good way to get in touch.
Larry:
Perfect. I’ll put that in the show notes as well. Well, thanks so much, Fran. Always fun to talk and this was a particularly fun conversation.
Fran:
Well, thank you, Larry. A pleasure too. Always fun.
