François Scharffe: An Agentic Harness for Knowledge Engineers – Episode 58

photo of François Scharffe, expert on agentic knowledge engineering and co-founder of The Knowledge Graph Conference
François Scharffe

As AI unleashes a host of new tools and agents to help build knowledge graphs, engineers like François Scharffe are tasked with harnessing these new capabilities and deploying them effectively.

François brings decades of AI experience to the goal he set for himself earlier this year: to automate his own job. He’s making good progress, but he’s also finding that there’s still plenty of need for him and other humans in the workflows he’s designing.

We talked about:

  • the upcoming 2027 edition of the Knowledge Graph Conference
  • his long history of working with various kinds of AI
  • the distinctions between knowledge engineering and software engineering
  • the rationales behind his choice of Pi as his coding harness
  • an overview of his harness set-up and knowledge engineering workflow
  • the benefits to knowledge graph building that AI brings
  • the variety of tools and agents he has built for tasks like requirements specification, data sourcing, ontology design, term extraction and re-use, entity resolution, data mappings, ETL pipelines, etc.
  • the evaluation processes he’s using to assess the benefits of his new workflows
  • the ongoing need for humans in the loop, particularly for tasks like ontology creation
  • how he has adapted his work style for the AI era, in particular the number of tasks he now does in parallel with AI
  • how he manages ongoing improvements to his system – tracking decision traces and implementing introspection steps as well as devising scenarios and using “critic” agents to improve processes
  • the reason he downplays prompt writing and review in his operation in favor of system evaluations
  • his scientific approach to system evaluation, aiming for reproducibility when moving from one model to another
  • the focus on agent memory in the 2027 Knowledge Graph Conference program

François’ bio

Francois Scharffe is a hands-on enterprise knowledge architect and semantic AI leader. He has built and stewarded production knowledge graphs across several domains, designed ontology, taxonomy, metadata, entity resolution and data integration capabilities, and connected structured knowledge systems with AI and agentic workflows. His career spans enterprise engineering, data and AI strategy consulting, product leadership, research, and community building. François is also a founder of the Knowledge Graph Conference, a leading forum for knowledge-centric AI technologies.

Connect with François online

  • LinkedIn
  • email: francois at knowledgegraph dot tech

Resources mentioned in the interview

Video

Here’s the video version of our conversation:

Podcast intro transcript

This is the Knowledge Graph Insights podcast, episode number 58. AI unleashes a whole host of new tools and agents to help engineers build knowledge graphs. Harnessing these new capabilities and deploying them effectively is the job of the modern knowledge engineer. François Scharffe brings decades of AI experience to the goal he set for himself earlier this year: to automate his own job. He’s making good progress, but he’s also finding that there’s still plenty of need for him and other humans in the workflows he’s designing.

Interview transcript

Larry:
All right. Hi, everyone. Welcome to episode number 58 of the Knowledge Graph Insights podcast. I’m really excited to welcome back François Scharffe. He was in the first episode of the podcast series a couple of years ago with his colleague Tom Deely. They run together the Knowledge Graph Conference. But François is also a practicing in-the-trenches knowledge engineer in New York City, and we’re going to talk today about some cool workflow and architectural stuff he’s developed. But welcome, François. Tell the folks a little bit more about what you’re up to these days.

François:
Hey, Larry. Great to be with you today. What am I up to today? A bunch of things. I mean, the number one thing is the Knowledge Graph Conference, which we’re preparing for May in New York. So we’re moving from Cornell Tech to Manhattan, Central Manhattan, Bryant Park. This is exciting, larger venues, more rooms, larger rooms. And we’re going to open the program, the call for presentation soon. So if you’re interested to discuss knowledge graphs, you have interesting projects to discuss, then yeah, keep an eye open for the call that’s going to come out soon.

François:
Yeah, but I think you invited me to talk more about my other activities around knowledge graphs. As you say, I’m a knowledge engineer. I started my career in academia, and I’ve always come back to it at some point, either teaching or doing research. But now for several years I’ve been engineering, so I’ve been a knowledge engineer, in the weeds of the practice, and recently got pretty excited to see how AI is changing the job. So happy to discuss that if you’re interested as well.

Larry:
Yeah, no. We had a quick chat. I was in New York last week, and we had a quick chat about it, and I just got unreasonably excited about what you’re up to because everybody’s wrestling with this stuff now, like how to incorporate LLM, generative AI and other AI tech into their engineering workflows of all kinds. But specifically for this audience, for a knowledge engineer to be doing the stuff that you’re doing…

Larry:
Well, I guess, first tell me a little bit like… I guess like any software engineer these days, you feel obligated to use these tools. And then you have to harness them in some way, and that’s led to this new term, I think arose within the last year, of the harness. And you’re working on a very specific kind of harness specifically for knowledge engineers. Can you talk a little bit what the impetus was for that, why you decided to approach your work this way, and then maybe just a quick overview of how it works?

François:
Yeah. Yeah, yeah. So, I mean, the first thing I want to say is I’m a knowledge engineer, but also a background in general machine learning and data science, natural language processing. And I’ve been following the evolution of neural networks to language models and large language models for years, being more or less involved practically with them, but close to the developments there. And remember GPT-2 coming out and the poems it was writing about unicorns and finding that so amazing. But how long ago was it? I can’t even remember. 2018, 2019.

Larry:
I was going to say that’s about eight or 10 years. I remember.

François:
Right.

Larry:
Yeah.

François:
So when there started to be a big excitement about AI, I was originally quite skeptical, still following but not really being so enthusiastic as to its possibilities to do everything, especially in my field, knowledge engineering, that… We can come back on the details of it, but it gets pretty complex in terms of the sort of workflows that are involved. But yeah, last December there was this realization that… I think it was Claude 4.8 or GPT 4.8. Sorry.

Larry:
I think it was Claude. If I remember right, all my engineer friends got excited about Claude, I think is the one that really like, “Okay, here we go.”

François:
Right. I think it was Claude. Yeah. And I was like, “All right. I think something has changed now. We passed a capability level, and let’s go and jump to that.” And so now being a knowledge engineer and being a software engineer, those are separate jobs. We’re pretty clear about it. And I think the models and the software coding agents, like Claude Code or Codex or others, they have large prompts that makes them specialize for coding, so for coding tasks, for code reviews, for code basis maintenance, et cetera, et cetera. And that told me that those were not necessarily the best tools for building an AI that does knowledge engineering.

François:
Because one reason is that this will take a lot of context space with specific instructions for coding that are not necessarily relevant for knowledge engineering. Although there is coding involved in knowledge engineering, it may not be as complex. You will write scripts or some programs, but it will be always in a part of a certain workflow in a repeated and self-contained way compared to general software engineering. So that was one thing.

François:
And then another was we need specific tools and specific workflows for knowledge engineering. But also, I think there was part of me that likes open source and building my own things and adopting Claude Code was for me like, “This is something we have no control over. We don’t even know what’s in it.” Now we know a little bit better, but at the time.

François:
So I was looking around and found this minimal coding harness called Pi. And there are others. There’s OpenCode, but I went for Pi. It’s very minimal and extensible and made for being extensible, for being extended. And so I went there and adopted it and was like, “Hey, I’m now an experienced professional, and I’ve been doing that job for years, and I have a lot of knowledge on what a knowledge engineer is. Why don’t I try to build an artificial knowledge engineer to automate my job?” And the New York came, and the New York… I woke up, and I was like, “All right. This year, I’m going to automate my job. I’m going to make myself useless.” So yeah, that was the goal with it.

François:
And yeah, harness, to go back to your question… So a harness is a minimal environment that sort of helps contain your wild LLM, your wide AI models. If you just talk to that model, interact directly with the model and asks it to do some knowledge engineering task, it will go left, it will go right, it will go where it decides to go like a wild animal, a wild horse, but not necessarily bring you from A to B in a straight line. Eventually you may reach your point, your destination, but you may take turns and maybe fall sometimes as well. So harness is getting all the things that allow you to steer that model for a certain context. Yeah.

Larry:
Yeah. Because you mentioned earlier the distinction between knowledge engineering and software engineering. And some of the stuff that distinguishes knowledge engineering is all the ontology modeling. It’s like this weird… Not weird, but this unique combination of knowledge engineering, data engineering, ontology modeling, and plus all the enterprise data management stuff that goes underneath it. And so your harness… And also, so there’s this diverse set of needs that are unique to knowledge engineers, plus the models are changing all the time. And so within that harness, you’re swapping out horses or something, right?

François:
Exactly. Right. Yeah. Maybe the harness is more like a carriage where you can swap your horses and you have a storage for your goods, et cetera, et cetera. Yeah. I mean, so the basic of it… And I think you mentioned another point about swapping the models. It was also a motivation for building my own harness.

François:
So Pi gives you these… To go back to some concrete things, it give you this minimal coding loop, agentic loop where you plug your model, and then it has access to your command line and can read and write files and execute commits. And that’s all you need for doing really anything. Based on that, you can take over a machine and code it and do whatever you want. And then you extend it by adding tools that do specific operations and skills that also do specific operations. And so I’m in an enterprise context, so I need tools that are able to access all the different elements, all the different services that I’m interacting with for my job. So that’s one thing.

François:
And then two, as you say, knowledge engineering is a certain workflow, and I’ve been developing my workflow. So depending on your context, where you work, what you’re doing, the workflow will vary. But if you’re considering building a knowledge graph, it’s back to basis. You’re taking multiple data sources, and then you’re integrating them. Some are structured, some are unstructured. There may be some processing involved in unstructured data in order to extract some structured knowledge out of it, different data sources that you access in different systems. So your harness has to be able to communicate with all the systems.

François:
And then you will integrate these sources together. So you will perform entity resolution in order to identify the things that are in common across sources. So we can take an example here. So if we say we have a knowledge graph where there are companies and persons, and so it’s a bunch of data sources that contain companies, a bunch of data sources that contain persons, you will integrate all these companies together, all these persons together, and then there will be relationships between these companies and persons. And so you will need some data entity resolution processes. You will need an ontology to represent what is in your knowledge graph.

François:
And then eventually, at some point, you will put the results in a knowledge graph. So there may be a graph database involved, there may be some materialization, some virtualization. And then downstream you will consume that data. So you will have queries to your knowledge graph that will be consumed by consumers that we’ve also seen evolving pretty dramatically from apps that query the graph to agents that talk to the graph and figure it out. And so there’s a bunch of things involved here as well that I’m looking a bit less at, but are as important as well.

Larry:
Yeah, that’s really interesting because it used to be just a couple of audiences. Or not a couple, but generically it was sort of applications and analytics were sort of the main consumers of all this stuff. And now we have AI themselves. So there’s this kind of… You’re building this stuff with AI, and you’re building it for AI at the same time. So it’s AI everywhere these days.

François:
Yeah. I think it’s a blessing for knowledge graphs. Because traditionally, it’s been hard to build a knowledge graph and it’s been hard to consume a knowledge graph. So if you take a random sample of front-end developers or even back-end developers, they don’t know how to spark it. They don’t know that. So you’re ending up writing middleware layers that convert, and sometimes it’s artificial. But now you just plug your agent and people just start to ask questions and your AI will figure out how to extract the knowledge from the graph. And we can talk about this a little more, about the consumption and things. There’s a lot onto what you need to provide in order to make that work.

Larry:
No, that’s really… Because all the usual stuff goes in there. All the competency questions and ontology stuff and the data engineering stuff goes in there. But then there’s the need to be ready for agentic architectures and workflows. This conversation … because that’s what you’re doing, but it’s also what you’re serving sort of. Yeah.

François:
It’s very meta. Yeah.

Larry:
Yeah.

François:
Yeah, yeah. So maybe going back to this harness, so where you have this basic agentic loop, then you add the tools that are able to communicate, to do your SQL queries, for example, accessing the SQL server, finding the right data in the tables, figure out their way. So you build these tools that are able to access all your source systems, that’s one. And then you build these skills that are encoding the procedural knowledge about how you are building your knowledge graph.

François:
And then you have specialized tools or sub-agents. So it will depend on the cases, but specialized tools, let’s say, that are specific for some part of that process. So you mentioned competency questions. So the first step will be… The way it works is that you start with requirements. So A, here’s a new data source. We want to add that to the graph and be able to query at the end, and maybe we only want this part of the data source, et cetera, et cetera.

François:
So the agent will take it. It will do a profiling of the data that will the first thing, analyze what’s there, look at the fields, and then write competency questions based on the requirements. So what is the end thing we want to be able to query for this new data source? Eventually combining it with other data in the graph. But basically write these questions plain, natural language.

François:
Then it will look at the ontology. What do we have in the graph already and what is in the ontology? What are the terms that support this data source? And then come up with a proposal with terms that can be reused, terms that can be extended, et cetera, et cetera. Once you have that, then the next step, the next logical step, is to do the ETL or write the mappings from this data source to your knowledge graph. So it will do that, write these mappings.

François:
So that will vary. That’s the part that will vary the most depending on the context you work on. So here you may work with a graph database, with a SQL database. You may materialize your data. You may virtualize it, leave it in the source, data source. And the way you also write this mapping can vary. There are some domain-specific languages for that. You can do some Python code or some code that will just read the source and then convert it. Depending on the platform you work on, that will vary a lot. But the task is always the same. “Here’s my source, data source, and here’s my ontology, map the fields.” And then do something with that, either transform it or pass it to a virtualization tool. So that’s what this task is about.

François:
And then you will also want to manage the entity resolution, so be able to distinguish if the companies that are already in the knowledge graph that are coming from that new source, say they are the same, manage identity, have them through a common URI and then merge the fields. So all these processes is encoded there, and you have specific tools for the different sort of tasks there.

Larry:
Almost all of that stuff you just mentioned is the tedious, just… Not boring, but the tedious grunt work associated with knowledge graph construction and design. And you mentioned at the start of this section how much this has helped in that. And those are great examples of that. How much faster are you now, both generically in this sort of LLM assisted era, but specifically with your new framework?

François:
Yeah. It is difficult to evaluate. And we’re actually starting now… I’m looking at evaluation processes for that because it’s a different way to work. There’s still human in the loop, heavily human in the loop. I mean, it’s been a phase. It’s been almost a year. And so the excitement at the beginning was like, “Wow. I’m out. This is all going to be the job. Here’s a task. Do the whole thing.”

François:
At the end, the things that I didn’t mention about the rest of the process is once you’re done, things are merging the ontology, the data is converted, then you take your competency questions. You transform them to SPARQL queries. You have validation steps. This is all you’ve been doing the whole job. But I’ve been burning myself as well there where things will work for a simple task that is pretty well-defined like that, but sometimes it gets trickier when you have an enterprise knowledge graph with dozens of data sources. And there are different steps inside the graph where you will have a bare data, but then you will generate some other data based on this data, et cetera, et cetera. So there will be consequences and a stream of actions that will be taking different sort of transformations. And the agent has to figure all of that.

François:
And I’ve been in these situations where it looked nice at first. And I’m sure I’m not the only one, but yeah. The AI does the whole thing and say it’s done and clean. But then when you look at it in details, there’s this little detail that doesn’t work, and then you’re starting to pull that string and then realize that it took some directions that were completely wrong. So the human in the loop, it’s still heavily there.

François:
So I mean that opens a whole new discussion about how to work with AI, how to be an engineer in that new setting, which is something that I’ve been experiencing first thing. And it’s not easy. It’s a readjustment to work. So instead of having one task that you focus on and you progress step by step, here you launch a task and then the agent will reason about it and progress through it, but you’re idle in that time.

François:
So usually I will work with a bunch of tasks in parallel and then jump from one to the other. And it’s cognitively tiring, and you have to be prepared for that sort of work. But I think, I mean, if you look into it, it’s not necessarily very different from, “Hey, I’m compiling my program,” in the ’80s or in the ’90s. Big program that compiles. You have this xkcd comic where the guys are fighting where their fake swords in the office like, “What are you doing?” “Well, it’s compiling.”

Larry:
I’ll find that xkcd in the show notes.

François:
Yeah. Yeah, yeah. So what want to say is that it’s not a new thing that a software engineer or a knowledge engineer has to wait for something to be done in order to be able to progress in their work and… But yeah, still, it’s still sort of a new thing. Yeah.

Larry:
Well, some of the stuff we talked about last week though, it seems like you’re building some of the resolution of some of these issues into your workflow. You talked about chasing decisions. You do a lot of logging. You’ve got this whole introspection thing you do. Over time, will that… That’s a different kind of human in the loop. It’s like you’re encoding in the system ways to account for those issues you just identified. How do you see that maybe… I guess it’s probably too early to say for sure, but how do you anticipate that kind of… like the learning within your system improving things over time?

François:
Yeah. Yeah, yeah, yeah. Yeah. So I mean, once you’ve set up the basic workflow and your tools, then the main job is to improve it and make it more reliable. So when the task starts, you have more confidence in what is being done, and you have less human supervision or more remote human supervision, and also you get from point A to point B faster. So that was the next step.

François:
So a bunch of things there. One is to keep traces of everything, so everything that is being done, all the conversation history, all the tool calls, all the intermediate steps, the data. Keep traces of that so you can analyze it later, eventually using the AI to analyze it. But one of the first tools that we built was the introspection. So after every task, the harness will trigger an introspection. It will analyze the task conversation, the logs, the tool calls, et cetera, et cetera, and look at things that went wrong, so if a tool was called and that resulted in an error for one reason or another.

François:
And it will compile these and then make a list of suggestions that you can either tackle right away if there are things that are flagrant that say, “Hey, this is a bug in my tool, just go and fix it.” Or add it to at list, like a backlog of things that we need to fix, and then you can go back on it and organize them and prioritize the fixes and the improvements, sometimes prompt fixes, like, “This thing has been done five times over the past three tasks, this error. And maybe if we added a sentence in the prompt from that tool, it wouldn’t do it.” So that way, every time you run a task, you introspect over what you did, and then you can improve the system, and the system over time has been improving through that clearly. Right. Yeah.

Larry:
Yeah. Hey, you reminded me… Go ahead.

François:
Yeah. So another thing that came after that was more on the process itself, so refining the process. So you’re saying, “Hey, I mentioned the main components. But then when you work, there are things that you do that you use in order to do a better job, or maybe some knowledge you have in yourself that is implicit. You don’t make it explicit, but it will work better for the AI.” And so that’s when things diverge as… The perfect knowledge engineer doesn’t work the way I work. The perfect AI knowledge engineer doesn’t work the way I work, the same way planes don’t fly like birds fly. So introspection and copying our processes work up to a certain point.

François:
So what we added was a way to make… So basically, whenever you start a task, you’re starting to… There’s data profiling, but you also need to first write a full plan, so have the harness write a full plan for what it’s going to do with the details of all the transformations, the exact queries that are going to be issued, everything, but as a plan, and then also build scenarios. What are all the possible scenarios that are there? Are there corner cases? Identify all of that. And so write that.

François:
And then, on top of that… So it will do that. And then on top of that, we implemented a critic pattern, so agent critic pattern. So once that plan is done and those data scenarios are written, it will trigger a subagent that then criticizes the plan and the example. So it looks at the plan, looks at the queries, like find bugs, find patterns. Sometimes a data scenario will expose something that is not covered by the plan and vice versa, so things will be…

François:
So there’s a loop of this critic agent that will give back a report that will be fixed where points will be addressed, and then another round of… We’re doing three rounds like that. And so those two things made a big improvement in terms of reliability and steering. And this is also somewhere where you have a point where you’re like, “Okay, let me read that plan myself so you’re making sure and you can add your knowledge there. So yeah, that’s been definitely a big improvement over the process to add those things.

Larry:
That reminded me of something else I wanted to ask you about because we talked about this a little bit last week. But one of the benefits of Pi, one of the things you really like, is the ability to quickly swap out different models. But a lot of what you’re talking about is prompting and skills writing and things like that that might be interpreted differently by different models. Have you figured out in your workflow how to keep it, I don’t know, durable as you change from model to model? Or are there just best practices emerging around how to write…

François:
Yeah, that’s a great question. I mean, I think the best move there was to say that we would not write any prompt. So for all the skills and tools that we’re building for that harness, we’re not writing the prompts. So it starts by doing a task, and then first step is, let’s say, building the ontology like, “Okay, so here’s how you do it, so just do it,” and have a conversation with the agent. And then at the end of it, it’s like, “All right. So we had that conversation. We had multiple back and forth and steering, and now we reached something where we’re happy. Now build a tool that will do that in a more systematic way.” And we’ll write these prompts itself based on what it gathered through the conversation. And so that’s how you piece by piece at these things.

François:
I’m not even reading these prompts, like barely, sometimes curious, but that’s… The evaluation of the system is what matters more than what’s in it. So, I mean, it may sound scary, but… So you have to put a lot of attention on testing things, like heavily testing so the tests become your reference. So yeah, that’s what we do. And then in terms of model variation, I think newer generations of models are pretty good. We never done some prompt engineering as it was fashionable in the past, but yeah. I think now it seems that it’s much less used because models are just good at following instructions. Yeah.

Larry:
Yeah. It’s so interesting. Hey, François, I can’t believe we’re coming up close to time already, but I feel like this has been a really good overview. I think people are going to get a lot out of this, so thank you. But before we wrap up, is there anything last, anything you want to make sure we share with folks or that you want to revisit from the conversation or…

François:
Yeah. So, I mean, now we’re… Just to conclude on that, the next thing we’re doing is evaluations, so swapping a model for another and being able to have reproducible tasks, which is pretty hard in that complex environment where you have multiple sources, they evolve over time. So you want to have a very scientifically reproducible environment where there’s only access to the same data. You can trigger the same exact task on the bit by bit, same data environment, everything environment, knowledge graph, ontology, et cetera. And then you just swap one parameter. You swap the model, you remove a tool, you change a tool, and you redo it, and then you measure the outcome, the spend, the tokens. Because I mean, knowledge engineering burns through a lot of tokens, and you want to be conscious about that.

François:
And I’m pretty optimistic about the directions that are taken by model evolution, intelligence, the price of intelligence, both monetary, but also in terms of energy consumption. We’re seeing a lot of engineering and research in AI where same compute capacity, much better intelligence, smaller models, et cetera, et cetera. So this is all pretty exciting, but you want to be able to measure that. If you switch from one model to the other, sometimes how do you measure what that will change?

François:
Yeah. So I mean, just to conclude on that topic. Otherwise, hey, it was great to talk to you, as always, Larry. I am excited for KGC 2027. I think it’s going to be a big one. We’re working on agent memory, I think is a big topic for knowledge graphs. So that’s also when we’re talking about it being meta, because this whole thing can be used to build agent memory and have this loop where the graph is self-fed, but the approaches we’re seeing on agent memory using knowledge graphs are still the, I would say, basic GraphRAG where it’s very basic concept extraction and mapping. There’s no deep knowledge engineering or entity resolution that is being done there. So I’m interested about how those two communities, like the traditional knowledge graph community and all agentic excitement there is. There are many startups working on that, like ways to do agent memory. And several of them use knowledge graphs, but, yeah, I’m excited about the interactions between the two communities. So that’s something we’re going to put some emphasis on.

Larry:
Yeah. And you’ve always been good at that in your programming, that kind of mushing up the ideas, so I’m really… And the new bigger venue, it’s going to be exciting. Yeah. Looking forward to that. Hey, one very last thing, François. If folks want to follow you or connect online, what’s the best place to find you?

François:
You can write me on LinkedIn. That’s a good way to get in touch. And also, francois@knowledgegraph.tech is a good email address.

Larry:
Okay, cool. I’ll put that in the show notes as well. Well, thanks so much, François. It’s always fun to talk, but this is a particularly enlightening talk, so thank you.

François:
Thank you for the invitation, Larry.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top