Karat’s Scott Bonneau on Engineers, Interviews and AI

[00:00:00] Welcome to PeopleTech, the podcast of WorkforceAI.News, I'm Mark Feffer. I'm talking today with Scott Bono, Executive Vice President of Product and Operations at CARO. They use technology to help employers hire engineers by combining the capabilities of tech

[00:00:29] with the insight and experience of actual people. We're going to talk about how AI fits into the interview process for both employers and candidates and things to bear in mind as you try to make

[00:00:41] technology work for you all on this edition of PeopleTech. Hey Scott, welcome. Let's get into the grantoron, meaning these are technology and interviews, and I want to start by asking what's going on with AI? Is it being used only by solutions providers like yourselves?

[00:01:02] Is it being used by the candidates and? Yeah, I think a better question is what isn't going on with AI these days. It seems like it is everywhere and it certainly has shaken up things with respect

[00:01:15] to engineering jobs in general and of course that has found its way into the interview processes as well. From the time that ChatGPT emerged about a year and a half or so ago, one of the things

[00:01:29] that became very obvious fairly quickly was that it was quite capable of solving problems in code. If there were problems that you had that amounted to translating an English language description of a problem into some functioning and working software, ChatGPT was pretty good

[00:01:48] at doing that right out of the gate even 18 months or so ago. Obviously, we've seen since then the deployment and development of several other tools, notably GitHub's co-pilot that has been adopted by a really significant number of organizations from large to small that are

[00:02:02] designed to make engineers more efficient and effective. It's not at all surprising that we would expect to see AI showing up in interviews. Two things I'll just touch on very briefly

[00:02:13] one is I think the industry more broadly is still very much coming to terms with what role they want to have AI playing in their own organizations in terms of how engineers use it to develop code

[00:02:25] on a day-to-day basis. We see the gamut from people that are fully embracing it to those that are staying at arm's length, particularly organizations that are in highly regulated industries like banking or healthcare or pharmaceuticals. Unsurprisingly, we are a little more

[00:02:40] cautious when it comes to these newer technologies. But then the question also becomes how do candidates leverage it during the interview processes? Our approach and our thinking is that this is

[00:02:50] something that is going to be a part of the world that engineers inhabit and the work that they do moving forward. We are working to create solutions and have worked to create solutions

[00:03:00] that allow employers the opportunity to decide how much or how little they want to include of allowing candidates to leverage things like chat, GPT during the interview process. We hear a lot about people in organizations starting to use AI. They're putting AI to this

[00:03:18] use to that use in terms of your own segment interviewing. What are some of the problems you're facing? Yeah absolutely. The most obvious one is well how can you be sure that you are

[00:03:30] testing a candidate's ability to solve a problem or you're testing an LLM's ability to solve a problem? This is I think one of the biggest challenges in particular with the offline types of assessments, offline code tests and the like is that systems like chat, TPT and particularly

[00:03:50] with the advancements that we've seen in GPT 3.5 and more recently with GPT 4, those technologies are getting more and more effective at taking natural language problem descriptions and translating it into working code. At the end of the day if you are in an interview

[00:04:06] process as an employer and you're trying to understand what candidates capabilities are, if the only thing that you have to look at is a piece of software that was produced asynchronously and then presented to you in the context of an interview, it would be very

[00:04:20] difficult to determine what was written by an LLM and what was written by a human. This is why I think it's very important and this has always really been important but it's becoming even more important now that interview processes for technical roles particularly

[00:04:33] engineering and programming type roles focus on understanding comprehension expansion, those various things that go into the everyday work of engineering that really can help you understand what is the candidate actually know versus what the particular tool of choice that they're using to help them along the way notes.

[00:04:54] So does all of this change the role of the hiring manager in the hiring process? It's an interesting question. What I would say is one of the things that I think is definitely going to change is what the day-to-day work of a software engineer is moving forward and

[00:05:13] if I could use a little bit of an analogy from another industry, I think if you think about programmers or coders or software engineers historically, earlier career more junior talent would be doing sort of more akin to what you might look at as like sort of copywriting

[00:05:35] or copy editing in the literary field. You have a thing that you're trying to convey, you need someone to put it into natural language and make it understandable and readable and convey its message and so on and so forth. And then folks that were in

[00:05:49] more senior positions were doing more editorial work pulling in content from several different copy editors or copywriters, for example, and making sure that there is univocality in what they're presenting that everything adheres to the expectations and

[00:06:01] standards of brand safety or whatever have you. And there's an analogy I think with respect to the engineering world where that's kind of the work of more senior developers or dev managers, technical leads, those types of things historically was taking software that was written by a

[00:06:15] number of people, making sure that it all played nicely together and so on and so forth. I think what we are going to see if you move just a little bit down the road, maybe, you know,

[00:06:23] 24 or 36 months down the road, a significant amount of code that gets checked in and run in production is going to be written by these systems. And the work of engineers, that does not mean,

[00:06:35] by the way, that I think that software engineering roles are going to disappear far from it and the world is going to continue to need more and more software. But the work is going to be

[00:06:42] less about producing that sort of baseline code and it is going to be more about integrating it, understanding and adapting it to include various concerns like security or privacy, accessibility, you know, UI making sure that everything is pretty incoherent from a design

[00:07:00] standpoint. And so I think we are going to see the nature of that work of sort of lower level earlier career programming jobs be a little bit less about Rote Program creation and a little bit more about thoughtful engineering and integrating the results from outputs of these systems,

[00:07:16] if that makes sense. Turning to the internal workings of carrot, do you have plans or thoughts about what you might do with AI within your own products? Yeah, absolutely. I mean, we,

[00:07:30] the way that I have encouraged our teams to think about the impact that AI is going to have on carrot specifically given the sector that we are in is sort of in three domains. So one is

[00:07:45] there is going to be a lot of hiring that's going to be done around AI roles, you know, very quickly, much like over the course of the last 15 years, every company that had any kind

[00:07:55] of software suddenly had to worry about, for example, mobile or the cloud, every, you know, company is going to be looking at ways to incorporate AI technology into, you know, into their application or their domain in some form or fashion. That's going to generate a

[00:08:09] lot of demand for people who have expertise in leveraging AI systems. So how can we help our customers assess for and hire great AI talent? That's sort of area number one. Area

[00:08:22] number two is going back to what we were speaking about earlier, how do we make sure that the assessments that we perform on behalf of our clients are actually testing the candidate's capabilities and not just the LLM's capability. So making sure that we have content and interview

[00:08:37] formats that are really designed to understand what the candidate is capable of, whether or not they happen to be leveraging, you know, any kind of AI tools in the process of the interview.

[00:08:48] And then the third is what are the ways that we can leverage AI internally to make ourselves more efficient and effective? And, you know, one of the ways that we have focused on that

[00:08:56] and we found really good success is in the process of generating new content and new questions for our interview processes. So we have an entire team that's dedicated to content generation. It contains people with, you know, IOS psychology and psychometrics backgrounds,

[00:09:12] for example, making sure that we are generating content that is actually valid and testing for the things that we are aiming to test for that it performs consistently well across candidates irrespective of their backgrounds and the like. And that has long been a sort of

[00:09:26] very labor intensive process. We have found that leveraging AI in the ideation and content creation process there can be quite a leg up. Now, just to be clear, we don't simply ask chatGPT to generate content and ship that out to candidates as part of an integrated process

[00:09:40] that involves subject matter experts in our content team and that type of thing. But we have found some really good efficiency gains by leveraging generated AI in our own internal processes. And what do you hear from customers about all this? I mean,

[00:09:54] are they excited by it? Are they nervous about it? Is it just kind of a thing to them that, you know, they'll deal with? I think there is a range of emotions with respect to AI.

[00:10:07] What I will say is nobody is disinterested in AI. Everyone is very interested in what is happening with AI. But I think whether or not it is something that a particular organization is aggressively pursuing or sort of staying cautious about it is largely a function

[00:10:25] of the domains that they're in. Unsurprisingly, what we see as a lot of the sort of more tech native digital, you know, digital native companies are doing a lot of things around AI and really embracing it. Some of the more traditional industries are obviously staying a little bit

[00:10:43] more cautious. But one thing that I think we will see over the course of the next 12 to 18 months is I think there's quite a bit of hype and excitement, and I think rightly so around what the possibilities

[00:10:56] of this technology are. But there are far fewer obvious applications of the technology where it is going to deliver a step function improvement and performance or results along those lines. But I think that is going to change. You know, as companies like Amazon and Google and Meta

[00:11:11] and OpenAI continue to develop their platforms, and as those ecosystems become similarly robust to what we see in like cloud computing or mobile, for example, it is going to become dramatically simpler and more accessible for companies to be integrating AI powered solutions into their applications.

[00:11:29] And that's, you know, as somebody that's been, you know, in this industry for 25 plus years now, that's very exciting. So with all of this said, and you mentioned before that there's

[00:11:39] a lot of hype out there, where do you think AI really is right now? Is it really early stages or does it live up to a lot of the hype or what? Yeah, it's a good question. So I think it will

[00:11:55] live up to a lot of the hype. I think unlike what we saw, I think a little bit with crypto over the, you know, in the late, you know, 20 teens and into the first couple years of the 2020s,

[00:12:09] where in my opinion, crypto is a little bit of a solution in search of a problem. I think AI has extreme promise. And I think AI is genuinely going to change the way that companies build

[00:12:23] software, among many other things that I have less direct experience with. But that said, I do think it is this particular type of technology, large language models and generated AI still very

[00:12:36] much is in its infancy. And I think we are going to see a similar pace of innovation and improvement. I mean, it's, it's you only have to go look at some of the things like the image and video

[00:12:48] generation tools that were starting to come out say at the beginning of 2023, which were, you know, you could tell that there was promise there, but they were very clearly very first generations of these things. But you look at them now, what's happening? And, you know,

[00:13:03] there are applications that are out there that are capable of generating, you know, 4k high res video that you really have to look for to see, you know, whether or not it was generated by

[00:13:16] AI or it was done by hand. And that's with, you know, 18 months of work, you know, if you fast forward another 18 months or another 36 months, I think we are going to see, you know, similarly substantial leaves. It's, it's quite a time to be, you know, in this particular

[00:13:31] space. Yeah. Well, Scott, thanks very much. It was great to talk with you and hope we'll talk again. Sounds great. Thanks, Mark. My guest today has been Scott Mono, Executive Vice President of Product and Operations at

[00:13:57] Carrot. And this has been PeopleTech, the podcast of workforceai.news. To keep up with AI technology and HR, subscribe to Workforce AI today. We're the most trusted source of news in the HR tech industry. Find us at www.workforceai.news. I'm Mark Feffer.