Ep 15: The Future of Skills Assessments and Hiring in the Age of AI with Dr. Charles Handler

[00:00:00] Hey, you with the podcast in your ear!

[00:00:02] Just a moment.

[00:00:03] Have you already activated your mobile happy hour?

[00:00:05] So you have an unlimited data volume every day for at least one hour.

[00:00:10] It's quite new and for everyone who is now getting mobile phone plus internet for at home on the TV

[00:00:16] Magenta 1 customers will activate the mobile happy hour in the My Magenta app

[00:00:21] and it's already off.

[00:00:22] The 5G network of Telekom

[00:00:23] So don't wait, now get in and activate the My Magenta app of Telekom

[00:00:39] Hey, it's Bob Pulver.

[00:00:41] In this episode of Elevate Your AIQ, I sat down with Dr. Charles Handler, President and Founder

[00:00:46] of RocketHire and host of the PsychTech at Work podcast.

[00:00:50] Charles and I delve into the intersection of AI, skills-based hiring, and the challenges

[00:00:54] of managing information overload.

[00:00:56] Discuss the critical role of defining skills in the hiring process, the balance between

[00:01:00] AI's benefits and risks, and the future of automated assessments and simulations.

[00:01:05] Charles also highlights the importance of policies for responsible AI use and the impact

[00:01:10] of AI on diversity and inclusion.

[00:01:12] Tune in to gain insights into AI's evolving role in HR and how to navigate this transformative

[00:01:17] technology in order to, you guessed it, elevate your AIQ.

[00:01:22] Thanks for listening.

[00:01:24] Hello everyone, welcome to another episode of Elevate Your AIQ.

[00:01:28] I'm your host Bob Pulver.

[00:01:29] With me today is my friend Dr. Charles Handler.

[00:01:32] Charles, no worry today.

[00:01:34] I'm doing great.

[00:01:35] Thanks so much for inviting me to be on the show, Bob.

[00:01:37] We always have such good, meaningful and interesting conversations.

[00:01:42] So I'm looking forward to that today.

[00:01:43] Absolutely.

[00:01:44] I appreciate you being here.

[00:01:45] Why don't you give the audience just a quick little background of just what you've been

[00:01:50] doing, how you wound up where you are today.

[00:01:53] And yeah, always interesting to hear the path that you've taken.

[00:01:56] Well, my mom met my dad in graduate school.

[00:02:00] So I have a doctorate in industrial organizational psychology and for, I don't know, 30 years,

[00:02:07] I've specialized in predictive hiring tools.

[00:02:10] I call them now.

[00:02:11] And I've worked in a lot of capacities.

[00:02:13] I have somewhat of a unique perspective just because I've worked in so many angles of

[00:02:18] this thing.

[00:02:18] So I have a pretty holistic view from legal compliance to technology side of it,

[00:02:23] which I'm really passionate about to just looking at the overall market as a market

[00:02:28] analyst and to helping enterprise companies on a global level create strategy

[00:02:35] and implement programs.

[00:02:38] I do a lot of the standard job analysis and validation work, building assessments.

[00:02:43] And I like that stuff, but I really also love the innovation.

[00:02:46] And I'd say innovation and nowadays AI, that's really been my passion area and

[00:02:51] where I've been advising and learning constantly, trying to keep up dog power.

[00:02:56] I don't know if it's just folks like us who have done a lot of different things in

[00:03:00] their career that makes it harder to keep up with everything or that's

[00:03:05] something that everyone goes through because we're all experiencing information

[00:03:09] overload.

[00:03:09] But I feel like sometimes being a generalist and doing a lot of different

[00:03:14] roles makes it worse just because you find a lot of different things of interest

[00:03:20] and also you start to connect dots across industries and domains that maybe

[00:03:26] other people just don't see.

[00:03:28] Yeah, I think no matter who you are and what you do, if you're paying attention

[00:03:33] to the world of AI and AI in your own field, I mean, I guess in a general

[00:03:38] or higher sense, we all have so much information coming at us all the time

[00:03:45] that it becomes very difficult to be able to keep up with anything.

[00:03:50] It's overwhelming and I subscribe to so many different, you know,

[00:03:55] sub stacks and newsletters and stuff.

[00:03:57] And I feel like I skim those and I feel like I find great nuggets

[00:04:02] and I feel lucky.

[00:04:03] I'm like, oh, I'm glad I didn't miss that one.

[00:04:06] But yet there are so many that I probably did miss.

[00:04:09] So it's all you can do is absorb what you can, find the people

[00:04:14] and the outlets that you like the most, you know, and kind of stay tuned to those

[00:04:18] things. And so I read a lot about just general AI and large language models

[00:04:22] specifically. I'm trying, I'm not trying.

[00:04:24] I am learning and absorbing, you know, like a sponge and I'll read technical

[00:04:28] papers that push my understanding and I don't even read the whole thing

[00:04:32] because I'm like, all right, I get the nature of this.

[00:04:35] There's 15 pages of prompting and stuff that's going on.

[00:04:38] But but just really working hard to stay on top and the large language

[00:04:42] model stuff changes so quickly.

[00:04:45] And even if you just use in chat, you know, you can experience it by just

[00:04:49] using chat GTP, you know, for Omni to me is it's amazing.

[00:04:54] It's getting better and better.

[00:04:56] Like it's doing a lot of really good things.

[00:04:58] And I've discovered lately a really good way to manage information overload

[00:05:03] is perplexity.

[00:05:05] I subscribe to the perplexity pro.

[00:05:07] So I've almost completely replaced my Googling with perplexity.

[00:05:12] But Google, perplexity and chat GTP all have their own unique things.

[00:05:19] They do best.

[00:05:20] So but I feel like when I need to research a topic, perplexity summarizes

[00:05:26] and pulls a bunch of articles with links and it's much more current.

[00:05:30] And I love it.

[00:05:31] So I would recommend that to anybody.

[00:05:32] I'm going to start my own sub stack.

[00:05:34] So I've been kind of studying what sub stacks all about.

[00:05:37] There's an overwhelming number of awesome authors on there.

[00:05:41] And one of the interesting models you see on there is the paid people

[00:05:45] monetizing their content through paid descriptions and stuff.

[00:05:49] I'm not quite ready to do that.

[00:05:50] I'm still trying to figure out Bob a name for my sub stack.

[00:05:53] I'm stuck on the name.

[00:05:54] I have all this content ready to go, and I can't think of a good name.

[00:05:57] So maybe we can brainstorm on that.

[00:06:00] So this is separate from from rocket hire.

[00:06:02] It wouldn't be just.

[00:06:03] Yeah, I guess I forgot to mention thanks for prompting me.

[00:06:06] I forgot to mention that.

[00:06:08] Yeah, I have a company rocket hire when we've been in business

[00:06:10] for 22 years, basically everything I described in what I do is done

[00:06:16] within the context of rocket hire.

[00:06:19] But it's really pretty much me and some associates.

[00:06:21] So as far as my market facing brand, I'm really pushing on the

[00:06:26] sub stack and the content stuff myself above above anything.

[00:06:30] So there's a lot of choices to be made there.

[00:06:33] But anybody can be a content creator now.

[00:06:36] And there's a lot of good stuff that people have to say.

[00:06:39] Right.

[00:06:39] So I don't know.

[00:06:41] How do you keep up with it?

[00:06:42] I mean, do you have some similar experiences than I do?

[00:06:45] I do.

[00:06:45] I was actually just talking to a couple of guys yesterday about the

[00:06:49] fact that I signed up for all these different newsletters,

[00:06:53] you know, the sub stacks and mediums.

[00:06:55] And so now, and of course, that content doesn't just reside there.

[00:06:59] I get the nudges through emails and now my inbox is filled

[00:07:03] with all these newsletters.

[00:07:05] And you're right.

[00:07:05] The problem is I don't I need curation on top of curation because

[00:07:11] it is sort of serendipitous.

[00:07:13] If I click on one, they happen to put in a good catchy title or lead headline

[00:07:19] and I'll go in and then three scrolls down.

[00:07:22] I see something that piques my interest or whatever.

[00:07:25] And so I'm reluctant to unsubscribe to any of these.

[00:07:29] They're all like you said, there's some really good writers

[00:07:32] and really bright people who are thinking about things in different ways.

[00:07:37] And so so it is hard.

[00:07:39] I mean, one of the tools I used to love just to try to keep up with social

[00:07:42] fees and social information.

[00:07:44] I think this came out right around the time that Flipboard came out,

[00:07:47] which is another great tool that I still use.

[00:07:49] But there was a tool called Nuzzle, this entrepreneur, Jonathan Abrams,

[00:07:53] said I'm up with basically using your social network and your social graph.

[00:07:57] You could look at the things that you've identified that you find of interest

[00:08:01] and then look at your first degree connections and the things they flagged

[00:08:06] of interest. And then if you have enough time in your day,

[00:08:09] you could go beyond that to a second degree or whatever.

[00:08:12] But yeah, at least you could kind of gauge and sort of only commit

[00:08:16] to the things that you thought were most relevant to you.

[00:08:20] But I think to your point, there is a lot.

[00:08:22] And even for those folks who are just trying to keep up,

[00:08:26] even without getting into the technical stuff as you're doing,

[00:08:30] it's still a lot because there are even abundance of choice.

[00:08:34] Like you, I mean, I updated my Chrome default search engine to be

[00:08:38] for complexity. So that way I don't really get any of the nonsense

[00:08:43] that was part of PageRank and all the SEO content.

[00:08:48] And it's much more around here's what I think you were asking for.

[00:08:52] And it tells me the sources and I can single click.

[00:08:55] I see top five sources of where this information came from and what have you.

[00:09:01] So I've found it to be invaluable.

[00:09:03] And I always felt like it was only a matter of time before search engines,

[00:09:07] not necessarily their demise, but certainly at a significant challenge

[00:09:11] to the way that we discover information and knowledge.

[00:09:17] And so, yeah, so I think that whole market is ripe for disruption.

[00:09:23] Has been searches dead, by the way.

[00:09:25] And I want to plug one more thing real quickly that I just thought of.

[00:09:29] My very favorite thing to listen to now is something called this week in tech or twit.

[00:09:34] It's it's about a two and a half hour long panel discussion

[00:09:39] led by this guy, Leo Laporte.

[00:09:42] But man, oh man, it is the I like addicted to it.

[00:09:46] I learned so much and there's so many good.

[00:09:48] I don't know if you ever heard of it before, but I highly recommend it.

[00:09:52] I like every week on Monday, it's three some hours, but it's during my commute,

[00:09:57] which is about 20 minutes.

[00:09:59] I listen to a little bit of it every time.

[00:10:01] So anyway, highly recommend that to your listener.

[00:10:04] Yeah, I've heard of it.

[00:10:05] I haven't actually listened to it, but it sounds like it's pretty informative.

[00:10:09] Yeah, it is.

[00:10:10] I guess if we correlate sort of absorbing all this information

[00:10:14] and translating that to actual sort of skills, I do you think about,

[00:10:19] you know, there's a lot of the chatter these days, at least from the marketing teams

[00:10:24] of solution providers is around skills based hiring, skills based organizations.

[00:10:29] So with you with your background in psychology and, you know,

[00:10:34] behavioral and psychometric assessments, I mean, how do you think about

[00:10:38] the half life of skills and how are we properly sort of assessing skills?

[00:10:44] Because I feel like you and I, you know, being on sort of the back nine

[00:10:48] of our careers, you know, we're we're doing a lot of knowledge work.

[00:10:52] We interact with a lot of knowledge workers.

[00:10:55] But if knowledge is is that your fingertips and you're, you know,

[00:11:00] codifying, you know, knowledge and feeding it into these AI tools or whatever.

[00:11:05] I mean, at some point you've got to say, well,

[00:11:09] if I know where to find the information, even if I don't have it in my own

[00:11:12] human brain, you know, do do AI skills and the ability to not just

[00:11:17] know about it in terms of like AI literacy, but to practical use, you know,

[00:11:24] to make yourself more effective and efficient, making better decisions,

[00:11:28] things like that. I mean, it seems like at some point, you know,

[00:11:31] having the skills from, you know, the actual knowledge and expertise

[00:11:35] that you've acquired over potentially decades.

[00:11:38] Absolutely. Well, have a lot to say about skills based hiring

[00:11:41] probably more than we can even cover here in this in this conversation,

[00:11:46] right, during the time we have.

[00:11:48] But what I would say is the very first thing you have to do in

[00:11:53] skills based hiring is define what a skill is.

[00:11:56] And we all have to share that same definition.

[00:11:59] Otherwise we're calling things skills that may not be skills or maybe

[00:12:03] everything that we're using as a signal for a job is a skill.

[00:12:06] It could be a hard skill, a soft skill.

[00:12:10] A lot of times you see knowledge slash skills, right?

[00:12:13] So sometimes knowledge is assumed to be a skill or that you have a skill

[00:12:17] because you have the knowledge.

[00:12:18] So one of the things people who are steeped in school, then measurement,

[00:12:23] which I am, is that you have to define very clearly and objectively

[00:12:27] what it is you're going to measure so everybody's on the same page.

[00:12:31] So first of all, I don't think we've done that as to what a skill is, right?

[00:12:36] Secondly, you know, there's different ways, right?

[00:12:39] So think about reducing friction in the hiring funnel.

[00:12:43] And you think about basically parsing a person out into a bunch of labels

[00:12:48] that you can compare them to other labels that would say they're good at a job

[00:12:53] or, you know, the fit for an organization or whatever you do.

[00:12:57] You know, there's a lot of friction there.

[00:12:58] So we've got inferential skills, right?

[00:13:01] We've got a cloud of words, ontologies.

[00:13:04] We take a job description.

[00:13:05] We take a resume.

[00:13:07] We may we may take a preponderance of stuff that's digital exhaust for a person,

[00:13:12] you know, on the web and infer skills.

[00:13:14] And you've got to do that in some sense because of the friction

[00:13:18] and how painful that friction is.

[00:13:20] But the whole thing can't work in my opinion unless you can verify those skills,

[00:13:25] right? And that's where it gets harder.

[00:13:27] So how do you verify skills at scale without any input from candidates

[00:13:32] without asking them to do anything?

[00:13:33] Because ultimately that's where the friction begins is,

[00:13:37] OK, now we've got to get 5,000 candidates to sit for a skills test.

[00:13:42] Nobody likes taking tests.

[00:13:43] So we're not going to be able to do that verification.

[00:13:46] And without that, we lack, you know, some accuracy.

[00:13:49] So whoever figures that out how to verify at scale will be, you know,

[00:13:53] a really well regarded person or a company or whatever.

[00:13:57] But, you know, at the same time, it's there's something like credentialing,

[00:14:02] right? So about 10 years ago, I don't know.

[00:14:05] Maybe it's more or maybe 12 years ago, credentialing was a big thing

[00:14:09] everybody was talking about, like, oh, we'll be able to verify skills of these credentials.

[00:14:13] And again, it's the same thing as defining a skill

[00:14:16] without kind of a universally accepted thing like MCSE, right?

[00:14:20] Microsoft Certified, what is it?

[00:14:22] Soft Solutions Engineer or Software Engineer, whatever, right?

[00:14:25] Those are things that are credible courses and you should know your stuff

[00:14:28] if you get one. So those are great.

[00:14:30] But and everybody kind of knows those, but but we don't have.

[00:14:34] I was just looking up, you know, skills, verification, credentialing companies today

[00:14:39] on ChatGDP asking it to list them for me.

[00:14:42] And before we move on, I need to let you know about my friend Mark Feffer

[00:14:47] and his show, People Tech.

[00:14:50] If you're looking for the latest on product development, marketing, funding,

[00:14:54] big deals happening in talent acquisition, HR, HCM,

[00:14:59] that's the show you need to listen to.

[00:15:02] Go to the work to find network, search up People Tech.

[00:15:05] Mark Feffer, you can find them anywhere.

[00:15:10] You know, there's a lot of them.

[00:15:11] There's a lot I hadn't heard of.

[00:15:13] Everybody's kind of jockeying for the same thing to be the universal standard.

[00:15:18] I haven't seen credentialing go like I thought it would

[00:15:21] because it makes so much sense to me.

[00:15:23] You do have it in like any kind of technical medical field, the legal field.

[00:15:28] You have to take, you know, a very, very high quality,

[00:15:32] well-referred or proctored, you know, exams to keep yourself current.

[00:15:37] So that's very valuable.

[00:15:39] But again, that's not the preponderance of the population.

[00:15:42] So the one last thing I'll say about skills-based hiring,

[00:15:46] it's very easy for companies to say they're going to do it.

[00:15:49] And by gum, it has a lot of benefits, you know, from an equity,

[00:15:54] fairness, hidden workforce you can discover.

[00:15:57] Good, good stuff.

[00:15:58] But organizations struggle to really implement it at scale.

[00:16:02] Part of it for the reasons I talked about.

[00:16:04] But I think there's also a, hey, we're going to adopt this kind of like

[00:16:08] D.E. and I was, you know, 15 years ago.

[00:16:11] We're just going to show you some videos.

[00:16:13] We're going to say we're doing it or not.

[00:16:15] And so I had the pleasure to work on a project and the frustration

[00:16:20] about, I don't know, seven or eight years ago

[00:16:23] called the Essential Competencies Project.

[00:16:25] And it was funded by the conference board.

[00:16:27] And I was working with somebody, we canvassed, I don't know,

[00:16:31] 20 enterprise companies.

[00:16:33] We ended up with like five.

[00:16:34] And the whole project was we're going to get an assessment provider.

[00:16:39] We're going to plug it into your hiring funnel for a particular job or jobs.

[00:16:44] And you're not going to see a resume.

[00:16:46] You're just going to see the assessment results.

[00:16:47] And then you're going to move that person forward based on those results

[00:16:51] in the initial ATS screening stuff, you know,

[00:16:54] that's more neutral.

[00:16:56] And it didn't work.

[00:16:57] We couldn't get one company, even though they had high level people

[00:17:01] who signed off when we got into the guts of, OK, now we've got to modify

[00:17:05] your hiring process.

[00:17:06] You can't use this tech stack you were always using.

[00:17:09] Your recruiters have to do this instead.

[00:17:12] Nobody did it.

[00:17:13] You know, that was a while back, but it taught me a real valuable lesson.

[00:17:18] You could even your leadership can want to do it.

[00:17:21] The change management is.

[00:17:22] And we know HR technology is entrenched, man.

[00:17:26] It's hard to get it changed.

[00:17:29] So anyway, that's my long winded take.

[00:17:31] Like I said, I could talk forever about it.

[00:17:33] I guess part of what's frustrating about that to me is on some level,

[00:17:37] I know that that's the right approach and it's disappointing

[00:17:41] that we haven't been able to fully, you know, figure that out

[00:17:44] and flush that out.

[00:17:45] But you're right.

[00:17:46] I mean, if you can't validate the skills, then you could pretty much

[00:17:48] say, and without the credentialing like a standard credential,

[00:17:52] then you're subject to potential, I guess, fraud, over embellishing

[00:17:59] on what they're capable of doing.

[00:18:02] And even some of the ways that some companies have approached it,

[00:18:07] your evaluation or other things, this person really have this level

[00:18:11] of this skill just gets really muddy.

[00:18:15] And then you fall prey to some of the human biases that are plaguing

[00:18:20] the entire talent lifecycle.

[00:18:22] So it can be tricky, but, you know, I guess the other big thing

[00:18:26] about why it's disappointing to me is I don't like the cat and mouse

[00:18:29] game of job descriptions to resumes because everything just

[00:18:35] gets foggy and manipulated.

[00:18:38] And it just seems kind of silly.

[00:18:40] I mean, I have had the chance to basically do like a rich text

[00:18:45] kind of assessment.

[00:18:47] And not only did they not ask for my resume, but it replaced

[00:18:50] the recruiter phone screen and probably derived insight from at

[00:18:56] least the first round interview.

[00:18:59] So in theory, it's a better way of assessing someone's

[00:19:02] potential to succeed on that team and in that role.

[00:19:06] You know, if only we could pass some of these not insignificant

[00:19:11] hurdles. Yeah, 100 percent, you know, hiring is about in my mind,

[00:19:16] it's about two things, accuracy and fairness and the extent that

[00:19:20] you have a signal and that signal is predictive of some kind

[00:19:24] of outcome that you want.

[00:19:26] The more noise that's there, the less that, you know, the

[00:19:30] less of a direct bit there is, the less accuracy you have

[00:19:34] often then what gets substituted is is biased predictors,

[00:19:39] biased signal and then you lose your fairness.

[00:19:41] So we're always juggling that over my whole career and over the

[00:19:44] last, you know, 50 years of IO psychology, we're juggling

[00:19:48] fairness and accuracy all the time, trying to make sure we have

[00:19:52] both without compromising either.

[00:19:54] It's always hard and it applies to any kind of hiring,

[00:19:57] skills based hiring, interviewing, you name it, those are

[00:20:01] the, you know, those are the guiding stars.

[00:20:03] With that, I mean, as AI has sort of entered the picture,

[00:20:07] you could argue, ATSs have had some level of AI learning and

[00:20:10] they're matching algorithms, et cetera for quite a while.

[00:20:15] But I guess in modern terminology where AI is used more

[00:20:20] much more sort of broadly and generically, you know, how are

[00:20:24] you thinking about, I guess, both the value and the

[00:20:27] opportunity as well as the risks when it comes to having

[00:20:31] AI as part of the hiring process?

[00:20:34] Well, I mean, it's inevitable, right?

[00:20:36] I mean, there's so many advantages and it's all about

[00:20:38] reducing friction, which is lots of people, hopefully, that

[00:20:42] you want to evaluate and finding the very best ones,

[00:20:47] finding that best fit, excuse me, like we were talking

[00:20:49] about. So that I was just writing about this.

[00:20:53] I mean, AI is great at reducing friction in that

[00:20:55] it can go out there without anybody overseeing it and

[00:20:59] say, I'm delivering you the best matches or I'm going

[00:21:02] to go out and find these people or I'm going

[00:21:04] to screen these people. But what's the substance

[00:21:06] it's using? I mean, if you're using skills on

[00:21:09] top of these, so to call empty calories, that's

[00:21:12] difficult. And, you know, if you're using resume

[00:21:15] job descriptions, you get into this kind of garbage

[00:21:18] in garbage out situation. So you're getting

[00:21:21] reduction of friction, but at the cost of accuracy

[00:21:23] and fairness. And that's the paradox of AI, in my

[00:21:26] opinion is like it can make things so much faster

[00:21:30] and easier for us. But that doesn't always

[00:21:33] mean that it's achieving other objectives, right?

[00:21:35] It's because you can't always have it all.

[00:21:39] And so I think we'll get better with that.

[00:21:42] You know, in some sense, machine learning is AI.

[00:21:44] We've been using machine learning for a long time

[00:21:46] as you alluded to, you know, ATS has have been

[00:21:49] using that resume. I mean, from the very

[00:21:51] beginning of resume parsing and matching

[00:21:55] in the early 2000s, once monster and job

[00:21:58] boards happened, companies had no infrastructure

[00:22:01] or ability to deal with that thing happening.

[00:22:04] Right? There were just so many resumes coming in.

[00:22:06] So you could log into Monster and manage your stuff,

[00:22:09] but it wasn't really going into your ATS or anything.

[00:22:11] It was a difficult, you know, situation.

[00:22:14] So technology has arisen to the challenge,

[00:22:17] but it's still a challenge because friction

[00:22:20] reduction is not accuracy.

[00:22:22] I think AI will get continually better and better.

[00:22:25] And I think we will learn how to train it

[00:22:29] better. So ultimately, it only knows what we know.

[00:22:31] So if we're training it with bias, then it's going to have bias.

[00:22:34] But since we have the opportunity

[00:22:37] to train it with unbiased information,

[00:22:40] theoretically, we can do that. Right?

[00:22:42] I mean, you get into large language models,

[00:22:45] they've ingested the entire Internet,

[00:22:46] they've ingested your life, my life, etc.

[00:22:50] It gets a lot harder to do that,

[00:22:52] but I still think it's possible.

[00:22:53] And I think that will be working in that direction.

[00:22:57] I believe when it comes to assessment,

[00:22:59] I feel pretty strongly about this.

[00:23:00] I just don't know the time horizon.

[00:23:02] We're going to have what I call a sinkhole

[00:23:05] in the middle of assessment.

[00:23:06] And what's going to happen is we're going to have all these

[00:23:09] automated inferential assessments based on,

[00:23:12] you know, again, digital exhaust

[00:23:17] and information where an applicant doesn't have to do anything.

[00:23:20] And then on the other side of it,

[00:23:21] I believe we're going to have super high fidelity simulation.

[00:23:24] So, oh, you want this job?

[00:23:26] OK, well, you know, log on here and do part of the job

[00:23:29] and you'll have something that looks like you and I doing

[00:23:32] this exact thing and it'll be scored.

[00:23:34] And it'll be highly realistic and people, I think,

[00:23:37] would enjoy that.

[00:23:39] Maybe it's part of an interview, I don't know.

[00:23:41] But but I think we'll have that because it makes so much sense.

[00:23:44] I think sitting there answering personality questions

[00:23:47] and like it scales and even multiple choice questions.

[00:23:51] The multiple choice questions in interesting one, Bob,

[00:23:54] because I'm like, you know, it's a very effective way

[00:23:57] to test the body of knowledge.

[00:23:59] I don't know how.

[00:24:00] I think maybe your digital exhaust could do it.

[00:24:03] Maybe the simulations can too.

[00:24:04] But it's it's a very, you know, standard way to go.

[00:24:08] So I think that one will be the hardest one or the one

[00:24:11] with the longest tail to extinguish.

[00:24:13] But what we will be able to do is probably continue

[00:24:15] to up our game with adaptive testing and, you know,

[00:24:19] that kind of stuff where we can have shorter deals

[00:24:22] or we can we can have LLM sit there and role play

[00:24:25] with you and ask questions instead of multiple choice

[00:24:27] or something. I don't know.

[00:24:29] But as we know it and we see it now,

[00:24:33] ultimately, it's not going to look like that.

[00:24:35] My guess across the board, 10 years,

[00:24:38] but it's impossible to know.

[00:24:40] I think, you know, just to see how someone,

[00:24:43] you know, operates potentially.

[00:24:46] Not just in a leadership position, but, you know,

[00:24:49] in pressure situations or how they resolve conflict

[00:24:54] and delegate authority, perhaps and things like that.

[00:24:57] It's just it's really kind of crazy.

[00:25:01] Yeah, yeah. About where this is is going

[00:25:03] and it's going so fast that even in a couple of years,

[00:25:06] you know, who knows?

[00:25:08] Yeah, yeah.

[00:25:09] So here's one thing I say all the time.

[00:25:12] Just think about it like this.

[00:25:14] If chat GTP never evolved past four Omni that we're using now,

[00:25:18] it would still be a miracle.

[00:25:20] We would still be able to do all kinds of shit with it.

[00:25:22] That that would benefit us all.

[00:25:24] So the baseline of just where we are in the moment

[00:25:27] is still incredible and game changing, and it's not going to stop.

[00:25:32] But I think, you know, you listen to these tech shows

[00:25:34] and I think the large language models themselves

[00:25:37] may have some limitations.

[00:25:39] Also, we're going to run out of stuff to train them on,

[00:25:43] although new stuff's always coming out.

[00:25:44] So I don't know if I buy that one.

[00:25:45] The energy and resources it takes to train one of these things

[00:25:49] is insane.

[00:25:51] And so it's not accessible by individuals,

[00:25:54] a small model or local model or whatever,

[00:25:56] but or modifying or, you know,

[00:25:58] achieve augmented generation fine tuning on an existing model.

[00:26:01] Sure. But creating your own is going to you don't have the GPUs.

[00:26:05] You don't have that you need a nuclear power plant.

[00:26:08] So, you know, people say there's other actual

[00:26:12] ways of structuring the engine

[00:26:15] that's doing the AI essentially that may take over.

[00:26:20] And you've also got the agent thing like you're talking about,

[00:26:22] where, you know, it's moved quickly into

[00:26:26] we're going to have multiple agents coming to do like Lang chain.

[00:26:29] I don't know if you're familiar with that.

[00:26:31] You know, that makes it makes so much sense, right?

[00:26:33] You're accomplishing one task with a bunch of AIs or little

[00:26:38] LLMs that are going off and doing individual things

[00:26:40] and coming back and producing a product.

[00:26:42] And so we'll see more and more of that.

[00:26:45] You know, somebody I'm not going to claim that I

[00:26:49] that I made this up, but it blew my mind

[00:26:52] because I never thought about it.

[00:26:53] My friend, Georgi Yankov, he's at DDI.

[00:26:56] He's a he's an I.O., but he's he's like

[00:26:59] clairvoyant in AI stuff, I think he was saying, yeah, you know,

[00:27:03] the future is people like myself and him.

[00:27:05] I.O.s we're going to be training the agents

[00:27:08] how to have good competencies, right?

[00:27:10] So in other words, we're going to train the agent

[00:27:13] how to display empathy or how to do this complex mix of traits

[00:27:19] and competencies that so it can interact more like a human,

[00:27:22] right, because it may not be able to train itself kind of thing.

[00:27:27] I get if I make sense there, right?

[00:27:29] So we may shift from evaluating people on these things

[00:27:32] to training the LLMs that are going to evaluate the people

[00:27:36] more accurately, transferring our knowledge as

[00:27:39] trained psychologists into the model.

[00:27:42] I mean, I'm doing that right now to a little bit

[00:27:45] training a role playing LLM, right?

[00:27:48] So we're we've created a whole structure of dialogue

[00:27:51] that's set up to elicit certain risk,

[00:27:54] poke you to display a certain competency, right?

[00:27:56] Like I get mad at you and I'm a customer, right?

[00:28:00] There goes your poking you on your empathy

[00:28:02] or your ability to maintain your composure or whatever.

[00:28:06] And then we measure that score that with another LLM.

[00:28:09] So that's a simplified version.

[00:28:12] I think we could probably talk about that for at least another hour,

[00:28:15] but it brings up an important point about the fact that

[00:28:18] it's not just that everyone is now a user

[00:28:22] of this type of AI, being generative AI

[00:28:28] because it's really it's part of the user interaction,

[00:28:31] the user interface and the experience.

[00:28:34] It's that we all now have the capability to actually build

[00:28:38] essentially new AI sort of personas, if you will,

[00:28:42] also at our fingertips.

[00:28:44] So whether you're a GPT plus customer building,

[00:28:46] custom GPTs or you're working at an organization

[00:28:49] that's a Microsoft shop and you've got co-pilots

[00:28:53] there, then you can custom you create custom co-pilot

[00:28:57] or you're on Amazon and you're playing around in party rock

[00:29:01] or just all of these foundational models

[00:29:04] that the producers of these LLMs are giving

[00:29:08] the average person the ability to actually be a builder,

[00:29:11] not just a user.

[00:29:13] It elevates the importance of responsible AI practices,

[00:29:19] which is something I wanted to talk to you about

[00:29:20] because you and I have talked in the past about audits

[00:29:24] and anti-bias or bias mitigating practices

[00:29:28] and things like that.

[00:29:30] So when we think about that,

[00:29:33] I think about how do we give people just enough

[00:29:38] because they don't need to know every acronym

[00:29:40] that we just cited.

[00:29:43] They don't need to know what RAG necessarily is

[00:29:45] and how to fine-tune models on top of going that methodology.

[00:29:52] But I do think that one of the core things

[00:29:54] that they do need to know is design, build, test

[00:29:58] and use it responsibly.

[00:30:01] And so I don't know if that means

[00:30:03] like we need compliance training on responsible AI,

[00:30:08] but it seems like for organizations that have been waiting

[00:30:13] to figure some of this out before setting policy

[00:30:17] and figuring out what's acceptable to use,

[00:30:20] which tools, do we let them use it?

[00:30:22] Do we also let them build it?

[00:30:24] There's all kinds of things that need to be considered.

[00:30:28] And so I guess I wonder,

[00:30:30] based on the companies that you've talked to

[00:30:33] and that you're working with,

[00:30:34] I mean, how are they appreciating the magnitude

[00:30:37] of the impact of what we're talking about

[00:30:41] and the need to make sure that we train it the right way

[00:30:46] with transparency and ethics and fairness

[00:30:49] and all of these attributes in mind?

[00:30:52] I think so.

[00:30:54] And I think what we have right now is a little bit of,

[00:30:56] hey, we're not ready to even implement a lot of this.

[00:31:01] So I make a fundamental distinction though too, right?

[00:31:05] And if you think about the totality of an organization,

[00:31:08] there's a lot of different areas of the business, right?

[00:31:11] So for instance, AI for supply chain and logistics,

[00:31:15] what's the ethical problem there?

[00:31:17] Like what is it possibly gonna do that's unfair

[00:31:20] or anything like you're just routing things

[00:31:22] or you're ordering things and you're optimizing.

[00:31:24] So those are really easy adoption use cases.

[00:31:28] They probably have a lot less hoops.

[00:31:29] When you start talking about people, right?

[00:31:32] Or even like financial data, sensitive stuff,

[00:31:35] people being some of the most,

[00:31:37] it gets a lot more difficult.

[00:31:39] And what I've seen and heard is companies saying,

[00:31:43] hey, we don't even have a policy for this stuff yet.

[00:31:45] Like let's take a little bit of a breather here

[00:31:49] and let's figure out what our policy is,

[00:31:52] how we're gonna manage this.

[00:31:53] At the same time, in hiring and recruitment,

[00:31:55] people are using a lot of these AI screening tools.

[00:31:59] So there's gotta be use cases

[00:32:01] where their company has a policy or not

[00:32:03] because we've been using those tools for a while.

[00:32:05] Maybe they come back and say,

[00:32:07] hey, we've got to look at everything

[00:32:08] which is what they should be doing.

[00:32:10] So a lot of companies are in the process

[00:32:13] of putting in place a chief AI officer, right?

[00:32:16] So there's a set of creating the environment

[00:32:20] to be prepared for using AI tools

[00:32:23] that I think is going on now.

[00:32:25] And I think the adoption curve is still very much in the,

[00:32:28] which I'm glad about.

[00:32:30] I did a project,

[00:32:32] I'm actually in the process of writing it up.

[00:32:34] It's pretty interesting.

[00:32:35] Colleague and I, we interviewed 20 IO psychologists

[00:32:38] at global enterprise companies

[00:32:40] about their testing programs and their AI policies, et cetera.

[00:32:45] And pretty much zero of them are using heavily

[00:32:48] using any AI type assessment tools, right?

[00:32:52] An interesting aside on that is we started that project

[00:32:55] like six months ago.

[00:32:57] We were gonna, we did parse apart,

[00:32:59] we interviewed transcripts from an AI note, AI note taker,

[00:33:03] parsed them apart into rows and columns of text

[00:33:07] so that we could analyze it ourselves better

[00:33:10] but that we could have chat GDP do it.

[00:33:13] But we started by just feeding 20 transcripts

[00:33:15] into PDFs and to chat GDP and asking it questions.

[00:33:19] I have no need to go and put it all in a grid.

[00:33:23] Like it's nailing it, it's incredible.

[00:33:26] Anyway, as an aside,

[00:33:27] like the kind of stuff we can do with that is insane.

[00:33:30] But I think that it's good

[00:33:33] that companies are being careful.

[00:33:34] I think that AI regulation is gonna be a good thing,

[00:33:39] noting there's a spectrum, New York city law

[00:33:42] kind of garbage, EEOC stuff is what it is

[00:33:48] and it applies no matter what.

[00:33:51] But the EU AI Act is gonna be just like GDPR in my opinion

[00:33:54] and anybody in the US who touches anybody in the EU

[00:33:58] with any kind of hiring thing

[00:33:59] is gonna have to go through a certified audit

[00:34:03] of all kinds of stuff.

[00:34:04] You're not gonna be able to get around that.

[00:34:06] So I feel like that's gonna be great

[00:34:09] and we've got to prepare for that inevitability.

[00:34:12] And that'll help us, but at the same time,

[00:34:16] internal governance is critical.

[00:34:19] Companies, if they really care,

[00:34:20] they're gonna have to have policies,

[00:34:21] they're gonna have to be careful.

[00:34:23] And I think right now there's a lot of questions

[00:34:27] and so where are you gonna find it

[00:34:30] where there's least sensitivity and more objectivity

[00:34:35] and the highest levels of friction in things?

[00:34:39] And I think a lot of companies aren't prepared

[00:34:42] to understand the HR tech side of this really super well.

[00:34:47] You know, originally, I think when you and I first talked

[00:34:50] in New York City bias law,

[00:34:52] I forget if it had gone into effect yet,

[00:34:55] we're approaching its anniversary date

[00:34:58] and people aren't knocking down our doors

[00:35:00] to get their audits done.

[00:35:03] No, right.

[00:35:04] Or nor anyone's.

[00:35:06] You know what we see is vendors getting some kind

[00:35:10] of New York City audit when vendors

[00:35:12] wouldn't even be audited under that,

[00:35:14] but basically saying here, look at our data.

[00:35:16] We don't have any ratios that are out of whack.

[00:35:20] But as you know, just like with the EOC,

[00:35:22] the vendor has, it's not about the vendor.

[00:35:24] It's about the local use of the tool.

[00:35:26] However, the EUAI acts gonna change that.

[00:35:29] Benders are gonna have to be accountable soon.

[00:35:31] Yeah, that's what I was gonna say.

[00:35:32] I do respect those who have taken the initiative

[00:35:36] to get out in front of this.

[00:35:38] I think for solution providers who have,

[00:35:40] they're either AI sort of native solutions

[00:35:43] or they've added AI capabilities to their solution

[00:35:47] just to go out and say, you know what?

[00:35:49] This is important for our clients

[00:35:51] and our current clients and our future clients

[00:35:55] that we are considered a trusted partner.

[00:35:58] And people can say that,

[00:35:59] but if you don't want it to just be lip service

[00:36:01] or based on some historical metric,

[00:36:04] if you really want them to be trustworthy,

[00:36:06] then why all else being equal,

[00:36:09] you had your short list of vendors

[00:36:10] and only one who had taken initiative

[00:36:14] to go ahead and get an independent audit,

[00:36:16] even though they weren't necessarily responsible for it.

[00:36:20] It just, I don't know,

[00:36:21] it gives people a little bit more comfort

[00:36:24] that this person's gonna,

[00:36:25] that this company is going to be with me

[00:36:28] and they're gonna stay on top of it.

[00:36:29] And if I do get audited as a user,

[00:36:32] as a town acquisition team, for example,

[00:36:34] that they're gonna be there to support me

[00:36:37] and if the auditor needs some of their data,

[00:36:41] maybe you don't have enough demographic data

[00:36:44] and you need to go back to the vendor

[00:36:45] to supplement it or whatever,

[00:36:48] I think that's important.

[00:36:49] I also think that just because

[00:36:52] I guess where I wanted to go was like,

[00:36:54] when I think of the concept of responsible AI

[00:36:57] and that being a sort of umbrella term

[00:36:59] similar to like human centric AI

[00:37:03] that I think is more used in research

[00:37:05] and academic communities.

[00:37:08] But if you maintain that

[00:37:10] and you think of that as the overarching theme

[00:37:13] and collection of these concepts around fairness

[00:37:17] and explainability and transparency

[00:37:19] and bias mitigation and all these other concepts

[00:37:22] that we supposedly care about,

[00:37:25] then you should want that wherever your solution

[00:37:28] is deployed across the town, life cycle.

[00:37:32] Because some might say, oh well,

[00:37:33] we're too early in the cycle, right?

[00:37:36] We're doing programmatic advertising, right?

[00:37:39] We're doing job boards or whatever.

[00:37:41] And so that stuff's never gonna be

[00:37:43] we're never gonna be on the hook for that.

[00:37:45] It's like, well, you know what?

[00:37:47] I disagree.

[00:37:49] You might not be the target of a lawsuit

[00:37:52] by a candidate, but you can't tell me

[00:37:55] that you're not using algorithms

[00:37:57] where maybe you're bias

[00:37:59] and who you're actually targeting

[00:38:01] with your programmatic advertising

[00:38:03] or who you even bite to come into your talent community.

[00:38:06] Or I mean, you could be not necessarily

[00:38:08] in a purposeful way,

[00:38:11] but you could be unnecessarily excluding

[00:38:13] certain populations from even seeing a job

[00:38:19] that they're capable of doing.

[00:38:22] And so if they can't even throw their hat in the ring,

[00:38:25] so to speak, then it seems like a problem unto itself.

[00:38:29] A million percent.

[00:38:30] I say and I have model like graphics

[00:38:32] and stuff I've made about this.

[00:38:35] So the great Satan of all this

[00:38:37] in my estimation is programmatic job ads

[00:38:40] and recommendation engines

[00:38:42] because hiring is a probability game.

[00:38:44] It's a funnel.

[00:38:46] And if you don't put something in the funnel

[00:38:48] that you want to come out the other side,

[00:38:49] you can't get blood from a rock.

[00:38:51] So if you're excluding people of diverse backgrounds

[00:38:57] from seeing job ads

[00:38:58] because the preponderance of people in that job

[00:39:01] are white males,

[00:39:03] so you're not gonna serve it up to that person.

[00:39:05] You'll never get that person.

[00:39:07] I mean, they may find it another way,

[00:39:08] but you're not attracting

[00:39:10] and getting in that person's face with your job.

[00:39:13] And so they'll never apply

[00:39:14] and they'll never get hired.

[00:39:15] And then bias compounds down the funnel.

[00:39:18] Every single decision at a layer in the funnel

[00:39:21] has a potential for bias.

[00:39:23] So by the time you get to the bottom of that funnel,

[00:39:26] you're not gonna be even cl...

[00:39:28] That's why we end up with such homogenous stuff.

[00:39:31] So one of the best ways in my mind to combat that.

[00:39:34] And I've seen this done really well

[00:39:36] like in Nike, I used to work for Nike a bunch.

[00:39:39] They would go out in the community

[00:39:40] with their recruitment efforts

[00:39:42] and build and engage people

[00:39:44] that they want to fill certain jobs to say,

[00:39:48] we want diversity.

[00:39:49] So we go find diversity,

[00:39:51] we engage diversity with our brand,

[00:39:52] we sponsor events

[00:39:54] and we have recruitment representatives on the ground

[00:39:58] meeting people, building relationships.

[00:40:01] It costs more money, it's harder to scale,

[00:40:03] but at the end of the day,

[00:40:05] that overcomes what we're just talking about, right?

[00:40:07] Like then you don't even have to worry about a job ad.

[00:40:09] You're going out and getting what it is

[00:40:12] that you wanna have

[00:40:13] and making sure they get in your funnel.

[00:40:15] So that is not sitting back

[00:40:17] and letting the AI do it for you, you know?

[00:40:20] Yeah, you hit on two really, really important points

[00:40:23] that I always try to impress upon people.

[00:40:26] One is if you care about DEI

[00:40:29] and you can say, well, it was well intended

[00:40:32] but badly marketed or whatever.

[00:40:35] If you care about diversity, equity, inclusion

[00:40:38] and belonging, then you have to support responsible AI

[00:40:41] because that's what's going to be,

[00:40:44] that's where we can actually keep tabs on all of this

[00:40:47] and that's what's gonna be the overarching

[00:40:51] and prevalent mechanism following those principles

[00:40:55] are gonna have those downstream effects on DEI

[00:40:58] and other kinds of programs similar to that.

[00:41:01] But before we go, Charles,

[00:41:03] I mean, there's one question I ask all my guests

[00:41:05] which is the title of this podcast

[00:41:07] is called Elevate Your AIQ,

[00:41:10] what comes to mind in terms of getting people sort of ready?

[00:41:14] Yeah, two things I did.

[00:41:16] So A, back to consuming information, right?

[00:41:19] I took a Coursera class on prompt engineering

[00:41:22] and then there's a really good Microsoft course

[00:41:25] that has like fundamentals of AI

[00:41:27] and it's a self-guided course.

[00:41:29] There's like six or eight modules, it's free.

[00:41:32] Can't remember where I found it.

[00:41:34] I think someone posted on LinkedIn about it.

[00:41:35] I can't remember the name of it

[00:41:37] but it's sponsored by Microsoft.

[00:41:39] It's either IBM or Microsoft.

[00:41:41] I can't remember which one.

[00:41:43] I think it's IBM actually.

[00:41:44] It's a really good course

[00:41:46] and I knew a lot of the stuff

[00:41:47] but there was stuff I didn't know

[00:41:49] and I took it over about a week.

[00:41:51] So there's all kinds of free high quality

[00:41:53] education tools out there.

[00:41:55] You know, you and I talk about a lot of concepts.

[00:41:57] They're kind of flying around.

[00:41:59] We tie them together on some themes

[00:42:00] but from a purely technical standpoint,

[00:42:03] what the hell's going on behind the scenes?

[00:42:05] How does this stuff work?

[00:42:07] I'm still learning about the transformer model

[00:42:10] and diffusion models of like how Chatchi DP works

[00:42:13] and if you understand that it's a prediction engine

[00:42:16] and all it's doing is predicting what the next word is,

[00:42:19] it blows your mind even more

[00:42:21] because you're like, wow, there's a lot of combinations.

[00:42:24] How does it know?

[00:42:25] But it's not a person.

[00:42:27] It doesn't have a personality.

[00:42:28] It's just the thing that can predict stuff.

[00:42:31] Supernatural math is kind of what I call it.

[00:42:33] But anyway, and I would suggest if you're getting into this

[00:42:37] that's a foundational and fundamental thing to do

[00:42:40] because then you can understand what transparency is

[00:42:43] and what explainability is at a deeper level.

[00:42:46] Love it, love it, awesome.

[00:42:48] That's what great to talk to you as always.

[00:42:50] Thanks so much for coming on

[00:42:51] and sharing your perspective and insights.

[00:42:55] Really, really appreciate it.

[00:42:56] Yeah, 100%.

[00:42:58] Thanks so much for inviting me.

[00:42:59] It's always a great conversation.

[00:43:01] Congrats on having a podcast as a fellow podcaster.

[00:43:05] I know how much work it is,

[00:43:06] but it's also super rewarding

[00:43:08] because we get to have these focused conversations.

[00:43:11] So always a pleasure, Bob.

[00:43:13] Awesome, much appreciated.

[00:43:14] That's it. Thanks everyone for listening

[00:43:16] and we'll see you next time.