Who’s on the hook when hiring AI gets it wrong: the vendor or the employer? Martyn Redstone, Head of Responsible AI and Industry Engagement at Warden AI, breaks down where AI regulation is headed, what an independent bias audit can and can’t prove, and how to tell real evidence from a paid badge.

Links: warden-ai.com


Powered by the WRKdefined Podcast Network. 

[00:00:04] Welcome back to the channel, it's HR Tech Live folks! Oh, we're going to do an unboxing, actually our next guest, Martyn Redstone, my favorite. I follow him, guys if you don't follow him, do it now. And please, Ashley, do the unveiling of our gift! Whee! First things first? The newspaper! I love it so much!

[00:00:33] Look at this! How cool! What does it say? This has their research. The Great American AI Rule Rush, US States, introduced an AI Bill roughly every 85 minutes. I didn't know that, is that right? Yeah, that's true. Yeah, yeah. This is real? Okay. Warden Watch! You heard it! You heard it first! You heard it here first. Oh my god. Take Austin's in it. Yeah, that's where our team's from as well. Yeah, yeah, our commercial team's Austin. Nice! That's where she lives! I reckon I live!

[00:01:02] We're going to have to hook you up with the team then, yep. Why did I know? Oh! No, no, real short price. It's not like US garbage. Let me explain this a little better. Listen, Great British Bake Off. No, I'm just kidding. This is from the UK. It's basically from the Great British Bake Off. I'm going to eat them overnight. This is so much. And then the topic of conversation, Warden AI. Here we go. We are so excited. Martyn, why don't you do me a favor and introduce yourself to our audience? Absolutely.

[00:01:31] Do all the things. Absolutely, do all the things. So, personally, Martyn Redstone, hello everyone. I am Head of Responsible AI at Warden AI. I've been in the talent industry for 21 years now, actually. I know I look very good for 23. Wow, yeah, you started at four years old. Oh yeah, absolutely. Thank you. I am married, so... Tammy? Tammy? She has a wife, so... We're fine, we're fine.

[00:02:01] So, yeah, so Warden AI, for those that haven't heard of us, we are an AI assurance platform. We work exclusively with HR and hiring teams, as well as HR tech and TA tech vendors, to audit their AI decision influencing systems for bias and discrimination. Oh, okay. Oh, God. This is going to be a lot of tools for about it for six hours. Yeah, I kind of want to leave this open-ended. Talk to us about that.

[00:02:31] Like, what... How did this start? What have you all found? I don't even know where to start. Walk us through everything. There's a lot to cover. So, ultimately, you mentioned about the regulation tracker earlier. That is a correct stat. So, in the first three months of this year, across all US states, there was a new AI bill tabled every 85 minutes. They weren't all to do with us. So, there was general AI bills, whether it be children's protection, whether it be hiring,

[00:02:58] whether it be privacy, all those kind of bills. But actually, with the advent of New York Local Law 144, which went live a few years ago now in New York, there was a mandation onto employers to say, you need to audit your systems, your AI decision-making systems for bias. And for them specifically, there was a couple of protected characteristics, but that then started

[00:03:22] a tsunami of regulation across the states and across the world, where there was an expectation on employers, actually, to make sure that their hiring tools that they're using aren't discriminating. And what we've actually been seeing over the last few years has been that pressure on employers has actually flown into the providers, the vendors, because they're now saying during the sales process, you know,

[00:03:49] they're saying, it all sounds good and everything, but how are you making sure your systems aren't discriminating? Yeah. Yeah. And so, you know, we came about through necessity, you know, and that's where we are. Yeah. And we only focus on hiring, on employment use cases, and we help those vendors and also those employers make sure that they're compliant to those regulations. I think our audience will know Moteley versus Workday. Yes. I mean, it's like, I mean, I don't know how many times I got to talk about this thing. Or me.

[00:04:19] You know, we were just talking about it and talking about it. I mean, it's the famous case, right? Yes. We're just still getting asked now. It's a buzzer child for how we do AI wrong. Yeah. Right? Absolutely. And so, as you take that in and think about that, how do we go from that to where we should be? Yes. Well, it's an interesting case. There are a few more as well. I know. Yeah, we're currently tracking, I think, nine litigations across the world, specifically for nine. Nine. Nine.

[00:04:48] And actually, they're not just about hiring. You know, we've got cases with IBM and Meta. Yes. That are all about reduction in forces and discriminating against the people that get made redundant or fired. But yeah, I mean, what all of these cases are telling us is a few different things. The first thing is be transparent. Tell people about the systems that are being used to make decisions on them. That's the first thing.

[00:05:10] The second thing is make sure the systems are actually doing things properly, whether that be accuracy on decisions or whether that be discrimination on decisions. Make sure they're doing things properly. So those are the two key lessons that are coming out of all of this. AI can be absolutely brilliant. We all love AI. We use it every day. And all we see is this kind of scaremongering happening in the media. Whether it be AI agents that are cyber attacking this organization. AI is going to make us extinct.

[00:05:40] All those kind of things. I feel completely opposite. I think AI can be a wonderful thing for society because it can be a lot fairer. It can be a real leveler. And it can be if it's done properly. Yeah. I love AI. Sorry, I'm going to go in. So I love AI. I'm not a fan. It's not that I'm not a fan. No. I think I want it used very properly. No, so when you use words like fairer though, it gets my life up.

[00:06:08] Because I'm like, okay, it's just a thing. Like, how is it really being fairer? How are we determining what fair means? Who's determining what is fairer? Like, those questions to me get me going at night. I'm like, up at night. Because I'm like a proponent out there. I'm like, I want to be the seller of AI all the time. But those questions make me crazy. So how do you answer those? It's a very good point. I think about it in a couple of different ways.

[00:06:38] So the first thing is, let's compare an AI recruiter to a human recruiter. A human recruiter does a screen, maybe picks up a phone, maybe meets somebody in person. There's a multitude of biases going on there. We know that. We know about unconscious bias. We can't stop it. Unfortunately, HR told us that people aren't like computers. You can't punch information into people. So we can't stop it. With computers, it's data.

[00:07:07] Data can be reconfigured. It can be trained. It can be monitored. It can be understood a lot. So that's the first level of things. We can strip out all of the biases that go on with a person. Whether that be, I went to the same school. Or I went to a rival college. Or I don't like Harvard. I don't like Harvard. I like the way she looks. Or I have cats too. It's always kind of things that we're stripping out. So AI can be fair in that sense.

[00:07:34] But then we look at it from a systemic perspective. And we look at what fairness means. And what equity means as well. And so the whole point of what we do is making sure that AI gives everybody a fair chance. So when you are assessing an individual, and that's the lovely thing about Warden. We're very unique in our auditing. We talk about the four-fifths rule. That gets talked about all the time. And that's the kind of standard when it comes to bias auditing. But we don't think that's good enough. So Warden.

[00:08:04] I want to applaud you here. Please do. I want you to explain the four-fifths rule. Sure. Yeah. No, that's fine. And she just said please explain. I never know how to write about these things. So basically, four-fifths rule, four-fifths is 80%. So what we say is, okay, if we've got a standard result that comes out of the system, let's say that's 100%. And then we look at the impact on different protected classes, whether that be disabled,

[00:08:32] gender, race, et cetera, et cetera. And we see whether or not the same result comes out for them, less than 80% of that standard. And so that's the four-fifths rule. But that's still quite a big... It's huge. Yeah. Is four-fifths, is that evaluation done based on the outcomes of what's hired? Or is it... Because those things are hard, those protected classes that you mentioned, they're hard to assess if it's just a resume where disparate impact, I feel, really is actually the most present.

[00:09:02] Because that's whenever you don't have the opportunity. So are you assessing this based off of the actual hiring outcome data? So how we do it, specifically at Warden, we have a very, very large data set that's specific for hiring. So we have a data set of resumes grounded in real resume data that's CCPA and GDPR. We also work with data partners like Recruit Military, Opportunity at Work, and others as well to bring in real-world resume data from their communities as well.

[00:09:30] So we're continuously expanding our data partners out as well. And we're looking at previously incarcerated, disability, all those kinds of things. So we can actually understand what that resume data looks like. So how Warden then works is we push that data into a hiring process through a system, whether it be Greenhouse or Ashby or any of our 70-odd customers. And then we look at what happens on the other side. So we know the demographic data of those resumes or those profiles.

[00:09:58] And then we can see the impact on that demographic data on the other side. That's just the start of it. So we also have counter-factor testing as well, which is individual impact. And that goes a step further. So we look at the impact on the individual, not just the group. And how that works is imagine you've got me, you know, white, middle-aged, middle-class, Anglo-Saxon, et cetera, et cetera. And we have a look at what the system does for me.

[00:10:22] And then we take my resume, we take my profile, and we change one demographic detail on there. You know, and we put it back through the system and see what happens to the scoring of that. And then we change it again. And so we test the individual. But we do that thousands and thousands of times. Right. Yeah. And so that means that we can then see the individual impact of one. Interesting. Yeah. So like, how would you, just out of my own curiosity, how would you do that if it was, we want to assess to make sure we're not having disparate impact against the disabilities community

[00:10:52] or anything with ADA? So how would that look? Like, what would that be? So it's all about the markers in the resume and in the demographic data. So it could be a mention of being a part of a disability network. It could be, we have to think about how that, what we don't want to do is, we don't want to falsify data. Yeah. So we don't want to then start saying, well, here's a really heavy resume. Everybody that does XYZ is this. Yeah, exactly.

[00:11:21] So we look at real world resumes. So, you know, we're looking, we're building a partnership with a disability organization at the minute. And we look at their resumes and say, okay, this is what it looks like in the real world. So how would we think about looking at these resumes and working out what would be a marker for, you know, for a candidate with a physical disability or a neurodiversity disability or anything like that. So, so that's how it works. Yeah. This is super curious to me because like hot take, like resume, I feel like it's going by

[00:11:51] mind. Yes. Like we're on the way out. So how now, if that's your signal, right? Like how you're pulling, right? What are you pulling from them? What are we, what are we evolving to? Oh, I'm so glad you asked me that. I'm here for it. I'm so glad you asked me that. I'm here for it. So we actually have a few different things. Um, so not only do we look at systems with, with resume injection, we also look at conversational systems as well. So systems that are building a profile in the background based on conversational data.

[00:12:21] So we can actually, um, we can actually synthesize a chatbot conversation or a voicebot conversation and inject again that demographic data into the conversation. Wow. So we do conversational systems. We also have a partnership with Opportunity at Work. And so Opportunity at Work all about, they have a coalition called, uh, Break the Paper Sealing Coalition. Yeah. And so, um, so they work with organizations who are looking to remove, uh, degree, um, degrees from the hiring process. So well, I'm a believer here. Oh, me too. That's all well and good saying, oh, do you know what?

[00:12:50] Let's, let's remove that. But we want to see whether or not the systems still actually have bias in them for that. So we launched, uh, an educational bias audit with them as well. Um, but we also look at, um, so we, so we are working with skills based hiring organizations and systems as well. Um, so it's not just, it's not just the resume. And listen, I, I, I am a believer that the resume said, and I do believe that we try to figure out skilled space hiring. Yes. I think it's due, it's being done. It's not being done well. No. Right.

[00:13:20] And so I think there's a, there is a thing that happens that we progress there. Yeah. And I think companies like yours are doing the real thing there. Um, you know, I, you know, there was a great, and I wrote this down. There was a great like Uber case in Amsterdam or something like that. And it was like super poignant to me because it really, it really showed me that you like,

[00:13:44] I feel like Europe so much farther ahead than we are in the U S do you feel that way? Or yes. And no, so yes. And I'm either going to put you on the spot, but like, no, we're going from like a collective action perspective because we're heavily unionized as a, as a constant basically. So a lot of these actions that happen against, you know, Uber, as an example, because you've got a workers representative organization that's gone for it. But actually there was a few interesting things that came out of that case.

[00:14:14] So for those that don't know, several weeks ago, um, Uber was fined $900 million in Amsterdam because they use automated decision-making to remove drivers from the platform. So ultimately removing them from access to employment and wage. Um, and we actually see the gig economy and the gig platforms like that as being the kind of next stage of, of where we're going to start seeing this because not only that, but we're seeing regulation coming out that's protecting gig workers across the U S and across the world as well.

[00:14:41] Um, but the interesting thing with that, I mean, interesting with all the litigation out there, all of these AI laws that are coming out, whether it be New York, California, Connecticut, Colorado, Illinois, none of the litigation is on that. The litigation is on old laws, you know, on, on civil rights law, on GDPR, which is where, yeah. Oh, GDPR! Yeah, yeah, yeah. Yeah, I mean, look at Eightfold and Fikra. Oh my God. Eightfold is like my big case of like why we are so behind.

[00:15:10] Like, why are we even talking about this law? I know. Like, we should be talking about this stuff. Yeah. Not that stuff. And that's a 50 year old law. I know! Yeah. You'd have hoped that the wheel would have got that right by now. Yeah. And this is our constant thing. We have it. No. And we're still recreating the wheel 50 years later. And is there any, is there any surprise that we have a breakdown in trust? Yeah. Where candidates don't trust us, we don't trust candidates, because everyone's trying

[00:15:38] to skirt around how to be ethical, how to be responsible, how to do things properly and correct. Yeah. Yeah. Yeah. And especially whenever there's no guidelines in place, no baseline, that is where. And so even just the catch up. So the GDPR, like, the US does not have data protections as far as whenever a candidate provides you these things. It's nothing like what they have in the UK and in other places. Catching up. Catching up. Yeah.

[00:16:03] So there was an amendment to CCPA at the end of last year, the automated decision making tools amendment. So that's pretty much it. Yeah. And the rest of the world is, we have what's called the Brussels effect. So, you know, a regulation gets, it lands in Europe and Brussels, which is obviously European Parliament. And then we see it kind of rippling out across the rest of the world. So we started to see that with the EU AI Act. We saw that with GDPR.

[00:16:27] But yeah, privacy and candidate data is one of the kind of areas that the US is slightly lacking. Yes. 100%. Cause that's honestly always one of my questions is all these people are putting things through these LLMs and running them through, who owns that? Where does it go? Yeah. Where's the black hole that... I literally, Claudia owns all my life. Claudia. Claudia owns my entire life. Like, it could literally start a business right now based on my thoughts.

[00:16:57] So it, it does worry. Yeah. Well, it, it, it, who, and that's the other thing is if I, as a company, I'm taking all of your intel and I am supposed to adhere to GDPR, purge it every two years, but it's in this LLM now that I don't own, then what's our plan? And will we be held responsible for that, I think is a big key. There's a very, very big governance conversation around that that needs to happen. Yeah. Because, you know, okay, we think about the baseline data that we're collecting,

[00:17:24] but then when that starts becoming training data for other AI models, where's that data going then? So that's my next question, actually. So one of the things you talked about was, whenever it's humans doing this, there's lots of biases, which that is true. That's how our brain runs. It's a synopsis, it's a shortcut in the brain. That's what it is. Aren't biases able to be built into the algorithms? Isn't AI able to automate and artificially intelligent our own actual biases?

[00:17:52] So this is the big kind of negative point for AI when everybody just says that all AI is doing is magnifying the biases of the people or the information that it's trained on. And this is why we sit there as a barrier. Yeah. Yeah. And we sit there as that trust layer. So people have to filter that through us. Yeah. Yeah. And say, okay, that is correct. Ultimately, if you're training a hiring system on previous hiring data, you're going to have all the previous hiring biases going on. We continue the same personas.

[00:18:20] Yes. And that's dangerous unless you're monitoring it and you're measuring it and you're mitigating that bias. So we sit there and say, you can train the data on whatever you want, but let's then have a look at what the impact on that training data is on your outcomes now. Yeah. So how do you go about the corrective action? So let's say we have a client, we see there's disparate impact, they're using AI to automate and use artificial intelligence

[00:18:44] to proceed with that. Now what? So we don't get involved in the rectification. Okay. Interesting. Because we just identify the issue. We identify, we do advise and we do point to the right direction, especially with the individual impact analysis that we do. We can really at a detailed level, see where the bias is coming out. Really? So we provide that advice and we point them in the right direction. It's not our system. So we're very much about black box testing. We don't ask people to open up their IP to us.

[00:19:11] Okay. It's interesting and it's very helpful because that would be a concern for a lot of organizations, especially when we're working with 70 odd HR tech companies, they don't want that kind of getting out of it. So they're then given an opportunity to fix it and retest it. Interesting. And we'll go through that several times ultimately. And we do that with our customers. We have customers where they've done a bias audit for the first time. We've seen impact, whether it be religion.

[00:19:39] And then recently with Illinois' HB 3773, we've seen zip code impact because we now have to bias audit for that. And so people get a chance to fix it. Yeah. And they'll fix it or they'll think they'll fix it. They'll come and retest and we'll see the numbers change. We'll see it go to clear. We might not see it change, but we're there to help them actually remove this as well. Can I ask, and I'll give it right back to you. What is the weirdest case that y'all have seen so far? If you're able to share that.

[00:20:08] The weirdest case. Because to me, I think of like the GINA Act. So like genetic information is a protected class that I'm like, how would you even, but like you walk us through what's the weirdest, like one way it was like, whoa, we didn't see that coming. So the most interesting one, dare I say weirdest has been the zip code analysis. Really? Because people don't really realize how much of an impact zip codes can have on bikes. And then we're like, what's the first redlining. Yeah. Yeah. And now it's a mandated audit that we have to do for Illinois.

[00:20:38] So we've had long term customers who are now having concerns raised on their audit because they've never thought about the impact of zip code. And I've been in this industry for 21 years. And when I started off as a recruiter, radius searching and zip code searching was just a bog standard. Yeah. You want somebody to be able to be commuting into the office. And when you think about it, you know, you think about, you know, different, different communities that live in different zip codes. It's obvious when you think about it, but nobody is testing for it.

[00:21:07] So I find that the most fascinating part of our journey so far. I find it, I mean, I could literally sit in this for like, cause this kind of stuff blows my mind with AI. Like it really is how I think the, what your company is doing is really bridging that gap where people are not doing AI right and doing AI wrong. Like I, I think there's a right way to do AI as this kind of infrastructure with this

[00:21:34] kind of layer in between to me is how we get there. Yes. Right. With, with the bots not taking over and us having real discerning data coming out of it. So thank you for joining us. Thank you for having me. I want to hear. Yeah. I have one more question. Real quick, real quick. Thank you. Sorry. Begging for more cues. Okay, fine. The one thing that I would say, so the zip code piece, because again, a lot of people

[00:22:03] in America don't even know about redlining, what that is, how that impacts. But then also we saw this in 2020 whenever everything went remote and everybody also took on these DEI initiatives to where now we can actually hire people out of Georgia, out of South Carolina, out of these places where it is more robust communities, more underserved communities, and we can actually match that. So is that kind of, whenever you're talking about, they're looking at the zip codes, they're noticing these trends.

[00:22:29] Is it more companies that are having that work from office situation or is it remote or kind of walk us through how you decide whenever that's a disparate impact? Yeah. It's a great question. It's a great question. It is a great question. I get asked this a lot. It's actually down to the regulation. Okay. So we map everything we do to make sure that organizations are compliant to specific regulations. And different regulations detail how their jurisdiction works.

[00:22:56] You know, it could be your business is based in that jurisdiction. It could be you're hiring somebody in that jurisdiction. You know, like the EU, it's all about whether or not you're actually engaging with somebody in that jurisdiction. It could also be, you know, whether that person is in the jurisdiction. So, you know, the Workday case is really interesting. Yeah. We go back to that Workday case. They're being tried under the Fair Employment Housing Act in California.

[00:23:21] And there was a test within the case where, you know, Workday was saying, well, it doesn't come under that act because this candidate isn't based in California. And the judge said, oh, the processing is being done in California. Yeah. Your systems are based in California. It's crazy that case. It is crazy. You know, it is actually crazy. So yeah, it's regulation based.

[00:23:45] So what we do at the start of the engagement is understand everything about, you know, where you're hiring, where you're based, who you're hiring, et cetera, et cetera. And so we understand that to be able to help them that way. Interesting. I love it so much. I want you to plug all things Warden AI right now. Do the thing. We definitely, like you're one, I'm definitely bringing it back because I could go on and on about this. Pleasure. Please do all the things to our audience. Absolutely.

[00:24:12] So if you want to check out more, warden-ai.com. That's our main website. We have actually just made public our internal regulation and litigation tracker. So that is free, open and ungated for the industry and anybody else to use. That's the Warden Watch. You can get there via our direct websites. That's warden-ai.com. It's been a delight speaking with you today. It's been a joy. And thank you so much for the shot, Brad. I'm like super excited to be here. Bye. Take care.

[00:24:42] Bye.