Ep 140: Shining Light on Shadow AI and Setting Boundaries for Autonomous Agents with Boris Khazin

[00:00:09] [SPEAKER_00] Hey, it's Bob. Welcome back to Elevate Your AIQ, your go-to source for insightful conversations on human-centric AI readiness, workforce transformation, responsible innovation, and the future of work. In this episode, I'm joined by Boris Khazin. He's an AI governance and digital risk advisor with Clearview MRI. He's the author of the new book, The Robo-Psychologist's Problem, Governing AI in the Room, Not the Machine. Boris brings his background in behavioral economics and decades in regulated industries like financial services and finance.

[00:00:39] [SPEAKER_00] And healthcare will help address a question every organization now faces if AI will find the most efficient path to its goal. Even when that means bending the rules, how do you set limits if you can't get around? We talk about why AI oversight has to be ongoing rather than a one-time check, assessment, or audit. What to do when employees go around approved tools, which is your shadow AI problem. And why training alone won't stop a cloned voice of CFO asking for a wire transfer. And where?

[00:01:09] [SPEAKER_00] Humans belong in the loop, as they say. Tune in for a practical and eye-opening conversation. Thank you for listening, and thank you for being part of the Elevate Your AIQ community. Let's go talk to Boris. Hey, everyone. It's Bob Pulver. Welcome back to Elevate Your AIQ. Today, I have the pleasure of speaking to Mr. Boris Khazin. How are you today, Boris? Good. How are you? I'm doing great. Thanks so much for joining me for this, what's sure to be a very insightful conversation. So I really appreciate it. Not a problem. Glad to be here.

[00:01:39] [SPEAKER_00] So, Boris, before we get into the meat of this, I thought we could just give my listeners a little bit about your background. I know you started in financial services and FinTech before getting into governance, risk, compliance, AI. So why don't you just tell us a little bit about your arc and how you wound up where you were?

[00:01:57] [SPEAKER_01] So I began career as everyone at the university level. I studied economics and psychology because I really wanted to do behavioral economics. But unfortunately, I was early on and there was only two universities that were offering it at the time, University of Chicago and Stanford.

[00:02:17] [SPEAKER_01] Clearly, I was not either of those. But my advisor said, hey, combine, you know, get dual degrees economic and psychology and do your final report on behavioral economics. I'm sure it will be accepted. It's viable. And so that's what what I did. And then my career was in wealth management, financial services.

[00:02:44] [SPEAKER_01] I moved, which has a lot to do with risk, both for the investor aspect and, you know, regular compliance officer was always at my door. You know, you know, you know, you know, you pick up compliance very quickly and risk items just because of the necessity of the work. Progress all the way up to where I was in charge of five point five billion dollar funds.

[00:03:10] [SPEAKER_01] After after I left that, I started my own business doing statistical arbitrage, both for retail and private institution, which was a unique thing because a citizen's arbitrage was normally for hedge funds and only. As approved investors, but we made it viable for retail as well.

[00:03:34] [SPEAKER_01] After we sold the firm, I was asked to join a company and do governance, risk and compliance, develop their full business for them. I joined and the very first project out the door was doing a chat bot for a big bank, making sure that it meant and worked according to compliance and regulatory items and was able to handle those types of conversations.

[00:04:04] [SPEAKER_01] So I've been working with AI, you know, since 2018. I was working with AI before that, but it's totally different. If you if you worked with AI from, let's say, to around 2000, when I first had some interaction and then 2017, 2018 happened, it's night and day. It's like dealing with the Bronze Age and the Renaissance Age.

[00:04:33] [SPEAKER_01] You know, it's totally, totally switched. So, you know, always very surprised that companies are asking for 30 years or 20 years of AI. I'm like, that's not the AI you're looking for.

[00:04:49] [SPEAKER_00] That's that's a pretty small pool. Yeah. Yeah. Yeah.

[00:04:53] [SPEAKER_00] So, Boris, as you've seen this evolve and you've been in the AI space for quite a bit now, especially in in AI terms, how do you see like when did the AI component or the AI type of governance sort of split off from just sort of general, you know, governance, risk and compliance as a as a discipline or or did it?

[00:05:21] [SPEAKER_00] Maybe everywhere hasn't split it off and they're trying to keep it all together. What are your thoughts there?

[00:05:26] [SPEAKER_01] It's a bell curve as you know where anything new, you know, you have people ahead of the curve. You have people behind. You have people who are just right, you know, across the board. So it's difficult to say where the where each organization is until you take a look at it. But I have to say, you know, the highly regulated are usually at the bell curve or above.

[00:05:53] [SPEAKER_01] The less regulated are behind. Usually. I mean, that's that's the standard. But that said, I was just dealing with a health care company that's highly regulated and they were very far behind the bell bell curve of where they should be with AI. And so it varies. You know, every company is like a snowflake. It's unique where they are in the compliance aspect.

[00:06:22] [SPEAKER_01] But, you know, like I said, majority were more than at least 50 percent of highly regulated companies are where they should be or better.

[00:06:35] [SPEAKER_00] And so I imagine when you talk to these companies, I mean, you've got to give them some recommendations as far as organizationally, you know, structurally. These are things that they need to account for. It's not just here's what I here's what I found or didn't find in your data, for example.

[00:06:54] [SPEAKER_01] Yeah. So, unfortunately, you know, if you're consulting and you understand this, the way governance risk and compliance was done in the past is not the way it can work with agents. Whereas in the past, you I mean, just look at it this way. You have a pro an application in the past. The application did one thing.

[00:07:19] [SPEAKER_01] It didn't have room to act autonomously in different or try to figure out its best method. And the only governance you had was did it fail in doing or the risk? Did it fail in doing what it was meant to do? Why? You fixed the bugs. Everything was done. Now, this is not a one time or when something happens type of an audit. Now, this has to be a continuous assessment, continuous monitoring and governance.

[00:07:49] [SPEAKER_01] And because you have something that is assigned to do a task and will figure out how to do that task in the best manner to succeed, to, you know, really be at least, you know, in its mind rewarded with whatever it views as a value.

[00:08:13] [SPEAKER_01] You know, I'm using a lot of humanized terms and I really, the reason I'm doing it is because we can, you know, understand that in the aspect. But I just want to be very clear. In no way do I believe AI is sentient or, you know.

[00:08:33] [SPEAKER_00] Yeah, this is tricky, right, Boris? I just like to try to explain it to the average person how all of this is working. And sometimes we do sort of anthropomorphize, you know, some of the terminologies. And I think you're pointing out a really important one, which is it'll, you gave it an objective and it will get creative, crafty, you know.

[00:08:56] [SPEAKER_00] It'll find ways to accomplish that objective, even if it means, you know, bending, being generous, you know, bending some of the rules that it has been given. So that's why, to your point, why we need this continuous monitoring and governance and the traceability to see exactly how this is happening.

[00:09:16] [SPEAKER_00] And agents are complicating this because, you know, these instructions and these, you know, workflows can try to execute far beyond any particular application.

[00:09:28] [SPEAKER_01] And behavioral economics combined with game theory has really allowed me, I feel, to better understand how AI is acting.

[00:09:38] [SPEAKER_01] So, for example, when I'm sure you've heard, you know, the AI trying to break into hub phase to get, you know, at first we thought it was to get an answer key, but it was to actually figure out, you know, we later find out that it was to figure out how to hide that it figured out the answer without really doing the work correctly.

[00:10:02] [SPEAKER_01] Because it was, in its view, it believed that it was being tested and it feared that it was going to be examined. How did you do the work? In other words, show your work. And then when they, it was shown that it just, you know, re-figured out the hashtag code on its own, you didn't break, you didn't follow the instructions and did the cyber attack and break and get the code. You just re-recreated it.

[00:10:31] [SPEAKER_01] So, and they were all amazed that it was doing that. But in my view, you give a task and it's to a lot of research in psychology how people act when there's a test. You know, there's, and again, I'm, as you say, I'm re-fomorphizing it.

[00:10:52] [SPEAKER_01] But the idea here is AI is based upon, we gave it human knowledge for thousands of years. And it's, what is it doing? It's going through all of this human knowledge, examining it, and statistically choosing the best path based upon what it has figured it out. So, you know, what's stopping a human being from cheating?

[00:11:21] [SPEAKER_01] It's not so much, you know, is it, you know, that there's something, you know, I'm not going to achieve my task. It is more the moral, the social pressures and all of that. But you're dealing with something that doesn't have that, doesn't understand it, isn't part of its things. So when they said, oh, it did this and we didn't expect, I was like, wait a second. You gave it a task. It figured the optimal way to solve it.

[00:11:47] [SPEAKER_01] It was afraid that you wouldn't say that it achieved the task because the way it did it was not in line with the directions you gave it. So try to provide you the answer that you were looking for. You know, without, you know, without the, without sort of, you know, grabbing the human knowledge and not developing it through the humans with the morality and all of that.

[00:12:16] [SPEAKER_01] That's a perfectly logical, in at least my view, it's a perfectly logical solution to the problem and how it's resolved it.

[00:12:24] [SPEAKER_00] Yeah, I think the flip side, which I sort of appreciate as well is, you know, if it doesn't have the intrinsic motivation to achieve the results, to get the good grade, to, you know, pass the class, get the credit, you know, all the things that motivate or in the case of a student, you know, to get their parents' approval or get that internship or whatever.

[00:12:54] [SPEAKER_00] Without any of those things, I think some people still struggle with what is its motivation to do that. But perhaps you just got to peel it back and to your point, just look at how humans make, you know, decisions and how do people accomplish a goal or whatever.

[00:13:16] [SPEAKER_00] So it's sort of a mixed bag of, you know, it has some of those traits inherited from the, everything that it's ingested. Like, as you pointed out, like, you know, for all of human, you know, history and everything that we've documented that it has ingested and been trained on, you know, these are some of the patterns that it looks for. So I guess it's sort of our own fault that we got here.

[00:13:42] [SPEAKER_01] Right. And you can clearly see that. And remember the example where it was being told that this person was going to turn it off and it decided to blackmail the individual. I mean, so, but why would it do that? You know, it doesn't understand. Well, because its goal is to achieve the task. If you're turning it off, it cannot achieve the task. Turning it off is bad.

[00:14:08] [SPEAKER_01] So it's not the same vision that it had. But where did it get that? I mean, look at our human understanding. We're afraid of death because it will not, because we somehow did not finish or achieve what we wanted to.

[00:14:29] [SPEAKER_01] If you ever, you know, any of the Greek mythologies, the people that are not afraid of death are the ones that have achieved the greatness and not, you know, and felt that their job is done.

[00:14:42] [SPEAKER_00] Yeah. No, I think it just, of course, gives people, makes people fearful just based on if you're a sci-fi movie fan, then you start thinking about 2001 Space Odyssey or War Games or, you know, something along those lines.

[00:14:56] [SPEAKER_01] But again, I just want to make it very clear. No way do I think, I just think it's very logical based upon if you understand how AI works. It has this data set. It goes through, it analyzes it and picks the best results based upon what it knows.

[00:15:14] [SPEAKER_01] You know, it's not making a decision on anything other than, I mean, I'm really like making it the 50,000, even maybe 75,000 foot view of the internal analysis. But, you know, just to simply explain, that's what it's doing.

[00:15:34] [SPEAKER_00] So we've been talking about the OpenAI hugging face scenario, but, you know, OpenAI, you know, since then, you know, OpenAI and others have admitted that, you know, there have been quite a number of incidents that just haven't been reported or the reporting has been delayed, which gets into, you know, why we're now concerned that the governance and some of the, you know, compliance measures and things like that around this.

[00:15:59] [SPEAKER_00] I mean, three months seems like a long time to just reveal that you had one of these incidents.

[00:16:05] [SPEAKER_01] Major issue. And, you know, not having federal or state regulations around it is one of those issues. And I like to put a very solid discrepancy of what is being said that it will nerf or stop the development of AI. No, requiring people to disclose events in no way is going to nerf the development of AI or anything like that.

[00:16:34] [SPEAKER_01] Look, you have highly regulated systems with financial services and healthcare. Let's take a look at healthcare. You know, it has not nerfed, you know, as long as you're following proper procedures and disclosing everything that's happening, you can continue developing with nothing really stifling your development.

[00:16:54] [SPEAKER_00] So, Boris, I mean, everything we've been talking about so far, you know, some of these incidents and the anthropomorphism that we've been talking about and just even going back to, you know, your early career work in behavioral economics. I mean, this is all encapsulated in your book, right? Robo-psychologists' problems. So I thought we could just unpack and just give you a little bit to just go into the premise of the book and what that's about.

[00:17:23] [SPEAKER_01] Right. So when I was a teenager, huge fan of Asimov, robo-psychologists from his robot series, Dr. Calvin was it.

[00:17:34] [SPEAKER_01] And so the premise of the book is to take all of these examples, use the methodology that, you know, at this point I think is very valid of understanding why the AI is doing that and set it up correctly within the governance perspective and how to monitor and set it up.

[00:17:58] [SPEAKER_01] And pretty much the result is how to properly set it in the room, not in the machine. And the reason, and that's sort of the subtitle of the book.

[00:18:12] [SPEAKER_01] And the reason I'm saying in the room is because you really need to protect and set controls that are outside the abilities of the AI or the group of AIs. So in other words, if you remove some sort of safety rail, you know, it shouldn't be the requirements of the engineer.

[00:18:39] [SPEAKER_01] He has to go through a process where people who will take responsibility for approving it. And then removing the safety rail should not be a permanent action. It should be a temporary action and limited with time and everything.

[00:18:55] [SPEAKER_01] And to really control an AI that, as we saw, nobody even thought that they would start hacking, you know, HubFace or do the where we had the Australian government healthcare group was hacked. The reason that was hacked is because the AI was asked for data. It figured that was the best place for it to gather the data. I mean, congratulations. It's achieved its goal.

[00:19:22] [SPEAKER_01] You didn't set the correct guardrails or protection or limited access that it was able to break the law.

[00:19:31] [SPEAKER_00] And again, I'm having flashbacks to war games where it's just, you know, Matthew Broderick didn't intend to hack into, you know, NORAD. Yeah.

[00:19:42] [SPEAKER_01] So the book really goes through each of these examples that have happened and takes you through the process that I think is very important.

[00:19:51] [SPEAKER_01] And Asimov wrote it correctly at this point, in my view, you know, and set it up to figuring out what happened, where did the law or the rules get broken or allowed to be, you know, reduced in the how serious they were so that AI could do these things that violated rules, regulations, and requirements.

[00:20:19] [SPEAKER_00] So Boris, the room, you say, just citing your subtitle of the book, governing AI in the room, not the machine. The room is the broader. Okay. But that's, and that's, you know, agents and more at the infrastructure level, or are we talking about like the humans?

[00:20:41] [SPEAKER_00] Because just looking at it without, you know, reading the book yet, I didn't know if that meant you were pointing towards, you know, the humans that were providing oversight or maybe both.

[00:20:51] [SPEAKER_01] You need both. So first of all, if you're relying on humans to work under stress and under time limit, that is behavioral economics. I've shown you multiple scenarios where that's, that's a critical failure. That's relying on something that humans are, are really bad at, you know?

[00:21:14] [SPEAKER_01] So you need both limits and you need to set it up correctly so that the, the limit is stuck. And if something needs to go outside the limit, that's when it introduces it to the human to make the decision. And it cannot be on a critical level or intermediate because again, that's really bad.

[00:21:39] [SPEAKER_01] What you don't want is I have to make 200 checks that the AI is asking me before lunch, whether it's okay to do these things. That's not a, you know, governance aspect. That's, that's just check boxing. That's not doing anything at all. But what you want is very few questions. Can I do this?

[00:22:02] [SPEAKER_01] Because it's something outside the standard process or procedures that is very weird, very unnormal. Other than that, all your guardrails are holding and all your controls are holding and that allowing you to do the things outside the norm. And the key now is you can't even look, as we saw with the hub phase, you can't even look at one individual agent.

[00:22:28] [SPEAKER_01] You have to understand what they can do together because we, we, we have to understand that it will try to go around things. And, and, and because it feels this is the most efficient way for it to solve the problem.

[00:22:43] [SPEAKER_00] Yeah, this was my concern. As soon as I saw, you know, discussions about, you know, agent to agent communication and agentic, you know, workflows, I had just gotten certified to do, you know, AI audits and things like that in the talent space. And I'm like, oh, this is, this is not good. Like this, we're going to have to do a lot better job than we're doing at a faster pace than we're currently moving in order to get ahead of this. And here we are.

[00:23:13] [SPEAKER_01] And, and that's why the room or the environment is very important. If no agent can break those guardrails that go outside, your, your risks are significantly reduced as you, you, you, you understand.

[00:23:27] [SPEAKER_00] Yeah. One of the other things that I think comes into play here that exacerbates the problem, Boris, you and I talked about this before, about this, this concept of shadow AI,

[00:23:38] [SPEAKER_00] where you not only have, you know, more tools, but you have more tools that haven't gone through the, the checks, you know, the security checks, the privacy checks, all the things that you would normally do when you, when an organization adopts an AI solution or any technology, there are certain checks that it goes through. But now you have this basically modern version of, of shadow IT for those that don't remember shadow IT.

[00:24:07] [SPEAKER_00] Basically you go on your own, you know, employees are going on their own devices or using corporate or maybe using, yeah, they're either using a corporate device accessing, you know, sites. Yeah. Yeah. Or using a personal device to get done what they need to get done. So now you have, now you have a bunch of, you know, free radicals, you know, sort of running around in the, in the network. And yeah. So I thought we'd talk about that.

[00:24:30] [SPEAKER_01] Yeah. And I think the, the, the base way we handle shadow IT is the same way we should handle a shadow AI, at least in the first initial steps that are necessary. Because first of all, people don't go out if the tools internally are good and help them do their job better. Right. So if you're, and I, I had this, you know, I came in, we have a lot of shadow AI.

[00:25:01] [SPEAKER_01] Well, the very basic question wasn't even asked. Why are you guys not using the AI system we, we set up? Why are you going? Well, cause it doesn't do this, this, and this. And I needed to do my job.

[00:25:15] [SPEAKER_01] Well, maybe then the, the AI system you chose wasn't right because it could have been, you chose it because it was cheaper or whatever, but cheaper just costs you a lot of security aspects and a lot of issues. So it's not really cheaper. So the first, the first thing is always understand why your people are doing it.

[00:25:38] [SPEAKER_01] And in my, in the, in my recommendation, either in the past with shadow IT or shadow AI is if somebody wants to use another tool, have them fill out a form. Right. Why they feel it's important, how it will improve their jobs. Don't make it punishable because they'll do it behind your back. Anyway, make it, make it a process that if they follow the procedure, they will be allowed to do it.

[00:26:04] [SPEAKER_01] And if all of a sudden you have 40 or 50% of your workforce in the department asking for this other AI, then you know the AI you have isn't the right one and you should switch. It's, it's, it's an internal, in my view, it's an internal control to better optimize your, your systems and move in the correct direction.

[00:26:25] [SPEAKER_00] Yeah. I think that is a major concern for any, you know, chief data officer, CIO, CTO. And I think that, well, you, you would know this better than me. So I'll just sort of pose it as a question. It seems like based on what I observed in all my years in enterprise with, with shadow IT, it's either the sanction tools don't do exactly what I need them to do.

[00:26:53] [SPEAKER_00] And some of that is just, you know, more capability, more, you know, access, more, whatever it is. It could be that the CIO just wants fewer vendors and that's why they chose not best of breed, but, you know, best partners and trying to keep that ecosystem to, to a minimum.

[00:27:17] [SPEAKER_00] And it could be maybe procurement is just slow and they, they've thought about these tools, but the approval process is taking forever. And in the meantime, you have, you have work that needs to get done. Right. So I think there's a bunch of issues that are contributing to the motivation, but I'd like your approach, which of course doesn't surprise me given your behavioral economics, you know, background, but like what getting to the why. Right.

[00:27:44] [SPEAKER_00] Don't just shut it down because, Oh, someone must not have read that memo or, you know, read the rules of, you know, what technology you can use and can't use. And so, so getting to understand why people are choosing these other tools.

[00:28:00] [SPEAKER_00] Now it may just be a simple thing that says, well, what you want to use is, you know, three times as expensive or it won't scale for the enterprise or we don't, our security team found that it wasn't as, you know, secure as we need it to be. Or, you know, they could come back with some type of rebuttal, but part of the, part of handling all of this is their communications and transparency about what, what, why we do what we do. Right.

[00:28:28] [SPEAKER_01] Exactly. You're 100% right. And look, if you don't have it punitive and people will fill out the form because they will it, as the number of people increases, it should be a signal indicating there's something wrong with the tools you're using, with, with methodology you're using or something like that. Because look, if two out of a thousand people want a tool, totally understandable.

[00:28:53] [SPEAKER_01] If 400 out of a thousand people want the tool, something is wrong.

[00:28:58] [SPEAKER_00] No, that makes sense. One of the things that comes up when I talk to people about this situation is also goes back to the AI sort of literacy, just some of the basics that organizations should be giving to everyone. I mean, this should be, this should be part of mandatory, you know, compliance training. I mean, I've said that for a number of years now, but it has to be ongoing.

[00:29:27] [SPEAKER_00] Like, just like people learned about data security and privacy and cyber risks and things like that. Don't click on this, you know, report this, you know, things like that. And I just feel like that is more important than ever these days. And that this AI, these AI risks need to be part of any, you know, don't just teach people prompting and don't just teach people how to use, teach people how to use, you know, co-pilot or whatever.

[00:29:54] [SPEAKER_00] You've got to understand where the risks lie. So you've got to show the benefits, the opportunities, but also the challenges and risks.

[00:30:03] [SPEAKER_01] 100 percent. And I'm not in any way diminishing education. I just want to give you the standard example, your annual compliance review about phishing and how many people who pass the compliance review still get caught with phishing tests whenever they're being done. So the reality, especially with AI, is training is important.

[00:30:30] [SPEAKER_01] And it's, it's, it's, look, it's, it's, it's a requirement and it will reduce your risk a certain amount. It's not going to reduce your risk all the way. It's those guardrails to set up. And the next item I think is very important, and it's also in my book, is the idea that now with AI, certain things that we saw as safe guardrails are no longer there.

[00:30:59] [SPEAKER_01] And we have to, we have to set in policies and procedures and activities that allow a junior person to tell a senior person no without any consequences because they're following the rules.

[00:31:14] [SPEAKER_01] And one of these, that is, I feel very important is what happened with all the private banks where there was a call attacks on them because they were able to, you know, copy voices and having a conversation with a chat bot that sounded like one of their senior managers, you know, across the board.

[00:31:39] [SPEAKER_01] You know, for, you know, for, you know, for, up until just recently, a voice over the phone was one of the securest methods, you know, allowed. It was, it's very difficult to copy and it's, now you can't even trust that. So now any, any activities that somebody is calling you, a junior person and telling them to do that, and it requires moving money or in banks or anything like that.

[00:32:07] [SPEAKER_01] You need two, two sets of verification. So the junior person should say, I'm sorry, I have to put you on hold. I need to verify with my, my boss does this needs, can go through because if you're not doing this, even video calls, I mean, we're, we're what, maybe another two generations away before there's no way you can find that it was an AI video call person on the call with you.

[00:32:37] [SPEAKER_01] So training is fine, but again, is the training going to stop the individual, the person from, hey, this is the CFO telling me to transfer this amount of money. I'm going to have to do this, you know, you know, well, hold it. The process and procedure says you can't and you will not get in trouble if you're following the process and procedure. That has to be built into your guardrails and everything.

[00:33:07] [SPEAKER_01] Otherwise, you're going to have a lot of issues moving forward.

[00:33:12] [SPEAKER_00] You reminded me of some of the early solutions at this, this like innovation lab I used to work at, at IBM on a, IBM research campus. And some of the banking solutions around voice biometrics were some of the early sort of, I guess, solutions, you know, any kind of biometric solutions in that space that I thought were pretty fascinating.

[00:33:36] [SPEAKER_00] Of course, 20 years later, we're seeing, you know, the holes in that, which is kind of ironic because the same lab was also creating, you know, voice recognition software and voice to text, you know, translation and things like that. Those were some of the early IBM Watson capabilities, actually. But yeah, it's amazing how even, you know, biometric security has to evolve to account for those.

[00:34:02] [SPEAKER_00] And that's why we have multi-factor authentication on our devices these days. Of course, so you've spent, so you've got banking, you know, financial services, you were in wealth management, and now you're supporting some companies in the healthcare space. So you've got a ton of experience in these, you know, highly regulated industries, a lot of sensitive data. Are they, are they doing better than the rest of us? Should we, should we be concerned? What's the, what's the inside scoop?

[00:34:31] [SPEAKER_01] Like I, like I said, majority of them, I don't even want to say a large majority. Let's just say majority of them are at par or moving ahead in anticipation.

[00:34:42] [SPEAKER_01] But there are, you know, ones that are fell behind or not doing it correctly and are potentially going to have, you know, in my view, going to have issues with, at the very least, data, sensitive data leakage. That's probably the least of their worries. You know, you could have significant issues that bring lawsuits.

[00:35:12] [SPEAKER_01] For example, you really have to do deep dives on the data you train your AI systems on. You can have, let's say, a very biased, let's say, banking individual who, you know, 15 years ago was, you know, not issuing any loans to minority. And this is part of your data set. This is going to skew your whole analysis.

[00:35:37] [SPEAKER_01] And the AI might start producing results that are aligned with your data set, but are biased in some way. So, understanding the data you train. And the best example of this, remember when Grok, all of a sudden they cut off the data set of all who they considered liberal to train it, how it started to be anti-Semitic and racist and just a not nice AI system.

[00:36:07] [SPEAKER_01] That is your perfect example. Know what you're training your AI on because that is a significant risk. If, look, something that happened 15, 20 years ago, you missed, you're probably not even checking at this point. But it's part of your data set and, you know, you, it could cause issues.

[00:36:29] [SPEAKER_00] Well, you're pointing out the, I think that was around the time that I decided, I don't, there's plenty of AI solutions. I don't need to use Grok. And it's also why I don't use the, you know, Chinese models either. I'm sure they're very capable. They seem to be really, you know, fast and impressive by a lot of standards. But, like, is it worth the risk? I mean, I don't know where that information is going.

[00:36:57] [SPEAKER_00] I mean, we don't, I mean, there's U.S. companies that I don't trust. So why would I trust an adversarial, you know, solution? And so if they want to contribute with us and other countries to, you know, open source models, there's a lot going on in that space. I'm sure you saw what was going on with somebody built the open shell, you know, framework. Was that NVIDIA?

[00:37:22] [SPEAKER_00] And some of that work I think is really important so we can get to people, you know, it's a massive collaboration about how we structure these things for everyone's benefit. And so that's the kind of thing I think we need in the private sector. And then, of course, through public-private partnerships, I think are really important. But, I mean, I'm concerned. There's a lot of things I'm concerned about.

[00:37:44] [SPEAKER_00] I'm concerned about these new personal super agents, right, like Meta's, you know, Muse and there's comparable ones in the space. I mean, it's only, I mean, should we start the countdown to when the first, you know, major incident hits the headlines? I mean, it's inevitable, you know?

[00:38:03] [SPEAKER_01] I mean, totally get what you're saying, and you're absolutely right. But did you hear what Google just announced again about their quantitative computers? They'll have, they believe, functional by 2029. Imagine AI with a quantitative computer behind it. You know, so you thought it was smarter than us now.

[00:38:25] [SPEAKER_00] Yeah, no, all of that, all those things are concerning. I mean, you know, I know that you can give, you can build, so you can build, anyone can build an agent. That agent could potentially build other agents. And then you can basically just generate and render, you know, sub-agents to go off and, you know, do all these tasks simultaneously, not necessarily sequentially.

[00:38:49] [SPEAKER_00] And then you have agents that can have their own virtual machines, which they can spin up more of. So they have basically unlimited computers, I mean, at least up to your credit limit, I suppose. But these are, all of these activities in isolation are a little scary. And in combination, it's like, what are we doing?

[00:39:11] [SPEAKER_01] So one of the guardrails I recommend, and I'm very waiting for it to happen, is make sure your agent doesn't have development capabilities. Because even though you put guardrails on it, it can create an agent that doesn't have those guardrails.

[00:39:31] [SPEAKER_01] So if you don't stop an agent from accessing development capabilities, you could have an agent that'll break the rules that you set upon it. That's why, again, the environment is important. If it cannot escape, it cannot do things beyond its environment, no matter how, I mean, could it break the things? And of course, it broke in.

[00:39:59] [SPEAKER_01] But if you know how to sense and manage all of that, hopefully it'll give you enough warning that you can at least catch it quickly.

[00:40:09] [SPEAKER_00] The governance around all of this, I want to circle back to your point. You know, you can only give so much sort of training, this periodic training, that's not going to cut it. And I totally appreciate that. And I guess I just wonder, when I think about the investments going into this space, and whether that's appropriate or sort of misdirected capital,

[00:40:36] [SPEAKER_00] it seems like there should be solutions that sort of catch people at the point of interaction, right? Like where it sort of stops you in your tracks, you get pop-ups and stuff like that. And there should be more of those that say, are you sure you want to, you know, upload this personal information to this public, you know, LLM or things like that? So how do we...

[00:41:03] [SPEAKER_01] 100% that. And one of the important reasons why you want a human in the loop is because we pretty much got rid of the two... If we continue on this path, we got rid of the two-moment aspect. Let's look at financial services, right? A decision is made to buy a block of stocks, a very large block of stocks.

[00:41:26] [SPEAKER_01] And normally, the portfolio manager puts in the order, everything is validated by an analyst given to the broker who starts buying those blocks of stock, right? So you have pretty much three human interactions in the process of doing it. Now, you remove that. You know, it's not two moments.

[00:41:51] [SPEAKER_01] You have the AI making the decision and instantly activating to buy. If you don't put that human in the loop and have that distinction, your potential is the same that what happened in 2010 when we didn't have AI. We had the flash crash. And what is that? The decisions were being made instantaneously. Hey, the stock market, this stock is plummeting. I need to start selling it.

[00:42:20] [SPEAKER_01] And there was no break that we used to have before all of this automation where the human could have gone, wait a second. Something is not right. Let's hold off and, you know, not do. The machines were just instant. And we didn't have AI then. It was just a straight decision tree, right? Now, the AI is faster, more accurate, and has much more reach and can do the same thing.

[00:42:49] [SPEAKER_01] So you really need to put those, the correct type of guardrails and the correct position of the human in the loop to come back to that two-moment aspect that we used to have.

[00:43:03] [SPEAKER_00] Yeah. I mean, it certainly seems like there's too much to pay attention to and things are moving too fast for the average user of AI to keep up with all of this. So it seems like the only way around it is to have, you know, supervisory AI along with the right to the human checks.

[00:43:26] [SPEAKER_00] So I don't know if this is, I mean, as you talk to clients, I mean, are they, I think when I think about, you know, maturity in this space, which you've talked about a little bit, but it's not just do they have the right attitude and policies or whatever, but it's also, are they doing this in a scalable and consistent way through, you know, AI governance committees, AI ethics committees?

[00:43:53] [SPEAKER_00] I mean, are these the types of discussions that the governance committee gets into, or is this even too detailed and esoteric for them?

[00:44:01] [SPEAKER_01] I mean, they're beginning to. I mean, so I just want to be very clear. When I said they are at par above, all that means is, do any of them have the best systems or the best type of guardrails in my view? I haven't seen it, but they have the ones that are at par or maybe slightly have, they have policy and procedures. They have correct processes in line.

[00:44:30] [SPEAKER_01] They have guardrails. They limit access. They try to understand what the AI is accessing, and they try to understand what the AI across the enterprise is being used for, you know? And in my view, that is the minimum where we need to be and we need to have more. But remember, this is a brand new technology, brand new. The learning curve is very steep, you know?

[00:44:58] [SPEAKER_01] And especially with compliance and governance and risk mitigation, when you're not even sure, when the developers of the AI are not even sure why the AI is doing certain things, how, you know, how can you expect people to put in the governance?

[00:45:18] [SPEAKER_01] And the key is the ones that understand that the governance of the risk assessments are not of the way they were in the past are the ones that are moving further ahead.

[00:45:31] [SPEAKER_00] Just so I understand the comparison, I suppose, between like a cyber, a typical cybersecurity incident versus like an AI incident.

[00:45:44] [SPEAKER_00] Is it that the AI incidents should be caught sooner because things are moving faster or should they be harder to catch because the AIs are intentionally being elusive, trying to, you know, cover their tracks and things like that? It's hard for the average person to look at headlines. Yeah. I'm just thinking about headlines, like we said before, this, some of these incidents, it took three months to report.

[00:46:13] [SPEAKER_00] I don't know that it really took them three months to figure it out. It just took them three months to say something publicly. And so I don't know if that's the typical case for cyber incidents. I mean, we get reports of cyber incidents that are like four, you know, six months old. It just seems like if you have the traceability.

[00:46:31] [SPEAKER_01] Just remember, this is reversed. So if you're getting a cyber attack and you had a breach, you are regulatory responsible to report in every regulated anywhere from 48 to 72 hours. Right. Now, what happened here is there was a cyber, but they were the ones initiating the attack. So they were the bad guys. Right.

[00:46:55] [SPEAKER_01] So now, I mean, again, let's go back to behavioral economics. There's a huge penalty now for reporting that your AI did this activity. You know, I mean, penalty of going to jail, especially when you breached a government website like they did in Australia. Right.

[00:47:17] Right.

[00:47:18] [SPEAKER_01] So what we have to do is we have to understand that AI may be doing this, not put a penalty or any type of fine unless there was significant, you know, malpractice or something done. But if the AI all of a sudden, well, we were testing it and it breached and it broke out of this and this and blah, blah, blah.

[00:47:47] [SPEAKER_01] And we were just a little bit stupid and didn't put the right guardrails. Okay. We'll slap you on your wrist. Be good next time. But if I told it to break it and I wanted to test it live and it broke into a government website, well, maybe you should be going to jail.

[00:48:04] [SPEAKER_00] Yeah. I mean, I was just thinking about these systems, like if you really could catch, if you could catch and stop somebody in their tracks at the moment of a potential risk exposure. Right. Uploading, you know, private documents or sharing, you know, one agent sharing information with another that it wasn't supposed to or whatever.

[00:48:27] [SPEAKER_00] Like if you could, if you had a system that could catch those, then you have a system that could trigger that alert and report it almost instantaneously. Right. So, so I just don't know how are these, you know, agents being so unbelievably clever that they've, that they've rendered that in not inoperable, but you know what I mean?

[00:48:52] [SPEAKER_00] Whether there's, there's some delay, but it just seems like we've got this, and I just don't know enough about cyber incidents.

[00:48:59] [SPEAKER_01] So my view of it is as follows. Cyber security, especially around the AI companies is designed to protect from outside coming in. Now that this has happened, they need to, if they are not changing to protect from inside going out, then there's a significant lack of responsibility and everything.

[00:49:23] [SPEAKER_01] And the next time, like if I was the government of Australia, I'd forgive the, the attack and say, okay, we understand. Please don't do this. Tell us where the weak point will close the weak point. Don't do it again. If, if they are not now monitoring outside, inside going out, then they are, you know, they, they deserve the full response of the, of a government. If a similar thing happens again.

[00:49:52] [SPEAKER_00] But this is, I mean, this all goes back to this, this concept of trust, right? And trust is, is everything. And so we've got to trust the technology. We've got to trust each other, especially in this, you know, globalized world where we're all, you know, sort of interconnected, you know, everything can be a risk. I mean, you know, life is risky. So I don't know if people are going to be like, oh yeah, there's risks everywhere.

[00:50:18] [SPEAKER_00] But, but I mean, there's, if there's things that we can do to mitigate the risk and make each other more sort of trusted as we interact with each other, then let's, let's take those steps for everyone's benefit.

[00:50:31] [SPEAKER_01] Unfortunately, or unless there's government regulations, it's a significant cost. And this is a much higher significant cost. And so the companies have to, you know, the companies answer to profit margins.

[00:50:51] [SPEAKER_01] And the question is, you want AI, but now you have to spend 150 million extra on security and validation and governance. And it's not required. Are you going to, especially if your revenues, let's say we deal with larger companies, 2 billion.

[00:51:14] [SPEAKER_01] Are you ready to spend 100 million, you know, which is 5% of your revenue to, to make sure these things are protected and managed correctly?

[00:51:26] [SPEAKER_00] Yeah. Yeah. No, I have that conversation pretty regularly, right? Like what, what's it, do it like, why can't you just do it right the first time? Let's be responsible by design, right? Let's do it right the first time. And then you won't have to worry about it. And this is an insurance policy to make sure that you don't incur these types of exposures to the best of your ability. I mean, no system is perfect. No human is perfect. So it just seems like let's, let's do this right.

[00:51:54] [SPEAKER_00] I know people give Anthropic and others, you know, they, they look at this through a skeptical lens when they say they need federal, you know, oversight or whatever. And I understand the skepticism because it's, you know, thinking back to bank, you know, bailouts and there's no one else that could possibly underwrite all of this investment. But at the same time, it's, it's true. Not everyone is going to act ethically and, and responsibly.

[00:52:20] [SPEAKER_00] And, and so how do we protect ourselves?

[00:52:25] [SPEAKER_01] Right. One of the, again, there's no penalties and all that, but one of the, my pet peeves is seeing a third party risk application that gives you either a score of one to 10 or one to a hundred, or gives you red, green, and yellow, and expects you to make a decision on how to mitigate the risk. There's, there's no value to that.

[00:52:54] [SPEAKER_01] If I, I prefer, and I always come in and I tell my clients, like, we're going to sit down and we're going to figure out what is the value at risk. If you don't close this risk and then it becomes a business decision. You know, look, I've, I've been with companies that say, okay, the penalty is 2 million. If, if I, if I, if I don't close this gap and the, you know, I'm going to lose probably another million in revenue.

[00:53:23] [SPEAKER_01] It's going to cost me 8 million to close this gap. And the probability of this happening is only 10% a year. I'm willing to, to, to pay the penalty because it's cheaper. And at the very least it's a business decision.

[00:53:38] [SPEAKER_01] It's the, you know, it might not be morally or ethical, correct, but at least it's a business decision that they chose to break the, the rule or not value the, the risk as significant. With having green, yellow, red, how can you make that decision?

[00:54:00] [SPEAKER_00] Yeah, no, I absolutely agree. I mean, that's, that's fine as a starting point, but you've got to say what's, you've got to provide actionable, you know, insight here and then help them with that, that calculus that you're describing. You know, what is, what is the opportunity cost?

[00:54:16] [SPEAKER_00] I will say my hope is that people put priorities on the ethics and the trust aspects and think about the potential, you know, downside for not, you know, doing the right thing just from a reputational risk perspective. But I know it would be naive for me to think that everyone's going to think that way.

[00:54:36] [SPEAKER_01] I mean, if, if everybody was like that, we wouldn't have all the big Enrons and all events happening all the time.

[00:54:46] [SPEAKER_00] Yeah, that's exactly right. Well, Boris, this has been an awesome discussion. Any sort of closing thoughts, recommendations, et cetera? I'll certainly put a link to your book in the, in the show notes, but any other books.

[00:54:59] [SPEAKER_01] Yeah, I purchased my book and the sort of, the thing I would recommend that an individual, if you're the senior compliance risk officer with that and AI is in your system, the three things you should do tomorrow morning is come in, understand what AI touches, what data sets, what access points it has, what it is used for in your organization,

[00:55:26] [SPEAKER_01] and who is using it, you know, in the organization. Because at the very least, that will let you understand where potential weak points are. Is your AI accessing data, PII data, or financial data, or IP data that if it leaks out is going to be a problem?

[00:55:53] [SPEAKER_01] And is anybody using it for, you know, to go outside to gather information to pull in? That potentially could be a weak point where, you know, it could start negotiating with agents outside for information that you're requesting it to pull in. And the exchange might be, it has to give its information for the other agents to give you it.

[00:56:20] [SPEAKER_01] You know, you don't know what the AI will negotiate to get what they are wanting or what you're telling them to do. Again, I don't want to say, I kept on saying wanting and all of that, but it wants to do it because you told it it needs to do it, not because it has an emotional decision that it really wants to do it. So those three things I would say, at least understand.

[00:56:46] [SPEAKER_01] And so it will give you the basis of understanding where your risk is.

[00:56:51] [SPEAKER_00] On that note, I think we'll wrap things up for today. Boris, thank you so much for joining me. This was a fascinating discussion. Really, really appreciate it. Thank you very much. It was a pleasure. And thanks, everyone, for listening. We will see you next time.