Trust fall

So we put a single question to the OGs of IT – 13 people who have lived through every shift from mainframe to cloud and back. Here is what they said.

David Cannon

For me, the most important consideration is governance. By governance, I'm referring to three things. First is data governance. Who has access to what data? Put a query into your tool of choice, and look at the number of sources it goes to, then compare, depending on who's asking. If you're a senior person with access to more data, you're going to get more data informing the reports and decisions. If you're more junior, or you don't have access to the right data, you're going to have a much narrower set informing what that tool does. In those cases, you may find the tool making incorrect or inappropriate decisions because somebody didn't have the information they need. Second, the person creating the workflows and setting the parameters for the AI's decisions, do they have the authority to do the things they're asking the AI to do? Because the AI doesn't check that. If there's somebody getting the tool to do things they don't have the authority to do, you suddenly find the organization is out of compliance, out of control, and open to malicious players. Finally, governance in terms of access. Who has access to the tools, and at what level? We should have learned that lesson from the cloud. I hope we don't repeat the mistakes we made there with who we're giving access to AI.

Doug Tedder

We need to become very clear about the why. Many organizations I work with lack even basic policies about how we execute our practices, how we participate in security. We tend to do security by punishment, which is the wrong way of doing things. What leaders should be thinking about is how we use this technology from a positive perspective, rather than the negative perspective most of our technology governance sits around. The second consideration is the quality of our data. Studies in the early days of AI have shown human bias sneaking into the algorithms, based on the data patterns we feed in. How are we going to police that? Many organizations are far from prepared.

James Finister

First, you have to have existing guardrails that are robust. Because if you don't, agentic AI is going to break them. But a big area to look at is ethics and culture. We need to be honest about the fact that people aren't interfacing with a human being, but still interfacing with an AI model. We need to monitor how it behaves, whether it shows behaviours that are unacceptable, such as being discriminatory. But is it overly optimistic? Does it start telling people, yes, we will fix this incident in 30 seconds, when we all know it's going to take several hours? I'd look at the cultural and ethical aspects as the main guardrails missing at the moment.

Jeffrey Tefertiller

When we think about governance, risk, compliance, security, I want to throw in ethics. There are two things I encourage every leader to remember. The first is your organization inherits all the governance, risk, security, ethics, and issues of the AI technology you are implementing. If there are governance, risk, or compliance issues, you are inheriting those. So you must do your due diligence and not assume anything. If there's one thing we know from the last couple of years, it's that the big tech bosses really don't care about what we think of as governance, security, and risk. Look at the security breaches. Most came from vendors. And second, like all security, governance, and ethics issues, they are dependent on people. So make sure you have exhaustive user training so your people use AI the way you want them to.

John P. Custy

The first consideration is risk management. Do we really understand the risks of what fails, in the scope of what you're starting with? Compliance to governance, which assumes you're going to have the governance in place. Transparency, there has to be far more of it, with people understanding why decisions are made. Security, how do we prevent unauthorized access? Then, audits and monitoring, which have to be ongoing. But pay attention so we know what's going on before something runs off the rails. Because when the customers call to tell us it's off the rails, we've gone too far, we've lost the trust, and all the other negative things follow. The real risk isn't just whether a customer comes back if it doesn't work. It's the trust customers have in the organization, in the brand. The reason customers rebuy has to do with trust. If something goes wrong, how do we recover from that?

John Santaferraro

This whole area of AI governance is in its infancy. But there's one area we have matured, and that is data governance. At the core of every AI implementation is data. So one place to begin is to make sure your data governance is in order, feed that into the AI. From there, the frameworks being developed are fairly extensive. They include governance of the models. How do you determine which model is best for your business? And infuse that model with your code of conduct, your code of ethics, and make sure that model is tuned to do the workloads you do within your organization. Then security, making sure that both the AI and the data is secure. And maybe the most important thing is auditability. AI can no longer operate in a black box. You've got to know exactly what it's doing, and reproduce how it came to its conclusion and took the actions it took. The final one is responsibility, because the AI has to have the same responsibility you put in place for employees. It may even come out of the HR department. That responsibility is incredibly important. It's just beginning to emerge.

Phyllis Drucker

The risk and security implications are huge with agentic AI. Giving agentic AI the keys to the candy store, right? What can it do that it shouldn't be doing? We've got to build those guardrails into the systems. So we're looking at auditability. We need to know what they've done. Appropriate access control, so we know what it's allowed to touch. High-risk operations need a human involved in decision making. And we need a way to validate our models to make sure we're building them correctly. Then, on data security, we've got to protect certain data from those models. We can't be giving away personal information, but we can give away device information about a person. And finally, ethical. Ethical is always going to be the thing people blame stuff on. Transparency, accountability, and the ability to explain what agentic AI can do coherently to others.

Ryan Schmierer

There's three things we have to look at. The first is data privacy and access controls. There's real concern today about AI creating an opportunity for people to bypass the access controls and gain access to data they shouldn't be sharing. Your data is the core of your organization, and you need to find a way to control that. The second is ensuring third party agents are safe. Consider the risks when you've got agents that can perform actions within your organization. How do you prevent hackers and malware coming in through the agent stack? We've got to have guardrails around that. The third is preventing uncontrolled AI actions. You need to keep the human in the driver's seat, because we're not ready for self-driving enterprises yet.

Simone Jo Moore

Governance, risk, and security have always been around in technology. But to take it to the level of agentic AI, governance needs to evolve from compliance to conscience. It needs to become more about stewardship. We need explainability, reversibility, and accountability loops built in throughout, not bolted on. That's why ethical readiness is just as vital as technical readiness. Ethical AI is powered by human choices, not code. Without governance, AI becomes cognitive parasitism.

Stephen Mann

Most tools will come with some form of AI guardrails, but they're only a small part of what's needed. Something the industry is calling AI governance is a subset of IT governance, a subset of enterprise governance. It's important to understand how much changes with the introduction of AI, because the impact on governance is far larger than many think. Sometimes even the basics of understanding the risks of AI, you'll be surprised how many there are. Secondly, traditional IT governance is too slow relative to AI adoption. Not only do we need to change it to address the risks of AI, but also to keep up with the speed of AI's change. Another thing worth calling out is accountability. One thing I'm not seeing data on yet is whether we are more forgiving of humans who make mistakes at our work versus technology making mistakes. We're already seeing organizations find themselves in positions they don't want to be in because they haven't invested sufficiently in AI governance. They understand they need it, but I'm not convinced that ambition has turned into action yet.

Kenneth Gonzalez

All three are really important. From a governance perspective, what level of decision making are you currently working at? Do you have a tiered structure in place so you're not creating unnecessary bureaucracy, but you're empowering people with the visibility, the information, and the responsibility to make decisions at an appropriate level? What I see is that organizations don't do that. We control it at either department or organizational level where we implement policies. That only gives us sets of controls that might be part of a governance structure. We really need to be driving decision-making down. On risk, what methods are organizations using for risk identification? How and when do those get rolled up and made visible to the next layer? What management and mitigation strategies are used, and how are they documented and communicated? We are notoriously bad at dealing with risk. There are more opportunities to introduce risk than we have time, money, or attention to manage. And security is the same character. What is your approach to identify and manage threats, both internal and external? We might have perimeter defenses, firewalls, and complex logic to deal with attacks from the outside. But today, we're seeing attacks from the inside, or vulnerabilities and holes in the security fabric, introduced because of human actions, whether intentional or accidental. How are you determining the likelihood of the impact? And how are these documented, communicated, and escalated? It's a very complex area, and the more time we invest in taking it seriously, the better off we're going to be.

Shane Carlson

When we think about what governance looks like for AI, especially agentic, this has been largely overlooked by a lot of leaders, those in a rush to get AI products to market. Companies are investing hundreds of millions, if not billions, in building AI tools. Very few of them are looking at how to observe this LLM or agent in a way that it's doing what it was designed to do, and staying within the parameters they expect. They're also not looking closely at security for these agents. How do you know your agents or your customers aren't trying to game the system and get it to do things you didn't expect? Things like prompt injection attacks, manipulation, getting it to give access to information sitting in your CRM. These are real examples of what's happening today with LLMs, agents, and AI. A lot of companies need to think about how to observe their LLMs and agents so they're doing what they need, and how to protect the business against risk through this new doorway into the business.

Mark Smalley

I focus on bounded, or safe, autonomy. Before you roll out any agentic AI, create a short one page document that answers a few questions. What can this agent do? What must remain human? How will we know if it's working? Who owns the outcomes? Accountability is a big issue. How will we retain our cognitive sovereignty? How do you keep it, and not outsource it to AI?