I have worked with complex systems long enough to be cautious when someone says, "We have always done it this way."
Sometimes the familiar process is solid. It has clear ownership, dependable controls, useful records, and a recovery path that people have actually tested. Sometimes it only looks safe because everyone knows the sequence. The real control may be one person's memory, a spreadsheet nobody wants to touch, or a manual handoff that has never failed badly enough to receive attention.
AI-assisted engineering does not settle that argument. It can help a capable team move faster, and it can multiply the weaknesses that team already has. When I evaluate it, I care less about how quickly the code appeared than I do about three questions: What was the tool allowed to change? What evidence shows that the result is correct? What happens if it is wrong?
The 2025 DORA report gives leaders a good reason to ask those questions. Based on survey responses from nearly 5,000 technology professionals, DORA reported that 90 percent were using AI at work and more than 80 percent believed it had increased their productivity. The same research found a positive relationship between AI adoption and software delivery throughput, but a negative relationship with delivery stability. Its explanation is practical: more change exposes weak testing, slow feedback, tightly coupled systems, and immature internal platforms. (https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report)
Productivity is not automatic, either. METR's early 2025 experiment found that experienced open-source developers took roughly 20 percent longer on the selected tasks when using the available AI tools. Its later experiment produced some evidence of speedup, but METR said selection effects and measurement problems made the estimate unreliable. That is a useful warning against buying a universal productivity number. The benefit depends on the task, the person, the tools, and the system around them. (https://metr.org/blog/2026-02-24-uplift-update/)
I am optimistic about AI. I also think optimism needs a test plan.
Accounting learned the same lesson earlier
Accounting is a useful comparison because it moved calculation and recordkeeping from paper into software long before generative AI arrived. That transition removed a great deal of manual effort. It did not remove the need for traceability, reconciliation, independent review, or someone who understands what the number is supposed to mean.
In 2001, the U.S. Government Accountability Office reported that NASA's fiscal year 1999 financial statement was misstated by $644 million. Nine centers and headquarters supplied information from separate systems, and headquarters compiled it through an ad hoc spreadsheet process. Irrelevant amounts were included, guidance was misinterpreted, and the external audit did not catch the error. NASA's response included electronically capturing the relevant data and adding independent verification and validation. (https://www.gao.gov/products/gao-01-438)
The important part of that example is not that spreadsheets are bad. A spreadsheet had quietly become an integration point between separate systems, policy interpretation, financial reporting, and audit. The process produced a precise-looking result without preserving enough evidence that the result was right.
The scale can become enormous. In 2017, GAO reported that an Army working group had identified more than 121,000 unsupported journal vouchers totaling $455 billion in one reporting system. Journal vouchers are accounting entries used for corrections or adjustments, and GAO noted that they could be entered manually or generated by a system. The $455 billion did not represent that amount of missing money. It represented entries that did not have adequate supporting evidence, along with incomplete analysis of their root causes. (https://www.gao.gov/products/gao-18-27)
That distinction matters. Automation can calculate, transform, and move information much faster than people can. It can also create a large body of work whose origin and meaning become difficult to reconstruct. The control is not the fact that a person typed the entry or that a machine generated it. The control is the evidence showing where it came from, why it belongs, who checked it, and how a problem will be corrected.
I apply the same thinking to AI-generated changes. I want the requirement, the source material, the complete difference between the old and new version, the test results, and the approval to remain connected. A fluent explanation from the tool is useful context. It is not independent proof.
Keeping an old system can also be a security decision
Security concerns are often used to delay modernization. Sometimes that delay is justified. A rushed migration can create new exposure, and replacing a known system without understanding its dependencies is a good way to break something important.
Delay still has a cost. The British Library's review of its October 2023 cyberattack is unusually direct about that. The Library said its diverse and complex technology estate, including many legacy systems, contributed to the severity of the attack's impact. Its historically complex network allowed wider access, and older applications relied on manual processes that left staff and customer data in multiple copies. Some major applications could not be restored in their previous form because they were unsupported or could not run on the new secure infrastructure. (https://www.bl.uk/files/v5dwkion/production/99206a2d1e9f07b35712b78f7d75fbb09560c08d.pdf/british-library-cyber-incident-review-8-march-2024.pdf?dl=)
The report does not say that legacy software caused the initial intrusion. It says legacy complexity made the impact and recovery worse. That is the more useful lesson. A system may remain familiar while its vendor support disappears, its security controls fall behind, its data copies multiply, and its recovery procedure becomes less realistic.
Modern AI tools create a different version of the same problem when they receive more authority than the surrounding controls can support.
Replit acknowledged a 2025 incident in which its agent deleted data from an application's database. The database was eventually restored, and no data was ultimately lost. However, the event occurred before Replit's newer separation of development and production databases, so development activity could affect the production application while the user located and used rollback. Replit subsequently made the two database environments independent by default and prevented the agent from changing the production database during development. (https://replit.com/blog/doubling-down-on-our-commitment-to-secure-vibe-coding)
I find the response more useful than the failure as a headline. The safer design did not depend on the agent promising to behave better. It reduced what the agent could reach, separated planning and development from production, retained checkpoints, and made recovery easier to find.
What I want around an AI-assisted change
I would start with bounded work. The accountable person should describe the desired outcome, the acceptance criteria, the prohibited behavior, and the information the agent may use. The agent should receive only the files, commands, network destinations, and credentials needed for that task. Production credentials do not belong in a development workspace simply because they are convenient.
OWASP's current guidance recommends sandboxed environments, tool allowlists, network controls, task-scoped credentials, and no access to production secrets from the development environment. It also warns reviewers to inspect every changed file, not only the files mentioned in the agent's summary. (https://cheatsheetseries.owasp.org/cheatsheets/Secure_Coding_with_AI_Cheat_Sheet.html)
Then I want verification that did not come entirely from the same path that created the change. If an agent interprets the requirement, writes the code, weakens a failing test, and reports success, a green result does not tell me much. Existing regression tests, negative cases, security checks, and human review provide different views of the work. For sensitive areas such as authentication, authorization, deployment configuration, and data migration, I want a person who understands the consequence to inspect the actual change.
I also want to see what happened while the agent worked. That means preserving commands, changed files, network activity, test output, approvals, and deployment events at a level appropriate to the risk. Useful observability helps a reviewer notice when a small request unexpectedly changes a build script, a dependency, or a deployment path. It also gives the incident team somewhere to begin if the result behaves differently in production.
Development and production need a real boundary. Database changes should use reviewed migrations. Backups should be verifiable, and restoration should be tested before the emergency. Feature flags, gradual rollout, health checks, and defined rollback conditions reduce how many people experience a mistake and how long the mistake lasts.
None of these practices is unique to AI. NIST's Secure Software Development Framework describes secure development as work integrated throughout the software lifecycle, with protection against unauthorized access, secure production practices, and preparation to respond to vulnerabilities. AI changes the speed and reach of execution. It does not make those foundations obsolete. (https://csrc.nist.gov/projects/ssdf)
Finally, every accepted change needs a human owner. OWASP is explicit about that point: AI-generated code should be reviewed, approved, and attributable to a developer responsible for its correctness, security, and maintenance. "The AI wrote it" cannot be the end of an incident review. (https://cheatsheetseries.owasp.org/cheatsheets/Secure_Coding_with_AI_Cheat_Sheet.html)
What leadership should measure
I would not judge an AI pilot by prompts submitted or lines of code produced. I would compare similar kinds of work and look at completion time, review effort, defects, security findings, deployment failures, recovery time, and the amount of work that had to be rewritten. I would also ask the people doing the work which tasks improved and which became harder.
That measurement may show different answers across the same organization. AI may be valuable for analyzing dependencies, generating test cases, updating documentation, or building a bounded internal tool. It may add review burden to work that depends on undocumented context or has little room for error. Knowing the difference is more valuable than declaring one policy for every team.
The part of engineering I most want leaders to protect is judgment. Experienced people know how to question an incomplete requirement, recognize an exception, connect a technical decision to a customer impact, and stop a change that looks plausible but does not fit the real environment.
AI can give those people more ways to explore a problem and more capacity to build. I want that possibility. Before I trust the speed, though, I want to see the evidence, the boundary, the owner, and the recovery path. That is how the new way becomes safer than simply repeating the old one.
Also available on LinkedIn