Human approval may not be enough to ensure manual control over AI-enabled decisions, witnesses told Congress Wednesday, as artificial intelligence takes on a greater role in military target identification and recommendations, and in turn, compresses time available to intervene.
The concern, critics say, is not limited to weapons that can select and engage targets on their own.
At the hearing, titled “Artificial Intelligence in the Military Domain: Implications for Human Rights,” witnesses told members of the Tom Lantos Human Rights Commission that AI can influence human decisions leading to a strike while leaving the individual formally responsible for the final call.
“The approval button alone does not demonstrate the control,” Anna Mysyshyn, a Ukrainian AI-governance researcher, told lawmakers.
The issue comes as the Pentagon pushes to use AI to accelerate military decision-making. The Pentagon’s 2023 AI adoption strategy identified “fast, precise and resilient kill chains” as a desired outcome, while its 2026 AI strategy calls for becoming an “AI-first” warfighting force by redesigning military workflows and operational concepts around current AI capabilities.
Mysyshyn questioned whether a person supervising multiple AI-enabled systems can recognize an incorrect recommendation and intervene fast enough before harm occurs. She cited Ukraine as an example of AI compressing decisions from hours to minutes.
In many scenarios, a human making the final decision does not necessarily have enough information, time or the ability to challenge the system’s recommendation, she said.
The Defense Department’s 2023 directive on autonomy in weapon systems requires autonomous and semi-autonomous weapons to allow commanders and operators to exercise appropriate levels of human judgment over the use of force.
Wednesday’s testimony focused on a broader question: whether that judgment remains meaningful if AI is influencing targeting faster than a human can authenticate.
Amanda Klasing, national director of government relations and advocacy at Amnesty International USA, warned that AI can compress the military “kill chain” in ways that make it difficult to determine whether meaningful human discernment actually occurred before a strike.
Joseph Chapa, a former military officer and scholar of military ethics, argued that the debate should not focus solely on how autonomous a system is. He urged policymakers to consider whether an AI system relies on statistical inference, what happens when it fails in a particular operational environment and whether its decisions can be audited afterward.
Statistical AI systems can fail differently from traditional software, Chapa said, including producing substantially different outputs from relatively small changes in inputs. More complex systems can also make it difficult to reconstruct how a particular recommendation was produced after the fact.
He called for greater AI education across the military, including specialized training for personnel working with targeting and intelligence systems.
Witnesses on Wednesday also pointed to a larger accountability problem: determining what happens when an AI-assisted decision results in civilian harm.
The Defense Department established its Civilian Harm Mitigation and Response Action Plan in 2022, including a Civilian Protection Center of Excellence intended to serve as a hub for analysis and training. Federal law requires the Pentagon to operate the center.
But a Defense Department inspector general review released in May assessed the department’s implementation of the civilian-harm framework and found problems including the loss or reassignment of personnel, the end of funding for a civilian-harm data-management platform and the failure of a senior steering committee to continue meeting.
That matters in the case of AI because, as Klasing noted, the military currently has limited ability to determine whether AI may contribute to improper targeting of civilians.
On the first day of the U.S.-Iran war, a Tomahawk cruise missile struck Shajareh Tayyebeh elementary school in Minab, southern Iran. At least 168 people were killed — more than 100 of them under the age of 12, according to UN and Iranian officials.
The school building sat fewer than 100 yards from a long-time Islamic Revolutionary Guard Corps naval installation and was previously located within the IRGC compound perimeter until a wall appeared between 2013 and 2016, according to an analysis of satellite imagery by Amnesty International.
By the time the U.S. and Israel launched their first strikes on Feb. 28, the school had been established several years prior. It was active on social media and had its own website, a Reuters investigation found.
More than 120 House Democrats sent a letter to the Pentagon on March 12, demanding clarity on whether artificial intelligence, “including the use of the Maven Smart System, [was] used to identify the Shajareh Tayyebeh school as a target.”
U.S. military investigators in the wake of the incident stated that the strike was likely carried out by U.S. forces. The Pentagon, however, has still not released its investigative findings.
Klasing proposed adding information about AI involvement to the Pentagon’s annual civilian-casualty reports required under Section 1057 of the National Defense Authorization Act. The existing reporting requirement calls for information about civilian casualty mitigation efforts and investigations, but does not specifically require the department to identify whether AI contributed to an incident.
Steve Feldstein, a senior fellow at the Carnegie Endowment for International Peace, said the Pentagon should collect data from military operations and conduct after-action reviews to determine whether AI-assisted systems accurately identified intended targets and whether errors contributed to civilian casualties.
“A bigger question is to what extent is the Pentagon collecting data and verifying the target accuracy of all these 1000s of strikes?” Feldstein said.
Feldstein also cautioned against treating AI itself as the source of every potential problem. AI can be used to help verify intelligence, detect changes in imagery and prioritize defensive responses to threats such as drones and ballistic missiles.
But if a human-influenced system is built around assumptions that certain demographic traits make someone a lawful target, Mysyshyn said, AI would not correct that underlying premise.
Feldstein similarly pointed to the rules humans establish before an AI system is used. If commanders determine that a particular level of civilian harm is acceptable in pursuit of a military objective, he said, the machine is operating within a human-defined framework.
The challenge is particularly acute when systems operate faster than people can meaningfully assess recommendations.
The commission opened Wednesday’s hearing with an example from Iraq. In 2004, Army squad leader Nick Ellison encountered two young men firing a machine gun from a car during a census patrol in Kirkuk.
Ellison made eye contact with one of the men, saw panic and ordered his soldiers to hold fire. The men turned out to be brothers of a newly married groom, and the gunfire was celebratory.
Mysyshyn said testing should examine bias, reliability, civilian protection, realistic time pressure, whether operators can interrupt systems and whether decisions can be traced throughout an AI system’s lifecycle.
Chapa said some AI systems may be difficult to audit after the fact, while Deborah Brown of Human Rights Watch said information about military AI use has often emerged through investigative reporting and whistleblowers rather than government disclosures.
The growing challenge, according to witnesses, is no longer making a machine pull a trigger, but whether the human who approves the trigger pull has enough time, information and authority to make a meaningful choice before disaster.
Military Times reporter Katie Livingstone contributed to this report.
Natalie Oliverio is an independent journalist and Navy veteran who covers military and veterans’ issues. Drawing on her experience as a talent executive and mentor, she writes about the policies, people and programs shaping the military community.
Read the full article here