DON26BZ01-NV023 TITLE: Risk-Aware Regenerative AI-based Multimodal Visual-Tactical (ISRT) (Observant-AI) – Monitor, Understand, Alert, and Assist

OUSW (R&E) CRITICAL TECHNOLOGY AREA(S): Applied Artificial Intelligence (AAI)

COMPONENT TECHNOLOGY PRIORITY AREA(S): Human-Machine Interfaces;Integrated Sensing and Cyber;Trusted AI and Autonomy

PROJECTED CMMC LEVEL REQUIREMENT: Level 2 (Self)

The technology within this topic is restricted under the International Traffic in Arms Regulation (ITAR), 22 CFR Parts 120-130, which controls the export and import of defense-related material and services, including export of sensitive technical data, or the Export Administration Regulation (EAR), 15 CFR Parts 730-774, which controls dual use items. Offerors must disclose any proposed use of foreign nationals (FNs), their country(ies) of origin, the type of visa or work permit possessed, and the statement of work (SOW) tasks intended for accomplishment by the FN(s) in accordance with the Announcement. Offerors are advised foreign nationals proposed to perform on this topic may be restricted due to the technical data under US Export Control Laws.

OBJECTIVE: Develop risk-aware artificial intelligence (AI)-based computing methods motivated by three naval challenge problems that enable insightful active cross-domain (Sea-Space-Air-Land-Cyber) situational awareness and AI-assisted course of action and countermeasures in real-time conditions, namely, "LIVE" machine self-teaching (i.e., Regenerative AI); contextual machine exploitation; contextual networking to gain insights from accessible all-source-intelligence (ASI) and multimodal sensors; and proactive AI-assisted targeteer and decision support to manned and unmanned assets. The Observant-AI is envisioned as a distributed system of mission-focused AI agents that self-organize and share insights via ad hoc networking. The agents autonomously form mission-oriented collaborative teams to process and fuse multidomain anomalous events and activities for real-time AI-generated visual-tactical understanding, monitoring, alerts, and related operational risks. It applies natural language explanations for human-AI interactions, course of action assistance, and reasoning about risky engagements. For example, submersible X is tracking you, change course to southwest, speed up…; Cargo-Ship Y is armed, Container Marking is…, Departing Port XYZ; 20 UAVs are shadowing, armed, turn around go south; Littoral Zone X, Torpedo-Mines, Bottom-Mines, Deep Fencing, Actively Guarded, Speed Boats, Risk-High Navigation, Need Minesweeper, Check with CENTCOM; etc.

DESCRIPTION: Problem scope and capability concerns. First, over the past three decades, advancements in AI and machine learning (ML) for applications in hybrid networked teaming of manned and unmanned systems and sensors have unlocked new possibilities across a range of naval operations for novel missions. On the other hand, the defensive and offensive effectiveness of these technologies against near-peer adversaries remains a significant challenge.

Second, current Naval ISRT operations follow rigorous protocols supported by wide-ranging wargaming scenarios to plan tactics, techniques, and procedures (TTPs) with contingencies as operations unfold. TTPs focus on various situational details, such as adversary strength, leadership temperament, past and present operational performance, logistics, and exploitation opportunities for friendly cross-domain actions and effects. These plans are vital to be followed. However, they are extremely vulnerable to human biases and omissions that undermine the assessment of evidence, statistical analysis, and the understanding of cause and effect.

Third, generative AI methods are being integrated into the operational planning process and can enrich the development of a range of ISRT strategies. However, it must start all over again if "Unknown-Unknown" events crash the ongoing TTPs. Also, generative AI needs high-quality training datasets; otherwise, it is prone to inaccuracies and biases.

This SBIR topic will develop Observant-AI agents as a class of regenerative AI that learn in real time, enables active visual and tactical monitoring of anomalous activities, and trigger I&W alerts in naval operations. The envisioned Observant-AI agents proactively enforce the fail-safe execution of approved ISRT operational plans. They exploit unexpected events in real-time by leveraging insights from all-source intelligence (ASI) and remote sensors (i.e., space assets). They generate and execute novel all-domain ISRT TTPs plans consistent with the approved plans to counter evolving adversarial intents and undesired events, LIVE. In other words, the Observant-AI agents enable fault-tolerant mission-focused reconfiguration by analyzing existing assets’ capabilities through novel tactical teaming arrangements from approved deployable capabilities (sensors, manned and unmanned weapon platforms, intelligence data sources, etc.). Observant-AI will automatically alert the chain of command at all levels with emerging or mission-altering observables that may interfere with operational objectives.

The goal of the effort is to perform a combination of offline and online predictive engagement modeling to plan for trusted AI-enabled TTPs that will strategically adjust plans in real time to adapt to emerging events and conditions. It will use Monte Carlo simulation to model the probability of various outcomes under countless AI-generated Red vs. Blue engagement (action-reaction) scenarios for offline TTP planning and mission success assessment. Regenerative AI will ensure Observant-AI can quickly adapt the blue’s creative ISRT strategies against near-peer adversaries (Red). Regenerative AI offers unique capabilities such as learning from sparse data and predicting complex interactions. It will achieve this objective by testing novel all-domain penetration strategies, including offensive cyber and information operations to find advantageous strategies, then running them against many emerging scenarios, identifying the vulnerability points and engagement risks, and modifying strategies to sustain their performance with acceptable risks.

Critical AI technology components and developments are as follows:

1. Contextual modeling: relational modeling, graph-based modeling, spatial modeling, logic-based modeling, uncertainty modeling, ontology-based modeling, hybrid context modeling.

2. Multidomain multimodal all-source intelligence data and signals: multi-level secure connectivity and access.

3. Data learning: decision tree classifier, multilayer perception classifier, collaborative filtering, frequent pattern mining, K-means, deep learning.

4. Data quality, data interoperability, data generation.

5. Data storage: signal-oriented database, graph-based database, associative database, text-oriented database.

6. Spatiotemporal synchronization methods for multimodal data across decentralized architectures.

7. Multimodal contextual signal processing and fusion.

8. Cross-domain contextual collaborative learning, inference, and recognition.

9. Contextual collaboration, adaptation, and teaming via ad-hoc networking.

10. Contextual reasoning, risk assessment, and risk reduction.

11. Contextual query, question-answering (Q&A), and natural language processing.

12. Contextual priority-based task management and balancing competing multifaceted ISRT operational objectives such as persistence, endurance, opportunistic collections, and targeting.

13. AI-risk escalation control methods that will not erode decisions across the integrated chain-of-command.

14. AI-assisted targeteer maneuvers and engagements.

Work produced in Phase II may become classified. Note: The prospective contractor(s) must be U.S. owned and operated with no foreign influence as defined by 32 U.S.C. § 2004.20 et seq., National Industrial Security Program Executive Agent and Operating Manual, unless acceptable mitigating procedures can and have been implemented and approved by the Defense Counterintelligence and Security Agency (DCSA) formerly Defense Security Service (DSS). The selected contractor must be able to acquire and maintain a secret level facility and Personnel Security Clearances. This will allow contractor personnel to perform on advanced phases of this project as set forth by DCSA and ONR in order to gain access to classified information pertaining to the national defense of the United States and its allies; this will be an inherent requirement. The selected company will be required to safeguard classified material during the advanced phases of this contract IAW the National Industrial Security Program Operating Manual (NISPOM), which can be found at Title 32, Part 2004.20 of the Code of Federal Regulations.

PHASE I: Determine the technical feasibility of designing and developing the Observant-AI technology described in the Description section. Draw key distinctions for the proposed design approach compared to the current state-of-the-art naval ISRT information exploitation systems. Ensure that the proposed design reveals a clear set of steps that naval ISRT platforms and joint services can capitalize on now for a transformative AI-based ISRT operation with dominant applications within the next five years. Motivate the design with three compelling challenge problems supported by relevant datasets. Consider challenge problems corresponding to cross-domain littoral operations and navigational risks countering anti-access/denied-access enforcement scenarios. (Note: The scenarios need to consider seaside terrain, weather, the maritime order-of-battle, the movement; engagement rules and doctrine; logistics and supply demands, etc. Hostile enforcements may include hypersonic missiles, ballistic missiles attacks from land or undersea, sea-skimming cruise missile attacks, anti-submarine track and trail, torpedo attacks, anti-space jamming, littoral mining, swarming weapons, and decoy and deception tactics.) Perform testing and demonstrations may use a mix of OSINT datasets, synthetic datasets from DON, MCML, AIS maritime traffic, commercial satellite imagery, or similar sources. Conduct end-to-end Observant-AI system performance assessment, including:

1. Cross-domain contextual collaborative learning, inference, and recognition.

2. Multi-agent cross-domain contextual collaborative teaming and adaptation via ad hoc networking.

3. Cause and effect sensitivity analysis on contextual understanding.

4. Confidence rate on AI-generated proactive TTP enhancements on engagement plans, options, and risk reduction associated with the ups and downs of encounters.

5. Confidence rate on AI-risk escalation controls enabling trust decisions across the integrated chain-of-command.

6. Efficiency gains in human responsiveness through timely decision-making, chain-of-actions, and resources spent.

Performance criteria must include sensitivity (true-positive rate), specificity (true-negative rate), precision (positive predictive value), miss rate (false negative rate), false discovery rate, and false omission rate.

Performance metrics (considering outcomes will depend on data quality):

1. Analytic Completeness: not just identifying and stopping hostile acts, but also how it occurred by synthesizing the entire chain of events what would have happened had it not been stopped < 95%

2. Uniqueness: Signature attributes definable and retrievable (who, what, why, where, when) < 95%

3. Validity: Supporting evidence < 95%

4. Consistency: Periodic signature updates of attributes from various sources that reinforce linkages < 95%

5. Accuracy: Overcoming noisy data < 95%

6. Accuracy metrics for ingesting and classifying multimodal data: structured data mining and interpretation - accuracy of 95% over 98% captured content; unstructured data mining and interpretation – accuracy of 90% over 95% captured content.

Deliverables include end-to-end initial prototype technology, T&E, demonstration, a plan for Phase II, and a final report.

PHASE II: Conduct proof-of-concept and prototype development incorporating the recommended candidate technology from Phase I. Demonstrate the operational effectiveness based on the following criteria: (a) prioritized sensor alerts, (b) prioritized threat escalation, (c) measured severity of events, and (d) measured analytic completeness. Test and demonstrate the improved capability based on the performance metrics detailed for Phase I with the following requirements: Analytic Completeness < 98%, Uniqueness < 98%, Validity < 98%, Consistency < 98%, and Accuracy < 98%. Provide the following deliverables: analytics, signal processing tools, models, prototypes, T&E and demonstration results, interface requirements, and final report. Final report will include a detailed design of the system and a plan for transition to the program of record in Phase III.

It is probable that the work under this effort will be classified under Phase II (see the Description section for details).

PHASE III DUAL USE APPLICATIONS: Advance these capabilities to TRL-7 and integrate the technology into the Maritime Tactical Command and Control POR, Marine Air-Ground Task Force Command and Control, or ISR processing platforms at the Marine Corps Information Operations Center. Once conceptually and technically validated, demonstrate dual-use applications of this technology in civilian law enforcement and security services.

REFERENCES:

1. Sezer, O.B.; Dogdu, E. and Ozbayoglu, A.M. "Context Aware Computing, Learning and Big Data in Internet of Things: A Survey." IEEE Internet of Things Journal, Volume: 5 (Issue: 1), Nov 2017, pp. 1-27. DOI:10.1109/JIOT.2017.2773600
2. Zhang, R.; McNeese, N.J; Freeman, G. and Musick, G. ""An ideal human" expectations of AI teammates in human-AI teaming." ACM on Human-Computer Interaction, 4(CSCW3), 2021, pp.1-25. https://guof.people.clemson.edu/papers/ai.pdf
3. Robinson, E.; Egel, D. and Bailey, G. "Machine Learning for Operational Decision-making in Competition and Conflict, A Demonstration Using the Conflict in Eastern Ukraine." RAND Corp, 2023. https://www.rand.org/pubs/research_reports/RRA815-1.html
4. Wilner, A.S. and Babb, C. "New Technologies and Deterrence: AI and Adversarial Behavior." Springer, Dec 2020. https://www.researchgate.net/publication/347338819_New_Technologies_and_Deterrence_Artificial_Intelligence_and_Adversarial_Behaviour#:~:text=Abstract,political%20decisions%2C%20and%20coercive%20action.
5. U.S. Air Force U.S. Space Force. "Department of the Air Force Role in Joint All Domain Operations (JADO), Air Force Doctrine Publication (AFDP) 3-99." Maxwell Air Force Base, AL. https://www.doctrine.af.mil/Doctrine-Publications/AFDP-3-99-DAF-Role-in-Jt-All-Domain-Ops-JADO/, 2020
6. Wojton, H.; Vickers, B.; Carter, K.; Sparrow, D.; Wilkins, L. and Fealing, C. "Characterizing Human Machine Teaming Metrics for Test and Evaluation." DATAWorks 2021, IDA Document NS D-21563, Alexandria, VA, Institute for Defense Analysis, 2021. https://testscience.org/wp-content/uploads/formidable/13/Vickers_Brian_Characterizing-Human-Machine-Teams_SlidesOnly.pdf

KEYWORDS: Risk-Aware, Regenerative, Artificial Intelligence, Machine Learning, Contextual, Multimodal, Cross-Domain, Visual-Tactical ISRT

TPOC 1
Allen Moshfegh
allen.moshfegh.civ@us.navy.mil

TPOC 2
Lodewijk Brand
lodewijk.w.brand.civ@us.navy.mil