Request for Founders

$ $ /$

The velocity and risks of AI

We are living through the most consequential technological transition in human history. And it is happening faster than almost anyone outside of a small group of people in Silicon Valley fully appreciates.

AI can be enormously positive. But the risks are real, and the world is not taking them seriously enough.

These areas concern us most:

Loss of human control

AI systems that pursue their own goals, and can't be turned off

Democratization of dangerous capabilities

Bioweapons, cyberattacks, and beyond

Massive social disruption

AI arms races, surveillance states, erosion of democracy

These areas concern us most:

Loss of human control

AI systems that pursue their own goals, and can't be turned off

Democratization of dangerous capabilities

Bioweapons, cyberattacks, and beyond

Massive social disruption

AI arms races, surveillance states, erosion of democracy

$ /$

Projects we want to fund

Technical AI Safety and Alignment Work

How can we build AI systems that reliably do what we intend, and how can we understand them well enough to know when they don't? We are looking to back new technical research organizations and companies advancing safety and alignment science.

1. Mechanistic interpretability: Understanding what is actually happening inside AI models

2. Scalable oversight: Methods for humans to supervise AI systems that are more capable than the humans evaluating them

3. AI control: Techniques to ensure powerful AI systems remain under meaningful human control even as their capabilities grow

4. Reinforcement learning and training for safety: Training regimes, reward signals, and curated datasets that reliably produce safe, prosocial AI behavior

5. Open-weight model safety: How do we make the world safe when anybody can access and modify open-weight models, potentially stripping them of any guardrails and safety features?

6. AI consciousness and moral status: If AI systems develop something like genuine experience or suffering, there will be profound implications for ethics and governance

Critical Cybersecurity

Attackers are already using AI to exploit security vulnerabilities. Meanwhile, new models like Claude Mythos have been deemed too dangerous to release publicly. The world’s critical systems are poorly protected and the window for action is closing quickly.

1. AI-native cyber defense: AI systems purpose-built for defense, trained on security-specific datasets, capable of recognizing and stopping novel attacks at machine speed.

2. Critical infrastructure hardening: Protecting critical software, power grids, financial networks, and communications infrastructure from AI-enabled attacks

3. AI agent (and multi-agent) security: Preventing individual agents from being manipulated into misusing their capabilities, and addressing the new threats that emerge when agents interact at scale

4. Hyper-secure AI compute: Military-grade physical and digital security for the facilities where frontier models are trained and stored

5. Open source intelligence and monitoring: Systematically tracking AI-relevant developments in the real world: datacenter buildouts, large training runs, AI misbehavior in the wild, and nation-state AI activity

Biosecurity and Pandemic Prevention

Advanced AI is lowering the knowledge barrier for engineering dangerous pathogens and bioweapons. The tools are growing more powerful and more accessible, our defenses remain thin, and the window for action is closing quickly.

1. Prevent misuse: Reducing the probability that dangerous biological capabilities are created, accessed, or misused through frontier model safeguards and evaluations, differential access to powerful biological models, and stronger governance of the supply chain in areas like DNA synthesis

2. Threat detection and surveillance: Enabling rapid detection, identification, attribution, and response to emerging or engineered pathogens through early warning systems, disease surveillance, and platforms for reconfigurable diagnostics

3. Societal defense and biohardening: Limiting the spread and impact on critical infrastructure and society after a threat emerges, through solutions that harden the built environment against biological threats (Far-UVC, air filtration) and platforms for rapid medical countermeasure design

4. Cross-cutting interventions: Spanning domains to enable the field through market shaping and demand generation, policy and advocacy for increased government funding and stockpiling, and research into neglected threat classes

Governance, Policy, Standards, and International Coordination

We need the legal, diplomatic, and organizational infrastructure that makes AI development accountable. We must create the standards, watchdogs, and international frameworks that turn safety from a voluntary commitment into a reliable and durable system.

1. Independent AI evaluators, auditors and accountability mechanisms: Rigorous third-party testing of frontier AI models before and after deployment. Independent organizations that test for dangerous capabilities and track whether frontier labs are following their safety commitments.

2. Compute governance: Policy and technical tools to ensure the most powerful AI hardware is prevented from reaching bad actors or being used for highly destructive purposes.

3. US-China AI dialogues and coordination: Track 1.5 and Track 2 diplomacy, academic exchanges, and joint research on shared risks.

4. International AI safety frameworks: The verification mechanisms, political coalitions, and institutional infrastructure needed for meaningful international agreements on AI risk.

5. Safety and security standards: Certifiable standards for AI systems that enterprises, insurers, and regulators can rely on. Think SOC 2 for AI safety.

6. AI insurance and risk markets: Insurance products and financial mechanisms that price AI risk accurately and create market incentives for safer development.

Sensemaking and Societal Resilience

A broader category for civilizational hardening – making societies, institutions, and infrastructure resilient to the full range of risks that advanced AI creates or amplifies.

1. Democratic resilience and anti-authoritarian technology: AI dramatically lowers the cost of surveillance, censorship, and social control. We need censorship circumvention tools, privacy-preserving infrastructure, and AI systems designed with democratic principles at their core.

2. Information integrity: Content provenance infrastructure, AI-enabled disinformation defense, and tools to detect and attribute AI-generated influence operations.

3. Fraud prevention and deepfake detection: Technical tools to verify the authenticity of media and prevent AI-enabled fraud, impersonation, and manipulation at scale.

4. Economic disruption and labor transition: Mass AI-driven unemployment is itself a pathway to catastrophic instability – one that could destabilize democracies and fuel authoritarian backlash.

Field-Building and Infrastructure

We need organizations dedicated to finding, developing, and routing exceptional talent toward the most important work – and to building the communications infrastructure the field currently lacks.

1. Talent pipelines into AI safety, critical cybersecurity, and biosecurity: Programs that identify exceptional people in adjacent fields and help them transition into AI safety and resilience work. We would gladly fund another Halcyon!

2. Recruiting infrastructure: A professional recruiting firm focused specifically on placing talent into AI safety organizations. This does not exist and the field badly needs it.

3. Public communications: Orgs and media that help the public understand what is happening at the AI frontier. Perhaps also a professional services firm serving as a dedicated communications partner to the organizations already doing important work.

Technical AI Safety and Alignment Work

How can we build AI systems that reliably do what we intend, and how can we understand them well enough to know when they don't? We are looking to back new technical research organizations and companies advancing safety and alignment science.

1. Mechanistic interpretability: Understanding what is actually happening inside AI models

2. Scalable oversight: Methods for humans to supervise AI systems that are more capable than the humans evaluating them

3. AI control: Techniques to ensure powerful AI systems remain under meaningful human control even as their capabilities grow

4. Reinforcement learning and training for safety: Training regimes, reward signals, and curated datasets that reliably produce safe, prosocial AI behavior

5. Open-weight model safety: How do we make the world safe when anybody can access and modify open-weight models, potentially stripping them of any guardrails and safety features?

6. AI consciousness and moral status: If AI systems develop something like genuine experience or suffering, there will be profound implications for ethics and governance

Governance, Policy, Standards, and International Coordination

We need the legal, diplomatic, and organizational infrastructure that makes AI development accountable. We must create the standards, watchdogs, and international frameworks that turn safety from a voluntary commitment into a reliable and durable system.

1. Independent AI evaluators, auditors and accountability mechanisms: Rigorous third-party testing of frontier AI models before and after deployment. Independent organizations that test for dangerous capabilities and track whether frontier labs are following their safety commitments.

2. Compute governance: Policy and technical tools to ensure the most powerful AI hardware is prevented from reaching bad actors or being used for highly destructive purposes.

3. US-China AI dialogues and coordination: Track 1.5 and Track 2 diplomacy, academic exchanges, and joint research on shared risks.

4. International AI safety frameworks: The verification mechanisms, political coalitions, and institutional infrastructure needed for meaningful international agreements on AI risk.

5. Safety and security standards: Certifiable standards for AI systems that enterprises, insurers, and regulators can rely on. Think SOC 2 for AI safety.

6. AI insurance and risk markets: Insurance products and financial mechanisms that price AI risk accurately and create market incentives for safer development.

Sensemaking and Societal Resilience

A broader category for civilizational hardening – making societies, institutions, and infrastructure resilient to the full range of risks that advanced AI creates or amplifies.

1. Democratic resilience and anti-authoritarian technology: AI dramatically lowers the cost of surveillance, censorship, and social control. We need censorship circumvention tools, privacy-preserving infrastructure, and AI systems designed with democratic principles at their core.

2. Information integrity: Content provenance infrastructure, AI-enabled disinformation defense, and tools to detect and attribute AI-generated influence operations.

3. Fraud prevention and deepfake detection: Technical tools to verify the authenticity of media and prevent AI-enabled fraud, impersonation, and manipulation at scale.

4. Economic disruption and labor transition: Mass AI-driven unemployment is itself a pathway to catastrophic instability – one that could destabilize democracies and fuel authoritarian backlash.

Field-Building and Infrastructure

We need organizations dedicated to finding, developing, and routing exceptional talent toward the most important work – and to building the communications infrastructure the field currently lacks.

1. Talent pipelines into AI safety, critical cybersecurity, and biosecurity: Programs that identify exceptional people in adjacent fields and help them transition into AI safety and resilience work. We would gladly fund another Halcyon!

2. Recruiting infrastructure: A professional recruiting firm focused specifically on placing talent into AI safety organizations. This does not exist and the field badly needs it.

3. Public communications: Orgs and media that help the public understand what is happening at the AI frontier. Perhaps also a professional services firm serving as a dedicated communications partner to the organizations already doing important work.

Biosecurity and Pandemic Prevention

Advanced AI is reducing the knowledge barrier for engineering dangerous pathogens and bioweapons. At a high level, we need solutions across four key domains: Prevent, Detect, Defend, and Cross-cutting.

1. Prevent: Reduce the probability that dangerous biological capabilities are created, accessed, or misused. Frontier model safeguards and evaluations, systems for differential access to powerful biological models, and improved governance and regulation of the supply chain in areas like DNA synthesis.

2. Detect: Enable rapid detection, identification, attribution, and response to emerging or engineered pathogens. Early warning systems, wastewater monitoring and platforms for rapid, reconfigurable diagnostics.

3. Defend: Limit the spread and impact on critical infrastructure and society after the emergence of a threat. Solutions to harden the built environment against biological threats (Far-UCV, glycol vapors, air filtration) and platforms for rapid medical countermeasure design.

4. Cross-cutting: Interventions that span domains and are best understood as field-level rather than domain-specific. Market shaping and demand generation initiatives, policy and advocacy for increased government funding and stockpiling, and research into neglected threat classes, such as integrating viruses and prions.

Critical Cybersecurity

Attackers are already using AI to exploit security vulnerabilities. Meanwhile, new models like Claude Mythos have been deemed too dangerous to release publicly. The world’s critical systems are poorly protected and the window for action is closing quickly.

1. AI-native cyber defense: AI systems purpose-built for defense, trained on security-specific datasets, capable of recognizing and stopping novel attacks at machine speed.

2. Critical infrastructure hardening: Protecting critical software, power grids, financial networks, and communications infrastructure from AI-enabled attacks

3. AI agent (and multi-agent) security: Preventing individual agents from being manipulated into misusing their capabilities, and addressing the new threats that emerge when agents interact at scale

4. Hyper-secure AI compute: Military-grade physical and digital security for the facilities where frontier models are trained and stored

5. Open source intelligence and monitoring: Systematically tracking AI-relevant developments in the real world: datacenter buildouts, large training runs, AI misbehavior in the wild, and nation-state AI activity

Who we're looking for

We want to back exceptional people ready to start and scale ultra-high-impact institutions.

Repeat founders and senior operators

Technical and research leaders

Policy, defense and intelligence veterans

If you’re ready to launch your next venture, let’s talk.

Get in touch: hello@halcyonfutures.org

$ $

Contact Us

/$ /$ /$