Open Research Questions
Fundamental questions that AIDDA is working to answer. If you're interested in these questions, we'd love to hear from you.
Active Research Questions
Can these frameworks produce algorithms which are robust to a range of problem types?
Investigating the capabilities of AI-discovered algorithms across diverse problem domains.
About This Question
This research question explores whether current frameworks can produce algorithms that maintain performance across varying problem sizes, structures, and constraints without requiring retraining or significant modification.
For AI-discovered algorithms to be practically useful, must they demonstrate to work reliably across a wide range of inputs and scenarios?
Do the algorithms these frameworks produce overfit to training data?
Examining whether AI-discovered algorithms memorize training instances rather than learning generalizable principles.
About This Question
Overfitting is a well-known problem in machine learning. When an AI system discovers an algorithm, it might encode specific patterns from the training distribution rather than discovering the underlying algorithmic principle. This research investigates detection methods, mitigation strategies, and the fundamental tension between training on specific instances and discovering general algorithms.
If AI-discovered algorithms merely memorize training patterns, they will fail on novel inputs and lack the reliability required for production systems. Understanding and preventing overfitting is essential for trustworthy algorithm discovery.
How far are we from the scenario of 'algorithm mining' where entities with no domain knowledge effectively turn compute into algorithm search?
Exploring the democratization of algorithm discovery and the compute-knowledge tradeoff in AI-driven research.
About This Question
The vision of 'algorithm mining' represents a paradigm shift where computational resources can substitute for deep domain expertise in discovering novel algorithms. See Richard Suttons: The Bitter Lesson. This research question examines how close we are to this reality, what barriers remain, and what implications it has for research democratization, industry competition, and the future of computer science. It explores the compute-knowledge frontier and whether we're approaching a world where anyone with sufficient compute can discover state-of-the-art algorithms.
Understanding this trajectory is crucial for anticipating how AI will transform research and industry. It raises important questions about access, equity, and the changing nature of expertise in algorithmic innovation.
How should we think about novelty of algorithmic method vs. optimization of known methods?
Investigating the balance between discovering fundamentally new algorithmic approaches and optimizing existing ones.
About This Question
This research question addresses a fundamental tension in algorithm discovery: should we prioritize discovering entirely novel algorithmic paradigms, or focus on optimizing well-understood methods and what entity (if any) should be attributed with the novel idea that led to the new method? Novel approaches may unlock breakthroughs but carry higher risk and verification burden, while optimization of known methods offers more predictable improvements but may hit diminishing returns. This question explores evaluation frameworks, resource allocation strategies, and the criteria for judging progress in algorithmic innovation.
As AI systems become more capable of both optimization and discovery, we need frameworks to guide research priorities and evaluate contributions. This question is central to defining what constitutes meaningful progress in algorithmic innovation.
How applicable are these frameworks to real world problems, expanding beyond benchmark settings?
Assessing the practical utility of AI-discovered algorithms in production environments and real-world constraints.
About This Question
While AI-driven algorithm discovery has shown impressive results on standardized benchmarks, the transition to real-world applications presents unique challenges. Real problems often involve messy data, changing requirements, resource constraints, interpretable coding. This research question evaluates how well current frameworks handle these complexities, identifies gaps between benchmark performance and practical utility, and develops methodologies for testing algorithms in realistic conditions.
For AI-discovered algorithms to achieve widespread adoption, they must demonstrate value in real production environments, not just controlled benchmarks. Understanding the barriers to real-world deployment is essential for directing future research. Companies are springing up that aim to offer AI-discovered algorithms as commercial products, and we will be interacting with them.
Is it important to develop frameworks which work well when confined to cheaper opensource LLMs? If so, what are the best practices to make this possible?
Exploring the feasibility and best practices for algorithm discovery using open-source and cost-effective language models.
About This Question
As large language models become central to algorithm discovery frameworks, there is growing concern about reliance on expensive proprietary APIs. This research question investigates whether effective algorithm discovery is possible using smaller, open-source models, and what techniques can bridge the capability gap. It explores distillation methods, specialized fine-tuning approaches, multi-model ensembles, and architectural innovations that can make open-source alternatives viable for algorithm discovery tasks.
Democratizing algorithm discovery requires reducing dependence on expensive proprietary models. If open-source alternatives can be made effective, it would enable broader participation in AI-driven research and reduce barriers for academic and independent researchers.