Summary of Networks Of Networks: Complexity Class Principles Applied to Compound Ai Systems Design, by Jared Quincy Davis et al.
Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design
by Jared Quincy Davis, Boris Hanin, Lingjiao Chen, Peter Bailis, Ion Stoica, Matei Zaharia
First submitted to arxiv on: 23 Jul 2024
Categories
- Main: Artificial Intelligence (cs.AI)
- Secondary: None
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Compound AI Systems (CAS) comprising multiple language model inference calls are gaining popularity, pushing the reliability and quality frontiers of monolithic models. In this paper, we introduce Networks of Networks (NoNs), organized around generating a proposed answer and verifying its correctness, a fundamental concept in complexity theory that extends to Language Models (LMs). We propose a verifier-based judge NoN with K generators as an instantiation of “best-of-K” or “judge-based” compound AI systems. Our experiments on synthetic tasks like prime factorization and core benchmarks such as the MMLU demonstrate notable performance gains, particularly in domains where verification is easier than generation. For instance, a simple NoN improves accuracy from 3.7% to 36.6% in factoring products of two 3-digit primes. On MMLU, a verifier-based judge construction with only 3 generators boosts accuracy over individual GPT-4-Turbo calls by 2.8%. This paper highlights the importance of considering verification complexity and provides key takeaways for ML practitioners. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This research explores how to make computer programs smarter by combining many language model “calls” together. The idea is that some problems are easier to solve if you have multiple people or tools working together, rather than just one. The researchers created a new system called Networks of Networks (NoNs) that can generate and verify answers. They tested this system on various tasks and found that it works well when the problem is easy to verify but harder to generate an answer for. This could be useful in areas like math and logic, where you need to make sure your answer is correct. |
Keywords
» Artificial intelligence » Gpt » Inference » Language model
