It took me a long time to understand why David Deutsch thinks LLMs can't be creative: First, an LLM’s output, for a given prompt, was already determined during training. Second, the output of any formal system is bounded, while humans can take any formal system and expand its range of outputs. Here are explanations of these two ideas, as well an explanation of why this reasoning may be wrong. Errors are mine, not David’s. 1. LLM outputs are pre-determined Everything that LLMs do is implicit in the program and weights. The same prompt and seed will generate the same output every time. LLMs look creative because the weights reflect a large amount of training data and the input/output spaces are large. But every possible output was created during training and is already reflected in the weights. The only creativity is in the prompt that queries the LLM for a pre-determined output. In short, you can think of a neural net as a compressed database of outputs that you can look up, if you can come up with the right prompt. Or think of the LLM as a paint brush that is made useful with a creative human prompt. 2. Programs can never be as creative as a person You can’t make a formal system (e.g. a programming language) that captures all human creativity, because humans can always create a different or bigger formal system that can do things that the first one can’t do. For example, humans can create a new formal system with new definitions for all the words; invent new words for previously inexplicit ideas; put words in useful orders that don't follow the rules; create new forms of media like music, dance, drawing; or extend the formal system with undecidable propositions, as in Gödel's incompleteness theorem. Human creativity is limitless, while programs in a formal system are bounded and give the same result every time. 3. Nevertheless, there is a program that can simulate a human Despite my argument that a formal system can’t capture all human creativity, there is a program that can simulate a human, in principle. If you simulate a human, you have an AGI that can simulate human creativity. We know this program exists because you can simulate any quantum mechanical system to arbitrary accuracy with a normal computer. Although I said that you can’t express human creativity in a formal system, this program does. This conflict is unresolved. David Deutsch suggests that this AGI would be a program whose outputs can’t be specified. It must also be able to disobey instructions and create new knowledge. If you can create a program that meets these requirements and resolves this conflict, your name will be mentioned alongside Turing, Gödel and Deutsch—even if your program takes the lifetime of the universe to come up with one idea. Summary 1. LLM outputs are pre-determined. 2. Programs can never be as creative as a person. 3. There is a program that can be as creative as a human by simulating one, which contradicts the previous point. Credit All ideas are from @DavidDeutschOxf and @ToKTeacher. Errors are mine, criticisms welcome.

Great book and narrator. Probably the best Apple book. Thx @asherhunt for the tip.