‘This preservation of favourable variations and the rejection of injurious variations, I call Natural Selection.’
Darwin (1859)
Abstract
The following is an essay on the origins of intelligent systems. It should be taken as the beginnings of a formal, causal theory for intelligence as observed in the most capable systems we interact with today - other people, corporations, etc. Although we are limited in how much detail we can provide, we believe that an accurate, causal understanding of intelligence in general unlocks the ability to simulate the intelligent systems we find in the real world. We aim towards simulations that possess a capacity to interact with each other in a natural way - in accordance with the goals and individual plans possessed by the live players they are simulating. This is to say that the butterfly effect is free to play itself out, without the fixing of scope of what interactions can or cannot happen nor an overreliance on heuristics. We are then given the freedom to explore the possible futures, and to plan around events we had not previously considered.
We believe that this accurate simulation of intelligent systems has been and will remain intractable if we continue to focus on the implementations themselves, rather than the selection mechanisms and problem environments that produced them. We conclude by arguing that these intelligent systems converge when faced with the same problems, and that we therefore have sufficient reason to believe that AI counterparts of real world actors (and thus simulations) are well within reach.
Introduction
We argue that the human brain, models and institutional intelligences (large corporations, governments, etc.) share a number of characteristic similarities. These similarities warrant an explanation which tends to look Darwinian in nature: each has their origins in an autonomous optimisation process, and under this view, we find explanations for the aforementioned characteristic similarities. By autonomous optimisation we mean a type of process, of which natural selection is the paradigm instance: optimisation of a lineage through persistence conditions, in which members of such a lineage (organisms, parameters, org structure) persist or fail against said conditions - be it a loss function, reproduction, or capital. It should be noted that we take phenomenal experience as of little relevance and beyond scope.
Similarity (1) - Shared Illegibility
The first similarity of interest is that each, models, institutions, brains etc, seem fundamentally really hard to investigate, and that each has failure modes that seem to only be diagnosable after the fact (mental disorders and brain composition[1][2], model hallucination and confabulation[3], supply chain failure[4] - etc). The former, we argue, is demonstrated by their treatment as black boxes in the disciplines that study them, and the slow progress of the adjacent disciplines that try to treat them differently. Under the autonomous optimisation interpretation of these objects, this basal illegibility if you will seems to be expected.
Each implement a number of different behaviours, whose respective mechanisms have been constructed by a very long search through implementation space - the claim being that such a search will naturally converge on illegible implementations:
Let I(F) be the set of implementations realising some behaviour F, and call an implementation legible iff there exists a model of it, much smaller than its full specification, that predicts its behaviour under intervention and is comprehensible to some individual. Three observations then seem most immediate. First, legibility is not total - Rule 110 is both deterministic yet its outputs are incompressible - there is no shortcut in computation for its outputs - so the assumption that a describable, deterministic function must have a describable mechanism does not hold. Further to this, counting arguments seem to show that for almost all functions the vast majority of their implementations are illegible - there is no upper bound on how many moving parts I can use to implement addition, for example. Second, engineered implementations seem highly restricted in the behaviours they can implement - by virtue of being restricted to the legible space of implementations for any F - so engineered artefacts are drawn, by construction, from a small region of I(F) - the legible region. Third, and we offer this as conjecture - legibility appears to be anti-correlated with the number of functions per unit of substrate (i.e efficiency), since a part serving many purposes has no clean description in any one function’s terms - which is to say that legibility is not generally preserved under intersection of the I(Fi).
One might worry that such a position is betting against scientific progress. This is a reasonable point, because there is a consistent trend of views of this type (that we have reached the limit if you will) being proved wrong. I would agree if the problem did not seem information theoretic in nature. I refer back to rule 110 - its outcomes are incompressible, there is no method for predicting its outputs that is any more efficient than just running the computation yourself. What’s worse is that rule 110 is deterministic, and it is also quite simple. This position should therefore be read as taking the bet that a compressed, interventional understanding of the brain, AI models and institutions is likely to be out of reach for the same reason that one for Rule 110 appears out of reach.
To strengthen the point, we also have reason to believe that there are cognitive functions (possessed by the intelligent systems of which we are concerned) that seem closer to Wittgensteinian family relations rather than generalisable scientific phenomena. And that we can therefore cast doubt on the idea that there is one explanation for ‘how the brain works’ that permits a general theory of intelligence. Consider the condition of Aphantasia, where the subject is unable to visualise objects in their mind. Taking it on assumption that those with aphantasia can validly conceptualise something like a triangle, it seems unlikely that the mechanism responsible for the Aphantasia-based conceptualisation is the same as our non aphantasia-based mechanism - when are we both conceptualising triangles. Because both differ in the phenomenological experience that they produce - their mechanisms simply must be different in some way. It is in this sense that ‘understanding’ or perhaps ‘conceptualisation’ starts to look closer to a group of mechanisms connected via family relation, and we therefore have reason to doubt the general brain mechanism approach and its bet on legibility.
This conclusion might look gloomy, but there is good news in that, by focussing on the processes from which these systems originate, we find a sufficiently scientific method of analysis for their functions is not out of sight. The rub is that each emerges from some form of autonomous optimisation in a particular problem environment - each being measured against persistence conditions that are themselves legible. The evidence for this can be found in the different systems’ convergence to platonic solutions to shared problems.
Similarity (2) - Convergence to Platonic Solutions of Shared Problems
Consider the octopus eye and the vertebrate eye (we encourage you to go look at them side by side). Both evolved independently, as the last common ancestor between vertebrates and cephalopods did not possess eyes. Given their striking structural similarity, we do not want to conclude that this example of convergence (and all the other instances) happened by chance. Rather evolutionary biologists suggest that such convergence is to be expected when selection pressures are shared between lineages, even when distantly related. The claim is that the octopus ancestor and the vertebrate ancestor needed to solve the problem: moving in the correct direction while only having access to electromagnetic radiation. And that given enough time, the optimising pressure of evolution will trend strongly towards the mechanism that is most fit for solving said problem. In this case this is the eye.
There are differences in the octopus and vertebrate eye of course, but we must not throw out the theoretical baby with the theoretical bathwater. To illustrate - we can make a calculator out of silicon chips and plastic, or we can make one out of wooden planks and rubber balls. If we accept that it is non-trivial to say that both actually are calculators - then it is non-trivial to say in the octopus and vertebrate case that both actually are eyes, as this teaches us something about them. Anyone who objects ‘they cannot be considered the same kind of system – they evolved independently’ would be considered pedantic at best; and as having arbitrary language priors at worst.
Keeping this in mind, the substantive claim of this section is that the same phenomenon has occurred, repeatedly, between instances of the ITS, and that the same style of explanation - an appeal to selection pressures and problem environments - is worth pursuing.
There are too many examples for them all to be listed here, but the most salient instance is convergence on modularity and specialised sub structures. The brain has overt modularity, the same has been shown for models (Zhang et al. 2023). Institutions are saliently modular by the same lights - with specialised departments taking the functional place of specialised brain regions. This case is somewhat trivial, but we might have the urge to say that - especially in the case of corporations - that this is not really convergence - as the modularity had been implemented by human decision makers.
This, in effect, is to deny the premise of institutions being modellable as intelligent systems on assumption - and has an air of ‘I can point to a different cause, so this mechanism isn't actually load bearing’. This being similar to saying something like ‘fundamental physical laws govern everything, so there's no sense in talking about biology or theories of intelligence on parsimony’. Further to this, we can support the autonomous optimisation view by considering what institutional structure would look like if modularity were not optimal. If modularity were not optimal, humans would have chosen something else. So appealing to human decision makers fails to tell us why institutions have the specific structure they possess - as the human decision makers are always tracking the optimal structure. It is simply trivial to say that humans are the ones to realise these changes in institutions - analogous to how it is trivially true to say that gene mutations realise adaptation in organisms. Institutions are not engineered static objects - they grow incrementally, with these variations being subject to the persistence conditions faced by the institution. Like the case with the vertebrate and octopus eye - these structures emerge because they are the optimal solutions for their respective problems.
What is observed is that each of the intelligent systems discussed has converged on modularity, and in the absence of the autonomous optimisation view we lack an explanation for why this has occurred to this level of robustness, despite significant differences in substrate. It is in this sense that the autonomous optimisation view describes these instances as being platonic solutions to shared problems, as they appear tightly coupled to the shared problems, and much less tightly coupled than the lineage facing them.
Closing Thoughts
We take the pattern to be clear, the autonomous optimisation view gives us both a neat explanation for the illegibility of these systems, and their convergence on the optimal solutions to particular problems. We take this to sponsor a particular kind of research direction - being the investigation of the relationship between problem environments, their aforementioned platonic solutions, and how this pertains to particularities of the implementation space that the respective lineages are exploring. Tangibly this includes capability first environment design, a taxonomy of problem-solution pairs, and the isolation of substrate dependent failure modes.
What is exciting about this view is the possibility for accurate AI counterparts of individuals and institutions alike - by the reconstruction of the environments that selected for them. The benefits of a digital implementation also follow - namely high testability and malleability. It is by these lights that the strategy layer discussed in the abstract falls within reach, and our greater ability to mold our own futures along with it.
Bibliography
- Kapur, S., Phillips, A.G. & Insel, T.R. (2012). Why has it taken so long for biological psychiatry to develop clinical tests and what to do about it? Molecular Psychiatry, 17, 1174–1179.
- Wiegert, J.S., Spehr, M. & Hanganu-Opatz, I.L. (2023). Systems neuroscience: A box full of tools to illuminate the black box of the brain. PLOS Biology, 21(7), e3002221.
- Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B. & Liu, T. (2024). A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems. arXiv:2311.05232.
- Lee, H.L., Padmanabhan, V. & Whang, S. (1997). Information Distortion in a Supply Chain: The Bullwhip Effect. Management Science, 43(4), 546–558.
main