In February, GitHub said “GitHub Copilot is behind an average of 46% of a developers’ code across all programming languages—and in Java, that number jumps to 61%”, up from 27% when Copilot for Individuals launched last June. Then last week it announced Copilot X, “our vision for the future of AI-powered software development”, built on GPT-4: a chat window in the editor, AI-written pull-request descriptions, a command-line helper that can “compose commands and loops, and throw around obscure find flags”, and a voice interface. The slope is steep, and Copilot X will apparently push it higher still.
The danger comes when the AIs are smart enough to improve themselves. When they do, the process happens very, very quickly, and if it’s unsupervised it can leave an AI with the power to outsmart us and an ill-defined purpose. Even if it doesn’t know why it’s doing something, it can do it nevertheless — and we won’t be able to stop it, because it can outthink us.
The same worry showed up in a more formal register the same day Copilot X launched. The Future of Life Institute published an open letter calling on “all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4”. Its central question is the one above, put more starkly: “Should we develop nonhuman minds that might eventually outnumber, outsmart, obsolete and replace us?”
We aren’t talking about years from now before they’re coding their own upgrade cycles. Especially if we program rewards for them to improve themselves.
The code won’t stop — it just won’t be authored by humans.
Won’t the AIs innovate new code as they respond to requests to solve new coding problems? Won’t new logic languages, or new hardware, be derived to help them keep pushing the bounds? I guess I don’t see a logical end point to their development.
