GPT-4 and Claude 3 fail logic puzzles

AI puzzles are tripping up the smartest models. A new set of tests shows even top systems like GPT-4 and Claude 3 stumble over logic games designed to probe reasoning beyond pattern matching. The puzzles—ranging from classic riddles to spatial reasoning challenges—were released today by researchers at MIT Technology Review and partners. The goal wasn’t just to highlight failures; it was to push AI toward deeper, more human-like understanding.

These puzzles aren’t arbitrary. They trace back to the very roots of AI. In 1959, Arthur Samuel coined the term “machine learning” by having a computer improve at checkers through self-play. Games and puzzles have long served as benchmarks: chess in the 1990s, Go in 2016, and now these logic challenges. Today’s AI excels at absorbing vast data but struggles when asked to infer, adapt, or solve problems it hasn’t seen before. Current models rely on statistical patterns, not true reasoning. That’s why these simple puzzles reveal such big gaps.

The tests include problems like the “Wason selection task,” a classic logic puzzle that trips up even highly educated humans. In one example, participants must flip cards to verify a rule like “If a card shows a vowel on one side, it must have an even number on the other.” Most people get it wrong. Now, researchers find that leading AI models get it wrong too—often worse than average humans. The results underline a persistent issue: AI can mimic intelligence but doesn’t yet possess it.

What happens next could reshape how AI is built. Researchers say the tests will guide new training methods, including better use of symbolic logic alongside neural networks. Some labs are already experimenting with “neuro-symbolic” models that combine deep learning with rule-based reasoning. The stakes are high. If AI can’t solve basic puzzles, it won’t solve complex real-world problems—from medical diagnosis to climate modeling. The next step isn’t just to pass these tests, but to use them to build systems that can truly think.