LLMs are solving MCAT, the bar test, SAT etc like they’re nothing. At this point their performance is super human. However they’ll often trip on super simple common sense questions, they’ll struggle with creative thinking.
Is this literally proof that standard tests are not a good measure of intelligence?
Citation needed that LLMs are passing these tests like they’re nothing.
LLMs don’t have intelligence, they are sentence generators. Sometimes those sentences are correct, sometimes they’re gobbledygook.
For instance, they fabricate real-looking but nevertheless totally fake citations in research papers https://www.nature.com/articles/s41598-023-41032-5
To your point we already know standardized tests are biased and poor tools to measure intelligence. Partly that’s because they don’t actually measure intelligence- they often measure rote knowledge. We don’t need LLMs to make that determination, we already can.
Talked about this a few times over the last few weeks but here we go again…
I am teaching myself to write and had been using chatgpt for super basic grammar assistance. Seemed like an ideal thing, toss a sentence I was iffy about into it and ask it what it thought. After all I wasn’t going to be asking it some college level shit. A few days ago I asked it about something I was questionable on. I honestly can’t remember the details but it completely ignored the part of the sentence I wasn’t sure about and told me something else was wrong. What it said was wrong was just…not wrong. The ‘correction’ it gave me was some shit a third grader would look at and say ‘uhhhhh…I’m gonna ask someone else now…’
That’s because LLMs aren’t intelligent. They’re just parrots that repeat what they’ve heard before. This stuff being sold as an “AI” with any “intelligence” is extremely misleading and causing people to think it’s going to be able to do things it can’t.
Case in point, you were using it and trusting it until it became very obvious it was wrong. How many people never get to that point? How much has it done wrong before then? Etc.
OP picked standardized tests that only require memorization because they have zero idea what a real IQ test like the WAIS is like.
Also how those IQ tests work. You kind of have to go in “blind” to get an accurate result. And LLM can’t do anything “blind” because you have to train them.
A chatbots can’t even take a real IQ test, if we trained a chatbots to take a real IQ test, it would be a pointless test
Actually, you can give chatbots a real IQ test, and the range of scores fall into roughly the same spread as how they rank on other measures, with the leading model scoring at 100:
https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-100-iq
Nobody is a blank slate. Everyone has knowledge from their past experience, and instincts from their genetics. AIs are the same. They are trained on various things just as humans have experienced various things, but they can be just as blind as each other on the contents of the test.
No, they wouldn’t.
Because real IQ tests arent just multiple choice exams
You would have to train it to handle the different tasks, and training it at the tasks would make it better at the tasks, raising their scores.
I don’t know if the issue is you don’t know about how IQ tests work, or what LLM can do.
But it’s probably both instead of one or the other.
LLMs have a good time with standardized tests like SAT precisely because they’re standardized, i.e. there’s enough information on the internet for them to parrot on them
Try something more complex and free-form and where a human might have to work a little more to break it down into actual little subtasks with their intelligence - and then solve it, LLMs in the best case scenario will just say they don’t know how to do it, and in the worst case scenario they’ll hallucinate some actual bullshit.
All standardized test is how well you prepared for that particular standardized test, doesn’t matter if it is the SAT, MCAT, or Leetcode. You aren’t suppose to think on the spot for these tests, you are suppose regurgitate everything you have rehearsed for weeks and months during the test.
And unthinking regurgitation is what LLMs do better than anything else.
I would argue that some code test questions can be solved spontaneously, but they are limited to easy to some early medium questions, or patterns that are common enough.
I guess this is more common in non FANG companies that don’t have to filter out candidates just because of the sheer number alone.
As someone that didn’t really have good coaching on the SAT, I 100% agree. I kinda fucked it up, but at 17, I wasn’t really used to studying for things outside of school and my parents didn’t get me into any study classes
For GRE though, I studied my ass off… got top 96 percentile scores.
Also went through the leetcode grind. Bombed the first job search I ever did and then later aced the hell out of it after studying really hard.
These tests are all about how diligently you studied and your study technique.
Standard tests don’t measure intelligence. They measure things like knowledge and skill. And ChatGPT is very knowledgeable and highly skilled.
IQ tests have the goal of measuring intelligence.
just a reminder that IQ tests may have the goal of measuring intelligence, but that says nothing of their precision and accuracy
Exactly. I chose my words very carefully.
IQ tests have the goal of measuring intelligence.
The range of LLM scores on IQ tests:
https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-100-iq
Yep, very knowledgeable, highly skilled, kind of a dumbass.
You meet enough doctors and lawyers and you tend to find that combination unremarkable.
No. It may be proof that standardized tests are not useful measures of LLM intelligence, but human brains operate differently from LLMs, so these tests may still be very useful measures of human intelligence.
Eh, yes and no. It might help illustrate the limitations of testing for some people, but it’s not really telling us anything new about them. It is meant to cheaply provide an indication of how a student is fairing and has never been considered by anyone serious as some kind of comprehensive measure of intelligence. Their flaws have been known for a long time.
such tests are not standardized tests of intelligence, they are standardized tests of specific-competencies.
Thomas Armstrong’s got a book “7 Kinds of Smart, revised”, on 9 intelligences ( he kept the same title, but added 2 more ).
Social/relational intelligence was not included in IQ because it is one that girls have, but us guys tend to not have, so the men who devised IQ … just never considered it to have any validity/significance.
Just as it is much easier to make a ML that can operate a commuter-train fuel-efficiently, than it is to get a human, with general function, to compete at that super-specialized task, each specialized-competency-test is going to become owned by some AI.
Full-self-driving being the possible exception, simply because there are waaaaay too many variables, & general competence seems to be required for that ( people deliberately driving into AI-managed vehicles, people throwing footballs at AI-managed vehicles, etc, it’s lunacy to think that AI’s going to get that kind of nonsense perfect.
I’d settle for 25% better-than-us. )
Just because an AI can do aviation-navigation more-perfectly than I can, doesn’t mean that the test should be taken off potential-pilots, though:
Full-electrical-system-failures do happen in aviation.
Carrington-event level of jamming is possible, in-flight.
-
Intelligence is “climbing the ladder efficiently”.
-
Wisdom is knowing when you’re climbing the wrong ladder, & figuring-out how to discover which ladder you’re supposed to be climbing.
Would you remove competence-at-soccer tests for pro sports-teams?
“Oh, James Windermere’s an excellent athlete to add to our soccer-club! Look at his triathelon ratings!”…
… “but he doesn’t even understand soccer??”
… “he doesn’t need to: we got rid of that requirement, because AI got better than humans, so we don’t need it anymore”.
idiotic, right?
It doesn’t matter if an AI is better than a human at a particular competency:
if a kind-of-work requires that competency, then test the human for it.
-
When I was at uni lecturers would often state that exams were thr worst measure of grasping the subject material but its all we have at the moment.
I saw this my self with some of my class mates testing very well but when discussing or problem solving outside of the class there was nothing there.
I think llms fall into this category but with way better recall.
When I was at uni lecturers would often state that exams were thr worst measure of grasping the subject material but its all we have at the moment.
It’s not all we have…
But it’s the only way a professor can run multiple classes of 100 students each.
But colleges are all about profit, so classes sizes are going to be huge.
The goal isn’t educating people, it’s making money.
So when they say “there’s no other option” they’re not mentioning the “and keep making as much money” at the end, it’s just implied.
Sure, tests are bad, but another option is that AI is simply better.
It’d better be. Why’d anyone want to create unintelligent artificial intelligence?
It kind of bothers me that we work hard on making AIs intelligent, and then when one actually starts performing well we go “oh, the test must be bad, let’s change it to make sure the AI still scores poorly compared to humans.” I agree that tests are generally bad but this makes one of the biases we build into them obvious.
deleted by creator
LLMs don’t “think” at all. They string together words based on where those words generally appear in context with other words based on input from humans.
Though I do agree that the output from a moron is often worth less than the output from an LLM
This is kind of how humans operate as well though. We just string words along based on what input is given.
We speak much too fast to be properly reflecting on it, we just regurgitate whatever comes too mind.
To be clear, I’m not saying LLM think but that the difference between our thinking and their output isn’t the chasm it’s made out to be.
The key difference is that your thinking feeds into your word choice. You also know when to mack up and allow your brain to actually process.
LLMs are (very crudely) a lobotomised speech center. They can chatter and use words, but there is no support structure behind them. The only “knowledge” they have access to is embedded into their training data. Once that is done, they have no ability to “think” about it further. It’s a practical example of a “Chinese Room” and many of the same philosophical arguments apply.
I fully agree that this is an important step for a true AI. It’s just a fragment however. Just like 4 wheels, and 2 axles don’t make a car.
deleted by creator
Disagree. We’re very good at using words to convey ideas. There’s no reason to believe that we speak much too fast to be properly reflecting on what we say—the speed with which we speak speaks to our proficiency with language, not a lack thereof. Many people do speak without reflecting on what they say, but to reduce all human speech down to that? Downright silly. I frequently spend seconds at a time looking for a word that has the exact meaning that will help to convey the thought that I’m trying to communicate. Yesterday, for example, I spent a whole 15 seconds or so trying to remember the word exacerbate.
An LLM is extremely good at stringing together stock words and phrases that make it sound like it’s conveying an idea, but it will never stop to think about the definition of a word that best conveys a real idea. This is the third draft of this comment. I’ve yet to see an LLM write, rewrite, then rewrite again it’s output.
deleted by creator
To me it isn’t just the lack of an ability to delete it’s own inputs, I mean outputs, it’s the fact that they work by little more than pattern recognition. Contrast that with humans, who use pattern recognition as well as an understanding of their own ideas to find the words they want to use.
Man, it is super hard writing without hitting backspace or rewriting anything. Autocorrect helped a ton, but I hate the way this comment looks lmao
This isn’t to say that I don’t think a neural network can be conscious, or self aware, it’s just that I’m unconvinced that they can right now. That is, that they can be. I’m gonna start hitting backspace again after this paragraph
deleted by creator
deleted by creator
There has been plenty of proof that standardized testing doesn’t work long before ChatGTP ever existed. Institutions will keep using them though because that’s what they’ve always done and change is hard
Long before. Even in 1930 the eugenics-motivated creator Carl Brigham recanted his original conclusions only years ago that had led to the development of the SAT, but by then the colleges had totally invested in a quick and easy way to score students, even if it was inaccurate. Change is hard, but I think the bigger influence here was money since it hadn’t been around that long at that point.
Not disagreeing with you; how do you suggest a way for admissions to reliably compare applicants with each other? A 3.5 at one school can mean something completely different than a 3.5 at another school.
Something like the SAT is far from perfect, but it is a way one number that means the same thing across applicants.
There shouldn’t even be admission based on what you score in some random test. My (non-US) university accepted everyone who applied, at least for my field of study. Does that mean many people drop out after a semester or two? Absolutely, but there are countless people completing their studies who would have never gotten a chance to do so otherwise. Why shouldn’t they be allowed to prove themselves?
No. It’s the opposite in fact. It shows that ChatGPT is not very intelligent. Just very well-read.
Intelligence cannot be measured. It’s a reification fallacy. Inelegance is colloquial and subjective.
If I told you that I had an instrument that could objectively measure beauty, you’d see the problem right away.
But intelligence is the capacity to solve problems. If you can solve problems quickly, you are by definition intelligent.
the ability to apply knowledge to manipulate one’s environment or to think abstractly as measured by objective criteria (such as tests)
https://www.merriam-webster.com/dictionary/intelligence
It can be measured by objective tests. It’s not subjective like beauty or humor.
The problem with AI doing these tests is that it has seen and memorized all the previous questions and answers. Many of the tests mentioned are not tests of reasoning, but recall: the bar exam, for example.
If any random person studied every previous question and answer, they would do well too. No one would be amazed that an answer key knew all the answers.
But intelligence is the capacity to solve problems. If you can solve problems quickly, you are by definition intelligent
To solve any problems? Because when I run a computer simulation from a random initial state, that’s technically the computer solving a problem it’s never seen before, and it is trillions of times faster than me. Does that mean the computer is trillions of times more intelligent than me?
the ability to apply knowledge to manipulate one’s environment or to think abstractly as measured by objective criteria (such as tests)
If we built a true super-genius AI but never let it leave a small container, is it not intelligent because WE never let it manipulate its environment? And regarding the tests in the Merriam Webster definition, I suspect it’s talking about “IQ tests”, which in practice are known to be at least partially not objective. Just as an example, it’s known that you can study for and improve your score on an IQ test. How does studying for a test increase your “ability to apply knowledge”? I can think of some potential pathways, but we’re basically back to it not being clearly defined.
In essence, what I’m trying to say is that even though we can write down some definition for “intelligence”, it’s still not a concept that even humans have a fantastic understanding of, even for other humans. When we try to think of types of non-human intelligence, our current models for intelligence fall apart even more. Not that I think current LLMs are actually “intelligent” by however you would define the term.
Does that mean the computer is trillions of times more intelligent than me?
And in addition, is an encyclopedia intelligent because it holds many answers?
This is a semantic argument.
Have you never felt smarter or dumber depending on the situation? If so, did your ability to think abstractly, apply knowledge, or manipulate your environment change? Intelligence is subjective (and colloquial) like beauty and humor.
