Does the Math SAT measure anything worthwhile?

Daniel Coleman recently announced that the College Board will begin a process of updating the SAT. Again.  The last update was in 2006. While that may seem like a lifetime ago to any teenagers faced with taking the test, it feels more like yesterday to the "old fogies" among us.  The announcement has spawned a rehash of the usual anti-test rhetoric, which tends to consist of variations on the following:

• The test doesn't measure the true abilities of students who know the material but don't "test well."
• college entrance exams are unnecessary since a college can learn everything they need to know from other parts of the application - particularly the student's GPA/transcript
• the SAT in particular is "not aligned with the curriculum"
• college entrance exams, and the SAT in particular, are too "coachable."  This gives an unfair advantage to students whose families can afford expensive courses or tutors.

The article lambasting the test in question generally ends with a call to end all testing that is standardized and/or "high stakes" immediately and directly.

My expertise is primarily in the math section.  For a discussion of the reading section, I'll refer you to this article by Erica Meltzer "Why good grades in English do not always correlate with high SAT Verbal scores."

As test prep coach, I am frequently called by a parent who assures me that her child's grades are top-notch, but he/she just doesn't "test well."  The parent seems to feel that the student knows the material, but can't show it on the test.  Usually I discover that the student understands the material at such a shallow level that he or she can only answer a question if it fits the same pattern as the questions that have always been asked in math class. Furthermore, this only works for more recent material.  Topics covered more than a year before have been forgotten because they were never thoroughly understood in the first place.  Once I've worked with the student for a few hours, I've uncovered a list of "holes" in his or her math understanding.  Salman Khan discussed this phenomenon in a Ted Talk.  (Watch the Ted Talk here.  The part about holes in understanding is in minutes 8 through 10) A child can make an A or a B on a test without mastering about 15% of the material. Since math is cumulative, these holes add up.  Eventually the student will hit a wall. Suddenly, after years of A's and B's, the student is struggling to pass math.  The angry parents will often blame the teacher since, "he was doing well until this year."

The Math SAT shines a big spotlight on all of those holes.  Let's look at an example that should be accessible even to those readers who may be math-phobes:  even and odd numbers.  At the most basic level of understanding, a student can recite the definitions of even and odd numbers and identify a particular integer as being even or odd.  At a deeper level, a student should be able to apply the definition in order to answer a novel problem, such as "a and b are both even numbers. Which of the following could be the product of a and b?"  If you truly understand what it means to be an even number, you will realize that, of the choices given, the answer is the only one divisible by 4.

Critics see questions such as the above and wail, "The test is not aligned to the curriculum!"  Really? Since when are odd and even numbers not part of the curriculum? Apparently saying that is someone's attempt to translate, "The problem doesn't look exactly like the homework," into educationalese.  Of course the problem doesn't look exactly like the homework! That was the point.

Sometimes, on one of the earlier questions, an alert student will realize that she doesn't need to perform the tedious arithmetic called for in the question because the answer will have to be even and there is only one even answer choice.  Critics wail, "The test is too easy to 'game'!  A student can get the correct answer without even working the problem!"  Having a deeper understanding of odd and even numbers isn't "gaming" the test. That IS the test! Do you really think the folks at the College Board are so stupid that after decades they haven't figured out that to prevent a student from doing that they should include more than one even answer choice? They wrote it that way on purpose.  A student with a deep understanding of math can breeze through that question and have more time for the difficult questions at the end that are more at her level.  Meanwhile, however, other students can still get a correct answer. They just have to slog through the arithmetic. Test questions such as these are brilliantly designed.

"The test is too coachable!"  Hah. As a test prep coach I wish it were.  One mom took on a year-long project of attempting to get a 2400 on the SAT.  Her study regimen included just about every program or technique ever invented.  She even had a tutor at one point. Her math score barely budged. (Visit her blog on the subject here.) I have students who make dramatic gains on the math section after seeing me, and others who don't. Some students have a thorough understanding of the material, but they've never seen the types of test questions that would test a deeper knowledge.  Once you show them what these questions look like they are off and running. I don't try to explain Bloom's Taxonomy.  They think they are "gaming" the test.  Some of them, bless their honest little souls, anxiously inquire if using their new awareness is "cheating."

It is true that the students who come to see me often make dramatic gains.  And it is true that most of them have paid about $300 (total, not per hour) to do so. However, the solution to that problem is not to "fix" or eliminate the SAT. The solution is to include questions that test a deeper level of understanding on the students' every day math tests!

Critical thinking skills are enormously difficult to measure.  We want our students to learn critical thinking skills, and we would love to have a good measure to decide whether or not they are, but most tests, and in particular most teacher-made tests, assess learning at a fairly superficial level.  If I were in the admissions department at an engineering school, I would absolutely want to see a student's math SAT score. Even if the child has managed A's in math up to this point, I would want to be alerted to the holes in his understanding at the earlier levels that might keep him from continuing to perform at that level. I only hope that the SAT continues to be a test that can fill that role.