Dr. Dwight Galster
I found an article
by
Chris Correa called "More on the Elite Myth" after reading a post by
Jenny D, "Our Kids Aren't As Good As Their Kids." It's about international
comparisons and how our top students compare.
I've given this issue some thought, as well. Beginning with anecdotal evidence,
it certainly seems that our foreign college students (primarily Asian but also
notably some "Eastern Block" European) in Math and Engineering tend to
outperform the best local students mathematically. It is also evident that the
winners of competitions in Math and Science are mostly first or second
generation Asian immigrants.
This topic really deserves a closer look, beyond looking at the 95%ile. I am
concerned that the percentile may not be the best way to determine what we want
to know. Ideally, I would like to know the 95%ile score for all countries
combined, then the percentage of each country's students above that score. In
addition, it seems important to know more precisely the shape of the upper tail
of the distribution. Having information on more percentiles would be very
useful.
My study of this begins with the
TIMMS (2003) rather than the
PISA (2003). TIMMS tested 4th and 8th graders (I am only looking at 8th
grade results), while PISA tests 15-year olds, whose grade levels ranged from 7
to 11. The TIMMS reports data for 46 countries, while PISA reports data for 39,
of which only 17 are common to both studies. Thus the two studies, taken
together, include 68 countries. TIMMS includes subscales in five content areas,
Number, Algebra, Measurement, Geometry, and Data. PISA sub-divides by Literacy
and Problem Solving. The literacy scale is further divided into four subscales
for Quantity, Space and Shape, Change and Relationship, and Uncertainty. There
does not appear to be a direct mapping between the categories used in the two
studies. PISA also provides percentages of students in six achievement levels
for each subscale, giving us another way (besides percentiles) to compare our
elite students.
In the TIMMS study, 20% of the countries have mean combined scores higher than
the US, while 25% of the countries have a higher 95%ile on the combined scale.
This suggests we are losing ground as we get into the higher ability levels. The
US mean is 504, while Singapore holds the top position with 605. We could say
that Singapore's mean is 20% higher than ours, but since I don't know what kind
of scale these scores come from, I hesitate to try to interpret that. (Another
table shows that on average Singapore's students got 45% more questions right).
Looking at the 95%ile, the US has 635, while the top country (Chinese Taipei) is
at 733. Again, we could say that is about 15% higher than ours, but
interpretation is problematic. It would be much more useful to know what
percentage of Chinese students scored above 635, or alternatively, what
percentage of US students scored above 733. In the subscale means, the US is
12th in Number, 11th in Algebra, 21st in Measure, 24th in Geometry, and 13th in
Data.
In the PISA study, 28 (72%) of the countries have mean literacy scores higher
than the US, while 26 (67%) of the countries have a higher 95%ile. The US mean
is 483, while Hong Kong holds the top position with 550 (14% higher). Looking at
the 95%ile, the US has 638, while the top country (Hong Kong) is at 700 (10%
higher). The same problems with interpretation apply as discussed above.
However, when we look at the percent of students who achieved a Level 6
proficiency in Literacy, the picture is much clearer and more shocking. In the
US, only 2% of the students achieved this level. In Hong Kong, it was 10.5%. Not
only that, but there are 16 countries that have 4% or more in this group (double
or more the US figure). Canada and Australia have almost three times as many as
the US, and Korea, Japan, and Belgium have four times as many. This is strong
evidence that we are failing to provide an adequate math education for our best
students.
Further research: I would like to try to combine data from the two studies, and
also look at comparisons at the lower end of the distribution.
Dr. Dwight Galster
I have obtained the PISA 2003 data. Analysis of PISA data is
a daunting task. Before even beginning that analysis, one must study the
technical theory and philosphy of the PISA, which I can assure you, is anything
but simple or straightforward. I still don't understand it fully, but I have
been able to determine how the national scores are derived, and thus can study
in more detail the distribution of the upper tail, as I proposed doing in the
previous article.
The concern we are addressing here, is that not only is the USA falling behind
in the average performance of students in Math, but that our "brightest and
best" are falling behind as well. My studies of the upper tail of the
distributions support that hypothesis.
I calculated the percentile scores of all countries in the PISA survey for the
following percentiles: 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 99.5, 99.75, and
99.9. (PISA reports mainly on 38 countries. There were actually 41 country
codes, which include regions such as Hong Kong. In this study, I used all of
them, and refer to them as countries for convenience.) Then for each percentile
score thus obtained, I found the nearest percentile (to the tenth's place) for
that score in each country's data. Thus, I could calculate the estimated percent
of students in each country who exceeded those levels. I checked so many levels
because I wanted to know if there were any differences in the shape of the upper
tail between countries. However, the differences revealed by PISA can be
adequately summarized by looking at just the 90th and 95th percentiles.
The 90th percentile score of all surveyed countries is 604 (hereafter "P90"),
and the 95th percentile score is 640 (hereafter "P95"). The US students track
very close to the international profile all along the scale. Eleven percent of
US students exceed P90, and 4.8% exceed P95. By contrast, the top ranked country
(Hong Kong) has 31.8 percent above P90, and 18.4% above P95. What this means, is
that when we look at the top 10% of math students in the "world" (where "world"
means those countries for which the survey is representative), the US is
contributing a slightly "above average" share (11%), but Hong Kong contributes
three times the "average" share. When we look at P95, the US is a shade "below
average," while Hong Kong is producing three and a half times as many at that
level. Even Canada produces more than twice as many students at both the P90 and
P95 level as the world average.
Country %0ver P90 %Over P95 HongKong 31.8 18.4 Belgium 27.5 16.0 Netherlands 27.3 14.8 Korea 25.8 14.5 Liechtenstein 25.6 14.9 Japan 25.1 14.2 Finland 24.4 12.9 Switzerland 22.4 12.2 NewZealand 21.9 12.4 Canada 21.3 11.0 Australia 20.6 11.0 Macao 18.7 9.9 CzechRep 18.6 10.1 Germany 17.0 8.2 Sweden 17.0 8.7 Denmark 16.9 8.2 Iceland 16.3 7.3 France 15.7 7.9 Austria 15.3 7.4 UK 15.2 7.8 Slovokia 12.9 6.4 Ireland 12.0 5.4 Norway 12.0 5.5 Hungary 11.4 5.1 USA 11.0 4.8 Luxembourg 11.0 5.1 Poland 10.5 4.7 Spain 8.9 3.4 Latvia 8.4 3.5 Russia 7.8 3.5 Italy 7.5 3.6 Portugal 5.8 2.2 Turkey 5.8 3.9 Greece 4.4 1.9 Uruguay 3.0 1.2 Yugoslavia 2.5 1.0 Thailand 1.8 0.6 Brazil 1.1 0.6 Mexico 0.4 0.1 Indonesia 0.3 - Tunisia 0.2 -