Does Admissions Exam for Elite High Schools Measure Up? No One Knows

The Specialized High Schools Admissions Test, or SHSAT, is the sole means of admission to the city’s prestigious high schools.

Q. What does the admissions test for New York City’s elite high schools actually measure?

A. No one really knows.

“It’s really a test of itself,” said David C. Bloomfield, a professor of educational leadership, law and policy at Brooklyn College and The CUNY Graduate Center. “The test ought to be aimed at identifying who will be a more successful student at these schools, but it’s just a self-contained evaluation to see who does well on the test.”

The Specialized High Schools Admissions Test, known as the SHSAT, is the sole means of admission to the city’s prestigious specialized high schools, which include Stuyvesant High School, the Bronx High School of Science, and six other schools, with the results overwhelmingly favoring Asian and white students over black and Hispanic ones.

[Read more questions and answers about the SHSAT]

Mayor Bill de Blasio has proposed getting rid of the exam to increase diversity at the schools and shifting to a system that uses class rank and state test scores instead.

Unlike other high-stakes admission tests, such as the SAT and the ACT that are used for college, and the SSAT and the ISEE that are used for admission to private schools, the SHSAT has not undergone an extensive vetting process known as predictive validity testing, which provides statistical evidence that a test is actually doing what it claims to do: In the case of the SHSAT, it would be identifying the students who can thrive in the accelerated academics of the specialized schools.

Proponents of the test say that it must be working since the vast majority of students admitted to the specialized schools not only graduate but also go on to successful college and work careers.

For the college admissions tests, validity testing might compare students’ test scores with their freshman year grades and sometimes other measures, such as graduation rates, to look for a correlation. Ed Colby, a spokesman for ACT, the nonprofit organization that developed the ACT test, said that hundreds of validity studies for the ACT “confirm and show the test is predictive and does provide information the colleges can use to select students.”

Validity studies are considered a key requirement for any high-stakes test. Testing standards published in 2014 by three leading professional groups — the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education — specifically cite the need for such validity evidence.

“It’s highly unusual and it’s out of compliance with the industry standard,” said Dr. Barbara Plake, a professor emeritus of psychometrics at the University of Nebraska at Lincoln, of the lack of validity tests for the SHSAT. “It’s unproven.”

The SHSAT resulted from a 1971 state law, the Hecht-Calandra Act, which called for a test-based admission process for the three specialized high schools at the time. The first test under the law was created by Columbia University professors specifically to screen applicants for Stuyvesant High School and the Bronx High School of Science, according to city education officials. In 1983, the test was taken over by American Guidance Systems, a company that was later acquired by Pearson. Pearson has a $13.4 million, six-year contract with the city.

Many parents and teachers have long contended that the SHSAT is an assessment of students’ test-taking skills, honed by extensive test preparation, more than their potential to succeed at the specialized schools.

Pian Rockfeld, an English teacher at the High School of American Studies at Lehman College in the Bronx, one of the smaller specialized schools, has proctored the SHSAT. She said she could always tell who had taken prep courses. The students would draw diagrams to decipher confusing questions that left others stumped, or if they were good in math, they would start midway through the test on the math section to take advantage of a quirk in the scoring process that rewards students who score extremely high on one part of the exam rather than those with high but more balanced scores across subjects.

“The test does not assess at all how hard a student works, or the creative and independent thinking that a student would need to thrive in our high school,” Ms. Rockfeld said. “I’m always wondering what kids we’re missing by using this test.”

In a 2012 civil rights complaint filed with the U.S. Department of Education over the test, which is still pending, the NAACP Legal Defense and Educational Fund and other groups cited the lack of evidence showing that the SHSAT was even “a valid test of skills and knowledge that are integral to the educational mission” of the specialized high schools.

Rachel Kleinman, a lawyer for the NAACP Legal Defense Fund, said that the lack of validity testing for the SHSAT underscored the larger problem of using a single test to screen students. “If you’re going to have a single test, at the very least it should be a valid test,” she said. “This is just not the right way to assess merit.”

In a recent interview, Richard A. Carranza, New York City’s schools chancellor, expressed doubts that the SHSAT by itself is a good measure of student ability. “It’s not necessarily valid or reliable in terms of identifying student competencies to be successful in the specialized high school environment,” he said. “It’s not. It’s just an obstacle. It’s something you have to endure to be able to go to one of the specialized schools. It’s not aligned to state standards. It’s not necessarily aligned to anything, except it’s just a tough test that you have to prep for.”

Scott Overland, a spokesman for Pearson, deferred questions about the SHSAT to the city’s Education Department, which he said “is involved in all aspects of the test development process and provides final approval before students take the SHSAT.”

City education officials have not addressed why the test’s validity has not been established. They said they hired a consulting firm, Metis Associates, to conduct a predictive validity test in 2013 in response to the civil rights complaint. But city officials declined to release that study on Tuesday, citing legal concerns over the pending complaint. They noted that various controls were adopted “on the front end” that are another form of validity testing, known as “construct validity.” Improvements were made to the test, including redesigning it last year to better reflect the middle school curriculum, officials said. A frequent criticism of the SHSAT had been that it covered material that was not taught in classrooms, forcing students to hire private tutors or try to learn on their own. Officials said the scoring process did not change.

In 2005, Josh Feinman, an economist and a 1980 graduate of Stuyvesant who uncovered the quirk in the SHSAT scoring process, looked into the research behind the test when his daughter was preparing to take it (she was admitted to Bronx Science). Though the lack of validity studies is a concern, he said, there is far less evidence to support the mayor’s proposal.

“You’re replacing something that has not been thoroughly vetted — but that most people would say has done a pretty good job of selecting strong students — with something that you have picked out of thin air,” he said.

But Jonathan Taylor, a research analyst at Hunter College, called the SHSAT “a black box” because city education officials have never explicitly stated what it means to be successful at the specialized high schools, or what the test is actually supposed to be measuring.

Mr. Taylor undertook his own validity testing using data obtained from the city’s Education Department, tracking 22,576 eighth-grade students who took the SHSAT in 2013, including 3,732 who went on to attend the specialized high schools. He compared their SHSAT scores with their grade point averages in ninth grade. He also compared their grade point averages from seventh grade.

Mr. Taylor found that the highest SHSAT scores — more than 600 out of a possible 800 — did predict who would do well at the specialized high schools, but the results were less clear for lower scores around the individual admission cutoffs for each school, which ranged from 559 for Stuyvesant to 479 for Brooklyn Latin School (the highest score for any school that year was 701). Overall, he concluded, the grade point averages from seventh grade were a much better predictor of achievement at the specialized schools than SHSAT scores, a finding that would lead to the admission of more girls and also slightly more black and Hispanic students.

Under Mr. de Blasio’s plan, the top 7 percent of students from every middle school in the city would be admitted, which would significantly increase the number of black and Hispanic students at the schools, according to city officials.

“The bottom line is there should be multiple criteria,” said Mr. Taylor. “The SHSAT is not good enough to justify its use as the sole criterion.”

In Other News

© 2020 US News. All Rights Reserved.