For GRE-Maker ETS, standardized testing is no longer enough

Educational Testing Service (ETS), the organization behind standardized tests like the GRE and the Test of English as a Foreign Language (TOEFL), is looking to expand beyond its current products by offering a longitudinal data service.. The company wants to create a “report card” for employees, which would provide employers with a free way to screen a large number of candidates and use ETS’ own AI models to help identify which applications show the most promise. 

During an interview at the ASU+GSV conference, Kara McWilliams, the Vice President of Product Innovation and Development at ETS, called it “de-risk(ing) the hiring process”  and providing employers with a more accurate way to select the best candidates for their organizations.

It is no secret that hiring is a bit of a crap-shoot. Candidates who look good on paper might not work out because of a culture mis-match, because the candidate is exaggerating their skills, or because they lack soft-skills that used to be assumed by hiring managers.

Historically this has been hard to tackle because these things are difficult to quantify and test for. It is easy to have someone prove python efficiency, but not so easy to ask a potential manager to prove that they can de-escalate a tense situation with a subordinate. But some of this might be changing and ETS is interested in developing a service to help.

However, there are several stumbling blocks that ETS will need to overcome to make this service a success. One major obstacle is data. ETS needs to collect a large amount of data to make the service useful. To get to the point where this is useful requires network effects and a large number of candidates in the system. They would almost have to become a “necessary piece” of a job application for a specific industry to ensure that they could get enough candidates onto their platform. I don’t know how they get good hiring insights without candidate data and I can’t see a reason why candidates would be motivated to provide data if they weren’t required to as part of an application process. What motivation is there to provide information which could be used to rank employees in a way that could be detrimental to their career?

Organizations like ETS,  ACT or CollegeBoard know that they need to maintain relevance as things change and they have set their sights on hiring. Many graduate programs chose to go test-optional or test-blind even before the pandemic, and now standardized testing as an educational entrance requirement has become a center point for debate on equity, diversity, and inclusion. As ETS sees it, with their present tests they only engage students/workers at one point in their journey; when they are applying to grad school or proving their english skills. ETS wants to be able to generate longitudinal data, create “insight systems” with diagnostic, formative, and pulse assessments over time.

Part of this is because, while some universities are deciding to walk away from existing standardized tests, the demand for testing is still there.

According to McWilliams, “They don’t want that test, but they need measurement… If we can provide the right insights to institutions and demonstrate how they are equitable and ethical, that is really what they need to make those decisions. Employers are the same way.”

The idea of the “report-card” is that ETS maintains records on a whole slew of employees, which would allow it to use its own AI models to guess which 3-5 applications show the most promise and to provide questions which might fill in data holes in their models.

There is promise here, offering employers a free way to screen large numbers of candidates that could provide a demonstrably better employee choice than the methods they are using now. And then prompting employers to ask specific questions that would fill in some of the holes in ETS’ data so that the candidate could be more accurately screened in the future.

The second concern is the “Big Brother” question. It is hard not to equate this in at least a small way to China’s controversial social credit system. Of course the promise is that it is more of a “matching” tool than a ranking tool, but almost every matching tool invariably becomes a ranking tool because some candidates and some jobs are a lot more desirable than others.

While there is some worry here about what kind of data may be carried between companies, the ship has sailed on data you create by using a work computer. The assumption should be, and is for many employees, that any interactions or employee data generated from interacting with business owned equipment can be tracked, and some companies are already trying to put that to use by measuring it against productivity. It’s not a reason not to be concerned, just the reality of the current situation.

And the final problem is the same thing that is hurting their GRE business right now. Standardized tests like the GRE have been criticized for favoring white, upper-class men, and it is not clear how ETS will create an algorithm that does not reinforce existing biases. The company will need to do a lot of work to convince people that it is genuinely committed to equity and that its algorithms are not reinforcing the existing hierarchy.

And frankly ETS may not be well positioned to make this case. As the purveyors of the GRE they have held up some of these racial inequities. They are unfortunately not coming to the table with a blank slate like a potential new company would be. They will have to do a lot of work to convince people that they are really, genuinely committed to equity before that narrative shifts,

Do you want a company deciding your “employee score” the way companies do your credit score right now? Of course this is a large model so it won’t be one “score”, there will be a complicated matrix designed to match you with a number of areas where you rank high and a number where you rank low, but the end result will be more numerical than it is now. Which isn’t necessarily the worst thing for the world at large, credit scores have allowed banks to lend with more confidence, but it is hard to see how an individual employee/ candidate could see any immediate or short term benefit.

ETS is planning on incorporating ways to help an employee fill in gaps and upskill in some of the areas where they are lacking. While it would probably be easy to promise that a 10 hour MOOC will ensure a higher rating in some way on the platform, it is important to remember that this kind of upskilling is very hard on its own. There were dozens of companies at ASU+GSV this year planning on just working on upskilling and credentialing for soft skills and even they are not always confident they can do it.

So while traffic school can keep a ticket off your records, there isn’t currently a lot of data that says you will be a better driver afterwards. You might be able to take a class on leadership which could boost a number within their system, but turning you into an actually better leader is such a complicated and multidimensional task, I am not convinced they could do it without at least acquiring someone.

Despite these challenges, there are several reasons why ETS’ longitudinal data service could be successful. For one thing, it could provide employers with a way to screen a large number of candidates quickly and accurately, which could be especially valuable in a tight labor market. Additionally, the service could help employees fill in gaps and upskill in areas where they are lacking, which could be beneficial for both the employee and the employer.

The service could be seen as a way for employers to avoid doing the hard work of evaluating candidates themselves. Some employers may see the service as a shortcut that allows them to avoid the time and effort that is required to find the best candidates for their organizations. But it could help to level the playing field for candidates from underrepresented groups. By providing employers with a way to quantify soft skills and other factors that are often difficult to measure, ETS’ service could help to identify candidates who possess the skills and abilities that are required for success in a particular job, regardless of their race, gender, or socioeconomic background.

So while this concept of an “employee report card” is interesting and a little disconcerting, it might be coming for us whether we want it or not. But even if it does, ETS has a lot of ground to make up to put themselves in a place where they will be the people to manage it.