Best Practices

Rocket-Hire is 10: Reflections on a decade of testing

Friday, May 20th, 2011

We are very excited to announce our Tenth anniversary!!!  That’s right, through good times and bad we at Rocket-Hire continue to work hard to promote the benefits of best practices based screening and assessment programs.  For testing geeks like us, the past decade has seen some very very exciting innovations.  It is easy to lose the forest for the trees, as our daily efforts to implement assessment often keep us focused on the issues that still hold us back. In reflecting on the past decade as a thought leader for the assessment world, I have worked hard to refocus on the big picture and zooming out to the treetops has presented a view that is extremely positive and encouraging.  This vantage has reminded me that we have seen some quantum leaps in the testing game that have made the use of pre-employment screening and assessment an even bigger value add then ever.  Here is a quick review of the big picture when it comes to innovation and progress in our industry over the past decade.

#1: Test usage has crossed a major plateau- Ten years ago the testing industry was in total plateau mode.  Uptake was at the same level as it had been for decades with a handful of firms, mostly test publishers and consulting firms offering administratively heavy tests in two modes.  Ten years ago one could either buy a test off the shelf and drop it in place, sometimes doing validation work to support its use or sometimes not; or one could hire a consulting firm to do an expensive local validation study using their own content.  These options and the universal truth that testing required a good deal of resources to administer and manage, cost a butt load of money, and provided a cold war-sih icky feel to those taking the tests; served to really keep testing down.  We are way past all this now!!  Test uptake and the available revenue from selling tests has skyrocketed based solely on our friend, technology.  This technology enabled shift is the #1 big picture trend in the past decade.  This shift has been facilitated by several other important trends (discussed below).

#2: Data shows us the truth- Ease of administration and increased uptake have allowed us to capture millions of data points.  This information has greatly accelerated our understanding of what job performance is and how to accurately measure it.  We really do know how to accurately measure the traits that drive important work outcomes such as customer service and how to predict which applicants are most likely to achieve these outcomes.  This knowledge serves as the basis for increasing speed and accuracy in testing.

#3: Methods of demonstrating validity are changing- Please note, I am not saying that the concept of validity itself is changing.  I am saying that we have increasingly powerful tools to help us configure job relevant assessment content for local situations (thanks to Trend #2-above).  Most vendors have begun to bake a good deal of flexibility into the process and tools used to configure assessment content, building on the data they have harvested and then allowing end users to lightly customize their specific measurement model.  In a way this is the holy grail for validity as we begin to see criterion, content, and transportation of validity strategies merge to show us what content is correct for a given situation.  Isn’t this kinda what Landy (1986) was talking about when he rejected stamp collecting in favor of identifying ways to show how the rubber meets the road when it comes to showing a relationship between predictor and criterion space?

#4: Remote, unproctored testing is here to stay- Like it or not, there is no way to beat the convenience of remote testing.  I have served on more panels then I can remember on this topic over the past decade and all have reached the same conclusion, we do not have any strong evidence that remote testing is a problem.  This does not mean we can ignore the fact that we need to be vigilent.  Again, technology is our friend as we enter the age of IRT driven adaptive testing and increasing security tools such as bio-metrics.  The interesting thing in the decade to come will be the acceptance for remote testing via smart phones.  The jury is still out on this one.

#5: Candidate experience is becoming a key driver- A decade ago it was still common to see 2 and 3 hundred item long tests that asked question with no perceptible link to the job performance domain.  This is no longer the case as we begin to explore ways to increase simulations and games that make the assessment experience transparent and can easily be woven into employee branding.  This will be one of the most significant trends over the next decade as we begin to put the radio buttons of decades old personality tests in the rear view mirror.  If nothing else, the next generation of job applicants will begin to demand this type of treatment, and I really believe that this population is going to begin the redefinition of terms like “job” and “career” forcing us to adapt our hiring and assessment processes.

#6: Assessment is becoming an integral part of the employee lifecycle- We continue to see progress (albeit slow) towards a more unified vision of what talent is and how it fits within the organization.  Assessment has long been used as a tool for succession planning and development.  But there has been no continuity with the information collected during the hiring process.  Most of this time this info is basically industrial waste, going down the drain and taking value with it.  The rise of a talent management mindset has started to help promote a more strategic focus that covers the entire employee lifecycle.

All of the trends above have combined to open the door for increasing levels of value from assessment based on new levels of efficiency and effectiveness.  I encourage our readers to take a moment to reflect on just how far we have come.  Don’t even get me started talking about what we can expect to see in the next decade.  Almost daily I am seeing testing firms leveraging cool new technologies to help meet the end goal of providing realistic, accurate and efficient ways to predict applicant performance.   Just prepare to have your mind blown wide open.  The strong forward march of technology is going to make all aspects of our lives extremely interesting (and maybe a bit scary?).

Enhanced by Zemanta

Sharing is Caring- A Word About Access to Your Assessment Data

Monday, February 7th, 2011

I have recently been working with several clients who have been less than happy with their current pre-employment assessment vendor.  One of the common themes underlying their concerns is the fact that their assessment provider will not allow them access to the assessment data collected from applicants.

While data ownership is definitely a common bone of contention when creating assessment contracts, no one should be denied access to their data.  If your vendor is not willing to share this information with you, it raises a serious red flag.

Most of the vendor whom I have seen using a no share policy use a “proprietary” methodology and suggest that outside data analysts just don’t have the know how to make sense of the data correctly.  This is one reason to avoid vendors with super secret methods.  Vendors should be able to provide test data that can be used in external, 3rd party validation studies. This is important because statistics are easily manipulated and the procedures used are often not explained with enough clarity to allow for duplication.  Vendors have agendas to show their clients results and often this can lead to conclusions that cannot be fully trusted.

Be sure to ask potential vendors about their data sharing and ownership policies and strongly consider steering clear of those vendors who are not willing to share.  Barney the Dinosaur speaks the truth when he says “Sharing is Caring”  It’s hard to believe that a vendor who won’t share, really cares about you as a client.

Federal Government Hiring Made Easier-ish

Tuesday, May 25th, 2010

A recent Washington Post piece by V. Dion Haynes highlights both the Federal Government’s attempts to accelerate Federal hiring and a non-profit effort to help applicants. The Complete Idiot’s Guide to Getting Government Jobs has been released on the heals of an Obama administration directive to cut typical application processing times in half and make it easier to apply for an upcoming half-million job openings.

So will a 332-page book help? Probably, but there’s something that’s both frightening and disappointing about the need for a book that enlightens a process that’s normally quite simple (judging from most corporate application procedures) in a format that often describes otherwise complex skills such as C# Programming or financial planning.

Improving the inconsistent, complicated Federal hiring process is definitely worthwhile, but one troubling aspect of the Administration’s initiative is a movement from assessment of relevant knowledge, skills, abilities, and other characteristics (KSAO’s) to greater reliance on job experience. Unlike our own complex and individualistic career patterns, however, assessment is meant to cut straight to what’s required for the job. By attempting to quantify a vast array of job histories, this directive may make the applicant scoring and screening process actually more complex for the hiring manager.

On the other hand, anyone who has successfully navigated the Federal hiring process may have already displayed high cognitive ability, tenacity, attention-to-detail and dependability, so maybe most newly-hired Federal employees will have a lot of those KSAOs covered anyway!

New Assessment Trends Report- Worth Reading

Wednesday, April 14th, 2010

I highly recommend that anyone interested in learning about trends in pre-employment assessment usage take a look at PreVisor’s newest report.

This report is an excellent compliment to our own yearly assessment usage survey, the results of which are available from our website.  PreVisor’s report confirms some of our major findings and adds some insight in areas that we did not cover.  The key finding of interest across both studies is that…

  • While almost all companies report that quality of hire is very important to them, a relatively small number actually evaluate it.

What’s it going to take for companies to get with the program and understand how important it is to understand the value of their pre-employment assessment programs?

Some key findings not included in our report include:

  • Companies have use the downturn to concentrate on developing existing talent as opposed to hiring new talent.  Most companies have placed plans to adopt new assessment tools on hold until the economy improves (happening now!!!).
  • Most companies are still working on determining exactly how to to use social media as part of a formal recruiting strategy.
  • While applicant reactions to the recruiting process are seen as critical, few companies actually have any formal programs for evaluating them.
  • While career development is seen as a high priority, few companies surveyed actually  have any formal promotional processes.

Sum total, the results of PreVisor’s study confirm a “wait and see” attitude when it comes to hiring and hiring related initiatives.  Results also reinforce the lack of formalized evaluation for initiatives that are deemed important.

We look forward to the results of the 2010 survey and expect that they will reflect the results of economic recovery in terms of an increase in the use and evaluation of hiring related assessments.

Common Misconceptions About Assessments

Thursday, January 28th, 2010

One of the most important factors in determining the “success” of an assessment program involves the extent to which the assessments used meet the expectations of their users. Unfortunately, expectations for assessment are often a bit misguided due to a slick sales job or a misunderstanding regarding how to use assessments correctly.  Using assessment in a manner that will allow results to align with expectations requires a bit of effort and understanding.  Getting results starts with ensuring that you clearly understand the traits required for successful job performance and choose quality measures of these traits.  Beyond this, the most important aspect of using assessments correctly is a clear understanding of what assessments can and cannot tell us.

What assessments can tell us:

  • The basics about certain work related attributes- Assessment is great for providing a rough sketch of an applicant.  This is especially true when it comes to certain personality characteristics or mental abilities.  Expecting more than just a rough sketch from most pre-employment assessments is unrealistic.
  • Mastery of specific knowledge areas- Assessment is also good for measuring someone’s knowledge or skill level when it comes to a specific body of information.  Knowledge of a specific computer program is a good example.
  • How applicant compares to others on certain important work related attributes- Assessment can provide a good measuring stick for where an applicant is vs. other applicants or a general population of similar individuals.  This information is very helpful when evaluating applicants relative to one another.
  • The best and the worst- Assessment is a great way to identify those applicants who are very likely not to work out.  That is they are good for providing red flags on certain attributes that are required for job success.  Assessment can also help provide insight into those who rise to the top when it comes to potential and ability.

So, assessment is a really good way to get a snapshot of applicants relative to certain traits required for success but it is important that users do not expect it to tell the whole story all the time. There is plenty of important information that assessments generally do not tell us.

What assessments do not tell us:

  • Everything we need to know- Overreliance on assessment results can be problematic. Assessments are best used as rough indicators of specific things that candidates bring to the table.  Job performance is a complex composite of many different aspects.  It is best to try and use a variety of assessments and other data collection methods (i.e., resumes, interviews) to build a more complete picture of an applicant.  At the end of the day, it is the experience and skill of the hiring professional that should be used to make a decision and assessments are just one piece of information that helps them do their job.
  • Past job performance- Assessments do not tell you how well an applicant performed in a past role.  Of course there are other ways to gather this information.  Just don’t expect assessments to provide it.
  • Contextual issues that may impact performance- Assessments do not tell you about the many life-related factors that may cause someone to perform poorly or leave a job.  We have all performed at less than ideal levels or had problems with a job due to things such as commutes, difficult bosses, low salary, etc. These factors are just as likely to cause problems.
  • How traits translate into work behaviors- Just because an assessment provides insight into certain traits that may be important for job performance does not mean that it always tells you how an individual will actually behave when they are on the job.  It is important to understand that assessments use past data to make broad based predictions, not highly specific insight.

Work within your expectations

Remember humans are complex and interactions with other humans within a work context are often hard to predict.  Assessment is a far from perfect science and expecting too much from an assessment can set you up for disappointment.  However, assessment still has tremendous value when used correctly because it can provide meaningful data to help expert hiring personnel make informed decisions.


“Assessments 101″ White Paper

Friday, January 8th, 2010

A new white paper from Princeton One provides an overview of the various types of assessments currently available.  While it may be a bit basic for those who have a working knowledge of assessment, it is a great starting place for those who are interested in an easy to understand summary.

Great Guide To Best Practices for Pre-Employment Assessment

Thursday, December 10th, 2009

I recently came across a really nice summary of best practices published by PreVisor, a vendor of pre-employment assessment tools.  I think this guide is worth sharing.

Rocket-Hire is a vendor neutral consultancy and as such we choose not to promote specific products and services offered by assessment vendors.  We do however, work hard to help promote an understanding of best practices for pre-employment assessment.  Assessment is a complex topic that is often confusing for those who are just getting started.  There is a lack of quality information available to help those looking for quick info on how to properly use testing.

Please understand that the purpose of sharing this post is to help promote quality resources about best practices for assessment, not to promote PreVisor.

Enjoy!!