How are our programming and software tests developed?

General Overview

  • Programming tests include: JavaScript, Python, C#, HTML5, WordPress Developer, PHP, React, Flutter, Angular, Django, Laravel, Node.js, CSS, C++ and more.
  • Software tests include: Google Cloud Platform, Microsoft Azure, Microsoft Excel, Microsoft PowerPoint, Microsoft Word, Google Ads, Hubspot, Salesforce, Shopify, WooCommerce, Google Analytics, Facebook Ads and more. 
  • Questions follow a multiple-choice format with one correct answer and two-to-four distractors each.
  • Tests target a moderate level of proficiency.
  • Total number of questions ranges from 12 to 20.
  • 10-minute time limit per test.

Test Development 

These tests were developed by industry experts with proven track records, supported by further subject matter experts and our team of organizational psychologists and psychometricians. The test writers used our purpose-built platform to share input regarding the skill’s definition, scope and pace of change, relevance to specific job roles, and other details. The gathered data, together with the content of the US Department of Labour O*NET, was used to inform content and construct validity. 

The finalized items were field-tested by making the tests temporarily accessible via Practice Aptitude Tests, a testing platform where job seekers can practice psychometric tests similar to those they might face when applying for a job. Using pilot data and feedback from practice test-takers, the items were evaluated empirically and revised to improve reliability, validity, and fairness, the three most important aspects of evaluating a test. As the database of responses grows, the tests are periodically analyzed and updated to improve them in an iterative fashion over time. 


Scoring

Test scores are calculated as a simple percentage of questions correct out of the total number of questions (ranging from 0-100%). Results are provided together with a global average and distribution of scores for each test from a representative sample of job candidates that are kept up to date. This provides objective data for candidate comparison across the talent pool.


Implementation Guidance

Programming and software skills tests are most relevant when there is a certain threshold of proficiency required from employees to perform their jobs successfully. They are especially important when the skill is needed from day one and training would be difficult to provide. We highly recommend that these tests are used in a battery of tests selected based on information gathered by analyzing the job in question. 


Test Security

Access to the test and its questions is tightly controlled by our password-protected test bank. Subject matter experts involved in its development and review sign non-disclosure agreements. Additionally, questions are randomly assigned to candidates, controlling for the overall difficulty of the test; thus, no two candidates see the same test but their results remain comparable. 


Key Pilot Statistics

  • Samples consisted of a minimum of 100 potential job seekers per test.
  • Sample demographics across tests were as follows:
    • Average age was 31.8 years;
    • Genders included female (55.7%), male (43.4%) and other (0.9%);
    • Ethnicities included Asian (38.3%), White (29.9%), Black or African American (19.6%), Mixed or Multiple Ethnicities (8.4%), and Other Ethnicity (3.7%);
    • Countries included India (25.5%), United States (12.3%), Great Britain (11.3%), Nigeria (8.5%), Russia (2.8%) and 10 others.
  • Average score was 50% (SD = 16.56).
  • Average time to complete each test was 474.3 seconds (7 minutes 53 seconds). 
  • Average completion rate of 93.7%.
  • Items range in difficulty such that roughly 20% of test-takers answered the most difficult questions correctly and roughly 80% answered the easiest questions correctly.
  • Approximately 71% felt the questions were relevant to measuring knowledge of the language in question and 63% of test-takers believed each test to be at a reasonable level of difficulty.

*Most recent data on the test’s performance is available through the Neuroworx platform which is updated regularly.