Prediction of Computer Programmer Training and Job Performance Using the AABP Test

 GARLAND Y. DeNELSKY² and MICHAEL G. McKEE

Central Intelligence Agency

    The rapid growth of the field of computer programming in the past 10 years has generated a need to identify individuals with the talents to profit from training and become productive programmers. This need for early identification of potential programmers has been approached in two main ways by personnel psychologists: (1) creation of special tests designed to measure the specific aptitudes believed essential to computer programming, and (2) application of pre-existing tests. Unfortunately, the number of published validation studies which have resulted from the use of both types of tests seems disproportionately small in relation to the massive use of these tests as programmer selection devices for government, industry, and specialized training schools.

    Of the several aptitude tests specifically designed to select potential programmers, various versions of the IBM Programmers Aptitude Test have been most frequently subjected to controlled validity studies. In nearly all instances, these studies reported relationships between test scores and training performance.  

    Significant correlations between the Programmers Aptitude Test and programmer training have been reported by McNamara and Hughes (1961), Oliver and Willis (1963), Katz (1962), Hollenbeck and McNamara (1965) and Bauer, Mehrens, and Vinsonhaler (1968). Of these studies, only McNamara and Hughes (1961) conducted research relating test scores to actual job performance; they reported correlations of .44 and .36 between Programmers Aptitude Test Scores and supervisors’ ratings of job performance. Other specially-designed aptitude tests that have shown at least some promise for predicting programmer training performance include the Logical Analysis Device (McNamara and Hughes, 1961), the Computer Usage Company Programmer Aptitude Test (Hollenbeck and McNamara, 1965) and the Computer Programmer Aptitude Battery (Perry, 1967a).

    More general purpose tests which have been successfully used to predict programmer training performance include the Army’s General Technical Aptitude Area (Katz, 1962), the Wonderlic Personnel Test (Biamonte, 1965), the mathematics test from the Navy Officer Classification Battery (Meyer, 1965), the College Qualification Test (Bauer, Mehrens and Vinsonhaler, 1968) and the Strong Vocational Interest Blank (Bauer, Mehrens and Vinsonhaler, 1968). As with the IBM Programmers Aptitude Test, reports of relationships between test and job performance for these other tests are considerably scarcer. Seiler (1965) reported correlations ranging from .35 to .62 between supervisor’s ratings of workers in electronic data processing jobs and combinations of three nor more tests from the General Aptitude Test Battery, and Perry (1967b) found that an occupational key constructed using SVIB procedures was related to job satisfaction but not to relative salary of computer programmers.

 The AABP Test

    One widely-used test of programming aptitude-the Aptitude Assessment Battery: Programming (AABP) - is absent in the validity literature. Despite its lack of published validity research, its author claims that it has been administered to over 3600 persons in 283 different companies, institutions and government agencies (Wolfe, 1970). According to the brochure published by Programming Specialists, Incorporated, which describes the AABP, the AABP is an untimed test which attempts to evaluate the applicant’s aptitude for protracted concentrations on a long sequence of steps. Both the process of problem-solving as well as the answers to the items form the basis for evaluating individual performance. The scoring and interpretation of the AABP are typically performed by the distributor of the test.

    The AABP contains only five problems, each of which is apparently designed to closely resemble the type of activity engaged in by a programmer writing a program. These problems are quite detailed and involve manipulation of precisely-defined symbols, rigid adherence to instructions, tight logical reasoning and the use of flow charts.  The test is designed to measure “the applicant’s capacity for sustained concentration and accuracy in following instructions. . .(the) test permits an evaluation of his ability (1) to draw deductions with the aid of some simple calculations, (2) to understand the explanation of a somewhat complicated instruction of a kind given in programming reference manuals, (3) to understand a complicated statement of specifications, written succinctly and precisely without further illustration, and (4) to reason with symbols in accordance with specific definitions” (from brochure published by Programming Specialist, Incorporated).

    The author claims that this test penalizes neither the capable but slow worker nor the applicant whose mathematical background is not extensive. An average of about three hours is needed to complete the test; nearly all persons finish within five hours (which is the recommended time limit, if one is used). Examination of the test, suggests that it places a premium on sustained effort, compulsive attention to details, and deciphering of complex instructions (which are often presented without benefit of examples or illustrations). The publisher’s scoring and interpretation of the AABP provides a numerical (percentage) score, and adjectival rating estimating the individual’s overall potential for computer programming and a brief narrative statement describing his probable strengths and weaknesses as a programmer.

     Despite the wide use of the AABP reported by Wolfe (1970), no reliability or validity evidence has either been published or alluded to by its author. No manual for the test exists nor have there been any published reviews of the instrument.

     The purpose of the present paper is to report the relationships between programmer aptitude as measured by the AABP and both the training and job performance of computer programmers in a governmental agency. It is anticipated that these results will be of interest to those who are using this test without benefit of any validity data, as well as to those who may be considering its use but are reluctant to do so without some evidence of its efficacy as a measure of programming aptitude.

 Method

Measurement of Training Performance

    Ninety-three students, 71 men and 22 women, from seven runnings of an in-house computer programming course were selected for study. The average age of these students was 28 years, with a range in age from 20 to 52. 4% had earned advanced degrees, 46% possessed undergraduate degrees, 24% had some college experience and 12% had attended one or more business or technical schools; the remainder were high school graduates. The course, which was 15 weeks in length, was designed to produce professional programmers capable of writing programs in both PL/1 and ALC programming languages. Evaluation of students in these classes was based upon weekly tests (which accounted for about 70% of the students’ grades) and evaluation of the programs written by the students (which accounted for the remaining 30%). Based upon their total grades, students were given a rank representing their final standing among the graduates in their class. The AABP was administered to the 93 students prior to their admission to the programming course. Not all students in the seven training classes took the AABP; 35 persons who completed the course had not taken this test. While the test results entered into the selection decision for many of the students in this course, these results were not known by the instructors who taught and evaluated the students.

 Measurement of Programming Performance

    A total of 57 programmers, 36 men and 21 women, were included in this portion of the study; none of these people were present in the training sample described above. The average age of this group was 25 years, with a range in age from 20 to 50.  The number of months of programming experience of individuals in this group ranged from 1-78 with an average of slightly over 15 months. A variety of programming languages was used by these programmers in their daily work, with FORTRAN being most commonly used followed by PL/1, COBOL and ALC. Each subject was rated by his immediate supervisor on job performance as a programmer, future programming potential, potential for systems analysis and potential for management.  All ratings were made on a 7-point modified “letter-grade” scale, with ratings of A, B, C, D, and F permitted. Two intermediate categories (“Between A and B” and "Between B and C”) were added in an attempt to expand the variance of the ratings.  In nearly all cases, the programmers were rated before they were tested; thus this portion of the study was a test of the concurrent validity of the AABP.

 Results

 Training Performance

    For each of the seven programming classes, the class rank standings assigned to the members of each class were normalized according to the normalized-rank method (Guilford, 1954). This score transformation was utilized since rankings form a perfectly rectangular distribution; a normalized distribution of rankings is more likely to approximate the real distribution of performances among members of a class. Table 1 presents the correlations between AABP scores and normalized class ranks for each of the seven programming classes. Correlations ranged from .33 to .66 with a median of .44. When all classes were combined, the correlation between AABP numerical scores and normalized class ranks was .40.

 TABLE 1

Correlations between AABP Scores and Normalized Class Ranks in Seven Programming Classes

    Note.-For the total sample, mean AABP numerical score was 78.3 with a standard deviation of 14.5
    ^aFor most classes, the actual number of students was larger than the N which represent the number of students in each class who had taken the AABP.
     * p < .05.
    ** p < .01.

    When the AABP test is scored by its distributor, in addition to the numerical score a subjective adjectival rating is provided estimating the individual’s overall potential for computer programming. This adjectival rating is highly related to the numerical score on the AABP, although the rating takes into account additional subjective factors including the examinee’s “method, conception and approach which . . .may indicate sufficient superior aptitude to compensate for his lack of accuracy in the less challenging problems” (from the brochure of Programming Specialists, Incorporated, describing the AABP). That the adjectival rating is highly related but not completely equivalent to the numerical score is evidenced by a correlation of .94 between adjectival rating and numerical score for the overall training sample of 93 persons. Table 1 presents the correlations between these adjectival ratings and normalized class ranks for each of the seven programming classes, which range from .22 to .66 with a median of .44. When all classes were combined, the correlation between AABP adjectival ratings and normalized class ranks was .39, essentially the same as the .40 obtained with AABP numerical scores.

    Table 2 presents the relationship between AABP adjectival ratings and performance in programming training in expectancy table form. Individuals in all seven classes were divided into two groups - those who finished in the upper half of their classes and those who finished in the lower half. At one extreme, seven out of nine of those with the highest ratings on the AABP finished in the top half of their classes, while at the other extreme, only four out of 20 of those with the lowest ratings did so. Inspection of Table 2 suggests that  the AABP  adjectival  ratings were related to training outcomes in a manner consistent with the commonly-accepted meanings of these adjectives.

 TABLE 2

Relationship between AABP Adjectival Ratings and Instructors’ Rankings in Programming Courses

 TABLE 3

Correlations between AABP Scores and Supervisors’ Ratings of Performance and Potential for 57 Programmers

    Note.-For the total sample, mean AABP score was 81.4 with a standard deviation of 15.1
    ^aSome supervisors declined to rate individuals on all dimensions.
     * p < .05.
    ** p < .01.

Programming Performance

    Table 3 presents the correlations between AABP results and supervisors’ ratings of current performance and future potential for 57 programmers. Moderate-sized correlations, ranging from .40 to .46 were obtained between the AABP adjectival ratings and ratings of current job performance, programming potential, and systems analysis potential, with a smaller correlation (.30) emerging between the AABP and potential for management. Slightly smaller correlations were obtained between AABP numerical scores and ratings of job performance and potential, although in no instance was one of these correlations statistically significantly smaller than the corresponding correlation between AABP adjectival ratings and ratings of job performance and potential.

 Discussion

    The results of this study suggest that the AABP is a reasonably effective device for selection of computer programmers. Using independent samples, AABP test scores correlated significantly with both programming training and job performance as a programmer. No evidence was obtained supporting the tests distributor’s implied hypothesis that AABP adjectival ratings, which take into account subjective factors not considered in the numerical scores, are significantly superior to the numerical scores at predicting training or job performance of programmers. Since the two types of scores are so highly correlated, and since they correlate about equally well with training and job performance, they would appear to be interchangeable.

    Somewhat surprisingly, the AABP test was found to be as highly related to job as to training performance. This finding is in contrast to the research reported for the IBM Programmers Aptitude Test which has produced correlations ranging from the .30’s to the .50’s and .60’s with training performance (McNamara and Hughes, 1961; Katz, 1962; Hollenbeck and McNamara, 1965; Bauer, Mehrens and Vinsonhaler, 1968) but significantly lower relationships with actual job performance as a programmer (McNamara and Hughes, 1961). Since the correlations obtained in this study between the AABP and ratings of job performance and potential are comparable in size to those reported by McNamara and Hughes (1961), it may be tentatively concluded that AABP scores are as related to performance as a programmer as are scores from the Programmers Aptitude Test. It should be mentioned again, however, that the present study was a test of the concurrent validity of the AABP with regard to job performance and potential; its predictive validity in this domain remains to be established.  Meanwhile, the present study provided reassurance that the AABP - a widely used test despite its previous lack of any published validity research - appears to be a reasonably valid measure of computer programming aptitude.

     ¹The views expressed in this article are those of the authors and do not necessarily reflect an official position of the Central Intelligence Agency.  The authors gratefully acknowledge the assistance of Mr. Ed Ryan who provided training performance information used in the study.
    This paper is based upon a speech presented June 16, 1972 in Toronto, Ontario at the Tenth Annual Computer Personnel Research Conference (sponsored by the Association for Computing Machinery – Special Interest Group on Computer Personnel Research).
    ² Both authors are now at the Cleveland Clinic.  Requests for reprints should be sent to the Department of Psychiatry, Cleveland Clinic, Cleveland, Ohio, 44106.
   

REFERENCES

Biamonte, A. J. A study of the effect of attitudes on the learning of computer programming. In M. H.        Gotterer (Ed.), Proceedings of the third annual computer research conference.  Silver Spring, Md.:  Computer Personnel Research Group 1965. Cited by R. Bauer, W. A. Mehrens and J. F. Vinsonhaler, Predicting performance in a computer programming course.  Educational and Psychological Measurement, 1968, 28, 1159-1164.

Guilford, J. P. Psychometric Methods.  (2^nd ed.) New York:  McGraw-Hill, 1954.

Bauer, R, Mehrens, W. A., and Vinsonhaler, J. F. Predicting performance in a computer programming course.  Educational and Psychological Measurement, 1968, 28, 1159-1164.

Hollenbeck, G. P. and McNamara, W. J. CUCPAT and programming aptitude.  PERSONNEL PSYCHOLOGY, 1965, 18, 101-106.

Katz, A.  Prediction of success in automatic data processing course. U. S. Army Personnel Research Office Technical Note, 1962, No. 126.

McNamara, W. J. and Hughes, J. L. A review of research on the selection of computer programmers.  PERSONNEL PSYCHOLOGY, 1961, 14, 39-51.

Meyer, J. K. an experimental comparison of instructional techniques for use in teaching computer program flow chart design. USN PRA Technical Bulletin, 1965, No. 65-10.

Oliver, T. C. and Willis, W. K. A study of the validity of the programmer aptitude test.  Educational and Psychological Measurement, 1963, 23, 823-835.

Perry, D. K.  Evaluation of tests for improvement of programmer trainee selection. SDC Technical Memorandum 3570, System Development Corporation, 1967. (a)

Perry, D. K.  Vocational interests and success of computer programmers. PERSONNEL PSYCHOLOGY, 1967, 20, 517-524. (b)

Seiler, J. Abilities for ADP occupations. In M. H. Gotterer (Ed.), Proceedings of the third annual computer research conference.  Silver Spring, Md.: Computer Personnel Research Group, 1965. cited by R. Bauer, W. A. Mehrens and J. F. Vinsonhaler, Predicting performance in a computer programming course.  Educational and Psychological Measurement, 1968, 28, 1159-1164.  P. 1160.

Wolfe, J. M.  A new look at programming aptitudes. Business Automation, 1970, 17, 36-45.