There is a typically contentious article up on Slashdot this morning discussing the subject of testing IT job candidates, which is worth a read for the CIO responsible for hiring and firing such staff.

My take is that it is too blanket a conversation to be having, although the matter is clearly an important one on both sides of the table. But the fact is that there are some jobs which you can reasonably test people on before they are hired, and others were only their experience and whatever Socratic dialogue you can come up with during the interview are worth much. The higher you go in the hierarchy, the less you can rely on tests. It's reasonable to sit a tech support candidate down in front of a computer (an unplugged computer; always my favorite) and say, "Figure out what is wrong with this and fix it." It's more difficult to do with an IT manager who may be responsible for fixing a whole department.

The problem is that technology is broad and oddly enough, less codified in some ways than other professions where testing is the norm (medicine, law, accounting, etc). For a given problem in this field, there may be a dozen equally acceptable technical solutions, and more emerging each day as the state of the art advances. It's extremely difficult for any test to judge multiple right answers, or to select among the gradations of right to find the ideal answer. Beyond that, while you can test knowledge in such a fashion, it's very difficult to test the qualities that really matters in such a dynamic environment, such as adaptability and intelligence. You may bring in a candidate who has never worked on systems similar to yours before and he may fail every knowledge question on them that you can ask; yet he may be a better candidate than the one who knows the system inside and out, because he may be able to learn faster and adapt better... and if you think your systems are going to remain static for long, you must be new here. The ability to walk in cold on a new job and pick it all up from first or second principles is by far the most impressive and valuable skill in information technology today; it's also the one that is least likely to be demonstrated in formal testing.