Intelligent Tutoring Systems (e.g., in physics or algebra) typically use crystalized human expert knowledge and skills as the target of instruction. The "intelligence" in the systems comes from knowing how to solve problems and/or from correctly diagnosing deviations from expert performance. In more open-ended domains like writing instruction, our current abilities in AI are not up to the task of solving the problem on their own nor automatically diagnosing deviations from expert performance (nor are they likely to be up to these tasks in the near future). In those domains, one can layer a tutoring system on top of reciprocal evaluation - peer students diagnosing problems and suggesting solutions, and the computer assists the diagnosis and suggestion process rather than directly implementing it itself. In this situation, can one apply artificial intelligence into the process, and what would "intelligence" mean in this situation?
I argue that one can think of AI as usefully playing three roles: gating, weighting, and shaping of human intelligence. All three of these roles are different from the typical role it plays in intelligent tutoring. In the talk, I will give examples from a web-based system for scaffolding reciprocal evaluation in writing instruction called SWoRD that we have tested and evaluated in a number of university courses.