The emergence of large-scale classes and distance education has created
problems that cannot be adequately addressed by the traditional student-teacher
based model of interaction. Students in large-scale classes and distance
education environments do not receive the same kind of interaction with the
instructor they may have received in smaller, more traditional classroom
settings. Our experiences have resulted in "lessons learned" and solutions that
address issues that arise in distance and large-scale education. This paper
presents these solutions and an in-progress system that facilitates the
interaction and information flow of students, raters, and instructors.
Our university has made the CS1 class part of its core curriculum for all students and the CS2 class as required curriculum for many students; as a result, the CS1 and CS2 class sizes have grown at enormous rates. Within the last four years our College of Computing's CS1 class has grown from approximately 100 students per year to over 1400 students per year. Four years ago, the class required one professor lecturing and evaluating all the students' work; now the class requires a lecturing professor and between 30 and 40 teaching assistants (TAs).
As a result of such massification of the classes, we have learned that the old model of student-teacher interaction that works well for small classes does not work well for large classes.
Distance education is characterized by geographic displacement of the students and instructors, and large-scale classes often separate students from the instructor due to lack of time or resources. Whether the students are hours or minutes away from an instructor, a few hours a week is not adequate for one instructor to meet with hundreds of students. Human intimacy between students and teachers is often lost in large classes.
Since students do not get the same interaction with the instructor in the classroom, among other things it is extremely important that the student get detailed and constructive feedback on their graded assignments. Price and Petre make the point that detailed feedback on assignments is critical in distance education [6], but the need for detailed feedback also applies to large-scale classes.
In an effort to compensate for the loss of interactions between students and
teachers found in small-classes, the system described in this paper provides
instructors and students with detailed information about student performance,
facilitates multiple people evaluating students' work, and helps administrate
large classes. It is our opinion that distance education and large-scale classes
can benefit from using this system.
For the purpose of clarity in this paper, we define marking student work as
the process of noticing errors, providing corrections, and assigning numerical
grades. We will use the terms TA and rater to denote an assistant to the
instructor who evaluates s tudent work.
Figure 1 shows how the students and the teacher interact in a traditional
small classroom. Instructors create assignments and students complete the
assignments and submit them to the instructor. The instructor then marks the
assignments, records the grades, and returns the graded assignments to the
students; students learn by reviewing their mistakes. This process of assignment
flow is indicated by the solid arrow in Figure 1.
There is a lot more that is involved in the learning process than just the flow of assignments; there is a very human aspect to the nature of learning. We believe instructors internalize the impressions students make by observing student behavioral cues in the classroom and performance on assignments, and students internalize the feedback they get from the instructor and other students.
The dashed arrows in Figure 1 represent the students' and instructors'
interactions and internal reflection. This introspection may or may not
consciously occur, but it may help instructors improve the courses they teach
and may help students learn.
In a large-scale class or distance education, the instructor no longer has the same interaction with the students; most of the human interaction aspect of the small class is lost, and the instructor often loses valuable information about how well students are learning the course material. The instructor must now distribute criteria to multiple raters; since the instructor is no longer marking the students' papers, the instructor often does not have the opportunity to obtain qualitative information about student performance. TAs often submit a numerical grade to the instructor, giving the instructor no indication as to the types of errors students make and topics students do not understand.
In larger classes, students may have a feeling of anonymity, and instructors may not receive adequate behavioral cues feedback from the students during lecture. Students miss opportunities to learn because they lose the human interaction with the instruc tor and other students.
Figure 2 depicts how the instructor in the large classroom may not get
qualitative feedback on student performance; instructors typically only get
information-poor grade summaries from TAs; these grade summaries do not offer
the instructor the same type of detailed information that the instructor was
able to gather in the small class environment. 
In addition to removing much of the communication typical in small classes, large-scale classes and distance education also introduce new problems. Distributing criteria to multiple raters also raises the issue of consistency and inter-rater reliability. Since different raters interpret the instructor's criteria slightly differently, each rater can grade the same student's work entirely differently, often with as much as three or four letter grades disparity (i.e. a range of up to a 40% of the weight of the assignment) [5]. Given the problem of inter-rater reliability, the information that instructors receive may be inconsistent and little more than noise.
This loss of qualitative information about student performance makes it more
difficult, if not impossible, for instructors to adjust their class to suit
students' needs; it also makes it more difficult for instructors to improve the
assignments and other instructional material.
Our system supports the large-scale class environment by facilitating the interactions between students, raters, and instructors; we use newsgroups and anonymous surveys to bolster the communication in the class, and an on-line marking program and an information repository to improve re-use and quality of the class and feedback. Our system also facilitates the administration of the class by allowing students and raters to submit, test execute, confirm receipt of, and retrieve assignments electronically similar to the system described by Dawson-Howe [1].
The student-teacher interactions are reduced, and the social context of the traditional classroom is affected in large-scale and distance education. As a result, it is important to emphasize the interaction between students in an effort to compensate for the lack of instructor interaction [2].
Our classes use multiple newsgroups as a medium in which students and instructional staff interact. Three newsgroups were established: an announcement newsgroup that contains important course-related information (changes in assignments, etc.), a question and answer newsgroup that contains students' course-related questions to which other students and TAs post answers, and a general discussion newsgroup that allows students to discuss anything - course-related or not. These three newsgroups were necessary to delineate required reading (the announcement newsgroup) from help and general conversation messages.
Anonymous surveys provide a means of capturing information from students that is prevalent in small classes but usually lost in large classes. Anonymous surveys establish a safe environment for students to tell the truth about their feelings and problems with the class. From these surveys we can learn useful information about student perception of the class, fairness of assignments and grades, how long students worked on an assignment or studied for a test, and other such useful information from the students.
These surveys also serve as a check and balance for raters. If a rater is not getting graded assignments back to students in a timely manner or is not grading fairly, then the instructor can be notified of the problem situation by anonymous student feedback. Without the surveys, a problem situation could persist for months before the instructor is finally alerted.
Since instructors rely on raters for proper performance feedback, it is critical that TAs are consistent in their rating of student work [4]. As a result, we have developed an on-line grading environment that allows an instructor to explicitly specify th e criteria to a rater.
An expert instructor can designate standardized and appropriate responses which are automatically added to students' work as a rater recognizes errors in the students' work; the system re-uses examples and corrections imbedded in the criteria by the expert instructor. By separating the task of evaluating a student's response from penalty distribution (i.e. taking off points), the rater only has to worry about one task. The rater sees the aspects of an assignment that the instructor wants evaluated in the student work. Points are imbedded in the system so that the raters do not take off points; the raters are noticing errors in student work, and the grading program automatically takes off the correct amount of points for each error. We have found that this on-line system for marking decreases the variance among raters by up to 33%, improving inter-rater consistency and reliability.
In a traditional pen and paper methodology of marking, a rater often makes many notes and corrections on students' work, and the information about these marks is usually lost. If these marks are to be recorded and used later, then a rater must transcribe and categorize the marks into a database - a laborious and time consuming task. But an on-line marking program can automatically add the marks a rater makes to students' papers into the database.
The on-line marking program presents the raters with a series of dynamic questions about the student work; thus depending on how the rater answers one criteria question, a new series of questions can be asked for clarification. For example, if the rater notices an error in one aspect of a student's work, the system could then query the rater with a more detailed question to obtain a better assessment of the error.
In addition to improving the consistency of raters, an on-line marking system can also improve the speed in which assignments can be processed by administrators and raters [6].
We designed a repository-based system to give instructors information that they may not have in typical large-scale and distance education environments. In order to facilitate re-use of problems and criteria, the system tracks assignments, problems, and criteria. This system allows the expert instructor to create assignments and criteria for the assignments and store this information into a central database. Students then log into the system to retrieve their assignments. There is a valid time-period for each assignment that enforces students turn in assignments on time. The security of the system prevents students from accessing the criteria for the assignments. The student then submits the completed assignment into the database. Later, a rater can access the system and obtain the criteria and students' completed assignments.
It is possible to pre-process the students' work if desired. For example the students' programs can be pre-compiled and executed, and summary reports of compile and run-time errors can accompany the students' work to the rater. Pre-processing the student work is beneficial to the raters. Computers are better suited to find some errors, such as indentation errors. The system could automatically make note of the error and correct the problem before the rater ever sees the student's work. For example, the system could pre-processes the student work and remove any indentation errors; as a result the rater would be able to grade a more readable assignment [1]. Of course the instructor should be able to turn off this pre-processing feature if desired.
Once the rater has finished marking the students' work using the on-line marking program, the rater uploads the graded assignments to the repository, and performance information is automatically entered into the system based on the items that the rater noticed. Now a student or the instructor can query the repository for statistical information about performance on the assignment.
Figure 3 shows the new repository-based interaction system. Key elements of
large-scale classrooms such as newsgroups, TA help sessions, the anonymous
survey program, and the administrative program for handling submission and
retrieval of assignments are not shown in this figure, but are certainly a
necessary part of the entire classroom environment. 
The primary difference between the system described in Figure 3 and the old system described in Figure 2 is that the new system allows instructors and raters to query a central repository of statistical information concerning the class. Instructors can now ask the system such questions as, "Is there a correlation between students' perceptions of assignment difficulty and rater consistency?" The system gives the instructors the ability to obtain qualitative class information that they may not have in typical large-scale and distance education environments.
The dashed arrows in Figure 3 show that the system allows the instructor and
the raters to reflect on class statistics. Whereas in a small class the
instructor could do this because he was personally involved with all the
students, querying the repository is one of the only ways of obtaining such
information in a large-scale class. Raters can reflect on their performance in
evaluating student work; instructors can reflect on student knowledge and
improving the class; and students can reflect on detailed and meaningful
feedback on their assignments.
The new repository-based system for managing large classes opens up new possibilities in education. The system provides short-term benefits in improving communication and interaction in large classes and allowing customizable assignments; it provides mid-term benefits in helping optimize the class by providing information useful to instructors and raters; and it provides long-term benefits by providing an opportunity for research in new approaches for distance and large-scale education and investigation into learning issues.
Another key feature of the repository-based system is that student performance can be tracked with a fine level of detail. Since the raters are able to be more accurate and consistent in their marking of student work, the statistical data collected on student knowledge is more valid. Once good information about student performance is collected, then the system can automatically assign remedial problems to students who have not mastered important topics. For example, if a student performs poorly in the area of "procedure declarations" on assignment 1, then the system can automatically assign two or three remedial problems on "procedure declarations" in assignment 2 for that student.
Thus each student received individualized assignments that are tailored to his needs. Each assignment contains a number of problems that all students receive; students who are performing poorly in some topics will receive additional problems added to the end of their assignment; and students who are performing well on all the topics will be rewarded by having shorter assignments (i.e. not having to do the extra problems).
Since the raters will have to evaluate different problems for each student, it is impractical to distribute criteria and assignments on paper if the set of remedial problems is large. Without the on-line marking program, the set of remedial problems would have to remain small enough to distribute criteria to all raters. In preliminary studies, we were only able to distribute three or four different remedial problems per week using paper-based marking. But the on-line marking program allows raters to handle the large volume of different remedial problems for each student.
We have found that providing students with remedial problems does help students learn the material [7].
As the Figure 3 shows, statistics from the repository can be used by other, future classes. Future instructors can use the statistical information of student performance in past semesters to help plan their class.
Traditionally, if a student achieved a required grade, then the student passed a course and progressed onto more difficult topics. But a student can learn most of the material and still pass the class without learning the other important topics. Using the information in the repository, instructors can help students by making sure that students learn all required topics before progressing to a more difficult class. Our goal is to use this system to build a portfolio of student work and performance tha t can be used in the future.
Also, instructors can learn how their students performed on previous topics in previous classes. For example, if a student is having trouble with a topic in an upper-level class, an instructor can query the repository and find that the student did not learn a prerequisite topic in a previous class; the instructor can then assign remedial work to help the student.
This system allows inter- and intra-curricular collaboration and course re-use. Other universities can use course statistics collected using this system to learn from the instructors' experiences, effectively "borrowing" mature classes that have already used the system to improve how they teach in a large-scale class environment.
Once a repository of detailed and accurate student performance data is obtained, then instructors can use data mining to learn about trends in class performance and patterns in student learning. Discovering information would further be enhanced if multiple universities joined their repositories into a central data warehouse [3]. Without valid performance data, data mining is not possible; but this new system of retained and accurate information opens new frontiers in educational research.
From data mining performance repositories, instructors can plan classes and
distribute lecture time to topics based on statistics of previous semesters; for
example, if previous students do well on topic A and poorly on topic B, then the
instructor can plan to spend less class time on topic A and more class time on
Topic B. Instructors may also learn about topical dependencies and schedule
class lectures appropriately. Instructors may find learning patterns in students
that enable courses to be tailored to meet individual students' needs.
We have discovered some new issues and problems that arise when using this system which mainly center around raters not readily embracing change in the class. We found that raters often complain about the change in interface when using the on-line marking program, and raters do not like being constrained to using the computer to mark the students' work. We are looking into ways of improving the marking program to better suit raters' preferences. Also, the raters complained about having to issue a series of complex FTP commands to retrieve the needed student work and criteria files used by the on-line marking program; so we are changing the on-line marking program to be fully networked and retrieve the needed files with a "click of a button."
The system has the potential of allowing raters to "hide behind" the criteria. If students come to the raters complaining about the grades they received, then the raters could respond, "I was just following the criteria." But this line of argument misses the point entirely. The on-line marking system separates the marking of errors from the distribution of grades. The instructor defines the criteria and the points to deduct, and the rater notices the errors in the students' work. So when a student presents a complaint concerning a grade, a better response by the rater would be, "You did make the error, so if you want to talk about the fairness of the deduction, please talk with the instructor."
Another potential problem with the system is that raters may feel
disenfranchised by their seeming lack of control over the evaluation process.
But the system allows the raters to give immediate feedback to the instructor
concerning assignments and criteria if there is a problem. For example, what
happens when the criteria does not deal with a particular student's error?
Without the system, a rater probably notices the error and deducts an arbitrary
number of points and probably does not inform the instructor of the criteria's
shortfall; with the system, a rater notices the problem with the criteria, makes
a note to the instructor via a pop-up box, and then places a special marker in
the student's work. The instructor (or some other course administrator) can then
correct the fault in the criteria and the appropriate mark in the student's work
is automatically added. This improves the criteria for the future and assures
that the students are getting the correct marks and grades on their assignments.
This paper presented some of the problems that accompany the massification of
the small class environment. The system described in this paper allows
instructors to gain insight into students' performance and opinions that are
often lost in a traditional large-scale class environment. The system allows for
quality control of questions, criteria, and assignments. It also allows
instructors to collect more reliable and consistent performance evaluation from
raters by separating the objective assessment of student work from the
subjective differences in raters. Performance portfolios can be used to plan
future quarters and track students' progress in the university. The system is
not limited to CS1 and CS2 classes. We hope to expand the scope of the system to
allow other classes and universities to improve their distance and large-scale
classes. The system has the potential of opening new areas of research into
teaching and how students learn.
We are very grateful to all the TAs who took the time to use and provide
feedback on how to improve the components of this system.