1. Will you provide test cases for us to run our program on?  In particular,
    will you make public the test cases used by the auto-grader?

    No.  Writing test cases to verify a program is an integral part of
    writing and debugging the program.  Providing the specific test cases
    would be tantamount to doing this part of the project for you.

    The job of a grader (human or auto-grader) is to evaluate how well the
    student completed the project.  The purpose of a grader is clearly
    NOT to complete the project for the student.  For example, a grader
    evaluates how well a program works, but the grader doesn't re-write the
    program for the student.  In the same way, the grader evaluates how
    well the student verified the program, but this evaluation process
    shouldn't replace the verification performed by the student.

2. It's not fair to grade our program on input that you're not giving us!

    The range of input values that your program should work for is very clearly
    defined in the project handout.  It is not enumerated as specific test
    cases; rather it is defined as a universe of programs.  Your program should
    work for this entire universe of programs.  Since it is not possible to
    evaluate it on the entire universe of programs (which is infinite in size),
    we evaluate it on a few specific programs to give us a statistical sample
    of how it would perform over the entire universe.  You should be doing the
    same kind of evaluation yourself; conducting this evaluation is part of the
    project assignment.

3. If the auto-grader doesn't help me debug my project, what's it good for?

    The auto-grader's feedback is designed to inform the student of the
    project grade, so the student can make an informed decision about
    continuing to work on the project or not.  Any feedback other than the
    project grade should be considered as bonus.

4. Why do you limit the number of submissions that provide feedback?

    Even though the auto-grader is not intended to help debug, some students
    try to use it in this manner anyway.  Apparently, really frequent feedback
    about the numeric grade (plus any extra information it happens to provide)
    can be used clumsily in debugging.  The limit on number of submissions
    is intended to prevent use of the auto-grader in a trial-and-error manner.

    Trial-and-error debugging with a magical oracle is inefficient and
    unrealistic.  There are no auto-graders in the real world.  In the real
    world, you write test cases to debug your program, and you (hopefully
    methodically) track down the bug, fix it, and verify that it works on the
    test case.  Trying fixes at random should offend your sensibilities as a
    computer scientist.  Before you fix a bug, you should understand why it's a
    bug, and understand why the fix should work.  If you try a fix without
    completely understanding the bug and the fix, you're likely to get it to
    work for some but not all workloads, and the bugs that result will be even
    harder to find.

    The best way to debug a problem is to write a test case that triggers the
    bug, trace through the program to figure out why it failed, fix the
    bug, then re-run the test case to verify that the fix works.  This
    process is time consuming, so one submission per day should be plenty.
    You should catch most bugs without needing to submit to an auto-grader.

    The one-per-day submission policy is also intended to encourage you to
    start the project early.  Don't try to do the entire project on the due
    date!

    Finally, one submission per day is much more than you get in the real
    world (and in other classes, at least before we popularized auto-graders).
    Think of your submission to the auto-grader as shipping software to a
    customer -- how many buggy programs do you think a customer will tolerate
    before buying a competitor's program?  (Assume a non-monopolistic
    environment.  In a mononpolistic environment, there are no competitors, so
    some companies can ship buggy or inferior programs without fear of losing
    customers to competitors.  Does this happen today?)  :-)

5. Why are we required to write and submit a suite of test cases?

    First, you should have been writing a comprehensive suite of test cases in
    the normal course of writing your program.  So submitting this suite
    shouldn't be additional work.

    In practice, however, students tend to try to skip writing a good set of
    test cases.  This leads to poorly trained students.  Writing test suites
    and verifying programs is a major part of software development.  In fact,
    the main role of many software developers is to write test suites and
    verify programs.  As much manpower in industry is spent debugging and
    verifying programs as is spent writing them!

    Second, writing a good suite of test cases will help you debug your
    program.

    Third, writing a good suite of test cases will deepen your understanding
    of the program.  Thinking through the different possible behaviors the
    program can exhibit and designing test cases to generate these behaviors
    helps you thoroughly understand the program.

6. The auto-grader gave me a hint about some student test cases that exposed
    the student program as buggy.  What does this mean?

    The auto-grader runs your own test cases on your program.  This does not
    affect the grade; its purpose is to help students by informing you that
    your own test cases will trigger a bug in your program.  Ordinarily you
    should already know if your programs fails on your own test case, because
    you should have figured out the right answer for the test case in advance
    and compared your program's output to the right answer.  However, this may
    help alert you to a misunderstanding about the project specification.

    Once you know that your own test cases causes your program to fail, your
    job becomes much easier (most of my debugging time goes to constructing a
    test case that causes my program to fail repeatably).  You should figure
    out by hand the right answer (e.g. series of printStates) for this test
    case (you really should have done this while you wrote the test case), and
    see if your program matches this right answer.  If it does not match the
    right answer, you're well on your way to debugging the program.  If it
    DOES match the right answer, your "right answer" may be wrong.  If you
    can't figure out how your "right answer" could be wrong, see a TA or
    professor during office hours.

7. My test suite exposes most (or all) of the instructor buggy programs, my
    program works for all the test cases in the suite, but my program still
    isn't getting full credit.  How can this be?

    The set of instructor buggy programs is small (I can't generate as
    wide a variety of bugs as 100 students).  Your program apparently has
    a bug that is not represented in the set of instructor buggy programs,
    and your test suite is not exposing that bug.  Even though your test suite
    is getting full credit, you still need to write a test case that will
    expose your bug.

8. I'm trying to use submit377, but I get an error like
    submit377: Command not found.

    This is because the directory containing submit377 isn't in your search
    path.  In Unix, you can customize the set of directories that is searched
    to execute a command.  E.g. when you type "ls", the system searches for
    "ls" in several directories and finds it in "/bin/ls" or "/usr/bin/ls".
    You can type "echo $path" to see the current set of directories being
    searched.  If you change this path, you need to type "rehash" (if you're
    using csh or tcsh) to have the shell find the programs in the changed path.

    Your path variable is set when you login by your shell startup scripts
    (e.g. .cshrc and .login).
    
    On Edlab machines, add the appropriate directories to your
    path by adding the following line at the end of your .cshrc file.

set path = ($path /courses/cs300/cs377/cs377/bin)

9. I'm not getting replies from the auto-grader.

    First, check your mail on the edlab machines.  All your edlab mail
    goes to edlab.cs.umass.edu and you have to check it using an IMAP
    client (such as pine).  If you want the mail to go somewhere else,
    (like your @student.umass.edu account) you can place a .forward
    file in your home directory containing that email address.  For
    instance, I put mcorner@cs.umass.edu in my .forward file.  

    Second, it's possible the auto-grading machine is bogged down with
    lots of submissions; this tends to happen around project due dates
    for some inexplicable reason.  This might cause it to take a
    little longer than normal (e.g. a couple hours).  Be
    patient--think how long it takes to get feedback from human
    graders (about 2 weeks).

    Third, your program might have gone into an infinite loop on our test
    cases.  Of course, this doesn't happen for well-tested programs like
    EECS students submit :-)  If this does happen, the auto-grader will
    let your program run for a while before killing it.  So its response to
    you can take a while (e.g. 6 hours).

    Fourth, there might be a problem with the mail servers.  This is
    actually much more common than problems with the auto-grader.  Try
    sending mail to yourself, and see how long it takes for the mail
    to travel to your account.

10. Do we get one submission per day for all project parts, or do we get
    one submission per day PER project part?

    You can submit each project part once per day.  E.g. you can submit 1t
    (thread library), 1d (disk scheduler) in one day (without using any bonus
    submission).

11. How do I access the newsgroup from off campus?

Go here: http://www-edlab.cs.umass.edu/ and read the section on
"Usenet NEWS".  It will tell you how to access the newsgroup off campus.