From Brown University Robotics
CS148 Reports and Bugs in Scientific Writing
The purpose of the project reports is to gain familiarity with scientific writing. Now that you have completed your first assignment, we would like to address some general issues that commonly arise during grading. Almost everyone would benefit from improvement in at least one of these areas, but it is also important that everyone is aware of what we expect from the report.
With your grade, you received comments from the course staff about your report. Each abbreviation (listed under "Bugs in CS 148 Writing") listed in your comments section is a "bug in writing" the course staff identified in your report. The course staff would be happy to discuss our assessments
and help improve your writing skills, which will be beneficial for any career in computing.
Why Do I Need to Write for a CS Class?
Simple answer: the scientific method, or how could anyone understand and verify what you have done?
Your training in CS to this point has likely focused on engineering software systems (i.e., programming) and understanding what is logically possible (i.e., mathematics). As such, evaluation of your work is a relatively straightforward by a formal procedure: a mathematical proof is either logically correct or not, test cases for the (deterministic) output of a piece of software. However, formal evaluation with robots is not truly possible because they act in the (debatably nondeterministic) physical world. As such, your robot's behavior is not completely governed by the logic of your controller and is affected by the physical world. Therefore, an empirical study is necessary to support any claims about properties of your controller for acting in the physical world, as common for the natural sciences. That is, we use the scientific method for evaluation, which can be roughly stated as "don't trust what I say, reproduce and verify my results."
Robotics as a Formal and Natural Science
With your CS background, your programs have "made up the rules" that govern purely digital (virtual) domains in a logically valid manner. However, CS, as a "formal science", is distinct and complementary to the "natural sciences" that focus on "understanding the rules" that govern our world (or "laws of nature"). Examples of natural sciences include:
- physics strives to understand the true nature of the universe
- chemistry strives to understand the nature of matter
- biology strives to understand the nature of living organisms
Robotics can be considered both a formal and natural science and a discipline of engineering, as well as a social science. The creation of a robotic system is both an effort in engineering (to produce the robot's hardware/embodiment) and formal science (for defining the logical behavior of a robot's perception, decision making, and motion control). Once a robotic system is created, it becomes a phenomena of the natural world, whose properties must be studied empirically through the scientific method.
Bugs in CS 148 Writing
A quick list of shorthand we used to identify bugs in writing, focused towards technical content for CS 148. Please refer to Dupre's excellent Bugs in Writing book a more complete description of proper technical writing.
- VJ: Unsubstantiated value language. Value judgments should not be used. That is, use specific language that is unambiguous and can be supported objectively. Avoid using subjective phrases such as "it worked well", "the problem is simple and straightforward", or "the robot behaved intelligently". Such descriptions are subjective and their meaning can vary based on the reader's interpretation. More importantly, value judgments have no specific definition and cannot be measured. In general, do not use vague or relative language to describe the properties of your work. Avoid narrating over your own observations from the experiment; let your data do the talking. Videos are nice and very helpful, but alone are not sufficient for quantitatively evaluating your core hypothesis.
- QB: Excessive complaining. Complaining is not necessary and counter-productive. Constructive feedback about the assignment, equipment, ROS, etc. is always welcome. However, excessive or unbalanced criticism becomes subjective will not be tolerated. Again, let your data do the talking.
- CO: Compositional organization needs improvement. DO NOT rigidly follow the questions in the project rubric, treating it like a survey. The report should be a written composition in paragraph form, more similar to an essay. The report should be properly sectioned into: Introduction, Approach, Experiment, Discussion and Conclusion. Specifically, the Introduction should state what you are testing and why it is important. The hypothesis must be clearly stated in the Introduction. The Experiment should state the metrics you will use to validate or invalidate your hypothesis. The Conclusion should be a summary of your central thesis, whether its valid or invalid based on what you measured in the Experiments section, and what was learned from the project. Secondly, we also expect basic compositional skills such as spelling, grammar and documentation organization. Additionally, illustrative and interesting visuals (pictures, illustrations and movies) and tables describing the trials/metrics help us quickly understand your methods and results.
- RD: Lack of reproducibility of methods. Reproducible description of the methods employed. We do not expect excessive detail or verbosity. However, it is expected that any capable upper-level CS ugrad student should be able to reproduce your system (within reason) and results from the description in the report. You should be sure to describe the inputs and outputs of your computational problem, how inputs are algorithmically transformed into outputs (e.g., pseudocode, flow diagram, textual description), and any important parameters or other details in your specific implementation. This composition roughly follows Marr's three levels of analysis.
- RS: Do not state your work is reproducible. Related to VJ, it is irrelevant to make claims about the reproducibility of your work. Reproducibility is established by others' use of your description to produce a similar implementation that will yield similar results. When grading, we use reproducibility as a criteria for the completeness of your description, in terms how well another student (not necessarily you) could implement this description.
- DB: Description in report conflicts with implementation code and/or robot behavior.
- CL: Overly colloquial language. avoid informal loose language "we felt our system should X" should be "our algorithm was designed with X property". We do not really care how you feel and are focused on what you did. From The Elements of Style:
Vigorous writing is concise. A sentence should contain no unnecessary words, a paragraph no unnecessary sentences, for the same reason that a drawing should have no unnecessary lines and a machine no unnecessary parts. This requires not that the writer make all his sentences short, or that he avoid all detail and treat his subjects only in outline, but that every word tell.
- TH: Lack of a testable hypothesis. Your report lacked a central testable hypothesis with a quantitative metric that evaluated (validated or invalidated) through measurable evidence. In other words, what is the question your are evaluating with the experiments? We do not provide a specific hypothesis and expect you generate this question on your own. Qualitative descriptions or subjective value judgments (e.g,. "it worked well") are insufficient for evaluating a hypothesis. Individual trials should be evaluated using quantitative metrics, such as time to completion, "success" rate (do not forget to define success), or number of collisions. Aggregate results from multiple trials should also be stated using statistics such as mean and variance across trials. Tables with aggregate results are especially valuable to readers of your report.
- HC: Conclusion not related to hypothesis The conclusion must tie into the original hypothesis stated in the introduction. Think of the conclusion as the introduction looking back on the experiment and summarizing what was learned.
- PR: Proof reading your write up is important. Spelling, etc
- CS: Incorrect submission format. We expect correct submission of your report, namely to be in HTML (or pdf) format and in the correct location. Please refer to the missive and assignment description for proper submission instructions.
- more common bugs are likely to arise
Features in CS 148 Writing
To balance the positive with the negative, we also have a list of shorthand to compliment "features" in your reports:
- CC: clear and concise report.
- TR: thorough analysis that goes beyond the course staff's expectations.
- II: interesting/creative idea that seeded discussion among the course staff.
- ND: illustrative diagrams, figures, and/or visualization
- VA: videos appreciated by the course staff.
If you have questions about scientific writing and our report evaluation procedures, please consult with the course staff.