Add draft tutorial text

src/docs/text/randoop-dec-2016.tex

@@ -0,0 +1,287 @@
+\documentclass[11pt, oneside]{article} % use "amsart" instead of "article" for AMSLaTeX format
+\usepackage{geometry}                        % See geometry.pdf to learn the layout options. There are lots.
+\geometry{letterpaper}
+\usepackage{fullpage}
+\usepackage{url}
+
+\usepackage[parfill]{parskip}        % Activate to begin paragraphs with an empty line rather than an indent
+\usepackage{graphicx}        % Use pdf, png, jpg, or eps§ with pdflatex; use eps in DVI mode
+                % TeX will automatically convert eps --> pdf in pdflatex    
+\usepackage{amssymb}
+
+%
+\newcommand{\code}[1]{{\texttt{#1}}}
+\newcommand{\cmd}[1]{{\texttt{#1}}}
+
+
+\title{Randoop Tutorial}
+%\date{December 7, 2016}              % Activate to display a given date or no date
+
+\begin{document}
+\maketitle
+%\section{}
+%\subsection{}
+
+\section{Introduction}
+Randoop is a random test generator for classes written in Java.
+When Randoop is given a collection of classes, it will generate tests that either identify suspected failures in error-revealing tests, or capture behavior as regression tests.
+This tutorial introduces how Randoop works, and how to work with Randoop to control what happens during generation.
+
+\section{Getting Started}
+The tutorial involves generating files, and so we recommend that you create a directory where you are going to work:
+\begin{verbatim}
+  mkdir randoop-tutorial
+\end{verbatim}
+We'll do everything in that directory, and the tutorial assumes all of the files are there.
+
+In order to follow along, you will need the Randoop jar file, and the input classes for which tests will be generated.
+So, to get started, you'll need to download the following files:
+\begin{itemize}
+\item the Randoop jar file \texttt{randoop-all-3.0.8.jar} from the latest release of Randoop located at \url{https://github.com/randoop/randoop/releases/latest}.
+\item the sample jar file \texttt{tutorial-examples.jar} from the latest release of the Randoop tutorial located at \url{https://github.com/randoop/tutorial-examples/releases/latest}.
+\end{itemize}
+Place these files in your working directory
+\begin{verbatim}
+  mv [download-directory]/randoop-all-*.jar randoop-tutorial/
+  mv [download-directory]/tutorial-examples.jar randoop-tutorial/
+\end{verbatim}
+where \texttt{[download-directory]} is the directory to which the jar files were downloaded.
+And, then change into the directory you created
+\begin{verbatim}
+  cd randoop-tutorial
+\end{verbatim}
+
+To make sure that everything is setup correctly, run the command
+\begin{verbatim}
+ java -ea -cp tutorial-examples.jar:randoop-all-3.0.8.jar randoop.main.Main help
+\end{verbatim}
+This should print the following
+\begin{verbatim}
+Randoop for Java version 3.0.8.
+
+Randoop  is a command-line tool that creates unit tests for Java.
+It accepts one of the commands listed below. For the user manual,
+please visit https://randoop.github.io/randoop/manual/index.html
+
+Type `help' followed by a command name to see documentation.
+
+
+Commands:
+
+gentests -- Generates unit tests for a set of classes.
+
+help -- Displays a help message for a given command.
+\end{verbatim}
+
+
+
+\section{Generating Tests}
+To generate tests, run the following command 
+\begin{verbatim}
+ java -ea -cp tutorial-examples.jar:randoop-all-3.0.8.jar randoop.main.Main gentests \
+ --testclass=math.Int --junit-output-dir=out/int --inputlimit=20
+\end{verbatim}
+This will generate a small number of tests using the constructors and methods of the \texttt{math.Int} class; writing them to the subdirectory \texttt{out/int}.
+
+After running the command, you will see two files in the directory \texttt{out/int}:
+ \texttt{RegressionTest.java} and \texttt{RegressionTest0.java}.
+The file \texttt{RegressionTest0.java} contains the generated tests, while  \texttt{RegressionTest.java}  describes the JUnit4 test suite consisting of  \texttt{RegressionTest0}.
+
+\paragraph{A note on classpaths:} 
+Most of the questions that users have about problems with Randoop involve the Java classpath.
+The classpath is a list of directories and jar files that indicate where the compiled class files are located.
+We will always use the same classpath for this tutorial, which is
+\begin{verbatim}
+  tutorial-examples.jar:randoop-all-3.0.8.jar
+\end{verbatim}
+for a Unix/Linux environment.
+For Windows, write the classpath as
+\begin{verbatim}
+  "tutorial-examples.jar;randoop-all-3.0.8.jar"
+\end{verbatim}
+with the semicolon instead of the colon between path elements.
+
+If you run the above command without the \texttt{tutorial-examples.jar} file using the command
+\begin{verbatim}
+java -ea -cp randoop-all-3.0.8.jar randoop.main.Main gentests --testclass=math.Int \
+ --junit-output-dir=out/int --inputlimit=20
+\end{verbatim}
+Then you will get an error message like this one
+\begin{verbatim}
+Error: No class with name "math.Int" found on the classpath
+       This is most likely a problem with the classpath. It may be wrong, or
+       it is formatted incorrectly on the command line. The other possibility
+       is that the wrong class name is given.
+\end{verbatim}
+As mentioned this message will also occur when the class name is given incorrectly, but a malformed classpath is often the culprit.
+
+
+\subsection{The generated sequences}
+What Randoop did for this command is generate 20 code sequences from which attempted to make tests. 
+Randoop first selects a constructor or method, and then chooses values for arguments from those available .
+For instance, during this run, Randoop selected the constructor \code{math.Int(int)} with an argument of \code{1} to form the sequence
+\begin{verbatim}
+   math.Int int1 = new math.Int(1);
+\end{verbatim}
+Since this sequence runs without an error, the value it creates could be used to form a new sequence when \code{math.Int.getIntValue()} is selected:
+\begin{verbatim}
+   math.Int int1 = new math.Int(1);
+   int int2 = int1.getIntValue();
+\end{verbatim}
+With each new selection of a constructor or method, the generated sequences grow longer because Randoop will use the values created by sequences generated earlier. 
+For instance, \code{RegressionTest0.test6()} has the sequence
+\begin{verbatim}
+  math.Int int1 = new math.Int(1);
+  math.Int int3 = new math.Int(1);
+  int i4 = int3.getIntValue();
+  math.Int int6 = new math.Int(1);
+  int i7 = int6.getIntValue();
+  math.Int int8 = int3.add(int6);
+  math.Int int9 = int1.add(int6);
+  java.lang.String str10 = int1.toString();
+\end{verbatim}
+which is build for a selection of \code{math.Int.toString()} with an argument chosen from a longer sequence (built for a call to \code{math.Int.add(Int)}).
+(Note that the generated sequence doesn't necessarily use the most recently created value in the input sequence.)
+
+When a sequence is generated, it is executed using Java reflection, and with the results evaluated to determine if the sequence has an error or not. 
+If not, the sequence is considered a regression test, otherwise it is considered an error-revealing test.
+An exception is not necessarily a error, since some are expected; for instance, a \code{NullPointerException} can reasonably be expected when \code{null} is passed as an argument to a method. 
+
+When writing the test classes, Randoop only includes the longest sequence containing that sequence.
+For instance, for the three sequences above, since the first two belong to the longer sequence in \code{test6()}, neither will be written.
+
+The evaluation of the results of the sequences also results in assertions being attached to the sequence. 
+For sequences that become regression tests, these are assertions about values that are observed in the sequence.
+For \code{test6()} these are
+\begin{verbatim}
+  org.junit.Assert.assertTrue(i4 == 1);
+  org.junit.Assert.assertTrue(i7 == 1);
+  org.junit.Assert.assertNotNull(int8);
+  org.junit.Assert.assertNotNull(int9);
+  org.junit.Assert.assertTrue("'" + str10 + "' != '" + "1"+ "'", str10.equals("1"));
+\end{verbatim}
+which check the primitive and \code{String} values, and that returned objects are not \code{null}.
+
+
+\subsection{Refining the assertions}
+We can direct the generation of assertions somewhat by providing methods that should be used as \emph{observers}.
+These should be side-effect free methods that return attributes of the object that can be tested in assertions.
+
+In the class \code{math.Int}, the method \code{math.Int.getIntValue()} is a suitable observer.
+To use it in test generation the method needs to be given in a file, so run the command
+\begin{verbatim}
+  echo "math.Int.getIntValue()" > intobservers.txt
+\end{verbatim}
+Now when Randoop is run, we specify this file
+\begin{verbatim}
+java -ea -cp tutorial-examples.jar:randoop-all-3.0.8.jar randoop.main.Main gentests \
+ --testclass=math.Int --junit-output-dir=out/int --inputlimit=20 \
+ --observers=intobservers.txt
+\end{verbatim}
+This results in tests that show the behavior of the class more directly, as in
+\begin{verbatim}
+public void test10() throws Throwable {
+    math.Int int1 = new math.Int((int)'4');
+    math.Int int3 = new math.Int(100);
+    math.Int int4 = int1.add(int3);
+    org.junit.Assert.assertTrue(int1.getIntValue() == 52);
+    org.junit.Assert.assertTrue(int3.getIntValue() == 100);
+    org.junit.Assert.assertNotNull(int4);
+    org.junit.Assert.assertTrue(int4.getIntValue() == 152);
+  }
+\end{verbatim}
+which shows the result of the add operation of two distinct values.
+
+
+\subsection{Setting Limits}
+Randoop provides different ways to limit how many tests will be generated.
+The approach we recommend is the one we have seen so far, which is to set \cmd{--inputlimit}.
+This determines the maximum number of sequences that Randoop will consider as a test.
+In the example class that we have been using, this works well, but sometimes the definition of the class means that a generation limit cannot be reached.
+
+In these situations, Randoop uses a time limit, set with \cmd{--timelimit}, to determine when to stop.
+But, the default value is relatively large, so we should set a time limit for how long we are willing to let Randoop run each sequence. 
+Choosing a time limit that is too small can result in Randoop being non-deterministic between runs, but it turns out a minute is often enough for most circumstances.
+So we can change our command to 
+\begin{verbatim}
+java -ea -cp tutorial-examples.jar:randoop-all-3.0.8.jar randoop.main.Main gentests \
+ --testclass=math.Int --junit-output-dir=out/int --inputlimit=200 \
+ --observers=intobservers.txt --timelimit=60
+\end{verbatim}
+choosing \cmd{--inputlimit} to be a larger value to increase the diversity of the tests.
+
+
+\subsection{Contract Violations}
+As we mentioned in the introduction, Randoop is able to discover test sequences that exhibit apparent errors. 
+To see an example of this, run Randoop using this command
+\begin{verbatim}
+java -ea -cp tutorial-examples.jar:randoop-all-3.0.8.jar randoop.main.Main gentests \
+ --testclass=math2.Int --junit-output-dir=out/int2 --inputlimit=20  \
+ --observers=intobservers.txt --timelimit=60
+\end{verbatim}
+which uses a version of the \code{Int} class that has a buggy \code{equals} method (the previous version had none), and does not override \code{Object.hashCode()}.
+As a result, this run gives three error-revealing tests, all based on the failure of the contract that says that hash codes should be the same for two objects when they are equal (in other words, satisfy \code{equals}).
+One of these tests is
+\begin{verbatim}
+public void test1() throws Throwable {
+    math2.Int int1 = new math2.Int((int)(short)-1);
+    math2.Int int3 = new math2.Int((int)(short)-1);
+    java.lang.String str4 = int3.toString();
+    math2.Int int5 = int1.add(int3);
+    boolean b7 = int3.equals((java.lang.Object)0.0f);
+    
+    // Checks the contract:  equals-hashcode on int1 and int5
+    org.junit.Assert.assertTrue("Contract failed: equals-hashcode on int1 and int5", 
+        int1.equals(int5) ? int1.hashCode() == int5.hashCode() : true);
+} 
+\end{verbatim}
+The final assertion documents the fact that the objects \code{int1} and \code{int5} satisfy \code{int1.equals(int5)} but that \code{int1.hashCode() == int5.hashCode()} fails.
+In addition to this contract, Randoop checks a number of other contracts, each based on the JDK documentation.
+
+Note that error-revealing tests are intended to document failures, and should always fail when run. If, for some reason, these tests are not useful, they can be disabled by setting the flag \cmd{--no-error-revealing-tests}. 
+
+
+\section{The Difficult}
+
+There are methods for which randomly generated tests are problematic.
+Randoop has ways to deal with most of these.
+
+\subsection{Omitting methods}
+One way to deal with a method that we don't want to have included in a test is to omit it.
+The class \code{math3.Int} has a method \code{dontCallMe()} that calls \code{System.exit()}.
+Run this command 
+\begin{verbatim}
+java -ea -cp tutorial-examples.jar:randoop-all-3.0.8.jar randoop.main.Main gentests  \
+--testclass=math3.Int --junit-output-dir=out/int3 --inputlimit=20  \
+--observers=intobservers.txt --timelimit=60
+\end{verbatim}
+Don't worry if you don't find the output directory -- it is not there!
+Randoop used reflection to execute the method that then called \code{System.exit()}, and so Randoop stopped.
+
+The following command uses the \cmd{--omitmethods} command-line argument to indicate that the method should be skipped when selecting methods during test generation.
+\begin{verbatim}
+java -ea -cp tutorial-examples.jar:randoop-all-3.0.8.jar randoop.main.Main gentests  \
+ --testclass=math3.Int --junit-output-dir=out/int3 --inputlimit=20 \
+ --observers=intobservers.txt --timelimit=60 --omitmethods="dontCallMe\(\)"
+\end{verbatim}
+Run Randoop with this command should terminate as before.
+The argument  \cmd{--omitmethods} takes a regular expression of all of the method and constructor names that should be omitted.
+
+\subsection{Avoiding behaviors}
+Omitting methods works when we know we don't want particular methods of the classes under test to be called, but not so well when we we know what methods we want to avoid, but not which classes under test they are called from.
+In this situation, we can mock these undesirable methods so that they aren't actually called.
+
+To do this, you need another jar file 
+
+
+
+
+
+\subsection{Skipping Flaky Tests}
+Flaky tests are a general problem in testing.
+These are tests that can change behavior between runs, having difficult to find dependencies on global state in classes or the testing environment. 
+With Randoop, it is possible to observe flaky behaviors of sequences between runs inside Randoop, between the run in Randoop and at the command line, and between runs on the command line.
+At the moment, Randoop is only able to handle the first case, though by default Randoop will simply print an error message and halt.
+However, when given the option \cmd{--ignore-flaky-tests}, Randoop will treat a flaky test as invalid and discard it.
+
+\end{document}