Merge pull request #92 from SamuraiWTF/next

Next 4.2.0

README.md

@@ -1,50 +1,91 @@
-# samuraiwtf
+<p align="center">
+  <img alt="SamuraiWTF Logo" src="https://tiny.si/images/SamuraiWTFLogo.png" height="200"/>
+</p>
 
-**Want to Contribute? See section at the end of this readme**
+<p align="center">
+  <a href="https://professionallyevil.slack.com/messages/samuraiwtf"> <img alt="Slack" src="https://img.shields.io/badge/chat-ProfessionallyEvil-%238c0000.svg?logo=slack" /></a>
+  <a href="https://github.com/SamuraiWTF/samuraiwtf/releases"> <img alt="Github" src="https://img.shields.io/github/downloads/SamuraiWTF/samuraiwtf/total.svg?label=Github%20Downloads"/></a>
+  <a href="https://github.com/SamuraiWTF/samuraiwtf"> <img alt="SourceForge" src="https://img.shields.io/sourceforge/dt/samurai.svg?label=%28Deprecated%29%20%20SourceForge%20Downloads"/></a>
+  <a href="https://github.com/SamuraiWTF/samuraiwtf/releases"> <img alt="Latest version" src="https://img.shields.io/github/release/SamuraiWTF/samuraiwtf.svg" /></a>
+  <a href="https://github.com/SamuraiWTF/samuraiwtf/blob/master/LICENSE" > <img alt="License" src="https://img.shields.io/badge/license-GPLv3-blue.svg" /></a>
+  <a href="https://twitter.com/intent/tweet?via=secureideas&hashtags=SamuraiWTF%2CProfessionallyEvil&url=https%3A%2F%2Fsamurai.wtf"> <img alt="Twitter Hashtag" src="https://img.shields.io/badge/%23SamuraiWTF-tweet%20about%20us-lightgrey.svg?logo=twitter&style=social" /></a>
+  <a href="https://twitter.com/intent/follow?screen_name=secureideas" > <img alt="Twitter handle" src="https://img.shields.io/twitter/follow/secureideas.svg?label=Follow%20%40secureideas%20for%20updates&style=social" /></a>
+</p>
 
-The purpose behind this project is to migrate the SamuraiWTF (http://www.samurai-wtf.org), which until now has been maintained as a monolithic virtual machine, to a "packageable" distribution system. The current direction of choice is Vagrant with a VirtualBox provider, which is the effort in this master branch.  Alternative efforts can be found in other branches.
+----
+
+**Want to chat with us? Come message us in the [Professionally Evil slack][samurai-slack-url].**
 
-**To download an OVA to import a full virtual machine, visit https://tiny.si/samurai. 
+**Want to Contribute? See [here](#Contributors)**
+
+The purpose behind this project is to migrate the SamuraiWTF (http://www.samurai-wtf.org), which until now has been maintained as a monolithic virtual machine, to a "packageable" distribution system. The current direction of choice is Vagrant with a VirtualBox provider, which is the effort in this master branch.  Alternative efforts can be found in other branches.
 
-A [video tutorial](https://www.youtube.com/watch?v=3a3qOFubfGg) is available showing how to install from OVA.
+**NOTE:** for getting started quickly you can follow the ova installation [here](#OVA)
 
 ## Prerequisites
 - Vagrant - https://www.vagrantup.com/
 - Virtualization Software - The base vagrant box used supports virtualbox, vmware, and parallels, but testing at this time has been solely on virtualbox - https://www.virtualbox.org/
 - vagrant-vbguest plugin for vagrant (virtualbox only) - this automatically installs guest extensions which provide support for higher display resolutions, as well as other conveniences like clipboard sharing - https://github.com/dotless-de/vagrant-vbguest
+- vagrant-reload plugin - this facilitates a necessary reboot during initial provisioning (can be installed with `vagrant plugin install vagrant-reload`).
 - Disable Hyper-V (Windows and Virtualbox only) - follow the Resolution instructions provided by Microsoft to disable and enable Hyper-V (requires reboot) - https://support.microsoft.com/en-us/help/3204980/virtualization-applications-do-not-work-together-with-hyper-v-device-g
 
 ## Initial Install
-1. Make sure you have the prereqs listed above. Webpwnized has made some helpful [YouTube video instructionals](https://www.youtube.com/watch?v=MCqpTpxNSlA&list=PLZOToVAK85Mru8ye3up3VR_jXms56OFE5) for getting Vagrant and VirtualBox  with vbguest plugin installed in case you have not done so before.
-2. Clone this repository.
-3. From a command-line terminal in the project directory, run the command `vagrant up`. Then sit back and wait for it to finish. NOTE: The Guest VM's window will open with the CLI while provisioning is still ongoing. It's best to leave it alone until the vagrant up command fully completes.
-4. Immediately after the first time start up it is recommend you do a restart using vagrant reload.
+### OVA
+1. Make sure you have the Virtualization Software and Disabled Hyper-V from the prereqs [above](#Prerequisites)
+2. Download the OVA to import a full virtual machine, here: https://tiny.si/samurai.
+3. Watch this [video tutorial](https://www.youtube.com/watch?v=3a3qOFubfGg), made by [webpwnized](https://twitter.com/webpwnized), which shows you how to install SamuraiWTF using the OVA.
 
-## Running Separate User / Target Systems
-- Just running the `vagrant up` will build the primary target, which is a single VM with both the user environment and the targets.  
-- You can run `vagrant up userenv` and `vagrant up target` to build seperate virtual machines for those purposes.)
+### Vagrant
+1. Make sure you have the prereqs listed [above](#Prerequisites). Webpwnized has made some helpful [YouTube video instructionals](https://www.youtube.com/watch?v=MCqpTpxNSlA&list=PLZOToVAK85Mru8ye3up3VR_jXms56OFE5) for getting Vagrant and VirtualBox with vbguest plugin installed in case you have not done so before.
+2. Clone this repository.
+3. From a command-line terminal in the project directory, run the command `vagrant up`. Then sit back and wait for it to finish.
+4. (Optional) If you want to understand this process a little more, we have a video that discusses what is happening with more detail. It is listed as a free course on our training site: https://training.secureideas.com/course/foldingsteel/
 
 **NOTE: The Guest VM's window will open with the CLI while provisioning is still ongoing. It's best to leave it alone until the `vagrant up` command fully completes.**
 
-### Provisioning Scripts
-The main Vagrant provisioning script for SamuraiWTF is *install/userenv_bootstrap.sh*.  A standalone targets provisioning script is in *install/target_bootstrap.sh*.  Changes for the system, targets, or tools installation or initialization for SamuraiWTF are all handled within these scripts.
+#### Provisioning Scripts
+SamuraiWTF is provisioned through the ansible-local Vagrant provisioner.  Provisioning is organized into the following playbooks, all found in the install folder:
+
+- bootstrap.yml: Prepare the environment to install tools and targets.  A reboot occurs after this playbook runs.
+- tools.yml:  Install all tools for the SamuraiWTF environment. Tool installation tasks are found in the install/tools folder.
+- targets.yml:  Install all the targets for the SamuraiWTF environment.  Target installation tasks are found in the install/targets folder.
+- user.yml:  Finalize any configuration for the samurai user.
+
+## Development
+Once you have a running environment, use the ansible playbooks to define any changes or additions to the installation.  To test the change, there is no need to re-run the entire provisioning through Vagrant every time.  Instead, you can just run one of the ansible playbooks from the vagrant folder inside the guest (i.e. first do `vagrant ssh`).  For example, if you are adjusting a tool, you would type:
+
+```
+cd /vagrant
+ansible-playbook -i /etc/ansible/hosts install/tools.yml
+```
+
+_Note that this is run as the vagrant (non-privileged) user, and does not require sudo.  Ansible will call sudo internally if needed._
+
+### Development Guidelines
+
+- Our integration branch is the one called `next`. That's where all new features and bug fixes go for testing before a release.  The `master` branch should be kept stable at all times.
+- Larger changes should be done in separate feature branches.  Make sure to merge `next` into your feature branch, then PR the feature branch to merge into `next`.
+- If you break `next` or `master`, fix it (with help if necessary). It's best to run a full test build (i.e. `vagrant destroy`, `vagrant up`) and make sure tools ard targets are working before pushing changes.
+
+### Expected Errors
+Sometimes there are some expected errors during the build process.  
+
+- There is one scary error in the _Install Docker_ task for VirtualBox provisioning that looks like it fails miserably.  At the bottom of that error output you will see the message `...ignoring` .  This is because our ansible playbook is expecting the error and moves on to the next task. Docker should still have installed but it had trouble getting to the right version of certain a kernel library due to an upgrade of guest additions. This should resolve itself once the system reboots (which should happen automatically).
 
 ## Production VM Notes:
 Once you load the VM, the username and password are:
 
 - Username: samurai
 - Password: samurai
 
-The menus are available via a right click on the desktop.
+The menus are available in the top-left corner of the desktop.
 
-Once you log in the target systems need to be provisioned. (Working on doing this during the build!)
+Once you log in, there are a couple of things that might need to be adjusted manually.
 
 First, load the Chrome bookmarks by starting *Chrome*.  Then select the *three dots* menu and select *Bookmarks*.
 From the sub menu, select *Import bookmarks and settings*.  In the window that opens, select *Bookmarks HTML File*.
 A file selector window will open.  Select the *chrome_bookmarks.html* file in the samurai home directory.
 
-Some of the target environments need to be initialized before use.  Use their setup or Reset DB links to do this.
-
 ## Virtualbox Display
 - To automatically adjust the display resolution, do the following:
 	- Select Virtualbox Menu -> View
@@ -64,3 +105,5 @@ Contributors are very welcome and the contribution process is standard:
   * submit a pull request
 
 Substantial or *Regular* contributors may also be brought in as full team members. This includes those who have made substantial contributions to previous versions of SamuraiWTF with the assumption they will continue to do so.
+
+[samurai-slack-url]: https://professionallyevil.slack.com/messages/samuraiwtf

Vagrantfile

@@ -6,66 +6,74 @@ Vagrant.configure("2") do |config|
 #shared settings
   config.vm.box = "bento/debian-9"
 
-  config.vm.synced_folder "./config", "/tmp/config"
+#  config.vm.synced_folder "./config", "/tmp/config"
 
 # Single Machine
 # Primary build
   config.vm.define "samuraiwtf", primary: true do |samuraiwtf|
-    samuraiwtf.vm.host_name = "SamuraiWTF"
+    samuraiwtf.vm.host_name = "SamuraiWTF"    
 
     samuraiwtf.vm.provider "virtualbox" do |vb|
     # Display the VirtualBox GUI when booting the machine
       vb.gui = true
-      vb.name = "SamuraiWTF-4.1.1"
+      vb.name = "SamuraiWTF-4.2.0"
     # Customize the amount of memory on the VM:
       vb.memory = "4096"
       vb.customize ["modifyvm", :id, "--vram", "128"]
-      vb.customize ["modifyvm", :id, "--cpus", "2"]
+      vb.customize ["modifyvm", :id, "--cpus", "2"]      
 
-      samuraiwtf.vm.provision :shell, inline: "shutdown -r +1"
+      # samuraiwtf.vbguest.auto_update = false
+      # samuraiwtf.vm.provision :shell, inline: "shutdown -r +1"
     end
 
-    samuraiwtf.vm.provision :shell, path: "install/shared_before.sh"
-    samuraiwtf.vm.provision :shell, path: "install/userenv_bootstrap.sh"
-    samuraiwtf.vm.provision :shell, path: "install/target_bootstrap.sh"
-    samuraiwtf.vm.provision :shell, path: "install/local_targets.sh"
+    # Make sure VBGuestAdditions is up-to-date and certain pre-requisite packages are installed.  Then restart (reload) so we are using the right
+    # version of VBGuestAdditions before continuing.
+    samuraiwtf.vm.provision :shell, inline: "apt-get update && apt-get -y install aufs-tools cgroupfs-mount mate-desktop-environment lightdm python-pip ansible"
+    samuraiwtf.vm.provision :reload 
 
-  end
-
-#Additional build options.  Sepearate virtual machines
-#attack machine
-  config.vm.define "userenv", autostart: false do |userenv|
-    userenv.vm.host_name = "samuraiwtf"
-
-    userenv.vm.provider "virtualbox" do |vb|
-    # Display the VirtualBox GUI when booting the machine
-      vb.gui = true
-      vb.name = "SamuraiWTF User Environment"
-    # Customize the amount of memory on the VM:
-      vb.memory = "2048"
-      vb.customize ["modifyvm", :id, "--vram", "16"]
+    samuraiwtf.vm.provision "ansible_local", run: "once" do |ansible1|      
+      ansible1.playbook = "install/samuraiwtf.yml"
+      ansible1.version = "latest"
+      ansible1.install_mode = "pip"
+      ansible1.compatibility_mode = "2.0"
     end
 
-    userenv.vm.provision :shell, path: "install/shared_before.sh"
-    userenv.vm.provision :shell, path: "install/userenv_bootstrap.sh"
   end
 
-#target server
-  config.vm.define "target", autostart: false do |target|
-    target.vm.host_name = "samuraitargets"
-    #for debugging mainly
-    target.vm.network "private_network", ip: "192.168.42.42"
-    target.vm.hostname = "samurai-wtf"
-    #config.hostsupdater.aliases = ["juice-shop.wtf","dojo-basic.wtf"]
-
-    target.vm.provider "virtualbox" do |vb|
-      vb.name = "Samurai Target Server"
-      vb.memory = "2048"
-    end
-
-    target.vm.provision :shell, path: "install/shared_before.sh"
-    target.vm.provision :shell, path: "install/target_bootstrap.sh"
-  end
+#Additional build options.  Sepearate virtual machines
+#attack machine
+#   config.vm.define "userenv", autostart: false do |userenv|
+#     userenv.vm.host_name = "samuraiwtf"
+
+#     userenv.vm.provider "virtualbox" do |vb|
+#     # Display the VirtualBox GUI when booting the machine
+#       vb.gui = true
+#       vb.name = "SamuraiWTF User Environment"
+#     # Customize the amount of memory on the VM:
+#       vb.memory = "2048"
+#       vb.customize ["modifyvm", :id, "--vram", "16"]
+#     end
+
+#     userenv.vm.provision :shell, path: "install/shared_before.sh"
+#     userenv.vm.provision :shell, path: "install/userenv_bootstrap.sh"
+#   end
+
+# #target server
+#   config.vm.define "target", autostart: false do |target|
+#     target.vm.host_name = "samuraitargets"
+#     #for debugging mainly
+#     target.vm.network "private_network", ip: "192.168.42.42"
+#     target.vm.hostname = "samurai-wtf"
+#     #config.hostsupdater.aliases = ["juice-shop.wtf","dojo-basic.wtf"]
+
+#     target.vm.provider "virtualbox" do |vb|
+#       vb.name = "Samurai Target Server"
+#       vb.memory = "2048"
+#     end
+
+#     target.vm.provision :shell, path: "install/shared_before.sh"
+#     target.vm.provision :shell, path: "install/target_bootstrap.sh"
+#   end
 
   # forwarded port mapping
   # currently none

config/home/bookmarks.html

@@ -9,12 +9,11 @@ <H1>Bookmarks</H1>
     <DT><H3 ADD_DATE="1515013973" LAST_MODIFIED="1515602701" PERSONAL_TOOLBAR_FOLDER="true">Bookmarks bar</H3>
     <DL><p>
         <DT><A HREF="http://dojo-basic.wtf/" ADD_DATE="1515014571" ICON="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAABu0lEQVQ4jZXRu2uTURjH8U9S0wupl76W2NpWEwW1RlBUClIncXBzsYKDq4s4+Q8UHJ109EJB6+DkIDiogw6FdpCIIGIUOjQlSptS2tDapn1flwS0vin4gwPnHL7P83suxKsn5q83DmzZ8u5HCkt7GL7EvTOMzLFcpYg2HMBCE2MwhNaH5Kb5Ps3sA07Vzc5thXfEJPiEq69YuUZPC6kCJ3ASL7Zz/kv3uR0RRURPuduMi6vgInqfZDI7L1SrUXsYGu/qSimXr2MRL5slSOIsvuJtJZ2+cXNwMJFAaXZ2AeMIcB5TqDWCGtqHD5hBV6lUCmu12spmGG6Uy+UlZFDBRP0OEjEtDCHI5/Mfd6XTz6NEYvfa+vrlQqFwGJt432wejSpOj5Ic6++f+NHWFlZSqfBZX9+X0Wy2HceQaxacRB5eZzLDYX0DjTMZBFfq3BG0xg0xxGc4FIb5EJVUSoS9tZq+avV4nSv+6Rq3RmPd3cvfstnoZzqdgGB1NTq6uLisWPyHjU0wl06/G5mZ+VUIgvZaMmlofj4sBMGbZr3Haqqj49ZaIrFRI5zs7LzzX8ENPR4Y2P8olzu4HfMbvkaMPFN3VXkAAAAASUVORK5CYII=">dojo-basic</A>
-        <DT><A HREF="http://dojo-basic.wtf/reset-db.php" ADD_DATE="1515078852" ICON="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAABu0lEQVQ4jZXRu2uTURjH8U9S0wupl76W2NpWEwW1RlBUClIncXBzsYKDq4s4+Q8UHJ109EJB6+DkIDiogw6FdpCIIGIUOjQlSptS2tDapn1flwS0vin4gwPnHL7P83suxKsn5q83DmzZ8u5HCkt7GL7EvTOMzLFcpYg2HMBCE2MwhNaH5Kb5Ps3sA07Vzc5thXfEJPiEq69YuUZPC6kCJ3ASL7Zz/kv3uR0RRURPuduMi6vgInqfZDI7L1SrUXsYGu/qSimXr2MRL5slSOIsvuJtJZ2+cXNwMJFAaXZ2AeMIcB5TqDWCGtqHD5hBV6lUCmu12spmGG6Uy+UlZFDBRP0OEjEtDCHI5/Mfd6XTz6NEYvfa+vrlQqFwGJt432wejSpOj5Ic6++f+NHWFlZSqfBZX9+X0Wy2HceQaxacRB5eZzLDYX0DjTMZBFfq3BG0xg0xxGc4FIb5EJVUSoS9tZq+avV4nSv+6Rq3RmPd3cvfstnoZzqdgGB1NTq6uLisWPyHjU0wl06/G5mZ+VUIgvZaMmlofj4sBMGbZr3Haqqj49ZaIrFRI5zs7LzzX8ENPR4Y2P8olzu4HfMbvkaMPFN3VXkAAAAASUVORK5CYII=">Reset dojo-basic</A>
-        <DT><A HREF="http://localhost/vulnscripts/" ADD_DATE="1515602695">Vuln Scripts</A>
+        <DT><A HREF="http://dojo-basic.wtf/reset-db.php" ADD_DATE="1515078852" ICON="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAABu0lEQVQ4jZXRu2uTURjH8U9S0wupl76W2NpWEwW1RlBUClIncXBzsYKDq4s4+Q8UHJ109EJB6+DkIDiogw6FdpCIIGIUOjQlSptS2tDapn1flwS0vin4gwPnHL7P83suxKsn5q83DmzZ8u5HCkt7GL7EvTOMzLFcpYg2HMBCE2MwhNaH5Kb5Ps3sA07Vzc5thXfEJPiEq69YuUZPC6kCJ3ASL7Zz/kv3uR0RRURPuduMi6vgInqfZDI7L1SrUXsYGu/qSimXr2MRL5slSOIsvuJtJZ2+cXNwMJFAaXZ2AeMIcB5TqDWCGtqHD5hBV6lUCmu12spmGG6Uy+UlZFDBRP0OEjEtDCHI5/Mfd6XTz6NEYvfa+vrlQqFwGJt432wejSpOj5Ic6++f+NHWFlZSqfBZX9+X0Wy2HceQaxacRB5eZzLDYX0DjTMZBFfq3BG0xg0xxGc4FIb5EJVUSoS9tZq+avV4nSv+6Rq3RmPd3cvfstnoZzqdgGB1NTq6uLisWPyHjU0wl06/G5mZ+VUIgvZaMmlofj4sBMGbZr3Haqqj49ZaIrFRI5zs7LzzX8ENPR4Y2P8olzu4HfMbvkaMPFN3VXkAAAAASUVORK5CYII=">Reset dojo-basic</A>       
         <DT><A HREF="http://mutillidae.wtf/" ADD_DATE="1515602695">Mutillidae</A>
         <DT><A HREF="http://dvwa.wtf/" ADD_DATE="1515602695">DVWA</A>
         <DT><A HREF="http://juice-shop.wtf/#/search" ADD_DATE="1515014560" ICON="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAACQ0lEQVQ4jY2SzUtUYRTGn/Pe9975uk4zfiRpoRamZNBCiUyQxkVgEbVxH0FBm1a1TmjTqk1I1CroHwhxW7kKpqZC0Eqkmgo/rp/jfN25H+97WiUGztDZnuf8eM7DAzQYzg2aPHcxwTwp6mmoESA/PTbe0Z24a0TIgcaqIMxQ//Sb/RpZ7/jVo+HOtY3wXqI1zKRMiS/TDtIROco/rk1Qz8v8X92B1nJPB+OJZuuBERMZZ92HV9MIbANBixwKXDW5krsSb+iA08ljUmMcBLi1EOsbPnrPpQAwdovB5cKmPg5gvj5AqSoJwwUY0aYuWAmFxOGTILMN2PrsqdWPbmMHEbOKUJdYM1Kdw4hHK6D0GOD/hNaLZTcIKg0zMMy4y4wCADAzSJhA4AClt1C+U9S+qDYElOyST4QdsAJYgaABEQfarkOZ/YXtbNZrCMhkZkMho1vprjHE030g9oDqPKAq0GF1+91jBHUzYEDMnTl6JFwtx3y1DTeWB8QOqLkdqvgaQXGpcmpfAf9p4sLAgOUnvZsgvoOOSHcYh6XPp9B+wkazLVB1PJQ/uZuGY02hJh72PJ+t7TlYuDBg+0HttmEZ982kmSAm6CKDsxV4+RCOx0AphLSSrdHe0zeCte8vAHyTAJA929/iB7UnZNBVs8kkklSGhpYRwxIhRbHsg4gYhBWyI+9D59eM2nCX9174OtLXVEN4CYJMIcSW8tUuC/YNSx6Sthw1JI1IWy7B4qmCYS4OPfsQ4H+HAfo9MRzL3Ro0D9r/AYU7+H7GSFGwAAAAAElFTkSuQmCC">Juice Shop</A>
         <DT><A HREF="http://dojo-scavenger.wtf/" ADD_DATE="1515014578" ICON="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAA70lEQVQ4jc3SPU7DQBDF8Z9Z21iIBpCg4CgUFBSchJYCJHo4C5wCLpAbREKIngIivkSMHVPYQWRjJSl50hQ7mvfX7pvlv2oHlxjiFudYX9UccIcmqiskqwD28dYDuMdmPLzWc97Gdw94C7vLAAlyjHsA4+4GM54YADW+evqlngxiQJPneaVNP9awgy8GlGX5WRTFQBvcbx8DfET9eQCeq6p67YanescILzEg1gZOQgiP5tf4gFPtlmaU4ADXaZo+ZVkWG+Ma4QbH0xdcYBJCaEIIy8x/a4KzFEdI6nou4GVKcJhqv+50xwsDiswN9n4AXSVV4VRFzn0AAAAASUVORK5CYII=">dojo-scavenger</A>
-        <DT><A HREF="http://client.cors.dem/" ADD_DATE="1515014560">CORS Demonstrator</A>
+        <DT><A HREF="http://cors-dojo.wtf/" ADD_DATE="1515014560">CORS Demonstrator</A>
     </DL><p>
 </DL><p>