Perceptron

config.json

@@ -1001,6 +1001,21 @@
         "practices": [],
         "prerequisites": [],
         "difficulty": 4
+      },
+      {
+        "slug": "perceptron",
+        "name": "Perceptron",
+        "uuid": "b43a938a-7bd2-4fe4-b16c-731e2e25e747",
+        "practices": [],
+        "prerequisites": [],
+        "difficulty": 3,
+        "topics": [
+          "machine learning",
+          "loops",
+          "arrays",
+          "logic",
+          "math"
+        ]
       }
     ]
   },

exercises/practice/perceptron/.docs/instructions.md

@@ -0,0 +1,27 @@
+# Instructions
+
+#### Updating
+Checking if an object is on one side of a hyperplane or another can be done by checking the normal vector which points to the object. The value will be positive, negative or zero, so all of the objects from a class should have normal vectors with the same sign. A zero value means the object is on the hyperplane, which we don't want to allow since its ambiguous. Checking the sign of a normal to a hyperplane might sound like it could be complicated, but it's actually quite easy. Simply plug in the coordinates for the object into the equation for the hyperplane and check the sign of the result. For example, we can look at two objects `v₁, v₂` in relation to the hyperplane `[w₀, w₁, w₂] = [1, 1, 1]`:
+
+`v₁ = [x₁, y₁] = [2, 2]`
+
+`w₀ + w₁⋅x₁ + w₂⋅y₁ = 1 + 1⋅2 + 1⋅2 = 5 > 0`
+
+
+`v₂ = [x₂, y₂] = [-2, -2]` 
+
+`w₀ + w₁⋅x₂ + w₂⋅y₂ = 1 + 1⋅(-2) + 1⋅(-2) = -3 < 0`
+
+If `v₁` and `v₂` have the labels `1` and `-1` (like we will be using), then the hyperplane `[1, 1, 1]` is a valid decision boundary for them since the signs match. 
+
+Now that we know how to tell which side of the hyperplane an object lies on, we can look at how perceptron updates a hyperplane. If an object is on the correct side of the hyperplane, no update is performed on the weights. However, if we find an object on the wrong side, the update rule for the weights is:
+
+`[w₀', w₁', w₂'] = [w₀ + 1ₗ, w₁ + x⋅1ₗ, w₂ + y⋅1ₗ]`
+
+Where `1ₗ ∈ {1, -1}`, according to the class of the object (i.e. its label), `x, y` are the coordinates of the object, the `wᵢ` are the weights of the hyperplane and the `wᵢ'` are the weights of the updated hyperplane.
+
+This update is repeated for each object in turn, and then the whole process repeated until there are no updates made to the hyperplane. All objects passing without an update means they have been successfully separated and you can return your decision boundary!
+
+Notes: 
+- Although the perceptron algorithm is deterministic, a decision boundary depends on initialization and is not unique in general, so the tests accept any hyperplane which fully separates the objects.
+- The tests here will only include linearly separable classes, so a decision boundary will always be possible (i.e. no need to worry about non-separable classes).

exercises/practice/perceptron/.docs/introduction.md

@@ -0,0 +1,20 @@
+# Introduction
+
+### Perceptron
+[Perceptron](https://en.wikipedia.org/wiki/Perceptron) is one of the oldest and bestestly named machine learning algorithms out there. Since it is also quite simple to implement, it's a favorite place to start a machine learning journey. Perceptron is what is known as a linear classifier, which means that, if we have two labled classes of objects, for example in 2D space, it will search for a line that can be drawn to separate them. If such a line exists, Perceptron is guaranteed to find one. See Perceptron in action separating black and white dots below!
+
+<p align="center">
+<a title="Miquel Perelló Nieto, CC BY 4.0 &lt;https://creativecommons.org/licenses/by/4.0&gt;, via Wikimedia Commons" href="https://commons.wikimedia.org/wiki/File:Perceptron_training_without_bias.gif"><img width="512" alt="Perceptron training without bias" src="https://upload.wikimedia.org/wikipedia/commons/a/aa/Perceptron_training_without_bias.gif"></a>
+</p>
+
+### Details
+The basic idea is fairly straightforward. As illustrated above, we cycle through the objects and check if they are on the correct side of our guess at a line. If one is not, we make a correction and continue checking the objects against the corrected line. Eventually the line is adjusted to correctly separate all the objects and we have what is called a decision boundary!
+
+Why is this of any use? The decision boundary found can then help us in predicting what a new, unlabeled, object would likely be classified as by seeing which side of the boundary it is on.
+
+#### A Brief Word on Hyperplanes
+What we have been calling a line in 2D can be generalized to something called a [hyperplane](https://en.wikipedia.org/wiki/Hyperplane), which is a convenient representation, and, if you follow the classic Perceptron algorithm, you will have to pick an initial hyperplane to start with. How do you pick your starting hyperplane? It's up to you! Be creative! Or not... Actually perceptron's convergence times are sensitive to conditions such as the initial hyperplane and even the order the objects are looped through, so you might not want to go too wild.
+
+We will be playing in a two dimensional space, so our separating hyperplane will simply be a 1D line. You might remember the standard equation for a line as `y = ax+b`, where `a,b ∈ ℝ`, however, in order to help generalize the idea to higher dimensions, it's convenient to reformulate this equation as `w₀ + w₁x + w₂y = 0`. This is the form of the hyperplane we will be using, so your output should be `[w₀, w₁, w₂]`. In machine learning, the `{w₀, w₁, w₂}` are usually referred to as weights. 
+
+Scaling a hyperplane by a positive value gives an equivalent hyperplane (e.g. `[w₀, w₁, w₂] ≈ [c⋅w₀, c⋅w₁, c⋅w₂]` with `c > 0`), since an infinite line scaled by a value is just the same infinite line. However, it should be noted that there is a difference between the normal vectors (the green arrow in illustration above) associated with hyperplanes scaled by a negative value (e.g. `[w₀, w₁, w₂]` vs `[-w₀, -w₁, -w₂]`) in that the normal to the negative hyperplane points in opposite direction of that of the positive one. By convention, the perceptron normal points towards the class defined as positive.

exercises/practice/perceptron/.meta/config.json

@@ -0,0 +1,20 @@
+{
+  "authors": [
+    "depial"
+  ],
+  "contributors": [
+    "cmcaine"
+  ],
+  "files": {
+    "solution": [
+      "perceptron.jl"
+    ],
+    "test": [
+      "runtests.jl"
+    ],
+    "example": [
+      ".meta/example.jl"
+    ]
+  },
+  "blurb": "Write your own machine learning algorithm"
+}

exercises/practice/perceptron/.meta/example.jl

@@ -0,0 +1,33 @@
+function classify(point, hyperplane)
+    # Takes a single point and a hyperplane
+    # Classifies which nomrmal of hyperplane is associated with the point
+    # Returns 1 for positive normal, -1 for negative normal and 0 for a point on the hyperplane
+    sign(hyperplane' * point)
+end
+
+function update(point, label, hyperplane)
+    # Takes one point, its label and a hyperplane
+    # Updates the hyperplane conditional on classification not matching the label
+    # Returns a vector, the Perceptron updated hyperplane
+    hyperplane + (classify(point, hyperplane) != label) * label * point
+end
+
+function step(points, labels, hyperplane)
+    # Takes a vector of points, a vector of their associated labels and a hyperplane
+    # Iteratively updates the hyperplane for each point/label pair
+    # Returns a tuple: (true/false, decision boundary/hyperplane) for valid/invalid decision boundary, respectively
+    decisionboundary = hyperplane
+    foreach(i -> hyperplane = update(points[i], labels[i], hyperplane), eachindex(points))
+    decisionboundary == hyperplane, hyperplane
+end
+
+function perceptron(points, labels)
+    # Takes a vector of linearly separable points and a vector of their associated labels
+    # Performs steps of the Perceptron algorithm until a valid decision boundary is found
+    # Returns a vector, a valid decision boundary for the provided population of labeled points
+    hyperplane, points = [0, 0, 0], vcat.(1, points)
+    while true
+        isdecisionboundary, hyperplane = step(points, labels, hyperplane)
+        isdecisionboundary && return hyperplane
+    end
+end

exercises/practice/perceptron/perceptron.jl

@@ -0,0 +1,3 @@
+function perceptron(points, labels)
+    # Perceptronize!
+end

exercises/practice/perceptron/runtests.jl

@@ -0,0 +1,92 @@
+using Test, Random
+
+include("perceptron.jl")
+
+@testset verbose = true "tests" begin
+    decisionboundary = perceptron([[-1,-1], [1, 0], [0, 1]], [-1, 1, 1])
+
+    @testset "Boundary is a vector of three weights" begin
+        @test length(decisionboundary) == 3
+    end
+
+    @testset "Weights are Real numbers" begin
+        @test eltype(decisionboundary) <: Real
+    end
+
+    function runtestset()
+
+        @testset "Low populations" begin
+
+            # Initial population
+            points = [[-1, 0], [0, -1], [1, 0], [0, 1]]
+            labels = [-1, -1, 1, 1]
+            decisionboundary = perceptron(points, labels)
+            reference = [0, 1, 1] #A valid decision boundary need not match the reference
+            @testset "Initial population - Your decision boundary: $decisionboundary" begin
+                @test isvalidboundary(points, labels, decisionboundary)
+            end
+
+            #Initial population w/ opposite labels
+            points = [[-1, 0], [0, -1], [1, 0], [0, 1]]
+            labels = [1, 1, -1, -1]
+            decisionboundary = perceptron(points, labels)
+            reference = [0, -1, -1] #A valid decision boundary need not match the reference
+            @testset "Initial population w/ opposite labels - Your decision boundary: $decisionboundary" begin
+                @test isvalidboundary(points, labels, decisionboundary)
+            end
+
+            # Decision boundary cannot pass through origin
+            points = [[1, 0], [0, 1], [2, 1], [1, 2]]
+            labels = [-1, -1, 1, 1]
+            decisionboundary = perceptron(points, labels)
+            reference = [-2, 1, 1] #A valid decision boundary need not match the reference
+            @testset "Decision boundary cannot pass through origin - Your decision boundary: $decisionboundary" begin
+                @test isvalidboundary(points, labels, decisionboundary)
+            end
+
+            #Decision boundary nearly parallel with y-axis
+            points = [[0, 50], [0, -50], [1, 50], [1, -50]]
+            labels = [-1, -1, 1, 1]
+            decisionboundary = perceptron(points, labels)
+            reference = [-1, 2, 0] #A valid decision boundary need not match the reference
+            @testset "Decision boundary nearly parallel with y-axis - Your decision boundary: $decisionboundary" begin
+                @test isvalidboundary(points, labels, decisionboundary)
+            end
+
+        end
+
+        @testset "Increasing Populations" begin
+            for n in 10:50
+                points, labels = population(n, 25)
+                decisionboundary = perceptron(points, labels)
+                @testset "Population: $n points - Your decision boundary: $decisionboundary" begin
+                    @test isvalidboundary(points, labels, decisionboundary)
+                end
+            end
+        end
+
+    end
+
+
+
+    function population(n, bound)
+        v = !iszero(n % 10)
+        b, x, y = rand(-bound÷2:bound÷2), rand(-bound:bound), rand(-bound:bound)v
+        y_intercept = -b ÷ (iszero(y) ? 1 : y)
+        points, labels, hyperplane = [], [], [b, x, y]
+        while n > 0
+            point = [rand(-bound:bound), y_intercept + rand(-bound:bound)]
+            label = point' * [x, y] + b
+            if !iszero(label)
+                push!(points, point)
+                push!(labels, sign(label))
+                n -= 1
+            end
+        end
+        points, labels
+    end 
+    isvalidboundary(points, labels, db) = all(>(0), reduce(hcat, vcat.(1, points))' * db .* labels)
+
+    Random.seed!(42)
+    runtestset()
+end