Update Script to use StevenBlack's Hosts File

README.md

@@ -1,30 +1,35 @@
 # ad-blocker
+
 A simple ad-blocker for Synology devices
 
 ## Background
 
 The main goal of this project is to setup a simple ad-blocking service that will work on all LAN-connected devices. In order to keep things simple, an additional goal is to run this service on the Synology NAS that already provides many file and network services (including DNS) for the LAN. Because the DNS service is in active use on the Synology device, many solutions (like the very nice Pi-hole package) are not workable as there are conflicts over the DNS port. This solution has a minimal impact on the standard Synology system and so should be largely free of unwanted side effects and be "update friendly".
 
 There are several advantages of enabling a LAN-wide ad-blocking service over traditional browser plugins:
+
 * It's easier to maintain than ensuring all browsers have the appropriate plugins and remain updated on all devices
 * It works for mobile devices (phones, tablets, etc.) for both browser and apps
 * It is effective even on devices not allowed to be modified, e.g. school-owned tablets
 
 ## Requirements
+
 This project requires some familiarity with the basic Unix system and tools. Additionally, access to the Synology admin interface is requried. This project _does_ involve access to the internals to your system. As such, there is always the risk of an accident or mistake leading to irrecoverable data loss. **Be mindful when logged onto your system -- especially when performing any action as root or admin.**
 
 This project should be completed in 30-60 minutes.
 
 This service requires the following skills:
+
 * Editing files on a Unix system (e.g. `vi`, `nano`)
 * Setting up a DNS zone via the web-based DSM interface
 * SSH access
 * Standard Unix tools (`sudo`, `chown`, `chmod`, `mv`, `cd`, `ls`, `wget`, `cat`, etc.)
 * Administration/root access to the Synology device
 
-These instructions have been verified as working on a Synology DS1513+ running DSM 6.1-15047 Update 2. 
+These instructions have been verified as working on a Synology DS1513+ running DSM 6.1-15047 Update 2.
 
 ## DNS Service Setup
+
 1. Log in as adminstrator to the Synology DSM (administration interface)
 1. In the Package Center, open the "DNS Server" app.
 1. Select the "Zones" tab and create a new Master Zone.
@@ -40,24 +45,20 @@ These instructions have been verified as working on a Synology DS1513+ running D
 The Domain Name _must_ be `null.zone.file` and the Serial Format _must_ be set as `Date` as that is what the updater script requires. The blocked zones must reference a static zone configuration file and so the "Limit zone update" must be enabled with no values so that the resulting configuration file is generated with the line `allow-update {none;};`. The Master DNS Server should have the same IP address as your Synology device. (Don't fret over this; it will be overwritten later.)
 
 ## Script Installation
+
 1. SSH as the administrator to the Synology device
     * `ssh admin@synology.example.com`
-1. Navigate to the appropriate directory
-    * `cd /usr/local/bin`
-1. Download the `ad-blocker.sh` script
-    * `sudo wget -O ad-blocker.sh "https://raw.githubusercontent.com/steventblack/ad-blocker/master/ad-blocker.sh"`
-1. Change the owner and permissions of the script
-    * `sudo chown root:root ad-blocker.sh`
-    * `sudo chmod +x ad-blocker.sh`
-1. Verify the script executes properly
-    * `sudo ./ad-blocker.sh`
-    * Verify `/var/packages/DNSServer/target/named/etc/zone/data/ad-blocker.db` exists and has ~200k of data
-    * Verify `/var/packages/DNSServer/target/named/etc/zone/data/null.zone.file` has the line `include "/etc/zone/data/ad-blocker.db";`
-    * Verify `/var/packages/DNSServer/target/named/etc/zone/master/null.zone.file` is updated with the correct serial number
-
+2. Run the following command
+
+    ```bash
+        sudo -i
+        wget -O- https://raw.githubusercontent.com/0xW1sKy/ad-blocker/master/install.sh | sudo /bin/sh
+    ```
+
 The ad-blocking functionality should now be in effect. You can test the effectiveness by disabling any ad-blocking plugins in your browser and navigating to any ad-laden website to verify ads remain suppressed. Mobile devices should similarly be tested.
 
 ## Automated Block List Updating
+
 1. Log in as administrator to the Synology DSM (administration interface)
 1. Open up the "Control Panel" app.
 1. Select the "Task Scheduler" service.
@@ -71,15 +72,16 @@ The ad-blocking functionality should now be in effect. You can test the effectiv
     * First run time: `03:20`
     * Frequency: once a day
 1. For the "Task Settings" tab, fill in the fields as follow:
-    * Send run details by email: `<your email here>`
     * User defined script: `cd /tmp/; sudo /usr/local/bin/ad-blocker.sh`
 
 The run time should be set to run no more than once a day and be performed at an off-peak traffic time. The block lists don't change that frequently so be courteous to the provider. It is not strictly necessary to have the run details sent via email, but enabling it may help if there's a need to troubleshoot.
 
 ## Blacklist/Whitelist
+
 User-defined blacklist/whitelist functionality has been added to allow personalized rules to either enhance blocking or permit access. Templates for the configuration files of this functionality are automatically created upon the initial run of the `ad-blocker.sh` script.
 
 ### Blacklist
+
 A user-defined blacklist functionality is available to add custom domains into the block list. This may help fill in any "gaps" for domains not captured by the [yoyo.org](http://pgl.yoyo.org/adservers/) block list. There is no harm if a domain appears in both the user-specified black list and the main list as the scripts will detect the duplicate and skip over any redundant mentions.
 
 1. SSH as the administrator to the Synology device
@@ -96,6 +98,7 @@ A user-defined blacklist functionality is available to add custom domains into t
     * `sudo ./ad-blocker.sh`
 
 ### Whitelist
+
 The user-defined whitelist allows specified domains to continue to work despite their appearance in either the Block list or the blacklist. Note that the whitelist is applied last, regardless as to the the source of the domain.
 
 1. SSH as the administrator to the Synology device
@@ -112,7 +115,11 @@ The user-defined whitelist allows specified domains to continue to work despite
     * `sudo ./ad-blocker.sh`
 
 ## Caveats
+
 This solution works well for blocking the vast majority of ad providers. It should help speed up page rendering as well as provide a degree of privacy and security to your devices. However, it is not a panacea and you should continue to practice safe browsing habits. In particular, remember that this solution only applies to devices _within_ the LAN and so mobile devices may lose any protections it offers when using a different network.
 
 ## Thanks
-This solution utilizes the block list provided by [yoyo.org](http://pgl.yoyo.org/adservers/). A big thanks goes out to them for their hard work and continued maintainence.
+
+[Steven Black](https://github.com/steventblack)
+[Another Steven Black?](https://github.com/stevenblack)
+For real tho? is this the same guy or just a weird coincidence?

ad-blocker.sh

@@ -2,15 +2,16 @@
 
 ###########################
 # (C) 2017 Steven Black
+# Updated by Kyle Burk
 ###########################
 #
 # 2017-04-17 - 1.0.0 Initial release
 # 2017-04-18 - 1.1.0 Improved modularization; added functionality for white & black lists
 # 2017-04-25 - 1.1.1 Relocated conf dir to /usr/local/etc
-# 2017-04-25 - 1.1.2 Relocate script dir; update checks to create conf templates 
+# 2017-04-25 - 1.1.2 Relocate script dir; update checks to create conf templates
 # 2017-05-17 - 1.1.3 Remove local declarations for improved compatibility
 # 2017-05-17 - 1.1.4 Cleanup syntax as per shellcheck.net suggestions
-#
+# 2020-12-28 - 2.0.0 Modify to use host file lists, remove yoyo
 ###########################
 
 # routine for ensuring all necessary dependencies are found
@@ -38,23 +39,23 @@ check_conf () {
 
   # if no white list found, then create a template & instructions
   if [ ! -f "$WhiteList" ]; then
-    echo "No white list found; creating template" >&2
-	
-	{ echo "# White list of domains to remain unblocked for ad-blocker.sh"; 
-      echo "# Add one fully-qualified domain name per line"; 
-      echo "# Comments are indicated by a '#' as the first character"; 
-      echo "# example:"; 
+    printf "No white list found; creating template\n" >&2
+
+  { echo "# White list of domains to remain unblocked for ad-blocker.sh";
+      echo "# Add one fully-qualified domain name per line";
+      echo "# Comments are indicated by a '#' as the first character";
+      echo "# example:";
       echo "# ad.example.com"; } > "$WhiteList"
   fi
 
   # if no black list found, then create a template & instructions
   if [ ! -f "$BlackList" ]; then
-    echo "No black list found; creating template" >&2
+    printf "No black list found; creating template\n" >&2
 
-    { echo "# Black list of additional domains for ad-blocker.sh"; 
-      echo "# Add one fully-qualified domain name per line"; 
-      echo "# Comments are indicted by a '#' as the first character"; 
-      echo "# example:"; 
+    { echo "# Black list of additional domains for ad-blocker.sh";
+      echo "# Add one fully-qualified domain name per line";
+      echo "# Comments are indicted by a '#' as the first character";
+      echo "# example:";
       echo "# ad.example.com"; } > "$BlackList"
   fi
 }
@@ -64,24 +65,33 @@ check_conf () {
 check_user () {
   User=$(whoami)
   if [ "$User" != "DNSServer" ]; then
-    echo "Running as $User; switching to DNSServer" >&2
+    printf "Running as %s\n" "$User" >&2
+    printf "Switching to DNSServer\n" >&2
     su -m DNSServer "$0" "$@" || exit 1
-	exit 0
+  exit 0
   fi
 }
 
-# fetch the blocklist from yoyo.org and update the path element
+# function to fetch the blocklist and update the path element
 # for each entry to comply with the Synology setup
-fetch_blocklist () {
-  BlocklistURL="http://pgl.yoyo.org/as/serverlist.php?hostformat=bindconfig&showintro=0&mimetype=plaintext"
 
+fetch_blocklist () {
+  BlocklistURL="$1"
+  printf "Pulling blocklist from %s\n" "${BlocklistURL}" >&2
   # the "-O-" tells wget to send the file to standard out instead of a real file
   # this makes it suitable for piping and eliminates need for another temp file
-  wget -O- "$BlocklistURL" | \
-    sed -e 's/null.zone.file/\/etc\/zone\/master\/null.zone.file/g' > "/tmp/ad-blocker.new"
+  wget -qO- "$BlocklistURL" | \
+    sed -e 's/\s/ /g' | \
+    sed -s -e 's/ *$//g' | \
+    sed -s -r 's/([^#].*)?(#)+(.*)?/\1/g' | \
+    sed -r '/^\s*$/d' | \
+    sed -r 's/(.*)(\s)$/\1/g' | \
+    sed -r 's/(.*\s)?(.*)$/\2/g' | \
+    sed -r 's/(.*)+$/zone "\1" { type master; notify no; file "null.zone.file"; };/g' >> "/tmp/ad-blocker.new"
+  printf "\n" >> "/tmp/ad-blocker.new"
 }
 
-# user-specified list of domains to be blocked in addition to those from yoyo.org
+# user-specified list of domains to be blocked in addition
 apply_blacklist () {
   BlackList="${ConfDir}/ad-blocker-bl.conf"
   BlockList="/tmp/ad-blocker.new"
@@ -100,7 +110,7 @@ apply_blacklist () {
     if [ -z "$Domain" ]; then
       continue;
     fi
-	
+
     # if domain already listed then skip it and continue on to the next line
     # make sure you don't get a false positive with a partial match
     # by using the "-w" option on grep
@@ -115,17 +125,35 @@ apply_blacklist () {
   done < "$BlackList"
 }
 
+update_whitelist () {
+  WhiteListTmp="/tmp/ad-blocker-wl.tmp"
+  WhiteListURL="$1"
+  printf "Pulling whitelist from %s\n" "${WhiteListURL}" >&2
+  wget -qO- "$WhiteListURL" | \
+    sed -e 's/\s/ /g' | \
+    sed -s -e 's/ *$//g' | \
+    sed -s -r 's/([^#].*)?(#)+(.*)?/\1/g' | \
+    sed -r '/^\s*$/d' | \
+    sed -r 's/(.*)(\s)$/\1/g' | \
+    sed -r 's/(.*\s)?(.*)$/\2/g' >> "$WhiteListTmp"
+}
+
 # user-specified list of domains to remain unblocked
 apply_whitelist () {
   WhiteList="${ConfDir}/ad-blocker-wl.conf"
   BlockList="/tmp/ad-blocker.new"
   BlockListTmp="/tmp/ad-blocker.tmp"
-
+  WhiteListTmp="/tmp/ad-blocker-wl.tmp"
   # skip if the config doesn't exist
   if [ ! -f "$WhiteList" ]; then
     return 0
   fi
 
+  if [ -f "$WhiteListTmp" ]; then
+    cat "$WhiteListTmp" | sort | uniq -i | grep -v 'zone "" { type master; notify no; file "null.zone.file"; };' > "$WhiteList"
+  fi
+
+
   # process the whitelist skipping over any comment lines
   while read -r Line
   do
@@ -147,8 +175,11 @@ apply_whitelist () {
 update_zone_data () {
   ZoneDataFile="${ZoneDataDir}/null.zone.file"
   ZoneDataDB="${ZoneDataDir}/ad-blocker.db"
+  BlockListTmp="/tmp/ad-blocker.tmp"
   BlockList="/tmp/ad-blocker.new"
-
+  # Remove Dupes
+  cat "$BlockList" | sort | uniq -i | grep -v 'zone "" { type master; notify no; file "null.zone.file"; };' > "$BlockListTmp"
+  mv "$BlockListTmp" "$BlockList"
   # move the final version of the block list to the final location
   mv "$BlockList" "$ZoneDataDB"
 
@@ -166,8 +197,8 @@ update_zone_data () {
 
 # update the ZoneMasterFile with an new serial number
 update_zone_master () {
-  Now=$(date +"%Y%m%d")
-  ZoneMasterFile="${ZoneMasterDir}/null.zone.file" 
+  Now=$(date +"%Y%m%d%S")
+  ZoneMasterFile="${ZoneMasterDir}/null.zone.file"
 
   if [ -f "$ZoneMasterFile" ]; then
     rm -f "$ZoneMasterFile"
@@ -176,7 +207,7 @@ update_zone_master () {
   # rebuild the zone master file with the updated serial number
   { echo '$TTL 86400     ; one day';
     echo '@ IN SOA ns.null.zone.file. mail.null.zone.file. (';
-    echo '  '${Now}'00   ; serial number YYYYMMDDNN';
+    echo '  '${Now}'   ; serial number YYYYMMDDNN';
     echo '  86400        ; refresh 1 day';
     echo '  7200         ; retry 2 hours';
     echo '  864000       ; expire 10 days';
@@ -186,21 +217,83 @@ update_zone_master () {
     echo '* IN A   127.0.0.1'; } > "$ZoneMasterFile"
 
   # reload the server config to pick up the changes
-  "${RootDir}"/script/reload.sh
+  "${RootDir}"/script/reload.sh 'null.zone.file'
 }
 
+# manual_fixes () {
+#   ZoneDataDB="${ZoneDataDir}/ad-blocker.db"
+#   ZoneDataDBTmp="${ZoneDataDir}/ad-blocker.tmp"
+#   cat ad-blocker.db | \
+#     grep -v 'zone "format)">" { type master; notify no; file "null.zone.file"; };' | \
+#     grep -v 'zone "format)</title>" { type master; notify no; file "null.zone.file"; };' > ad-blocker.tmp
+#     'zone "href="rss/1.0/adservers.rss">" { type master; notify no; file "null.zone.file"; };'
+# }
+
 # Global vars for common paths
 ConfDir="/usr/local/etc"
 RootDir="/var/packages/DNSServer/target"
 ZoneDir="${RootDir}/named/etc/zone"
 ZoneDataDir="${ZoneDir}/data"
 ZoneMasterDir="${ZoneDir}/master"
+BlockLists=(
+  "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts"
+  "https://mirror1.malwaredomains.com/files/justdomains"
+  "https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt"
+  "https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt"
+  "https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt"
+  "https://hosts-file.net/psh.txt"
+  "https://raw.githubusercontent.com/quidsup/notrack/master/trackers.txt"
+  "https://v.firebog.net/hosts/Airelle-trc.txt"
+  "https://hosts-file.net/ad_servers.txt"
+  "https://adaway.org/hosts.txt"
+  "https://v.firebog.net/hosts/AdguardDNS.txt"
+  "https://v.firebog.net/hosts/Admiral.txt"
+  "https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt"
+  "https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt"
+  "https://v.firebog.net/hosts/Easylist.txt"
+  "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext"
+  "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts"
+  "https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts"
+  "https://raw.githubusercontent.com/jdlingyu/ad-wars/master/hosts"
+  "https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts_without_controversies.txt"
+  "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts"
+  "https://v.firebog.net/hosts/static/w3kbl.txt"
+  "https://raw.githubusercontent.com/matomo-org/referrer-spam-blacklist/master/spammers.txt"
+  "https://someonewhocares.org/hosts/zero/hosts"
+  "https://raw.githubusercontent.com/vokins/yhosts/master/hosts"
+  "https://winhelp2002.mvps.org/hosts.txt"
+  "https://v.firebog.net/hosts/neohostsbasic.txt"
+  "https://raw.githubusercontent.com/RooneyMcNibNug/pihole-stuff/master/SNAFU.txt"
+  "https://paulgb.github.io/BarbBlock/blacklists/hosts-file.txt"
+  "https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt"
+  "https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt"
+  "https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt"
+  "https://v.firebog.net/hosts/Prigent-Crypto.txt"
+  "https://mirror.cedia.org.ec/malwaredomains/immortal_domains.txt"
+  "https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt"
+  "https://phishing.army/download/phishing_army_blocklist_extended.txt"
+  "https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt"
+  "https://v.firebog.net/hosts/Shalla-mal.txt"
+  "https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt"
+  "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts"
+  "https://urlhaus.abuse.ch/downloads/hostfile/"
+  "https://v.firebog.net/hosts/Prigent-Malware.txt"
+  "https://raw.githubusercontent.com/HorusTeknoloji/TR-PhishingList/master/url-lists.txt"
+)
+WhiteLists=(
+  "https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt"
+)
 
 # Main Routine
 check_deps
 check_conf
 check_user "$@"
-fetch_blocklist
+for list in ${BlockLists[@]}; do
+  fetch_blocklist $list
+done
+for wlist in ${WhiteLists[@]}; do
+  update_whitelist $wlist
+done
 apply_blacklist
 apply_whitelist
 update_zone_data

install.sh

@@ -0,0 +1,19 @@
+wget -O /usr/local/bin/ad-blocker.sh "https://raw.githubusercontent.com/0xW1sKy/ad-blocker/master/ad-blocker.sh"
+chown root:root /usr/local/bin/ad-blocker.sh
+chmod 755 /usr/local/bin/ad-blocker.sh
+touch /usr/local/etc/ad-blocker-wl.conf
+touch /usr/local/etc/ad-blocker-bl.conf
+chown DNSServer:DNSServer /usr/local/etc/ad-blocker-wl.conf
+chmod 777 /usr/local/etc/ad-blocker-wl.conf
+chown DNSServer:DNSServer /usr/local/etc/ad-blocker-bl.conf
+chmod 777 /usr/local/etc/ad-blocker-bl.conf
+/bin/sh /usr/local/bin/ad-blocker.sh
+
+FILESIZE=$(du -h "/var/packages/DNSServer/target/named/etc/zone/data/ad-blocker.db" | cut -f1)
+INCLUDES=$(if grep -q 'include "/etc/zone/data/ad-blocker.db";' /var/packages/DNSServer/target/named/etc/zone/data/null.zone.file; then echo "True"; else echo "False"; fi)
+MODDATE=$(cut -c -8 <<<$(cat /var/packages/DNSServer/target/named/etc/zone/master/null.zone.file | grep 'serial number YYYYMMDDNN' | awk -F ';' '{print $1}' | sed -e 's/\s//g'))
+
+echo "Size of Database File: $FILESIZE"
+echo "Database reference in null.zone.file?: $INCLUDES"
+echo "null.zone.file Last Updated: $(date -d $MODDATE +"%A %B %d, %Y")"
+