Merge branch 'develop', version 0.7.3

CHANGELOG

@@ -1,3 +1,12 @@
+0.7.3 (2013-07-10)
+    * Handle 100-continue: do not forward expect header from client, ignore 100
+      continue response replied by some web servers
+    * For windows: add cow-hide.exe to run cow.exe as background process,
+      (provided by to xupefei)
+    * Filter sites covered by user specified domains on load
+    * Fix incorrectly changing header value to lower case: user name and
+      password can now contain upper case letters
+
 0.7.2 (2013-07-01)
     * Close idle server connections earlier: avoid opening too many sockets
     * Support authenticating multiple users (can limit port for each user)
@@ -46,7 +55,7 @@
 
     * Support parent HTTP proxy (such as goagent)
     * Work more automatically: because of this, updateBlocked, updateDirect,
-    autoRetry options and chou file are removed
+      autoRetry options and chou file are removed
     * Record direct/blocked visit count to make blocked/direct site handling
     more reliable
     * Builtin common blocked/direct site list
@@ -55,7 +64,7 @@
     * Support specifying host in blocked/direct file
     * User configurable timeout
     * Better windows support: connection reset, timeout and DNS error detection
-    tested and works on XP
+      tested and works on XP
     * Support listening multiple addresses
     * Support IP based and user password authentication
     * Various bug fixes
@@ -65,7 +74,7 @@
     * Performance improvement by better buffer usage
     * Allow specifying config file on command line
     * Better windows support: Config and domain list file on windows are put in the same
-    directory as COW's binary. And they all have txt extension for easy editing
+      directory as COW's binary. And they all have txt extension for easy editing
     * Bug fix: convert HTTP/1.0 response to HTTP/1.1
 
 0.3.4 (2012-12-09)
@@ -75,10 +84,10 @@
     * Allow specifying ssh server port in config file
     * Bug fix: crash when handling flush error
     * Bug fix: correctly handle web servers which use closed connection to
-    indicate end of response
+      indicate end of response
 
 0.3.3 (2012-12-05)
 
     * Keep HTTP CONNECT connection open. Avoid problems for Application which
-    uses long connection.
+      uses long connection.
     * Bug fix: crash when printing domain list inconsistency message.

README.md

@@ -1,34 +1,33 @@
 # COW (Climb Over the Wall) proxy
 
-COW 是一个利用二级代理帮助自动化翻墙的 HTTP 代理服务器。它能自动检测被墙网站，且仅对被墙网站使用二级代理。
+COW 是一个利用二级代理自动化穿越防火墙的 HTTP 代理服务器。它能自动检测被墙网站，仅对这些网站使用二级代理。
 
-当前版本：0.7.2 [CHANGELOG](CHANGELOG)
+当前版本：0.7.3 [CHANGELOG](CHANGELOG)
 [![Build Status](https://travis-ci.org/cyfdecyf/cow.png?branch=master)](https://travis-ci.org/cyfdecyf/cow)
 
-**如果要发 pull request，请在最新的 develop branch 上进行开发。**
-
-**0.7 版本配置文件更新说明**
-
-- 为支持指定多个 socks 代理，sshServer 配置语法有变化，[样例配置](doc/sample-config/rc)有详细说明
-- `socks`, `updateBlocked`, `updateDirect`, `autoRetry` 选项彻底移除，COW 遇到这些选项将**报错退出**
+**欢迎在 develop branch 进行开发并发送 pull request :)**
 
 ## 功能
 
-- 支持 HTTP, SOCKS5 和 [shadowsocks](https://github.com/shadowsocks/shadowsocks-go/) 作为二级代理
+COW 的设计目标是自动化，理想情况下用户无需关心哪些网站被封锁，可直连网站也不会因为使用二级代理而降低访问速度。作为 HTTP 代理，可以提供给移动设备使用；若部署在国内服务器上，可作为 APN 代理。
+
+- 支持 HTTP, SOCKS5 和 [shadowsocks](https://github.com/clowwindy/shadowsocks/wiki/Shadowsocks-%E4%BD%BF%E7%94%A8%E8%AF%B4%E6%98%8E) 作为二级代理
   - 可同时指定多个二级代理，支持简单的负载均衡
 - 自动检测网站是否被墙，仅对被墙网站使用二级代理
-  - 对未知网站，先尝试直接连接，失败后使用二级代理重试请求，2 分钟后再尝试直接
+  - **对未知网站，先尝试直接连接，失败后使用二级代理重试请求，2 分钟后再尝试直接**
   - 内置[常见被墙网站](site_blocked.go)，减少检测被墙所需时间（可手工添加）
-- 自动记录经常访问网站是否被墙
+- 自动记录经常访问网站能否直连
 - 提供 PAC 文件，直连网站绕过 COW
   - 内置[常见可直连网站](site_direct.go)，如国内社交、视频、银行、电商等网站（可手工添加）
 
 # 安装
 
-- **OS X, Linux:** 执行以下命令（也可用于更新，该安装脚本在 OS X 上可将 COW 设置为登录时启动）
+- **OS X, Linux:** 执行以下命令（也可用于更新）
 
         curl -s -L https://github.com/cyfdecyf/cow/raw/master/install-cow.sh | bash
 
+  - 该安装脚本在 OS X 上可将 COW 设置为登录时启动
+  - [Linux 启动脚本](doc/init.d/cow)，如何使用请参考注释（Debian 测试通过，其他 Linux 发行版应该也可使用）
 - **Windows:** 访问[这个网页](http://dl.chenyufei.info/cow/)下载
 - 如需其他平台二进制文件，请从源码安装
 
@@ -47,8 +46,10 @@ bug fix 和新功能在测试后会直接进入 master branch 而不等到发布
 启动 COW：
 
 - Unix 系统在命令行上执行 `cow &`
-  - [Linux 启动脚本](doc/init.d/cow) 在 Debian 上测试过，其他 Linux 发行版应该也可用
-- Windows 上执行 `cow-taskbar.exe` 即可
+- Windows
+  - 双击 `cow-taskbar.exe`，隐藏到托盘执行
+  - 双击 `cow-hide.exe`，隐藏为后台程序执行
+  - 以上两者都会启动 `cow.exe`
 
 PAC url 为 `http://<listen address>/pac`，也可将浏览器的 HTTP/HTTPS 代理设置为 `listen address` 使所有网站都通过 COW 访问。
 
@@ -58,7 +59,7 @@ PAC url 为 `http://<listen address>/pac`，也可将浏览器的 HTTP/HTTPS 代
 
 ## 手动指定被墙和直连网站
 
-**COW 的目标是自动化翻墙，一般情况下无需手工指定被墙和直连网站，该功能只是是为了处理特殊情况和性能优化。**
+**一般情况下无需手工指定被墙和直连网站，该功能只是是为了处理特殊情况和性能优化。**
 
 `~/.cow/blocked` 和 `~/.cow/direct` 可指定被墙和直连网站（`direct` 中的 host 会添加到 PAC）：
 
@@ -106,9 +107,11 @@ COW 默认配置下检测到被墙后，过两分钟再次尝试直连也是为
 贡献代码：
 
 @tevino: http parent proxy basic authentication
+@xupefei: 提供 cow-hide.exe 以在 windows 上在后台执行 cow.exe
 
-Bug report:
+Bug reporter:
 
-@glacjay, @fantasticfears, @hieixu, @Blaskyy, @lucifer9, @zellux, @JayXon, @graminc
+GitHub users: glacjay, trawor, Blaskyy, lucifer9, zellux, xream, hieixu, fantasticfears, perrywky, JayXon, graminc, WingGao, polong, dallascao
+Twitter users: @shao222
 
-@glacjay 对 0.3 版本的 COW 提出了让它更加自动化的建议，使我重新考虑 COW 的设计目标并且改进成 0.5 版本中的工作方式。
+@glacjay 对 0.3 版本的 COW 提出了让它更加自动化的建议，使我重新考虑 COW 的设计目标并且改进成 0.5 版本之后的工作方式。

auth.go

@@ -314,7 +314,7 @@ func authUserPasswd(conn *clientConn, r *Request) (err error) {
 		if err == nil {
 			return
 		} else if err != errAuthRequired {
-			sendErrorPage(conn, errCodeBadReq, "Bad authorization request", err.Error())
+			sendErrorPage(conn, statusBadReq, "Bad authorization request", err.Error())
 			return
 		}
 		// auth required to through the following

config.go

@@ -14,7 +14,7 @@ import (
 )
 
 const (
-	version           = "0.7.2"
+	version           = "0.7.3"
 	defaultListenAddr = "127.0.0.1:7777"
 )
 

doc/init.d/cow

@@ -1,21 +1,19 @@
 #!/bin/bash
-# Start/stop cow.
-#
 ### BEGIN INIT INFO
 # Provides:          cow
-# Required-Start:
-# Required-Stop:
-# Should-Start:
-# Should-Stop:
+# Required-Start:    $network
+# Required-Stop:     $network
 # Default-Start:     2 3 4 5
-# Default-Stop:
+# Default-Stop:      0 1 6
 # Short-Description: COW: Climb Over the Wall http proxy
 # Description:       Automatically detect blocked site and use parent proxy.
 ### END INIT INFO
 
+# Put this script under /etc/init.d/, then run "update-rc.d cow defaults".
+
 # Note: this script requires sudo in order to run COW as the specified
 # user. Please change the following variables in order to use this script.
-# COW will search for rc/direct/block/stat file under ~/.cow/ directory.
+# COW will search for rc/direct/block/stat file under user's $HOME/.cow/ directory.
 BIN=/usr/local/bin/cow
 USER=usr
 GROUP=grp

doc/logrotate.d/cow

@@ -0,0 +1,9 @@
+/var/log/cow {
+  rotate 4
+  weekly
+  compress
+  missingok
+  postrotate
+    /etc/init.d/cow restart
+  endscript
+}

estimate_timeout.go

@@ -53,10 +53,10 @@ func estimateTimeout() {
 	}
 	if est > config.DialTimeout {
 		dialTimeout = est
-		info.Println("new dial timeout:", dialTimeout)
+		debug.Println("new dial timeout:", dialTimeout)
 	} else if dialTimeout != config.DialTimeout {
 		dialTimeout = config.DialTimeout
-		info.Println("new dial timeout:", dialTimeout)
+		debug.Println("new dial timeout:", dialTimeout)
 	}
 
 	start = time.Now()
@@ -81,10 +81,10 @@ func estimateTimeout() {
 	}
 	if est > time.Duration(config.ReadTimeout) {
 		readTimeout = est
-		info.Println("new read timeout:", readTimeout)
+		debug.Println("new read timeout:", readTimeout)
 	} else if readTimeout != config.ReadTimeout {
 		readTimeout = config.ReadTimeout
-		info.Println("new read timeout:", readTimeout)
+		debug.Println("new read timeout:", readTimeout)
 	}
 	return
 onErr:

http.go

@@ -18,6 +18,7 @@ type Header struct {
 	ProxyAuthorization  string
 	Chunking            bool
 	ConnectionKeepAlive bool
+	ExpectContinue      bool
 }
 
 type rqState byte
@@ -125,6 +126,15 @@ func (r *Request) proxyRequestLine() []byte {
 	return r.raw.Bytes()[0:r.reqLnStart]
 }
 
+const (
+	statusCodeContinue = 100
+)
+
+const (
+	statusBadReq       = "400 Bad Request"
+	statusExpectFailed = "417 Expectation Failed"
+)
+
 type Response struct {
 	Status int
 	Reason []byte
@@ -289,6 +299,7 @@ const (
 	headerTrailer            = "trailer"
 	headerTransferEncoding   = "transfer-encoding"
 	headerUpgrade            = "upgrade"
+	headerExpect             = "expect"
 
 	fullHeaderConnection       = "Connection: keep-alive\r\n"
 	fullHeaderTransferEncoding = "Transfer-Encoding: chunked\r\n"
@@ -302,6 +313,7 @@ var headerParser = map[string]HeaderParserFunc{
 	headerProxyAuthorization: (*Header).parseProxyAuthorization,
 	headerProxyConnection:    (*Header).parseConnection,
 	headerTransferEncoding:   (*Header).parseTransferEncoding,
+	headerExpect:             (*Header).parseExpect,
 }
 
 var hopByHopHeader = map[string]bool{
@@ -315,9 +327,18 @@ var hopByHopHeader = map[string]bool{
 	headerUpgrade:            true,
 }
 
+// Note: Value bytes passed to header parse function are in the buffer
+// associated with bufio and will be modified. It will also be stored in the
+// raw request buffer, so becareful when modifying the value bytes. (Change
+// case only when the spec says it is case insensitive.)
+//
+// If Header needs to hold raw value, make a copy. For example,
+// parseProxyAuthorization does this.
+
 type HeaderParserFunc func(*Header, []byte, *bytes.Buffer) error
 
 func (h *Header) parseConnection(s []byte, raw *bytes.Buffer) error {
+	ASCIIToLowerInplace(s)
 	h.ConnectionKeepAlive = bytes.Contains(s, []byte("keep-alive"))
 	raw.WriteString(fullHeaderConnection)
 	return nil
@@ -329,6 +350,7 @@ func (h *Header) parseContentLength(s []byte, raw *bytes.Buffer) (err error) {
 }
 
 func (h *Header) parseKeepAlive(s []byte, raw *bytes.Buffer) (err error) {
+	ASCIIToLowerInplace(s)
 	id := bytes.Index(s, []byte("timeout="))
 	if id != -1 {
 		id += len("timeout=")
@@ -347,6 +369,7 @@ func (h *Header) parseProxyAuthorization(s []byte, raw *bytes.Buffer) error {
 }
 
 func (h *Header) parseTransferEncoding(s []byte, raw *bytes.Buffer) error {
+	ASCIIToLowerInplace(s)
 	// For transfer-encoding: identify, it's the same as specifying neither
 	// content-length nor transfer-encoding.
 	h.Chunking = bytes.Contains(s, []byte("chunked"))
@@ -359,6 +382,23 @@ func (h *Header) parseTransferEncoding(s []byte, raw *bytes.Buffer) error {
 	return nil
 }
 
+// For now, cow does not fully support 100-continue. It will return "417
+// expectation failed" if a request contains expect header. This is one of the
+// strategies supported by polipo, which is easiest to implement in cow.
+// TODO If we see lots of expect 100-continue usage, provide full support.
+
+func (h *Header) parseExpect(s []byte, raw *bytes.Buffer) error {
+	ASCIIToLowerInplace(s)
+	errl.Printf("Expect header: %s\n", s) // put here to see if expect header is widely used
+	h.ExpectContinue = true
+	/*
+		if bytes.Contains(s, []byte("100-continue")) {
+			h.ExpectContinue = true
+		}
+	*/
+	return nil
+}
+
 func splitHeader(s []byte) (name, val []byte, err error) {
 	var f [][]byte
 	if f = bytes.SplitN(s, []byte{':'}, 2); len(f) != 2 {
@@ -405,7 +445,7 @@ func (h *Header) parseHeader(reader *bufio.Reader, raw *bytes.Buffer, url *URL)
 			if len(val) == 0 {
 				continue
 			}
-			parseFunc(h, ASCIIToLower(val), raw)
+			parseFunc(h, val, raw)
 		} else {
 			// mark this header as not of interest to proxy
 			lastLine = nil
@@ -563,6 +603,13 @@ func parseResponse(sv *serverConn, r *Request, rp *Response) (err error) {
 		errl.Printf("Reading response header: %v %v\n", err, r)
 		return err
 	}
+
+	if rp.Status == statusCodeContinue && !r.ExpectContinue {
+		// not expecting 100-continue, just ignore it and read final response
+		errl.Println("Ignore server 100 response for", r)
+		return parseResponse(sv, r, rp)
+	}
+
 	// Connection close, no content length specification
 	// Use chunked encoding to pass content back to client
 	if !rp.ConnectionKeepAlive && !rp.Chunking && rp.ContLen == -1 {

install-cow.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-version=0.7.2
+version=0.7.3
 
 arch=`uname -m`
 case $arch in
@@ -72,7 +72,8 @@ fi
 
 # Download COW binary
 bin=cow-$os$arch-$version
-tmpbin=/tmp/cow
+tmpdir=`mktemp -d /tmp/cow.XXXXXX`
+tmpbin=$tmpdir/cow
 binary_url="http://dl.chenyufei.info/cow/$bin.gz"
 echo "Downloading cow binary $binary_url to $tmpbin.gz"
 curl -L "$binary_url" -o $tmpbin.gz || \
@@ -84,7 +85,9 @@ chmod +x $tmpbin ||
 # Download sample config file if no configuration directory present
 doc_base="https://github.com/cyfdecyf/cow/raw/master/doc"
 config_dir="$HOME/.cow"
+is_update=true
 if [ ! -e $config_dir ]; then
+    is_update=false
     sample_config_base="${doc_base}/sample-config"
     echo "Downloading sample config file to $config_dir"
     mkdir -p $config_dir || exit_on_fail "Can't create $config_dir directory"
@@ -115,9 +118,15 @@ else
     sudo mv $tmpbin $install_dir
 fi
 exit_on_fail "Failed to move $tmpbin to $install_dir"
+rmdir $tmpdir
 
 # Done
 echo
-echo "Installation finished."
-echo "Please edit $config_dir/rc according to your own settings."
-echo 'After that, execute "cow &" to start cow and run in background.'
+if $is_update; then
+    echo "Update finished."
+else
+    echo "Installation finished."
+    echo "Please edit $config_dir/rc according to your own settings."
+    echo 'After that, execute "cow &" to start cow and run in background.'
+fi
+