Skip to content

Commit

Permalink
Increase JSON scraper max buffer to 1MB (#112)
Browse files Browse the repository at this point in the history
Sometimes applications, especially Java based, can produce huge log
lines. The executor does not use much memory so extra megabyte should
not be a problem and it gives us some reserve of memory (so we can
avoid some bufio.ErrTooLong errors). Default max buffer for
bufio.Scanner was 64 kilobytes.
  • Loading branch information
medzin authored Apr 26, 2018
1 parent a27cc69 commit d5127b7
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions servicelog/scraper/json.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@ import (
"github.com/allegro/mesos-executor/servicelog"
)

const (
kilobyte = 1024
megabyte = 1024 * kilobyte
)

// JSON is a scraper for logs represented as JSON objects.
type JSON struct {
KeyFilter Filter
Expand All @@ -21,6 +26,7 @@ type JSON struct {
// as the passed reader does not return an io.EOF error.
func (j *JSON) StartScraping(reader io.Reader) <-chan servicelog.Entry {
scanner := bufio.NewScanner(reader)
scanner.Buffer(make([]byte, 64*kilobyte), megabyte)
logEntries := make(chan servicelog.Entry)

go func() {
Expand Down

0 comments on commit d5127b7

Please sign in to comment.