DNAQuery is a command line utility to take LogDNA archives and load them into BigQuery. This allows long-term, queryable storage of logs in BigQuery (which is considerably more cost effective). In our use case, we want real-time access to many different logs and long-term storage of a subset of those logs. This approach has helped us find a balance between access and cost.
go get -u github.com/lifechurch/dnaquery
DNAQuery has been tested on Go v1.11
- a LogDNA account with GCS Archiving enabled
- a Google Cloud Platform account
cp example.toml dnaquery.toml
Edit dnaquery.toml
All of these settings are currently required.
[storage]
LogDirectory = 'logs' # temp local directory to download logs to and store results, this directory will be created if it doesn't exist
[[apps]]
Name = "production-app" # name of app
Regex = '^([\d.]+) - \[([^\]]*)\] - - \[([^\]]*)\] "([^"]*)" (\d+) (\d+) "([^"]*)" "([^"]*)" (\d+) ([\d.]+) ([\d.:]+) (\d+) ([\d.]+) (\d+)$' # regex used to pull parts of logs out, currently we don't use any named capture groups
TimeGroup = 3 # the number of the capture group (1-based) in above regex that holds the time component of the log
TimeFormat = "2/Jan/2006:15:04:05 -0700" # the format of the time field using https://golang.org/pkg/time/#Parse
[[apps.excludes]] # an array of tables for exclusions
Group = 4 # the number of the capture group in above regex to be used in the exclusion check
Contains = "ping" # excludes logs if the value here is contained in the string in the above capture group
[gcp] # Google Cloud Project settings
ProjectID = "gcpproj" # name of GCP project
CredentialsFile = "gcp_credentials.json" # relative or absolute path to the credentials file downloaded from GCP
Bucket = "logdna_to_bq" # name of bucket in Google Cloud Storage to save results for ingestion into BigQuery, bucket will need to be created before first run
Dataset = "logdna" # BigQuery dataset
TemplateTable = "logdna" # currently DNAQuery uses a template table. More details below.
LogBucket = "logs" # name of bucket set in LogDNA archive setup above
LogPrefix = "a7112abc9d" # each archive file starts with a prefix specific to your LogDNA account
Currently the schema for the BigQuery table is specified by creating a table with the schema that matches the regex. This is likely to change in the future as we may need a different schema for each app above.
dnaquery --date 2017-11-20
This project uses dep for dependency management
go get -u github.com/golang/dep/cmd/dep
dep ensure -update
go test -v ./...
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests.
We use SemVer for versioning. For the versions available, see the tags on this repository.
- Chris Vaughn - Initial work - chrisvaughn
See also the list of contributors who participated in this project.
This project is licensed under the MIT License - see the LICENSE.md file for details