Skip to content

Commit

Permalink
Merge branch 'main' into phillip/--help-fix
Browse files Browse the repository at this point in the history
  • Loading branch information
phillip-stephens authored Jan 10, 2025
2 parents 5514a6b + 1dc7268 commit e45ba99
Show file tree
Hide file tree
Showing 8 changed files with 543 additions and 264 deletions.
122 changes: 122 additions & 0 deletions .github/workflows/check_tlsa_integration_test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
name: Update TLSA records in integration tests
# Our integration tests run against "_25._tcp.mail.ietf.org" which periodically changes its TLSA records.
# This action uses 'dig' to make a PR whenever the TLSA record is updated.

on:
schedule:
- cron: "0 12 * * *" # Runs daily at 12:00 UTC
workflow_dispatch: # Allows manual runs

jobs:
update-tlsa:
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Install required tools
run: sudo apt-get update && sudo apt-get install -y dnsutils jq

- name: Query TLSA record
id: fetch_tlsa
run: |
URL="_25._tcp.mail.ietf.org"
DIG_OUTPUT=$(dig -t TLSA "$URL" +short)
if [ -z "$DIG_OUTPUT" ]; then
echo "TLSA record not found."
exit 1
fi
# Start JSON array
echo "[" > tlsa.json
FIRST=true
# Parse dig output
echo "$DIG_OUTPUT" | while read -r line; do
CERT_USAGE=$(echo "$line" | awk '{print $1}')
SELECTOR=$(echo "$line" | awk '{print $2}')
MATCHING_TYPE=$(echo "$line" | awk '{print $3}')
CERT=$(echo "$line" | awk '{print $4 $5}' | tr '[:upper:]' '[:lower:]')
# Add a comma before each entry except the first
if [ "$FIRST" = true ]; then
FIRST=false
else
echo "," >> tlsa.json
fi
# Write JSON entry
echo "{" \
"\"type\": \"TLSA\"," \
"\"class\": \"IN\"," \
"\"name\": \"$URL\"," \
"\"cert_usage\": $CERT_USAGE," \
"\"selector\": $SELECTOR," \
"\"matching_type\": $MATCHING_TYPE," \
"\"certificate\": \"$CERT\"" \
"}" >> tlsa.json
done
echo "]" >> tlsa.json
echo "Parsed TLSA records:"
cat tlsa.json | jq .
- name: Update test file
id: update_test
run: |
TEST_FILE="testing/integration_tests.py"
# Pretty-format the JSON content
TLSA_ANSWERS=$(cat tlsa.json | jq .)
# Use `gawk` to preserve indentation in the Python file
gawk -v new_content="$TLSA_ANSWERS" '
BEGIN { RS = ""; ORS = "\n\n" }
/TLSA_ANSWERS = \[/ {
# Extract leading whitespace for indentation preservation
match($0, /^[[:space:]]*/)
indent = substr($0, RSTART, RLENGTH)
# Break JSON content into lines and add proper indentation
split(new_content, lines, "\n")
formatted_content = indent "TLSA_ANSWERS = ["
for (i = 2; i <= length(lines) - 1; i++) {
formatted_content = formatted_content "\n" indent " " lines[i]
}
formatted_content = formatted_content "\n" indent "]"
# Replace the matched block with the formatted JSON
$0 = formatted_content
}
1
' "$TEST_FILE" > temp_file && mv temp_file "$TEST_FILE"
# Re-format with `black` to ensure consistent style
pip3 install black
black "$TEST_FILE"
echo "Updated $TEST_FILE with properly indented TLSA records."
# Check if the file was updated
if ! git diff --exit-code "$TEST_FILE"; then
echo "File updated."
echo "file_updated=true" >> $GITHUB_ENV
else
echo "No changes detected."
fi
# Cleanup temp file
rm tlsa.json
- name: Create Pull Request
if: env.file_updated == 'true'
uses: peter-evans/create-pull-request@v5
with:
title: "Update TLSA records"
body: "This PR updates the TLSA records in the test file."
base: main
branch: update-tlsa-record-${{ github.run_id }}
delete-branch: false
author: "github-actions[bot] <github-actions[bot]@users.noreply.github.com>"
draft: true
20 changes: 20 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,14 @@ on:
branches:
- main
pull_request:
types: [opened, ready_for_review, synchronize]
workflow_dispatch:

jobs:
check-license:
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
runs-on: ubuntu-latest
steps:
- name: Checkout
Expand Down Expand Up @@ -68,3 +73,18 @@ jobs:
fi
- name: golangci-lint
uses: golangci/golangci-lint-action@v6.1.1

# Set up Python for black
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11' # Adjust to your required Python version

# Install black
- name: Install black
run: pip3 install black

# Check Python files with black
- name: Check Python Code Formatting
run: |
black --check .
11 changes: 4 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,21 +9,18 @@ optimized for performing lookups of a diverse set of names. We use
https://github.com/zmap/dns to construct and parse raw DNS packets.
For more information about ZDNS's architecture and performance, check out the following [paper](https://lizizhikevich.github.io/assets/papers/ZDNS.pdf) appearing at ACM's Internet Measurement Conference '22.

As an example, the following will perform MX lookups and a secondary A lookup
for the IPs of MX servers for the domains in the Alexa Top Million:

cat top-1m.csv | ./zdns MX --ipv4-lookup --alexa

> [!TIP]
> The [ZDNS Wiki](https://github.com/zmap/zdns/wiki) contains additional information on ZDNS and walks thru use-cases and examples.
Install
=======

ZDNS can be installed by checking out the repository and running `make zdns`.
ZDNS can be installed by checking out the repository and running `make install`.

```bash
git clone https://github.com/zmap/zdns.git
cd zdns
make zdns
make install
```

Usage
Expand Down
2 changes: 2 additions & 0 deletions makefile
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ lint:
goimports -w -local "github.com/zmap/zdns" ./
gofmt -s -w ./
golangci-lint run
@if ! command -v black >/dev/null 2>&1; then pip3 install black; fi
black --check ./

license-check:
./.github/workflows/check_license.sh
Expand Down
2 changes: 1 addition & 1 deletion src/cli/cli.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ type StatusHandler interface {
type GeneralOptions struct {
LookupAllNameServers bool `long:"all-nameservers" description:"Behavior is dependent on --iterative. In --iterative, --all-name-servers will query all root servers, then all gtld servers, etc. recording the responses at each layer. In non-iterative mode, the query will be sent to all external resolvers specified in --name-servers."`
CacheSize int `long:"cache-size" default:"10000" description:"how many items can be stored in internal recursive cache"`
GoMaxProcs int `long:"go-processes" default:"0" description:"number of OS processes (GOMAXPROCS by default)"`
GoMaxProcs int `long:"go-processes" default:"0" description:"number of OS processes to use, GOMAXPROCS if 0"`
IterationTimeout int `long:"iteration-timeout" default:"8" description:"timeout for a single iterative step in an iterative query, in seconds. Only applicable with --iterative"`
IterativeResolution bool `long:"iterative" description:"Perform own iteration instead of relying on recursive resolver"`
MaxDepth int `long:"max-depth" default:"10" description:"how deep should we recurse when performing iterative lookups"`
Expand Down
Loading

0 comments on commit e45ba99

Please sign in to comment.