Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rec: move to embedded web service written in Rust #15114

Merged
merged 38 commits into from
Feb 14, 2025

Conversation

omoerbeek
Copy link
Member

Short description

This PR introduces a Rust written embedded web service

Main new features

  • Using a well maintained libraries (hyper and rusttls), instead of YaHTTP. We still use a few existing YaHTTP classes to represent http related data on the C++ side, but header only.
  • The actual method implementing the various (REST) functionality did not change, they are now called from Rust code.
  • We support multiple listen addresses and incoming TLS. It is also possible to listen on some addresses for plaintext http, and on some others require https. In contrast: current code only supports one listening address and no https.
  • A few regression tests now use https.
  • The settings directory a been changed to the more general rec-rust-lib as it contains settings and web related code. The produced static lib is also renamed. We work in one crate as having multiple crates is a bit of a pain in combination with a static lib. See https://cxx.rs/build/other.html#linking-the-c-and-rust-together. To be revisited in the future.

Future areas of improvement:

  • TLS config is very basic (e.g. no encrypted private keys or TLS version options)
  • Routing is done in code, it would be better to setup a table by the app
  • Review of enabled features in new imported crates.
  • Move JSON handling to Rust?
  • http2 support? I do wonder if that would bring us anything in the Recusor context.

The new authentication and authorization code needs a good review to establish it's indeed equivalent to the existing code.

Checklist

I have:

  • read the CONTRIBUTING.md document
  • compiled this code
  • tested this code
  • included documentation (including possible behaviour changes)
  • documented the code
  • added or modified regression test(s)
  • added or modified unit test(s)

@omoerbeek omoerbeek added rec enhancement dependencies Pull requests that update a dependency file rust Pull requests that update Rust code labels Feb 4, 2025
@omoerbeek omoerbeek added this to the rec-5.3.0 milestone Feb 4, 2025
@coveralls
Copy link

coveralls commented Feb 4, 2025

Pull Request Test Coverage Report for Build 13197595043

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 159 of 176 (90.34%) changed or added relevant lines in 6 files are covered.
  • 58 unchanged lines in 15 files lost coverage.
  • Overall coverage increased (+0.06%) to 64.785%

Changes Missing Coverage Covered Lines Changed/Added Lines %
pdns/recursordist/rec-web-stubs.hh 0 1 0.0%
pdns/recursordist/rec-rust-lib/cxxsupport.cc 27 29 93.1%
pdns/recursordist/rec-main.cc 10 13 76.92%
pdns/recursordist/ws-recursor.cc 111 122 90.98%
Files with Coverage Reduction New Missed Lines %
pdns/webserver.hh 1 83.33%
modules/gpgsqlbackend/gpgsqlbackend.cc 1 88.62%
ext/json11/json11.cpp 1 64.49%
pdns/recursordist/ws-recursor.cc 1 66.74%
pdns/dnsdistdist/dnsdist.cc 2 68.73%
pdns/recursordist/syncres.hh 3 87.67%
pdns/recursordist/syncres.cc 3 80.23%
pdns/iputils.cc 3 56.07%
pdns/opensslsigners.cc 3 61.34%
pdns/rcpgenerator.cc 3 90.82%
Totals Coverage Status
Change from base Build 13177913388: 0.06%
Covered Lines: 128238
Relevant Lines: 166847

💛 - Coveralls

pdns/recursordist/ws-recursor.cc Dismissed Show dismissed Hide dismissed
Copy link
Contributor

@miodvallat miodvallat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take these with a large grain of salt due to my rusty skills.

pdns/recursordist/ws-recursor.cc Outdated Show resolved Hide resolved
pdns/recursordist/ws-recursor.cc Outdated Show resolved Hide resolved
pdns/recursordist/ws-recursor.cc Outdated Show resolved Hide resolved
pdns/recursordist/rec-rust-lib/docs-new-preamble-in.rst Outdated Show resolved Hide resolved
using Logger = ::Logr::Logger;
struct KeyValue;

template <typename A>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to be future-proof:

Suggested change
template <typename A>
template <typename AAAA>

pdns/recursordist/rec-rust-lib/rust/src/web.rs Outdated Show resolved Hide resolved
.build()?;

// For each listening address we spawn a tokio handler an then a single Posix thread is created that
// waits (forever) for all of them to complete by joining them all.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// waits (forever) for all of them to complete by joining them all.
// waits (forever) for all of them to complete by joining them all and in the darkness bind them all.

pdns/recursordist/rec-rust-lib/rust/src/web.rs Outdated Show resolved Hide resolved
@omoerbeek omoerbeek force-pushed the rec-rust-web branch 2 times, most recently from dd3b266 to a759843 Compare February 7, 2025 08:31
Copy link
Contributor

@miodvallat miodvallat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more typos

pdns/recursordist/rec-rust-lib/rust/src/web.rs Outdated Show resolved Hide resolved
pdns/recursordist/rec-rust-lib/rust/src/web.rs Outdated Show resolved Hide resolved
pdns/recursordist/rec-rust-lib/rust/src/web.rs Outdated Show resolved Hide resolved
value: String,
}

// Clippy does not seem to understand what cxx does and complains about needless_lifetimes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clippy does not seem to understand what cxx does we don't either...

Copy link
Member

@rgacogne rgacogne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a big PR but it looks good, nice work! I did not realize before that the logging was done in the rust code as well, I'll have to figure out how to handle that if I want to integrate this in DNSdist (but I'll wait until our structured logging code is ready anyway).

pdns/recursordist/rec-rust-lib/rust/build.rs Outdated Show resolved Hide resolved
pdns/recursordist/rec-rust-lib/rust/src/web.rs Outdated Show resolved Hide resolved
pdns/recursordist/rec-rust-lib/rust/src/web.rs Outdated Show resolved Hide resolved
.urls
.iter()
.position(|x| String::from("/") + x == uripath);
if pos.is_none() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not clear to me why we are doing this lookup?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Originally I had a plan to fully implement the file access in Rust, but that plan changed. Some remains are left, will remove.

pdns/recursordist/ws-recursor.cc Outdated Show resolved Hide resolved
regression-tests.api/runtests.py Outdated Show resolved Hide resolved
regression-tests.api/test_helper.py Outdated Show resolved Hide resolved
regression-tests.recursor-dnssec/test_Prometheus.py Outdated Show resolved Hide resolved
@omoerbeek
Copy link
Member Author

Rebased to fix conflicts

@omoerbeek
Copy link
Member Author

Hmm, I hate it when a rebase shows no issues but the result does.

omoerbeek and others added 2 commits February 12, 2025 10:54
Co-authored-by: Remi Gacogne <github@coredump.fr>
@omoerbeek
Copy link
Member Author

omoerbeek commented Feb 12, 2025

This is a big PR but it looks good, nice work! I did not realize before that the logging was done in the rust code as well, I'll have to figure out how to handle that if I want to integrate this in DNSdist (but I'll wait until our structured logging code is ready anyway).

I'll check if it is possible to do this via callbacks or whatever, so you can plug in your own logging function.

Answer: in misc.rs Logger (and thus SharedPtr<Logger>) are used as opaque C++ objects, so it should be ealtively easy to provide alternative implementations of `pdns::rust::misc::{withValue,log,error} in the C++ support code, probably with some #preprocessor magic.
But if dnsdist will move to stuctured logging anyway, this will probably not be needed. In recursor we have the luxury that old-style logging is no longer supported, so mapping logging to structured logging from Rust to C++ was pretty easy. Dunno if dnsdist wil have a two way logging thing for some time.

@omoerbeek omoerbeek merged commit cc7d8cd into PowerDNS:master Feb 14, 2025
83 checks passed
@omoerbeek omoerbeek deleted the rec-rust-web branch February 14, 2025 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file enhancement rec rust Pull requests that update Rust code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants