Skip to content

Commit

Permalink
Update benchmarks (#51)
Browse files Browse the repository at this point in the history
* Update irrd used in benchmark to latest version

* Only benchmark irrd parser, avoid matching object type and validating attribute data

* Update benchmark results

* Add vega graph and spec to benchmark

* Update README to reflect updated benchmarks
  • Loading branch information
SRv6d authored Dec 25, 2023
1 parent 66abe2e commit c3ee584
Show file tree
Hide file tree
Showing 6 changed files with 212 additions and 11 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

An [RFC 2622] conformant Routing Policy Specification Language (RPSL) parser with a focus on speed and correctness.

⚡️ Outperforms other parsers by a factor of 33-60x\
⚡️ 130-250x faster than other parsers\
📰 Complete implementation for multiline RPSL values\
💬 Able to parse objects directly from whois server responses\
🧠 Low memory footprint by leveraging zero-copy\
Expand Down
15 changes: 8 additions & 7 deletions docs/benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,14 @@ Benchmarks comparing performance to other RPSL parsers.

## Results

| Parser | Mean | Min | Max |
| ----------------- | ----------------- | --------- | --------- |
| **[rpsl-parser]** | 1.86 ms | 1.8642 ms | 1.8693 ms |
| [RPSL::Parser] | 61.8 ms ± 2.5 ms | 60.1 ms | 74.7 ms |
| [irrdnet/irrd] | 98.6 ms ± 3.4 ms | 91.2 ms | 110.9 ms |
| [RIPE-NCC/whois] | 114.7 ms ± 6.3 ms | 106.5 ms | 124.6 ms |
![graph](graph.svg)

| Parser | Mean | Min | Max |
| ---------------- | ----------------- | --------- | --------- |
| **rpsl-parser** | **444.61 µs** | 444.96 µs | 445.38 µs |
| [RPSL::Parser] | 61.8 ms ± 2.5 ms | 60.1 ms | 74.7 ms |
| [RIPE-NCC/whois] | 114.7 ms ± 6.3 ms | 106.5 ms | 124.6 ms |
| [irrdnet/irrd] | 114.8 ms ± 0.9 ms | 113.5 ms | 116.7 ms |

_Parsing of the AS3257 aut-num object on a 2022 M1 Max._

Expand All @@ -24,7 +26,6 @@ Benchmarks for the parser itself are done using `cargo bench`, while any externa
To run specific benchmarks, execute `/.run` in the directory of the respective parser.
It sets up dependencies and runs `sudo` so it is only recommended to be used in an isolated environment and is only tested on Ubuntu 22.04, other platforms might require additional dependencies to be installed.

[rpsl-parser]: https://github.com/srv6d/rpsl-parser
[RPSL::Parser]: https://metacpan.org/pod/RPSL::Parser
[irrdnet/irrd]: https://github.com/irrdnet/irrd
[RIPE-NCC/whois]: https://github.com/RIPE-NCC/whois
Expand Down
199 changes: 199 additions & 0 deletions docs/benchmark/graph-spec.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
{
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
"data": {
"values": [
{
"parser": "rpsl-parser",
"time": 0.44461,
"timeFormat": "444µs"
},
{
"parser": "RPSL::Parser",
"time": 61.8,
"timeFormat": "61.8ms"
},
{
"parser": "RIPE-NCC/whois",
"time": 114.7,
"timeFormat": "> 100ms"
},
{
"parser": "irrdnet/irrd",
"time": 114.8,
"timeFormat": "> 100ms"
}
]
},
"config": {
"params": [
{
"name": "defaultFont",
"value": "-apple-system,BlinkMacSystemFont,\"Segoe UI\",Helvetica,Arial,sans-serif,\"Apple Color Emoji\",\"Segoe UI Emoji\""
},
{
"name": "titleColor",
"value": "#333333"
},
{
"name": "labelColor",
"value": "#333333"
}
],
"header": {
"labelFont": {
"expr": "defaultFont"
},
"titleFont": {
"expr": "defaultFont"
},
"titleFontWeight": 500
},
"text": {
"font": {
"expr": "defaultFont"
},
"color": {
"expr": "labelColor"
}
},
"mark": {
"font": {
"expr": "defaultFont"
},
"color": {
"expr": "labelColor"
}
},
"title": {
"font": {
"expr": "defaultFont"
},
"subtitleFont": {
"expr": "defaultFont"
},
"fontWeight": 500
},
"axis": {
"labelColor": {
"expr": "labelColor"
},
"labelFont": {
"expr": "defaultFont"
},
"titleFont": {
"expr": "defaultFont"
},
"titleFontWeight": 500,
"titleColor": {
"expr": "titleColor"
},
"titleFontSize": 12
},
"legend": {
"titleFontWeight": 500,
"titleColor": {
"expr": "titleColor"
},
"titleFontSize": 12,
"labelColor": {
"expr": "labelColor"
},
"labelFont": {
"expr": "defaultFont"
},
"titleFont": {
"expr": "defaultFont"
}
},
"view": {
"stroke": null
},
"background": "transparent"
},
"background": "transparent",
"encoding": {
"y": {
"field": "parser",
"type": "nominal",
"axis": {
"grid": false,
"title": null,
"labelFontSize": 12,
"ticks": false,
"labelPadding": 10,
"domain": false
},
"sort": null
},
"x": {
"field": "time",
"type": "quantitative",
"axis": {
"title": null,
"labelExpr": "datum.value + 'ms'",
"tickCount": 3,
"tickSize": 0,
"labelPadding": 6,
"labelAlign": "center",
"labelFontSize": 12,
"tickColor": "rgba(127,127,127,0.25)",
"gridColor": "rgba(127,127,127,0.25)",
"domain": false
}
}
},
"height": 140,
"width": "container",
"layer": [
{
"mark": "bar",
"encoding": {
"size": {
"value": 13
},
"color": {
"value": "#E15759"
}
}
},
{
"transform": [
{
"filter": "datum.parser !== 'rpsl-parser'"
}
],
"mark": {
"type": "text",
"align": "left",
"baseline": "middle",
"dx": 6,
"fontSize": 12
},
"encoding": {
"text": {
"field": "timeFormat"
}
}
},
{
"transform": [
{
"filter": "datum.parser === 'rpsl-parser'"
}
],
"mark": {
"type": "text",
"align": "left",
"baseline": "middle",
"dx": 6,
"fontSize": 12,
"fontWeight": "bold"
},
"encoding": {
"text": {
"field": "timeFormat"
}
}
}
]
}
1 change: 1 addition & 0 deletions docs/benchmark/graph.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions docs/benchmark/irrdnet_irrd/main.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env python3

from irrd.rpsl.rpsl_objects import rpsl_object_from_text
from irrd.rpsl.parser import RPSLObject

AS3257 = """aut-num: AS3257
as-name: GTT-BACKBONE
Expand Down Expand Up @@ -9571,4 +9571,4 @@
source: RIPE
"""

rpsl_object_from_text(AS3257)
RPSLObject()._extract_attributes_values(AS3257)
2 changes: 1 addition & 1 deletion docs/benchmark/irrdnet_irrd/requirements.txt
Original file line number Diff line number Diff line change
@@ -1 +1 @@
irrd==4.3.0.post1
irrd==4.4.2

0 comments on commit c3ee584

Please sign in to comment.