Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V0.1.0 release #14

Merged
merged 3 commits into from
Oct 13, 2024
Merged

V0.1.0 release #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
789d399
update README and plugin metadata
haohanyang Oct 13, 2024
ef74bba
Update screenshot
haohanyang Oct 13, 2024
f01948a
update title
haohanyang Oct 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
288 changes: 164 additions & 124 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,97 +1,50 @@
# Grafana MongoDB data source
![example branch parameter](https://github.com/haohanyang/mongodb-datasource/actions/workflows/ci.yml/badge.svg?branch=master)
# Grafana MongoDB Data Source

This plugin provides a Grafana datasource for querying and visualizing data from MongoDB.
**Visualize your MongoDB data in Grafana with powerful aggregation queries.**

![ci](https://github.com/haohanyang/mongodb-datasource/actions/workflows/ci.yml/badge.svg?branch=master)

This plugin enables you to query and visualize data from your MongoDB databases directly within Grafana. Leverage the flexibility of MongoDB's aggregation pipeline to create insightful dashboards and panels.

![screenshot](/static/screenshot.png)

## Download
[Download the latest build](https://github.com/haohanyang/mongodb-datasource/actions/runs/11313289435/artifacts/2049966955)
## Features

- **Flexible Querying:** Craft precise queries using MongoDB's aggregation pipeline syntax in JSON or JavaScript.
- **Time Series & Table Data:** Visualize time-based data or display results in tabular format for various Grafana panels.
- **Secure Execution:** Execute JavaScript queries within a secure ShadowRealm sandbox to control access.
- **Legacy Plugin Compatibility:** Seamlessly migrate from the legacy plugin with support for its query syntax.
- **Up-to-date:** Use the latest Grafana Plugin SDKs and follow best practices.

## Use
### Query language
The query text should be a valid MongoDB Aggregate pipeline - an array consisting of MongoDB Aggregate operations. Your may use the Grafana's built-in variables `"$__from"` and `"$__to"` to query data based on the current panel's time range. The plugin supports JSON and JavaScript query languages. In JSON query, you need to enter the database in the UI. Here is an example of JSON query.
```json
[
{
"$match": {
"createdTime": {
"$gt": {
"$date": {
"$numberLong": "$__from"
}
},
"$lt": {
"$date": {
"$numberLong": "$__to"
}
}
}
}
}
]
```
In JavaScript query, you need to follow the format `db.<collection-name>.aggregate([<json-query>])`. As shows in the following example.
```js
db.transactions.aggregate([
{
"$match": {
"createdTime": {
"$gt": {
"$date": {
"$numberLong": "$__from"
}
},
"$lt": {
"$date": {
"$numberLong": "$__to"
}
}
}
}
}
]);
```
## Getting Started
1. **Download:** Obtain the latest plugin build from the [Release page](https://github.com/haohanyang/mongodb-datasource/releases) or [workflow artifacts](https://github.com/haohanyang/mongodb-datasource/actions?query=branch%3Amaster).

You can also use the `"$from"`, `"$to"` and `"$dateBucketCount"`(number of intervals in the time range) conventions originated from a legacy plugin.
```json
[
{
"$match": {
"createdTime": {
"$gt": "$from",
"$lt": "$to"
}
}
}
]
```
```json
{
"$bucketAuto": {
"groupBy": "$timestamp",
"buckets": "$dateBucketCount",
//
}
}
```
### Query type
#### Time series
Time series query type is suitable for time series panels. The query results should consist the following fields:
* `ts`
The timestamp
* `value`
Either integer or double. Currently all values should be of the same type.
* `name` (Optional)
Useful if you want to show time series data of different categories.

Here is an example of JSON query from [Sample AirBnB Listings Dataset](https://www.mongodb.com/docs/atlas/sample-data/sample-airbnb/). The query shows the number of created AirBnB reviews of apartment and house property type in each month during the selected time range.
2. **Install:**
- Extract the downloaded archive (`haohanyang-mongodb-datasource-<version>.zip`) into your Grafana plugins directory (`/var/lib/grafana/plugins` or similar).
- Ensure the plugin binaries (`mongodb-datasource/gpx_mongodb_datasource_*`) have execute permissions (`chmod +x`).
```bash
chmod 0755 mongodb-datasource/gpx_mongodb_datasource_*
```
- Configure the plugin as a data source within Grafana, providing your MongoDB connection details.

Refer to the [example docker-compose.prod.yaml](/docker-compose.prod.yaml) file for a production-ready setup.

3. **Start Querying:**
- Select your MongoDB data source in a Grafana panel.
- Use the query editor to write your aggregation pipeline queries (see Query Language below).

## Query Language

### JSON (Recommended)

Provide the collection name and your MongoDB aggregation pipeline in standard JSON format.

**Example:** Retrieve 10 AirBnB listings scraped within the selected time range:
```json
[
{
"$match": {
"first_review": {
"last_scraped": {
"$gt": {
"$date": {
"$numberLong": "$__from"
Expand All @@ -102,54 +55,141 @@ Here is an example of JSON query from [Sample AirBnB Listings Dataset](https://w
"$numberLong": "$__to"
}
}
},
"property_type": {
"$in": [
"Apartment",
"House"
]
}
}
},
{
"$group": {
"_id": {
"month": {
"$dateToString": {
"format": "%Y-%m",
"date": "$first_review"
}
},
"property_type": "$property_type"
},
"value": {
"$count": {}
}
}
},
{
"$project": {
"ts": {
"$toDate": "$_id.month"
},
"name": "$_id.property_type",
"value": 1
}
"$limit": 10
}
]
```
#### table
Table type is more flexible and doesn't require the output schema. This usually fits panels that don't require timestamp.

### JavaScript (Legacy & ShadowRealm)

## Install
* Download the packaged plugin `haohanyang-mongodb-datasource-<version>.zip` from [workflow artifacts](#download) to the root directory (where `docker-compose.yaml` exists) and extract files to folder `mongodb-datasource`
- **Legacy:** Maintain compatibility with the older plugin's syntax:
```javascript
db.listingsAndReviews.aggregate([ /* Your aggregation pipeline (JSON) */ ]);
```
This gives the same result as the previous JSON query.
```js
db.listingsAndReviews.aggregate([
{
"$match": {
"last_scraped": {
"$gt": {
"$date": {
"$numberLong": "$__from"
}
},
"$lt": {
"$date": {
"$numberLong": "$__to"
}
}
}
}
},
{
"$limit": 10
}
])
```
- **ShadowRealm (Secure):** Define an `aggregate()` function that returns your pipeline. The function executes within a [ShadowRealm](https://github.com/tc39/proposal-shadowrealm) sandboxed environment.
```javascript
function aggregate() {
// ... your logic based on template variables ...
return [ /* Your aggregation pipeline */ ];
}
```
In this example, only the admin user to can view the query result.
```js
function aggregate() {
const user = "${__user.login}"
let filter = {}
if (user !== "admin") {
filter = {
noop: true
}
}
return [
{
$match: filter
},
{
$limit: 10
}
]
}
```

```bash
unzip haohanyang-mongodb-datasource-<version>.zip -d mongodb-datasource
```
* Grant `0755`(`-rwxr-xr-x`) permission to binaries in `mongodb-datasource` directory.
```bash
chmod 0755 mongodb-datasource/gpx_mongodb_datasource_*
```
* Create/start the docker container as descriped in the [example docker compose file](/docker-compose.prod.yaml)
### Query Types

- **Time series:** For time-based visualizations. Your query must return documents with `ts` (timestamp) and `value` fields. An optional `name` field enables grouping by category.

The following query of [Sample AirBnB Listings Dataset](https://www.mongodb.com/docs/atlas/sample-data/sample-airbnb/) shows the number of AirBnB listings in each month that have the first review in the selected time range.
```json
[
{
"$match": {
"first_review": {
"$gt": {
"$date": {
"$numberLong": "$__from"
}
},
"$lt": {
"$date": {
"$numberLong": "$__to"
}
}
}
}
},
{
"$group": {
"_id": {
"month": {
"$dateToString": {
"format": "%Y-%m",
"date": "$first_review"
}
},
"property_type": "$property_type"
},
"value": {
"$count": {}
}
}
},
{
"$project": {
"ts": {
"$toDate": "$_id.month"
},
"name": "$_id.property_type",
"value": 1
}
}
]
```
- **Table:** For more flexible data display in tables, pie charts, etc. No specific output schema is required.

## Supported Data Types

| BSON Type | Support | Go Type | Notes |
|-----------------------|---------|-------------------|-------------------------------------------|
| Double | ✅ | float64 | |
| String | ✅ | string | |
| Object | ✅ | json.RawMessage | May be converted to string if necessary |
| Array | ✅ | json.RawMessage | May be converted to string if necessary |
| ObjectId | ✅ | string | |
| Boolean | ✅ | bool | |
| Date | ✅ | time.Time | |
| Null | ✅ | nil | |
| 32-bit integer | ✅ | int32 | May be converted to int64/float64 |
| 64-bit integer | ✅ | int64 | May be converted to float64 |

**Note:** Unsupported BSON types are not included in the table and will display as `"[Unsupported type]"`.

## License
Apache-2.0
4 changes: 3 additions & 1 deletion src/img/logo.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
16 changes: 11 additions & 5 deletions src/plugin.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,30 @@
{
"$schema": "https://raw.githubusercontent.com/grafana/grafana/main/docs/sources/developers/plugins/plugin.schema.json",
"type": "datasource",
"name": "Mongodb-Datasource",
"name": "MongoDB",
"id": "haohanyang-mongodb-datasource",
"metrics": true,
"backend": true,
"executable": "gpx_mongodb_datasource",
"info": {
"description": "",
"description": "MongoDB integration and data source",
"author": {
"name": "Haohan yang"
"name": "Haohan Yang"
},
"keywords": [
"datasource"
"datasource",
"mongodb"
],
"logos": {
"small": "img/logo.svg",
"large": "img/logo.svg"
},
"links": [],
"links": [
{
"name": "Source Code",
"url": "https://github.com/haohanyang/mongodb-datasource"
}
],
"screenshots": [],
"version": "%VERSION%",
"updated": "%TODAY%"
Expand Down
Binary file modified static/screenshot.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.