-
Notifications
You must be signed in to change notification settings - Fork 4
Diagnostic queries
Helpful queries for checking whether everything is working correctly.
The most useful high level diagnostic query checks that the latest synced header, transformed event, transformed diff, and synced diff are all close in terms of block time, indicating that the data synchronization layer is still storing headers and diffs, and the transformation layer is still processing them.
{
allHeaders(last: 1) {
nodes {
blockNumber
}
}
getMaxTransformedEventBlock
getMaxTransformedDiffBlock
allStorageDiffs(last: 1) {
nodes {
blockHeight
}
}
}Here's an example response:
{
"data": {
"allHeaders": {
"nodes": [
{
"blockNumber": "13766042"
}
]
},
"getMaxTransformedEventBlock": "13765997",
"getMaxTransformedDiffBlock": "13766041",
"allStorageDiffs": {
"nodes": [
{
"blockHeight": "13766041"
}
]
}
},
"meta": {
"graphqlQueryCost": 1
}
}The blockNumber nested in allHeaders and blockHeight nested in allStorageDiffs represent the block heights of the latest raw data. Here you can see that they are just one block apart. (Not every block will have an associated storage diff—only blocks that include a change to the MCD contracts we're following).
getMaxTransformedEventBlock and getMaxTransformedDiffBlock return the latest block for which we have a transformed event or diff. If these are lagging far behind the head of the chain (i.e. 250+ blocks), the transformation layer (vdb-mcd-execute) may have slowed or stopped.
Note that these transformed block heights are sometimes up to 200 blocks behind the head of the chain when there are periods of low activity in the Maker protocol (or potentially, periods of very high activity that crowd out Maker related transactions, although I don't think we've seen this in practice). If nothing's happening in the MCD contracts, we won't have relevant events or diffs to transform. In practice, the biggest period of inactivity we've seen in the wild is around 190 blocks, which triggered a false positive alert. (See the "diff lag alert" investigation here for more).
To diagnose issues with storage diff transformation, you may wish to check the status of recently synced/transformed diffs:
{
allStorageDiffs(last: 5) {
nodes {
status
blockHeight
}
}
}If storage diff syncing and diff transformation are both running smoothly, this query should mostly return diffs with TRANSFORMED status. (Note that in a real scenario, you probably want to load more than 5).
{
"data": {
"allStorageDiffs": {
"nodes": [
{
"status": "TRANSFORMED",
"blockHeight": "13766048"
},
{
"status": "TRANSFORMED",
"blockHeight": "13766048"
},
{
"status": "TRANSFORMED",
"blockHeight": "13766048"
},
{
"status": "TRANSFORMED",
"blockHeight": "13766048"
},
{
"status": "TRANSFORMED",
"blockHeight": "13766048"
}
]
}
},
"meta": {
"graphqlQueryCost": 1
}
}A few diffs in the NEW, PENDING, and NONCANONICAL states are nothing to worry about here as long as 90%+ are TRANSFORMED. However, if the majority of diffs are some other status, it likely indicates an error:
- If most diffs are
NEW, transformation has stopped or slowed. This could be because thevdb-mcd-transformerscontainer crashed or failed, or because of a slow or pathological DB insert locking or blocking. However, as long asNEWdiffs keep syncing from theextract-diffscontainer, they will be processed eventually. Restart thevdb-mcd-transformerscontainer or kill the query. - If most diffs are
UNRECOGNIZED, there is probably an application level error in one of the storage diff transformers. AnUNRECOGNIZEDdiff means we recieved a storage diff from a contract we're tracking, but cannot find a matching storage key. If transformer-level logic calculates the wrong storage key, we will seeUNRECOGNIZEDdiffs. - If most diffs are
NONCANONICAL, the statediffing geth node may be following a noncanonical fork. This is pretty unlikely in practice, but could happen if we fail to upgrade our statediffing geth nodes before a scheduled hard fork.
getUrnsByIlk returns all urns and their history for a specific collateral type. This is usually the easiest entry point to look at Urn data for a specific collateral, although there are a lot of vaults for large collateral types like ETH-A. If you know the urn's address, you can
load it directly using getUrn. See the comments below for how to embed attributes and interpret the data.
{
getUrnsByIlk(ilkIdentifier: "ETH-A", first: 1) {
nodes {
# Current parameters for this collateral type
ilk {
rate
art
spot
line
dust
chop
lump
flip
rho
duty
pip
mat
dunk
}
# Current debt/collateral for this vault
ink
art
# Address of this vault
urnIdentifier
# First 100 frobs associated with this vault. Check the totalCount
# attribute and adjust this upwards if there are more than 100
# events associated with this vault.
frobs(first: 100) {
totalCount
nodes {
# Change in ink and art from frob event
dink
dart
# Snapshot of the vault state at the time of the frob event.
urn {
ink
art
}
# Snapshot of the collateral parameters at the time of the
# frob event. (Note how these values are different than the
# "current" parameters at the top of this query).
ilk {
rate
art
spot
line
dust
chop
lump
flip
rho
duty
pip
mat
dunk
}
}
}
# Bite events associated with this vault. You can embed
# Urn/Ilk state snapshots similarly inside this query.
bites(first: 100) {
nodes {
bidId
ink
art
tab
}
}
}
}
# Get a single Urn. The same embedded attributes as above are available in this query.
getUrn(ilkIdentifier: "ETH-A", urnIdentifier: "0x3F1AB0825D403E5D87A4D710d685587321736c35") {
ink
art
frobs {
totalCount
}
}
}allClips returns liquidation auction history for a particular collateral type.
{
allClips(ilk: "YFI-A", first: 1) {
totalCount
nodes {
# Auction ID
saleId
# Block height at which the liquidation started
blockHeight
# Current collateral parameters. Note that these are
# not that useful when looking at a historical liquidation.
ilk {
spot
}
# Current urn state. For a completed liquidation auction,
# ink and art should both be zero.
urn {
ink
art
}
# Current auction parameters. For a completed auction,
# these should all be zero (they are set to zero when the
# auction is deleted from the storage mapping). For an
# active auction, these should have nonzero values.
tab
lot
usr
tic
top
# History of events related to this auction.
saleEvents(first: 100) {
totalCount
nodes {
# The type of event, either 'kick', 'redo', 'or 'take'.
# Depending on the type, one of `kickEvent`, `redoEvent`, or
# `takeEvent` will be populated
act
kickEvent {
top
tab
lot
usr
kpr
coin
}
takeEvent {
max
price
owe
tab
lot
usr
}
redoEvent {
top
tab
lot
usr
kpr
coin
}
}
}
}
}
}