v0.32.0-rc.0

v0.32.0-rc.0 is the first release candidate of the upcoming v0.32.0. This release candidate is intended for testnet use only.

$ docker pull graphprotocol/graph-node:v0.32.0-rc.0

What's New

• Derived fields getter: Derived fields can now be accessed from within the mapping code during indexing. (#4434)
• Sorting interfaces by child entity: Interfaces can now be sorted by non-derived child entities. (#4058)
• File data sources can now be spawned from handlers of other file data sources: This enables the use of file data sources for scenarios where a file data source needs to be spawned from another one. One practical application of this feature is in handling NFT metadata. In such cases, the metadata itself is stored as a file on IPFS and contains embedded IPFS CID for the actual file for the NFT. (#4713)
• Allow redeployment of grafted subgraphs even when graft_base is not available: This will allow renaming of already synced grafted subgraphs even when the graft base is not available, which previously failed due to graft-base validation errors. (#4695)
• history_blocks is now available in the index-node API. (#4662)
• Added a new subgraph features table in primary to easily track information like apiVersion, specVersion, features, and data source kinds used by subgraphs. (#4679)
• subgraphFeatures endpoint now includes data from subgraph_features table.
• ens_name_by_hash is now undeprecated: This reintroduces support for fetching ENS names by their hash, dependent on the availability of the underlying Rainbow Table (#4751).
• Deterministically failed subgraphs now return valid POIs for subsequent blocks after the block at which it failed. (#4774)
• eth-call logs now include block hash and block number: This enables easier debugging of eth-call issues. (#4718)
• Enabled support for substreams on already supported networks. (#4767)
• Add new GraphQL scalar type Int8. This new scalar type allows subgraph developers to represent 8-bit signed integers. (#4511)
• Add support for overriding module params for substreams-based subgraphs when params are provided in the subgraph manifest. (#4759)

Bug fixes

• Fixed PublicProofsOfIndexing returning the error Null value resolved for non-null field proofOfIndexing when fetching POIs for blocks that are not in the cache (#4768)
• Fixed an issue where Block stream would fail when switching back to an RPC-based block ingestor from a Firehose ingestor. (#4790)
• Fixed an issue where derived loaders were not working with entities with Bytes as IDs (#4773)
• Firehose connection test now retries for 30 secs before setting the provider status to Broken (#4754)
• Fixed the nonFatalErrors field not populating in the index node API. (#4615)
• Fixed graph-node panicking on the first startup when both Firehose and RPC providers are configured together. (#4680)
• Fixed block ingestor failing to startup with the error net version for chain mainnet has changed from 0 to 1 when switching from Firehose to an RPC provider. (#4692)
• Fixed Firehose endpoints getting rate-limited due to duplicated providers during connection pool initialization. (#4778)

Graphman

• Added two new graphman commands pause and resume: Instead of reassigning to a non-existent node these commands can now be used for pausing and resuming subgraphs. (#4642)
• Added a new graphman command restart to restart a subgraph. (#4742)

Full Changelog: v0.31.0...c350e4f

v0.31.0-rc.1

v0.31.0-rc.1 is the second release candidate of the upcoming v0.31.0.

$ docker pull graphprotocol/graph-node:v0.31.0-rc.1

Edit 2023-06-08: v0.31.0-rc.1 has been promoted to v0.31.0.

v0.31.0

$ docker pull graphprotocol/graph-node:v0.31.0

What's new

• Fulltext searches can now be combined with where filtering, further narrowing down search results. #4442
• Derived fields loader support. This feature requires subgraph authors to upgrade graph-cli to v0.51.0. #4434, graph-cli release notes
• Tweaked how RPC provider limiting rules are interpreted from configurations. In particular, node IDs that don't match any rules of a provider won't have access to said provider instead of having access to it for an unlimited number of subgraphs. Read the docs for more information. #4353
• Introduced WASM host function store.get_in_block, which is a much faster variant of store.get limited to entities created or updated in the current block. #4540
• The subgraph_deploy JSON-RPC method now accepts a history_blocks parameter, which indexers can use to set default amounts of history to keep. #4564
• IPFS requests for polling file data sources are not throttled anymore (also known as concurrency or burst limiting), only rate-limited. #4570, #4649
• Exponential requests backoff when retrying failed subgraphs is now "jittered", smoothing out request spikes. #4476
• Indexers can use thee new feature GRAPH_EXPERIMENTAL_SUBGRAPH_SETTINGS to define matching rules on subgraph names for reducing amounts of history stored. #4633
• RPC provider responses that decrease the chain head block number (non-monotonic) are now ignored, increasing resiliency against inconsistent provider data. #4354
• It's now possible to to have a Firehose-only chain with no RPC provider at all in the configuration. #4508, #4553
• The materialized views in the info schema (table_sizes, subgraph_sizes, and chain_sizes) that provide information about the size of various database objects are now automatically refreshed every 6 hours. #4461
• Subgraph error logs now contain the transaction hash when available. #4635
• Adapter selection now takes error rates into account, preferring adapters with lower error rates. #4468
• Removed support for GRAPH_ETHEREUM_IS_FIREHOSE_PREFERRED, REVERSIBLE_ORDER_BY_OFF, and GRAPH_STORE_CONNECTION_TRY_ALWAYS env. variables. #4375, #4436
• New env. var. GRAPH_ETH_CALL_NO_GAS which is a comma-separated list of chains' net_version for which to disable eth_call gas limits. The default value (421613) disables gas limits for Arbitrum Goerli. #4609

Performance improvements

• Batched writes: write operations to the database when indexing subgraphs are now accumulated into in-memory batches before persisting them, resulting in noticeably faster indexing and lower database CPU load. In case of unexpected issues, this feature can be turned off setting GRAPH_STORE_WRITE_BATCH_SIZE=0. #4606, #4632
• The entity cache that graph-node keeps around is much more efficient, meaning more cache entries fit in the same amount of memory resulting in a performance increase under a wide range of workloads. #4485, #4624
• Indexing speed improvements for subgraphs with more than 10k data sources. #4631
• BRIN indexes in PostgreSQL now use ..._minmax_multi_ops instead of _minmax_ops if available (PostgreSQL 14+), often resulting in more selective indexes. #4629
• GIN indexes are not created anymore for numeric arrays. #4628

Substreams

• The substreams protocol has been updated to sf.substreams.rpc.v2.Stream/Blocks. #4556
• Added support for adapter balancing a.k.a. provider failover to substreams. #4578
• Fixed faulty startBlock selection logic in substreams. #4463
• Several bug fixes were applied after breakage caused by an internal refactor of how entities are stored in-memory. #4604, #4572, #4612, #4664

Bug fixes

• Fixed a bug that would cause subgraphs to fail with a subgraph writer poisoned by previous error message following certain database errors. #4533
• Fixed a bug that would cause subgraphs to fail with a store error: no connection to the server message when database connection e.g. gets killed. #4435
• The subgraph_reassign JSON-RPC method doesn't fail anymore when multiple deployment copies are found: only the active copy is reassigned, the others are ignored. #4395
• Fixed a bug that would cause on_sync handlers on copied deployments to fail with the message Subgraph instance failed to run: deployment not found [...]. #4396
• Fixed a bug that would cause the copying or grafting of a subgraph while pruning it to incorrectly set earliest_block in the destination deployment. #4502
• Handler timeouts would sometimes be reported as deterministic errors with the error message Subgraph instance failed to run: Failed to call 'asc_type_id' with [...] wasm backtrace [...]; this error is now nondeterministic and recoverable. #4475
• Fixed faulty exponential request backoff behavior after many minutes of failed requests, caused by an overflow. #4421
• json.fromBytes and all BigInt operations now require more gas, protecting against malicious subgraphs. #4594, #4595
• Fixed a Rust panic on program startup when mixing Firehose and RPC providers. #4680

Graphman

• graphman rewind now requires block-number and block-hash to be passed as flags instead of arguments. #4400
• You can now use the new flag --start-block in graphman rewind to rewind a subgraph to the startBlock set in manifest, or to the genesis block if no startBlock is set. #4400
• The behavior for graphman prune has changed: running just graphman prune will mark the subgraph for ongoing pruning in addition to performing an initial pruning. To avoid ongoing pruning, use graphman prune --once (docs). #4429
• The env. var. GRAPH_STORE_HISTORY_COPY_THRESHOLD –which serves as a configuration setting for graphman prune– has been renamed to GRAPH_STORE_HISTORY_REBUILD_THRESHOLD. #4505
• You can now list all existing deployments via graphman info --all. #4347
• The command graphman chain call-cache remove now requires --remove-entire-cache as an explicit flag, protecting against accidental destructive command invocations. #4397
• graphman copy create accepts two new flags, --activate and --replace, which make moving of subgraphs across shards much easier. #4374
• The log level for graphman is now set via GRAPHMAN_LOG or command line instead of GRAPH_LOG. #4462
• graphman reassign now emits a warning when it suspects a typo in node IDs. #4377

Metrics and logging

• Subgraph syncing time metric deployment_sync_secs now stops updating once the subgraph has synced. #4489
• New endpoint_request metric to track error rates of different providers. #4490, #4504, [#4430](...

v0.31.0-rc.0

v0.31.0-rc.0 is the first release candidate of the upcoming v0.31.0. This release candidate is intended for testnet use only.

$ docker pull graphprotocol/graph-node:v0.31.0-rc.0

Edit 2023-06-08: v0.31.0-rc.0 is obsolete; v0.31.0-rc.1 is the latest release candidate.

v0.30.0-rc.0

v0.30.0-rc.0 is the first release candidate for the upcoming v0.30.0. This release candidate is intended for testnet use only.

$ docker pull graphprotocol/graph-node:v0.30.0-rc.0

Edit 2023-03-01: v0.30.0-rc.0 has been promoted to v0.30.0.

v0.30.0

$ docker pull graphprotocol/graph-node:v0.30.0

Database locale change

New graph-node installations now mandate PostgreSQL to use C locale and UTF-8 encoding. The official docker-compose.yml template has been updated accordingly. Pre-existing graph-node installations are not concerned with this change, but local development scripts and CI pipelines may have to adjust database initialization parameters. This can be done with initdb -E UTF8 --locale=C. #4163, #4151, #4201, #4340

What's new

• AND/OR filters. AND/OR logical operators in where filters have been one of graph-node's most awaited features. They do exactly what you would expect them to do, and are very powerful. #579, #4080, #4171
• IPFS file data sources. IPFS file data sources allow subgraph developers to query offchain information from IPFS directly in mappings. This feature is the culmination of much community and development efforts (GIP here). A future iteration of this feature will also include a so-called "Availability Chain", allowing IPFS file data sources to contribute to Proofs of Indexing. At the moment, entity updates that originate from these data sources' handlers do not contribute to PoIs. #4147, #4162, and many others!
• Sorting by child entities (a.k.a. nested sorting). You can now orderBy properties of child entities. #4058, #3737, #3096
• Added support for a Firehose-based block ingestor. Indexers that use the new Firehose-based block ingestor cannot automatically switch back to RPC. In order to downgrade, indexers must manually delete all blocks accumulated by Firehose in the database. For this reason, we suggest caution when switching over from RPC to Firehose. #4059, #4204, #4216
• Fields of type Bytes can now use less than and greater than filters. #4285
• "userinfo" is now allowed in IPFS URLs (e.g. https://foo:bar@example.com:5001/). #4252
• The default for GRAPH_IPFS_TIMEOUT is now 60 seconds instead of 30. #4324
• Forking options can now be set via env. vars. (GRAPH_START_BLOCK, GRAPH_FORK_BASE, GRAPH_DEBUG_FORK). #4308
• Allow retrieving GraphQL query tracing over HTTP if the env. var. GRAPH_GRAPHQL_TRACE_TOKEN is set and the header X-GraphTraceQuery is included. The query traces' JSON is the same as returned by graphman query. #4243
• Lots of visual and filtering improvements to #4232
• More aggressive in-memory caching of blocks close the chain head, potentially alleviating database load. #4215
• New counter Prometheus metric query_validation_error_counter, labelled by deployment ID and error code. #4230
  graph_elasticsearch_logs_sent
• Turned "Flushing logs to Elasticsearch" log into a Prometheus metric (graph_elasticsearch_logs_sent) to reduce log noise. #4333
• New materialized view info.chain_sizes, which works the same way as the already existing info.subgraph_sizes and info.table_sizes. #4318
• New graphman stats subcommands set-target and target to manage statistics targets for specific deployments (i.e. how much data PostgreSQL samples when analyzing a table). #4092

Fixes

• graph-node now has PID=1 when running inside the official Docker image. #4217
• More robust ipfs.cat logic during grafted subgraphs' manifest lookup. #4284
• Fixed a bug that caused some large multi-entity inserts to fail because of faulty chunk size calculation. #4250
• Subgraph pruning now automatically cancels ongoing autovacuum, to avoid getting stuck. #4167
• ens.getNameByHash now fails nondeterministically if ENS rainbow tables are not available locally. #4219
• Some kinds of subgraph failures were previously wrongly treated as unattestable (value parsing, enum and scalar coercion), i.e. nondeterministic. These subgraph failure modes are now flagged as fully-deterministic. #4278

v0.29.0

$ docker pull graphprotocol/graph-node:v0.29.0

Upgrade notes

• This release includes a determinism fix that affect a very small number of subgraphs on the network (we counted 2): if a subgraph manifest had one data source with no contract address, listening to the same events or calls of another data source that has a specified address, then the handlers for those would be called twice. After the fix, this will happen no more, and the handler will be called just once like it should.

  Affected subgraph deployments:

  • Qmccst5mbV5a6vT6VvJMLPKMAA1VRgT6NGbxkLL8eDRsE7
  • Qmd9nZKCH8UZU1pBzk7G8ECJr3jX3a2vAf3vowuTwFvrQg

  Here's an example manifest, taking a look at the data sources of name ERC721 and CryptoKitties, both listen to the Transfer(...) event. Considering a block where there's only one occurrence of this event, graph-node would duplicate it and call handleTransfer twice. Now this is fixed and it will be called only once per event/call that happened on chain.

  In the case you're indexing one of the impacted subgraphs, you should first upgrade the graph-node version, then rewind the affected subgraphs to the smallest startBlock of their subgraph manifest. To achieve that, you can use the following graphman rewind CLI command invocations:

  graphman rewind 0xd75c2a2412396b6b2b387323d84b9c19313dbba35ccb1493d94976fedb0b3a23 5774644 Qmccst5mbV5a6vT6VvJMLPKMAA1VRgT6NGbxkLL8eDRsE7
  graphman rewind 0xf23c45dd2e1ed77fce1919cdcc9df2de51e821763e9691f8adbb79a55ddb28a4 3914495 Qmd9nZKCH8UZU1pBzk7G8ECJr3jX3a2vAf3vowuTwFvrQg
  

  See #4055 for more information.

• This release fixes another determinism bug that affects a handful of subgraphs. The bug affects all subgraphs which have an apiVersion older than 0.0.5 using call handlers. While call handlers prior to 0.0.5 should be triggered by both failed and successful transactions, in some cases failed transactions would not trigger the handlers. This resulted in nondeterministic behavior. With this version of graph-node, call handlers with an apiVersion older than 0.0.5 will always be triggered by both successful and failed transactions. Behavior for apiVersion 0.0.5 onward is not affected.

  The affected subgraphs are:

  • QmNY7gDNXHECV8SXoEY7hbfg4BX1aDMxTBDiFuG4huaSGA
  • QmYzsCjrVwwXtdsNm3PZVNziLGmb9o513GUzkq5wwhgXDT
  • QmccAwofKfT9t4XKieDqwZre1UUZxuHw5ynB35BHwHAJDT
  • QmYUcrn9S1cuSZQGomLRyn8GbNHmX8viqxMykP8kKpghz6
  • QmecPw1iYuu85rtdYL2J2W9qcr6p8ijich9P5GbEAmmbW5
  • Qmaz1R8vcv9v3gUfksqiS9JUz7K9G8S5By3JYn8kTiiP5K

  In the case you're indexing one of the impacted subgraphs, you should first upgrade the graph-node version, then rewind the affected subgraphs to the smallest startBlock of their subgraph manifest. To achieve that, you can use the following graphman rewind CLI command invocations:

  graphman rewind 0x8aed5a76735b0890d4c0a9e752c88cba4f3d9d0e4e1dad70d6bd913cc279e4b8 7913573 QmNY7gDNXHECV8SXoEY7hbfg4BX1aDMxTBDiFuG4huaSGA
  graphman rewind 0x15da49d2177dde3012b67f12d25d9ae4cb56c3638c2ec011298e509be27be77a 7303699 QmYzsCjrVwwXtdsNm3PZVNziLGmb9o513GUzkq5wwhgXDT
  graphman rewind 0x1be8b52bd6b3c29b1ead63f6a8bef83d626d8b6795f6c2d89c302cbe32d59830 10613640 QmYUcrn9S1cuSZQGomLRyn8GbNHmX8viqxMykP8kKpghz6
  graphman rewind 0xef547ca70b451cc9fedbc039edcfd4a35de80fce76ed6ad31bc15a47850dfefd 10736242 QmecPw1iYuu85rtdYL2J2W9qcr6p8ijich9P5GbEAmmbW5
  graphman rewind 0x8fb3744b077831856203bbfe465e9cb681d0ed0c67db297faee54f0542f045d3 5533890 Qmaz1R8vcv9v3gUfksqiS9JUz7K9G8S5By3JYn8kTiiP5K
  
  # Note: the following subgraph (https://github.com/Data-Nexus/721-Marketplace) is extremely
  # write-heavy, and your node may OOM while rewinding. If you run into this issue suggest
  # you either rewind in smaller chunks of blocks, or simply redeploy it.
  graphman rewind 0xf9bc13c7864c997aaed501715a0627a25e748c2df4889cc9cbf4d077e7bca34a 4944642 QmccAwofKfT9t4XKieDqwZre1UUZxuHw5ynB35BHwHAJDT
  

  See #4149 for more information.

What's new

• Grafted subgraphs can now add their own data sources. #3989, #4027, #4030
• Add support for filtering by nested interfaces. #3677
• Add support for message handlers in Cosmos #3975
• Dynamic data sources for Firehose-backed subgraphs. #4075
• Various logging improvements. #4078, #4084, #4031, #4144, #3990
• Some DB queries now have GCP Cloud Insight -compliant tags that show where the query originated from. #4079
• New configuration variable GRAPH_STATIC_FILTERS_THRESHOLD to conditionally enable static filtering based on the number of dynamic data sources. #4008
• New configuration variable GRAPH_STORE_BATCH_TARGET_DURATION. #4133

Docker image

• The official Docker image now runs on Debian 11 "Bullseye". #4081
• We now ship envsubst with the official Docker image, allowing you to easily run templating logic on your configuration files. #3974

Graphman

We have a new documentation page for graphman, check it out here!

• Subgraph pruning with graphman! #3898, #4125, #4153, #4152, #4156, #4041
• New command graphman drop to hastily delete a subgraph deployment. #4035
• New command graphman chain call-cache for clearing the call cache for a given chain. #4066
• Add --delete-duplicates flag to graphman check-blocks by @tilacog in #3988

Performance

• Restarting a node now takes much less time because postgres_fdw user mappings are only rebuilt upon schema changes. If necessary, you can also use the new commands graphman database migrate and graphman database remap to respectively apply schema migrations or run remappings manually. #4009, #4076
• Database replicas now won't fall behind as much when copying subgraph data. #3966 #3986
• Block handlers optimization with Firehose >= 1.1.0. #3971
• Reduced the amount of data that a non-primary shard has to mirror from the primary shard. #4015
• We now use advisory locks to lock deployments' tables against concurrent writes. #4010

Bug fixes

• Fixed a bug that would cause some failed subgraphs to never restart. #3959
• Fixed a bug that would cause bad POIs for Firehose-backed subgraphs when processing CREATE calls. #4085
• Fixed a bug which would cause failure to redeploy a subgraph immediately after deletion. #4044
• Firehose connections are now load-balanced. #4083
• Determinism fixes. See above. #4055, #4149

Dependency updates

Dependency	updated to
anyhow	1.0.66
base64	0.13.1
clap	3.2.23
env_logger	0.9.1
iana-time-zone	0.1.47
itertools	0.10.5
jsonrpsee	0.15.1
num_cpus	1.14.0
openssl	0.10.42
pretty_assertions	1.3.0
proc-macro2	1.0.47
prometheus	0.13.3
protobuf-parse	3.2.0
semver	1.0.14
serde_plain	1.0.1
sha2	0.10.6
structopt	removed
tokio-stream	0.1.11
tokio-tungstenite	0.17.2
tower-test	d27ba65
url	2.3.1

v0.29.0-rc.0

v0.29.0-rc.0 is the first release candidate for the upcoming v0.29.0. This release candidate is intended for testnet use only.

$ docker pull graphprotocol/graph-node:v0.29.0-rc.0

Edit 2022-12-06: v0.29.0-rc.0 has been promoted to v0.29.0.

v0.28.2

Indexers are advised to migrate to v0.28.2 and entirely bypass v0.28.0 and v0.28.1.

Fixed a bug which would cause subgraphs to stop syncing under some graph-node deployment configurations. #4046, #4051

v0.28.1

Yanked. Please migrate to v0.28.2.