Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Index incoming tokens #6736

Open
wants to merge 11 commits into
base: feat/chain-go-sdk
Choose a base branch
from
Open

Index incoming tokens #6736

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a62d85b
Index incoming tokens
axenteoctavian Jan 23, 2025
340e464
new version of indexer
axenteoctavian Jan 28, 2025
804da76
sovereign deploy scripts update
axenteoctavian Jan 28, 2025
a171435
update function name
axenteoctavian Jan 28, 2025
4c17ba5
update function
axenteoctavian Jan 28, 2025
4380d51
added esdt prefix in arguments
axenteoctavian Jan 28, 2025
983c039
Merge branch 'feat/chain-go-sdk' into MX-16271-indexing-incoming-tokens
axenteoctavian Jan 31, 2025
a79b1a5
go.mod indexer update
axenteoctavian Jan 31, 2025
c8b2623
updated external.toml comments
axenteoctavian Feb 6, 2025
723364c
Merge branch 'feat/chain-go-sdk' into MX-16271-indexing-incoming-tokens
axenteoctavian Feb 6, 2025
7fe8b76
Merge branch 'feat/chain-go-sdk' into MX-16271-indexing-incoming-tokens
axenteoctavian Feb 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions cmd/node/config/external.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,17 @@
# ["rating", "transactions", "blocks", "validators", "miniblocks", "rounds", "accounts", "accountshistory", "receipts", "scresults", "accountsesdt", "accountsesdthistory", "epochinfo", "scdeploys", "tokens", "tags", "logs", "delegators", "operations", "esdts"]
EnabledIndexes = ["rating", "transactions", "blocks", "validators", "miniblocks", "rounds", "accounts", "accountshistory", "receipts", "scresults", "accountsesdt", "accountsesdthistory", "epochinfo", "scdeploys", "tokens", "tags", "logs", "delegators", "operations", "esdts"]

# MainChainElasticSearchConnector defines settings related to ElasticSearch such as login information or URL
[MainChainElasticSearchConnector]
# We do not recommend to activate this indexer on a validator node since
# the node might lose rating (even facing penalties) due to the fact that
# the indexer is called synchronously and might block due to external causes.
# Strongly suggested to activate this on a regular observer node.
Enabled = false
URL = "http://localhost:9201"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you give an example here in the comment with a link to the public es link from our mainnet?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not put it here, I don't think is necessary. The deploy scripts update this automatically

Username = ""
Password = ""

# EventNotifierConnector defines settings needed to configure and launch the event notifier component
# HTTP event notifier connector integration will be DEPRECATED in the following iterations
[EventNotifierConnector]
Expand Down
2 changes: 1 addition & 1 deletion cmd/sovereignnode/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ require (
github.com/multiversx/concurrent-map v0.1.4 // indirect
github.com/multiversx/mx-chain-communication-go v1.1.1 // indirect
github.com/multiversx/mx-chain-crypto-go v1.2.12 // indirect
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131122054-68d88c13fdac // indirect
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131125841-73868f5ca49c // indirect
github.com/multiversx/mx-chain-scenario-go v1.4.4 // indirect
github.com/multiversx/mx-chain-storage-go v1.0.19 // indirect
github.com/multiversx/mx-chain-vm-common-go v1.5.17-0.20241119132002-2fa80c5ec516 // indirect
Expand Down
4 changes: 2 additions & 2 deletions cmd/sovereignnode/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -389,8 +389,8 @@ github.com/multiversx/mx-chain-core-go v1.2.25-0.20250206111825-25fbb1b4851c h1:
github.com/multiversx/mx-chain-core-go v1.2.25-0.20250206111825-25fbb1b4851c/go.mod h1:P/YBoFnt25XUaCQ7Q/SD15vhnc9yV5JDhHxyFO9P8Z0=
github.com/multiversx/mx-chain-crypto-go v1.2.12 h1:zWip7rpUS4CGthJxfKn5MZfMfYPjVjIiCID6uX5BSOk=
github.com/multiversx/mx-chain-crypto-go v1.2.12/go.mod h1:HzcPpCm1zanNct/6h2rIh+MFrlXbjA5C8+uMyXj3LI4=
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131122054-68d88c13fdac h1:AaD3sEsepJTHeLqp20ZvMzTY1KakWXYK2qcJfqfEdZM=
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131122054-68d88c13fdac/go.mod h1:u3P9JM8JWPRcYW3X0Lsw06Nf6a6M7cS42XwT4Au/Ar8=
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131125841-73868f5ca49c h1:Qm84hhybDLTeS+BLebFYgmGJpkX+AUnzZsUC+u9KzBU=
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131125841-73868f5ca49c/go.mod h1:u3P9JM8JWPRcYW3X0Lsw06Nf6a6M7cS42XwT4Au/Ar8=
github.com/multiversx/mx-chain-logger-go v1.0.15 h1:HlNdK8etyJyL9NQ+6mIXyKPEBo+wRqOwi3n+m2QIHXc=
github.com/multiversx/mx-chain-logger-go v1.0.15/go.mod h1:t3PRKaWB1M+i6gUfD27KXgzLJJC+mAQiN+FLlL1yoGQ=
github.com/multiversx/mx-chain-scenario-go v1.4.4 h1:DVE2V+FPeyD/yWoC+KEfPK3jsFzHeruelESfpTlf460=
Expand Down
1 change: 1 addition & 0 deletions cmd/sovereignnode/sovereignNodeRunner.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1226,6 +1226,7 @@ func (snr *sovereignNodeRunner) CreateManagedStatusComponents(
StatusCoreComponents: managedStatusCoreComponents,
CryptoComponents: cryptoComponents,
IsSovereign: true,
ESDTPrefix: snr.configs.SystemSCConfig.ESDTSystemSCConfig.ESDTPrefix,
}

statusComponentsFactory, err := statusComp.NewStatusComponentsFactory(statArgs)
Expand Down
15 changes: 12 additions & 3 deletions config/externalConfig.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@ package config

// ExternalConfig will hold the configurations for external tools, such as Explorer or Elasticsearch
type ExternalConfig struct {
ElasticSearchConnector ElasticSearchConfig
EventNotifierConnector EventNotifierConfig
HostDriversConfig []HostDriversConfig
ElasticSearchConnector ElasticSearchConfig
MainChainElasticSearchConnector MainChainElasticSearchConfig
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a preference, if you add something new, can you please MainChainElasticSearchConnector at the end of the struct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used the sections order from the .toml file...

EventNotifierConnector EventNotifierConfig
HostDriversConfig []HostDriversConfig
}

// ElasticSearchConfig will hold the configuration for the elastic search
Expand All @@ -19,6 +20,14 @@ type ElasticSearchConfig struct {
EnabledIndexes []string
}

// MainChainElasticSearchConfig will hold the configuration for the main chain elastic search
type MainChainElasticSearchConfig struct {
Enabled bool
URL string
Username string
Password string
}

// EventNotifierConfig will hold the configuration for the events notifier driver
type EventNotifierConfig struct {
Enabled bool
Expand Down
13 changes: 13 additions & 0 deletions factory/status/statusComponents.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"github.com/multiversx/mx-chain-core-go/core"
"github.com/multiversx/mx-chain-core-go/core/check"
factoryMarshalizer "github.com/multiversx/mx-chain-core-go/marshal/factory"
esFactory "github.com/multiversx/mx-chain-es-indexer-go/process/elasticproc/factory"
indexerFactory "github.com/multiversx/mx-chain-es-indexer-go/process/factory"
logger "github.com/multiversx/mx-chain-logger-go"

Expand Down Expand Up @@ -47,6 +48,7 @@ type StatusComponentsFactoryArgs struct {
CryptoComponents factory.CryptoComponentsHolder
IsInImportMode bool
IsSovereign bool
ESDTPrefix string
}

type statusComponentsFactory struct {
Expand All @@ -64,6 +66,7 @@ type statusComponentsFactory struct {
cryptoComponents factory.CryptoComponentsHolder
isInImportMode bool
isSovereign bool
esdtPrefix string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why Pass esdPrefix all the way down to here? Don't you already have the whole config in statusComponentsFactory?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just the GeneralConfig, and I needed SystemSCConfig for the prefix.

}

var log = logger.GetOrCreate("factory")
Expand Down Expand Up @@ -109,6 +112,7 @@ func NewStatusComponentsFactory(args StatusComponentsFactoryArgs) (*statusCompon
isInImportMode: args.IsInImportMode,
cryptoComponents: args.CryptoComponents,
isSovereign: args.IsSovereign,
esdtPrefix: args.ESDTPrefix,
}, nil
}

Expand Down Expand Up @@ -218,6 +222,13 @@ func (scf *statusComponentsFactory) createOutportDriver() (outport.OutportHandle

func (scf *statusComponentsFactory) makeElasticIndexerArgs() indexerFactory.ArgsIndexerFactory {
elasticSearchConfig := scf.externalConfig.ElasticSearchConnector
mainChainElastic := esFactory.ElasticConfig{
Enabled: scf.externalConfig.MainChainElasticSearchConnector.Enabled,
Url: scf.externalConfig.MainChainElasticSearchConnector.URL,
UserName: scf.externalConfig.MainChainElasticSearchConnector.Username,
Password: scf.externalConfig.MainChainElasticSearchConnector.Password,
}

return indexerFactory.ArgsIndexerFactory{
Enabled: elasticSearchConfig.Enabled,
BulkRequestMaxSize: elasticSearchConfig.BulkRequestMaxSizeInBytes,
Expand All @@ -234,6 +245,8 @@ func (scf *statusComponentsFactory) makeElasticIndexerArgs() indexerFactory.Args
ImportDB: scf.isInImportMode,
HeaderMarshaller: scf.coreComponents.InternalMarshalizer(),
Sovereign: scf.isSovereign,
ESDTPrefix: scf.esdtPrefix,
MainChainElastic: mainChainElastic,
}
}

Expand Down
1 change: 1 addition & 0 deletions factory/status/statusComponents_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ func createMockStatusComponentsFactoryArgs() statusComp.StatusComponentsFactoryA
},
IsInImportMode: false,
IsSovereign: false,
ESDTPrefix: "",
}
}

Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ require (
github.com/multiversx/mx-chain-communication-go v1.1.1
github.com/multiversx/mx-chain-core-go v1.2.25-0.20250206111825-25fbb1b4851c
github.com/multiversx/mx-chain-crypto-go v1.2.12
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131122054-68d88c13fdac
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131125841-73868f5ca49c
github.com/multiversx/mx-chain-logger-go v1.0.15
github.com/multiversx/mx-chain-scenario-go v1.4.4
github.com/multiversx/mx-chain-storage-go v1.0.19
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -391,8 +391,8 @@ github.com/multiversx/mx-chain-core-go v1.2.25-0.20250206111825-25fbb1b4851c h1:
github.com/multiversx/mx-chain-core-go v1.2.25-0.20250206111825-25fbb1b4851c/go.mod h1:P/YBoFnt25XUaCQ7Q/SD15vhnc9yV5JDhHxyFO9P8Z0=
github.com/multiversx/mx-chain-crypto-go v1.2.12 h1:zWip7rpUS4CGthJxfKn5MZfMfYPjVjIiCID6uX5BSOk=
github.com/multiversx/mx-chain-crypto-go v1.2.12/go.mod h1:HzcPpCm1zanNct/6h2rIh+MFrlXbjA5C8+uMyXj3LI4=
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131122054-68d88c13fdac h1:AaD3sEsepJTHeLqp20ZvMzTY1KakWXYK2qcJfqfEdZM=
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131122054-68d88c13fdac/go.mod h1:u3P9JM8JWPRcYW3X0Lsw06Nf6a6M7cS42XwT4Au/Ar8=
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131125841-73868f5ca49c h1:Qm84hhybDLTeS+BLebFYgmGJpkX+AUnzZsUC+u9KzBU=
github.com/multiversx/mx-chain-es-indexer-go v1.7.15-0.20250131125841-73868f5ca49c/go.mod h1:u3P9JM8JWPRcYW3X0Lsw06Nf6a6M7cS42XwT4Au/Ar8=
github.com/multiversx/mx-chain-logger-go v1.0.15 h1:HlNdK8etyJyL9NQ+6mIXyKPEBo+wRqOwi3n+m2QIHXc=
github.com/multiversx/mx-chain-logger-go v1.0.15/go.mod h1:t3PRKaWB1M+i6gUfD27KXgzLJJC+mAQiN+FLlL1yoGQ=
github.com/multiversx/mx-chain-scenario-go v1.4.4 h1:DVE2V+FPeyD/yWoC+KEfPK3jsFzHeruelESfpTlf460=
Expand Down
1 change: 1 addition & 0 deletions node/nodeRunner.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1128,6 +1128,7 @@ func (nr *nodeRunner) CreateManagedStatusComponents(
StatusCoreComponents: managedStatusCoreComponents,
CryptoComponents: cryptoComponents,
IsSovereign: false,
ESDTPrefix: nr.configs.SystemSCConfig.ESDTSystemSCConfig.ESDTPrefix,
}

statusComponentsFactory, err := statusComp.NewStatusComponentsFactory(statArgs)
Expand Down
1 change: 1 addition & 0 deletions scripts/testnet/include/config.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ updateNodeConfig() {

if [ $USE_ELASTICSEARCH -eq 1 ]; then
sed -i '/^\[ElasticSearchConnector\]/,/^\[/ s/Enabled *= *false/Enabled = true/' external_observer.toml
sed -i '/^\[MainChainElasticSearchConnector\]/,/^\[/ s/Enabled *= *false/Enabled = true/' external_observer.toml
fi

sed -i '/^\[DbLookupExtensions\]/,/^\[/ s/Enabled *= *false/Enabled = true/' config_observer.toml
Expand Down
1 change: 1 addition & 0 deletions scripts/testnet/sovereignBridge/config/configs.cfg
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ WALLET="~/wallet.pem"
WALLET_SOVEREIGN="~/MultiversX/testnet/node/config/walletKey.pem"

#=========== NETWORK CONFIGURATION ===========
MAIN_CHAIN_ELASTIC=https://testnet-index.multiversx.com
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this config be automatically updated to mainnet/devnet elastic link if we change the script configuration chain ID?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CHAIN_ID is not changing the PROXY or MAIN_CHAIN_ELASTIC URL. It needs to be changed manually.

PROXY=https://testnet-gateway.multiversx.com
CHAIN_ID=T
PROXY_SOVEREIGN=http://localhost:${PORT_PROXY}
Expand Down
9 changes: 0 additions & 9 deletions scripts/testnet/sovereignBridge/config/deploy.snippets.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,15 +71,6 @@ sovereignStart() {
$TESTNET_DIR/sovereignStart.sh
}

# This function will restart sovereign:
# - stop sovereign nodes and services
# - deploy sovereign nodes with all services
sovereignRestart() {
stopSovereign

sovereignStart
}

# This function will reset sovereign:
# - stop sovereign nodes and services
# - deploy sovereign nodes with all services
Expand Down
2 changes: 1 addition & 1 deletion scripts/testnet/sovereignBridge/config/py.snippets.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ updateSovereignConfig() {
ESDT_PREFIX=$1
fi

python3 $SCRIPT_PATH/pyScripts/update_toml.py $ESDT_SAFE_ADDRESS $ESDT_SAFE_ADDRESS_SOVEREIGN $ESDT_PREFIX
python3 $SCRIPT_PATH/pyScripts/update_toml.py $ESDT_SAFE_ADDRESS $ESDT_SAFE_ADDRESS_SOVEREIGN $ESDT_PREFIX $MAIN_CHAIN_ELASTIC
}

generateRandomEsdtPrefix() {
Expand Down
31 changes: 30 additions & 1 deletion scripts/testnet/sovereignBridge/pyScripts/update_toml.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import os
import sys
import re
import sys


def update_subscribed_addresses(lines, section, identifier, main_chain_address) -> []:
Expand Down Expand Up @@ -68,11 +68,37 @@ def update_node_configs(config_path, esdt_prefix, sovereign_chain_address):
update_transfer_and_execute_address(config_path + "/config.toml", sovereign_chain_address)


def update_main_chain_elastic_url(lines, section, key, value):
updated_lines = []
section_found = False

for line in lines:
if line.startswith("[" + section + "]"):
section_found = True
if section_found and key in line:
line = re.sub(rf'({re.escape(key)}\s*=\s*)".*?"', rf'\1"{value}"', line)
section_found = False
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why set this to False if you don't use it afterwards?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will skip the second if and will not check key in line every time.

updated_lines.append(line)

return updated_lines


def update_external_config(file_path, main_chain_elastic):
with open(file_path, 'r') as file:
lines = file.readlines()

updated_lines = update_main_chain_elastic_url(lines, "MainChainElasticSearchConnector", "URL", main_chain_elastic)

with open(file_path, 'w') as file:
file.writelines(updated_lines)


def main():
# input arguments
main_chain_address = sys.argv[1]
sovereign_chain_address = sys.argv[2]
esdt_prefix = sys.argv[3]
main_chain_elastic = sys.argv[4]

current_path = os.getcwd()
project = 'mx-chain-go'
Expand All @@ -84,6 +110,9 @@ def main():
config_path = project_path + "/cmd/node/config"
update_node_configs(config_path, esdt_prefix, sovereign_chain_address)

external_path = project_path + "/cmd/node/config/external.toml"
update_external_config(external_path, main_chain_elastic)


if __name__ == "__main__":
main()