Skip to content

Latest commit

 

History

History
138 lines (123 loc) · 6.08 KB

Atlas API.md

File metadata and controls

138 lines (123 loc) · 6.08 KB

Important API/commands to fetch information and/or troubleshoot Atlas

1. Fetch Atlas server status

curl -u admin -X GET -H 'Content-Type: application/json' -H 'Cache-Control: no-cache' http://$(hostname -f):21000/api/atlas/admin/status

2. Fetch Atlas metrics

curl -u admin -X GET -H 'Content-Type: application/json' -H 'Cache-Control: no-cache' http://$(hostname -f):21000/api/atlas/admin/metrics

Example

curl -u admin -X GET -H 'Content-Type: application/json' -H 'Cache-Control: no-cache' http://c1232-node3.coelab.cloudera.com:21000/api/atlas/admin/metrics

3. Basic search with entity type

curl -u admin -X GET  --header 'Accept: application/json;charset=UTF-8' "http://$(hostname -f):21000/api/atlas/v2/search/basic?&typeName=<entity-type>" | python -m json.tool

Example

curl -u admin -X GET  --header 'Accept: application/json;charset=UTF-8' "http://$(hostname -f):21000/api/atlas/v2/search/basic?&typeName=hive_table" | python -m json.tool

4. Basic search type + tag

curl -u admin -X GET  --header 'Accept: application/json;charset=UTF-8' "http://$(hostname -f):21000/api/atlas/v2/search/basic?classification=<tag>&typeName=<entity-type>"

Example

curl -u admin -X GET  --header 'Accept: application/json;charset=UTF-8' "http://$(hostname -f):21000/api/atlas/v2/search/basic?classification=NewTag&typeName=hive_column"

5. Fetch Solr collection status

kinit -kt /etc/security/keytabs/ambari-infra-solr.service.keytab $(klist -kte /etc/security/keytabs/ambari-infra-solr.service.keytab | awk 'NR==4{print $4}')
curl -ivk --negotiate -u : "http://$(hostname -f):8886/solr/admin/collections?action=CLUSTERSTATUS&wt=json&indent=on"

6. Atlas-Kafka commands

a. Describe ATLAS_HOOK topic
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper `hostname -f`:2181 --topic ATLAS_HOOK

Expected result

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper `hostname -f`:2181 --topic ATLAS_HOOK
Topic:ATLAS_HOOK	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: ATLAS_HOOK	Partition: 0	Leader: 1001	Replicas: 1001	Isr: 1001
b. Describe ATLAS_ENTITIES topic
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper `hostname -f`:2181 --topic ATLAS_ENTITIES

Expected result

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper `hostname -f`:2181 --topic ATLAS_ENTITIES
Topic:ATLAS_ENTITIES	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: ATLAS_ENTITIES	Partition: 0	Leader: 1001	Replicas: 1001	Isr: 1001
c. Describe atlas consumer group (to check current consumer lag)
  • Without Kerberos
/usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh  --bootstrap-server `hostname -f`:6667 --describe --group atlas
  • With Kerberos (HDP-2.6.x)
/usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh  --bootstrap-server `hostname -f`:6667 --describe --group atlas --security-protocol <kafka-security-protocol>

Example

/usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh  --bootstrap-server `hostname -f`:6667 --describe --group atlas --security-protocol SASL_PLAINTEXT
  • With Kerberos (HDP-3.x)
/usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh  --bootstrap-server `hostname -f`:6667 --describe --group atlas --command-config cluster.config

# Contents of cluster.config file
cat > cluster.config
security.protocol=SASL_PLAINTEXT
CTRL+D

Expected result

/usr/hdp/current/kafka-broker/bin/kafka-consumer-groups.sh  --bootstrap-server `hostname -f`:6667 --describe --group atlas --command-config cluster.config

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
ATLAS_HOOK      0          12              12              0               consumer-1-367dc678-f9cd-4e8b-a0c7-0cc154a13b72 /172.25.36.16   consumer-1
[2020-05-03 12:06:32,999] WARN [Principal=null]: TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.KerberosLogin)
d. Dump messages from ATLAS_HOOK Kafka topic to standard output/text file
  • With Kerberos (HDP-3.x)
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server `hostname -f`:6667 --topic ATLAS_HOOK --consumer.config config.properties --from-beginning > /tmp/atlas_hook_kafka_dump.txt

# Contents of config.properties file
cat config.properties
security.protocol=SASL_PLAINTEXT
  • With Kerberos (HDP-2.x)
/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server `hostname -f`:6667 --topic ATLAS_HOOK --security-protocol <kafka-protocol> --from-beginning > /tmp/atlas_hook_kafka_dump.txt
  • In Ambari > Kafka > Configs > filter with 'protocol' to find the protocol being used by Kafka and replace accordingly.

7. Check Atlas version

curl -v -u admin http://localhost:21000/api/atlas/admin/version

Expected result:

curl -v -u admin http://localhost:21000/api/atlas/admin/version
{"Description":"Metadata Management and Data Governance Platform over Hadoop","Revision":"9b84b9688b91afe3fc58b1a16ecdaa0c190910fa","Version":"1.1.0.3.1.4.0-315","Name":"apache-atlas"}

8. Atlas Export API

This API exports a Hive database in json format.

curl -X POST -u admin:hadoop12345! -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d '{
    "itemsToExport": [
       { "typeName": "hive_db", "uniqueAttributes": { "qualifiedName": "<db-qualified>" } }
    ]
}' "https://<atlas-server>:21443/api/atlas/admin/export" > export.zip

9. Add a label to an Atlas entity*

curl -L -X POST 'http://atlas-server:31000/api/atlas/v2/entity/guid/{guid}/labels' -H 'Content-Type: application/json' --data-raw '["label1","label2","label3"]' -u {username}

10. Fetch lineage of an entity

curl -u admin -X GET  --header 'Accept: application/json;charset=UTF-8' http://$(hostname -f):21000/api/atlas/v2/lineage/{guid}
  • Only supported in Atlas 2.0