Creating a thick JAR file #344

blaze-ops · 2021-10-27T15:04:36Z

blaze-ops
Oct 27, 2021

Hello,

I am new to Spline and planning to use the spline-spark-agent to extract the lineage and send it to Kafka. Later, use Apache Atlas to view the results.

Following the document I used the "build-all.sh" to create a fat JAR and ran into following issue:

java.lang.RuntimeException: Could not generate model 'AgentInfo'
    at io.swagger.codegen.v3.DefaultGenerator.generateModels (DefaultGenerator.java:451)
    at io.swagger.codegen.v3.DefaultGenerator.generate (DefaultGenerator.java:779)
    at io.swagger.codegen.v3.maven.plugin.CodeGenMojo.execute_ (CodeGenMojo.java:554)
    at io.swagger.codegen.v3.maven.plugin.CodeGenMojo.execute (CodeGenMojo.java:319)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
    at java.lang.Thread.run (Thread.java:833)
Caused by: com.github.jknack.handlebars.HandlebarsException: /handlebars/scala/client/licenseInfo.mustache:2:6: java.lang.reflect.InaccessibleObjectException: Unable to make public boolean java.util.Collections$EmptyMap.isEmpty() accessible: module java.base does not "opens java.util" to unnamed module @34faab49
    /handlebars/scala/client/licenseInfo.mustache:2:6
    at java.lang.reflect.AccessibleObject.checkCanSetAccessible (AccessibleObject.java:354)
    at java.lang.reflect.AccessibleObject.checkCanSetAccessible (AccessibleObject.java:297)
    at java.lang.reflect.Method.checkCanSetAccessible (Method.java:199)
    at java.lang.reflect.Method.setAccessible (Method.java:193)
    at com.github.jknack.handlebars.context.MemberValueResolver.cache (MemberValueResolver.java:82)
    at com.github.jknack.handlebars.context.MemberValueResolver.resolve (MemberValueResolver.java:54)
    at com.github.jknack.handlebars.Context$CompositeValueResolver.resolve (Context.java:199)
    at com.github.jknack.handlebars.internal.path.PropertyPath.eval (PropertyPath.java:52)
    at com.github.jknack.handlebars.Context$PathExpressionChain.next (Context.java:361)
    at com.github.jknack.handlebars.Context$PathExpressionChain.eval (Context.java:381)
    at com.github.jknack.handlebars.Context.get (Context.java:621)
    at com.github.jknack.handlebars.Context$PartialCtx.get (Context.java:165)
    at com.github.jknack.handlebars.internal.Variable.value (Variable.java:183)
    at com.github.jknack.handlebars.internal.Variable.merge (Variable.java:160)
    at com.github.jknack.handlebars.internal.BaseTemplate.apply (BaseTemplate.java:130)
    at com.github.jknack.handlebars.internal.TemplateList.merge (TemplateList.java:95)
    at com.github.jknack.handlebars.internal.BaseTemplate.apply (BaseTemplate.java:130)
    at com.github.jknack.handlebars.internal.ForwardingTemplate.apply (ForwardingTemplate.java:86)
    at com.github.jknack.handlebars.internal.Partial.merge (Partial.java:228)
    at com.github.jknack.handlebars.internal.BaseTemplate.apply (BaseTemplate.java:130)
    at com.github.jknack.handlebars.internal.TemplateList.merge (TemplateList.java:95)
    at com.github.jknack.handlebars.internal.BaseTemplate.apply (BaseTemplate.java:130)
    at com.github.jknack.handlebars.internal.BaseTemplate.apply (BaseTemplate.java:118)
    at com.github.jknack.handlebars.internal.ForwardingTemplate.apply (ForwardingTemplate.java:72)
    at io.swagger.codegen.v3.templates.HandlebarTemplateEngine.getRendered (HandlebarTemplateEngine.java:26)
    at io.swagger.codegen.v3.DefaultGenerator.processTemplateToFile (DefaultGenerator.java:795)
    at io.swagger.codegen.v3.DefaultGenerator.generateModels (DefaultGenerator.java:438)
    at io.swagger.codegen.v3.DefaultGenerator.generate (DefaultGenerator.java:779)
    at io.swagger.codegen.v3.maven.plugin.CodeGenMojo.execute_ (CodeGenMojo.java:554)
    at io.swagger.codegen.v3.maven.plugin.CodeGenMojo.execute (CodeGenMojo.java:319)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
    at java.lang.Thread.run (Thread.java:833)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make public boolean java.util.Collections$EmptyMap.isEmpty() accessible: module java.base does not "opens java.util" to unnamed module @34faab49
    at java.lang.reflect.AccessibleObject.checkCanSetAccessible (AccessibleObject.java:354)
    at java.lang.reflect.AccessibleObject.checkCanSetAccessible (AccessibleObject.java:297)
    at java.lang.reflect.Method.checkCanSetAccessible (Method.java:199)
    at java.lang.reflect.Method.setAccessible (Method.java:193)
    at com.github.jknack.handlebars.context.MemberValueResolver.cache (MemberValueResolver.java:82)
    at com.github.jknack.handlebars.context.MemberValueResolver.resolve (MemberValueResolver.java:54)
    at com.github.jknack.handlebars.Context$CompositeValueResolver.resolve (Context.java:199)
    at com.github.jknack.handlebars.internal.path.PropertyPath.eval (PropertyPath.java:52)
    at com.github.jknack.handlebars.Context$PathExpressionChain.next (Context.java:361)
    at com.github.jknack.handlebars.Context$PathExpressionChain.eval (Context.java:381)
    at com.github.jknack.handlebars.Context.get (Context.java:621)
    at com.github.jknack.handlebars.Context$PartialCtx.get (Context.java:165)
    at com.github.jknack.handlebars.internal.Variable.value (Variable.java:183)
    at com.github.jknack.handlebars.internal.Variable.merge (Variable.java:160)
    at com.github.jknack.handlebars.internal.BaseTemplate.apply (BaseTemplate.java:130)
    at com.github.jknack.handlebars.internal.TemplateList.merge (TemplateList.java:95)
    at com.github.jknack.handlebars.internal.BaseTemplate.apply (BaseTemplate.java:130)
    at com.github.jknack.handlebars.internal.ForwardingTemplate.apply (ForwardingTemplate.java:86)
    at com.github.jknack.handlebars.internal.Partial.merge (Partial.java:228)
    at com.github.jknack.handlebars.internal.BaseTemplate.apply (BaseTemplate.java:130)
    at com.github.jknack.handlebars.internal.TemplateList.merge (TemplateList.java:95)
    at com.github.jknack.handlebars.internal.BaseTemplate.apply (BaseTemplate.java:130)
    at com.github.jknack.handlebars.internal.BaseTemplate.apply (BaseTemplate.java:118)
    at com.github.jknack.handlebars.internal.ForwardingTemplate.apply (ForwardingTemplate.java:72)
    at io.swagger.codegen.v3.templates.HandlebarTemplateEngine.getRendered (HandlebarTemplateEngine.java:26)
    at io.swagger.codegen.v3.DefaultGenerator.processTemplateToFile (DefaultGenerator.java:795)
    at io.swagger.codegen.v3.DefaultGenerator.generateModels (DefaultGenerator.java:438)
    at io.swagger.codegen.v3.DefaultGenerator.generate (DefaultGenerator.java:779)
    at io.swagger.codegen.v3.maven.plugin.CodeGenMojo.execute_ (CodeGenMojo.java:554)
    at io.swagger.codegen.v3.maven.plugin.CodeGenMojo.execute (CodeGenMojo.java:319)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
    at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
    at java.util.concurrent.FutureTask.run (FutureTask.java:264)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
    at java.lang.Thread.run (Thread.java:833)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spline Spark Agent 1.0.0-SNAPSHOT:
[INFO]
[INFO] Spline Spark Agent ................................. SUCCESS [  3.162 s]
[INFO] agent-core_2.11 .................................... FAILURE [  4.496 s]
[INFO] examples_2.11 ...................................... SKIPPED
[INFO] integration-tests_2.11 ............................. SKIPPED
[INFO] spark-2.2-spline-agent-bundle_2.11 ................. SKIPPED
[INFO] spark-2.3-spline-agent-bundle_2.11 ................. SKIPPED
[INFO] spark-2.4-spline-agent-bundle_2.11 ................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  8.098 s (Wall Clock)
[INFO] Finished at: 2021-10-26T23:27:08-04:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.swagger.codegen.v3:swagger-codegen-maven-plugin:3.0.21:generate (swagger.api.v1_0) on project agent-core_2.11: Code generation failed. See above for the full exception. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :agent-core_2.11

Can anyone please help me on the next steps forward

Answered by wajda

Oct 27, 2021

Make sure you use Java 1.8 for building.
Check mvn --version

View full answer

wajda · 2021-10-27T20:56:41Z

wajda
Oct 27, 2021
Maintainer

Make sure you use Java 1.8 for building.
Check mvn --version

0 replies

blaze-ops · 2021-10-29T03:23:34Z

blaze-ops
Oct 29, 2021
Author

@wajda, Thank you very much for your response it really helped me a lot. I was able to create the JAR file successfully but just wanted to confirm if I will be able to import that JAR file in AWS Glue to generate the lineage of pyspark jobs.

0 replies

wajda · 2021-10-29T13:23:44Z

wajda
Oct 29, 2021
Maintainer

We didn't try it with Glue yet, but as long as it's Spark it should work in theory. Try and let us know. We'll happily assist if there's any issue.

14 replies

wajda Nov 19, 2021
Maintainer

Regarding the 2nd part of the question - Spline vs Spline Spark agent. The later is the agent integrated into the Spark job or the driver. The former is the server that aggregates and stores lineage data sent by agents.

blaze-ops Nov 23, 2021
Author

@wajda,

Following the document I used the build-all.sh to create a fat JAR. Multiple JAR's were created but in the AWS Glue job I am using "spark-2.4-spline-agent-bundle_2.11-1.0.0-SNAPSHOT.jar".

Later while running the job doing a codeless initialization and sending the lineage to Kafka:

--conf spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener 
--conf spark.spline.lineageDispatcher=kafka 
--conf spark.spline.lineageDispatcher.kafka.producer.bootstrap.servers= <bootstrap servers>
--conf spark.spline.lineageDispatcher.kafka.topic= <Topic Name>

Initially there were some issues while sending the lineage to Kafka which got fixed later. Later, I have a AWS Glue job which reads the above output from Kafka topic and dumps is S3. This is the output that got generated.

Any idea how it generated a mixture of two Spline outputs. Am i missing any step?

wajda Nov 24, 2021
Maintainer

Is this what you receive from Kafka? Try logging the output to see what the agent actually sends:

--conf spark.spline.lineageDispatcher=composite
--conf spark.spline.lineageDispatcher.composite.dispatchers=kafka,logging
--conf spark.spline.lineageDispatcher.kafka.producer.bootstrap.servers= <bootstrap servers>
--conf spark.spline.lineageDispatcher.kafka.topic= <Topic Name>

Alternatively try using Spline REST Gateway instead of Kafka for comparison.

blaze-ops Dec 3, 2021
Author

@wajda

I tried again with the same conf and no changes it was finally able to send a valid message to Kafka.

I am planning to integrate it with Apache Atlas now, will keep you posted on same.

Thanks! once again

wajda Dec 3, 2021
Maintainer

You are welcome, but could you please create new tickets or discussions for separate topics to keep things organized on github. Thank you.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Creating a thick JAR file #344

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 14 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Creating a thick JAR file #344

blaze-ops Oct 27, 2021

Replies: 3 comments · 14 replies

wajda Oct 27, 2021 Maintainer

blaze-ops Oct 29, 2021 Author

wajda Oct 29, 2021 Maintainer

wajda Nov 19, 2021 Maintainer

blaze-ops Nov 23, 2021 Author

wajda Nov 24, 2021 Maintainer

blaze-ops Dec 3, 2021 Author

wajda Dec 3, 2021 Maintainer

blaze-ops
Oct 27, 2021

Replies: 3 comments 14 replies

wajda
Oct 27, 2021
Maintainer

blaze-ops
Oct 29, 2021
Author

wajda
Oct 29, 2021
Maintainer

wajda Nov 19, 2021
Maintainer

blaze-ops Nov 23, 2021
Author

wajda Nov 24, 2021
Maintainer

blaze-ops Dec 3, 2021
Author

wajda Dec 3, 2021
Maintainer