Throwing Logic-Related Exception

[dbeavon]

If I have some reason to throw a logic-related error from my .Net worker code, how do I accomplish that?

I want the failure to bubble out to the driver.

I do NOT want the same work (in a UDF) to be retried multiple times on several different worker nodes, for the sake fault tolerance. IE. the exception indicates a logical problem that won't go away no matter how many times attempts.

Sorry for the basic question. I'm hoping I can use the try/catch pattern but maybe there is another technique that I'm not aware of. I've been using spark from scala for about a year and I'm new to the .net side of spark. I'm not aware if there was a technique for accomplishing this on the scala side of things either.

My exceptions eventually bubble out to the driver either way, but I'd like it to happen more quickly, without repeated attempts on multiple workers.

Here is a callstack showing multiple attempts:

JvmException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 16.0 failed 4 times, most recent failure: Lost task 0.3 in stage 16.0 (TID 210, 172.30.11.206, executor 2): org.apache.spark.api.python.PythonException: System.Exception: DIE DIE DIE
at MyClassLibrary1.Class1.AppendStuff(String p_Input, String p_Stuff) in C:\Users\mystuff\source\repos\mySparkApp\MyClassLibrary1\Class1.cs:line 11
at MySparkApp.Program.<>c.

b__0_0(Nullable1 age, String name) in C:\Users\mystuff\source\repos\mySparkApp\Program.cs:line 76
at Microsoft.Spark.Sql.PicklingUdfWrapper3.Execute(Int32 splitIndex, Object[] input, Int32[] argOffsets) in /_/src/csharp/Microsoft.Spark/Sql/PicklingUdfWrapper.cs:line 78
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.SingleCommandRunner.Run(Int32 splitId, Object input) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 252
at Microsoft.Spark.Worker.Command.PicklingSqlCommandExecutor.ExecuteCore(Stream inputStream, Stream outputStream, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 152
at Microsoft.Spark.Worker.Command.SqlCommandExecutor.Execute(Version version, Stream inputStream, Stream outputStream, PythonEvalType evalType, SqlCommand[] commands) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\SqlCommandExecutor.cs:line 76
at Microsoft.Spark.Worker.Command.CommandExecutor.Execute(Stream inputStream, Stream outputStream, Int32 splitIndex, CommandPayload commandPayload) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\Command\CommandExecutor.cs:line 65
at Microsoft.Spark.Worker.TaskRunner.ProcessStream(Stream inputStream, Stream outputStream, Version version, Boolean& readComplete) in D:\a\1\s\src\csharp\Microsoft.Spark.Worker\TaskRunner.cs:line 154

[imback82]

My exceptions eventually bubble out to the driver either way, but I'd like it to happen more quickly, without repeated attempts on multiple workers.

You can change spark.task.maxFailures to 1 if you don't want repeated attempts:

  private[spark] val TASK_MAX_FAILURES =
    ConfigBuilder("spark.task.maxFailures")
      .version("0.8.0")
      .intConf
      .createWithDefault(4)

[dbeavon]

Thanks that is helpful. I've never used that. I'm wondering how I never came across it. I rarely have errors that recover on their own. For most of my errors, if I get it once, it happens all four times. The VM's won't normally run out of resources or lose power in Azure.

I was also hoping there might be some .Net Exception class that I could derive from. Something like that could be used to communicate a failure which shouldn't generate any subsequent retry. Or maybe some sort of dynamic dirty-flagging in spark that would instantly throw failures for all future retries, instead of starting any real work. That would allow us to subvert any time-consuming tasks that we already know are doomed to fail part-way thru.