Can you take derivative of complicated function whose symbolic form is not explicit or not known?

[unary-code]

Hi,

Can I run ForwardDiff.derivative(f,float(x)) on a weird recursive function like "eval_tree_array", instead of
a more explicit function like f(x) = sin(x)?

My code:

from pysr import PySRRegressor

objective = """
Pkg.add("Roots")
Pkg.add("ForwardDiff")

using QuadGK
using Roots
using ForwardDiff
function eval_loss(tree, dataset::Dataset{T,L}, options)::L where {T,L}
    MAX_THRESHOLD_ALLOWED = 10000

    prediction, flag = eval_tree_array(tree, dataset.X, options)
    !flag && return L(Inf)
    
    f(x) = x -> 1/(eval_tree_array(tree, reshape([x;], 1, 1), options))
    fp(x) = ForwardDiff.derivative(f,float(x))
    
    roots = find_zero((f, D(f)), 0.3, Roots.Newton())
    p = 3
    
    #my_root = -1
    d = 2
    # If the function f(x) represented by the tree "tree"
    # has a value f(x) close to positive infinity or negative infinity
    # when x is in [0, 1], then return L(Inf).
    
    #if my_root >= 0 and my_root <= 1
    #    return L(Inf)
    #end
    
    num_points = length(prediction)
    
    for i in 1:num_points
        cur_expr_val = prediction[i]
        if (cur_expr_val > MAX_THRESHOLD_ALLOWED)
            return Inf
        end
        if (cur_expr_val < 0)
            return Inf
        end
    end
    
    num_rectangles = 200
    
    # You make sure that this object is of type Matrix, not Vector and not Array.
    evenly_spaced_numbers = reshape([LinRange(0, 1, num_rectangles + 1);], 1, num_rectangles + 1)
    evenly_spaced_numbers = Float32.(evenly_spaced_numbers)
    
    prediction_on_evenly_spaced_numbers, flag_on_evenly_spaced_numbers = eval_tree_array(tree, evenly_spaced_numbers, options)
    
    
    integral, err = quadgk(x -> eval_tree_array(tree, reshape([x;], 1, 1), options), 0, 1, rtol=1e-8)

    norm_constant = 0
    prev_expr_val = -1
    for i in 0:num_rectangles
        cur_expr_val = prediction_on_evenly_spaced_numbers[i+1]
        if (cur_expr_val > MAX_THRESHOLD_ALLOWED)
            return Inf
        end
        if (cur_expr_val < 0)
            return Inf
        end
        
        cur_expr_val = cur_expr_val * cur_expr_val
        if (i > 0)
            cur_trapezoid_area = (cur_expr_val + prev_expr_val) / (2.0 * num_rectangles)
            norm_constant += cur_trapezoid_area
        end
        
        prev_expr_val = cur_expr_val
    end
    
    prediction = (prediction .* prediction)
    
    actual_probs = (prediction) / norm_constant
    
    # println("HELLO WORLD")
    
    # length(actual_probs) equals length(prediction)
    return exp(-1 * sum(log.(actual_probs)) / (length(prediction)))
end
"""

model = PySRRegressor(
    niterations=40,  # < Increase me for better results
    binary_operators=["+", "*", "-", "/"],
#     unary_operators=[
#         "exp",
#         # ^ Custom operator (julia syntax)
#     ],
    # ^ Define operator for SymPy as well
    #loss="loss(prediction, target) = (prediction - target)^2",
    full_objective=objective
    # ^ Custom loss function (julia syntax)
)

IMPORTANT PART is below (where I explain the error I got):
Later in my ipynb file, I run this code:

model.fit(X, y)

which gives me this error: "
RuntimeError: <PyCall.jlwrap (in a Julia function called from Python)
JULIA: MethodError: no method matching extract_derivative(::Type{ForwardDiff.Tag{var"#f#29"{Node{Float32}, Options{Int64, Optim.Options{Float64, Nothing}, L2DistLoss, typeof(eval_loss), StatsBase.Weights{Float64, Float64, Vector{Float64}}}}, Float64}}, ::var"#27#30"{Node{Float32}, Options{Int64, Optim.Options{Float64, Nothing}, L2DistLoss, typeof(eval_loss), StatsBase.Weights{Float64, Float64, Vector{Float64}}}})
".

For your understanding, my call to PySR package's model.fit(X, y) this itself will create a bunch of trees (each tree represents a symbolic function in terms of x like "cos(3.4 * x) / 2".
For our case, I only have 1 independent variable x.

Hmm, maybe I can ask the creators of PySR package if there is a method to convert a tree "tree" to a symbolic expression, a symbolic expression that I think your ForwardDiff function would be able to handle.

[thomvet]

You cannot auto-differentiate through non-Julia code with ForwardDiff.jl. Since you are calling Python this cannot work..

It isn't possible, because the number-type by ForwardDiff (Dual{...}) is unknown to the non-Julia code.

[KristofferC]

Yew, but see https://juliadiff.org/ForwardDiff.jl/stable/user/limitations/.