Skip to content

Jacobian regularisation for neural networks (PyTorch) and hyperparameter tuning with Skorch

License

Notifications You must be signed in to change notification settings

fkempf92/JacobianRegularisation

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

30 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

JacobianRegularisation

Description

The presented method extends the Jacobian regularisation published by facebookresearch (https://github.com/facebookresearch/jacobian_regularizer), which is targeted for image recognition applications. They primarily consider what I call element-wise Jacobian regularisation, due to the nature of input gradients in the domain of image recognition.

However, in asset pricing, the input gradients have a clear economic meaning and interpretation (depending on the application)and, hence, I also consider column-mean regularisation. More details on this can be found here: https://www.kcl.ac.uk/business/assets/pdf/dafm-working-papers/2022-papers/interpretable-machine-learning-modelling-for-asset-pricing.pdf

The network can be trained using Skorch, sustantially simplifying the hyperparameter tuning!

The presented example does not only allow for the tuning of standard hyperparemeters, such as weight and Jacobian regularisation, but also architectural parameters, such as the activation function, number of nodes, hidden layers, and network shape (including a constant vs. tapered shape).

Installing

Use the package manager pip to install jacobian like below.

pip install git+https://github.com/fkempf92/JacobianRegularisation.git

Usage

from jacobian import RegularizedNet, NeuralNetKK
from skorch.callbacks import EarlyStopping
from sklearn.model_selection import RandomizedSearchCV

# setup early stopping
es = EarlyStopping(patience=10, threshold=1e-4)
# setup neural network
net = RegularizedNet(
          module=NeuralNetKK,
          module__input_dim=input_dim,
          module__nodes=64,
          module__const_arch=False,
          module__hidden_layers=1,
          module__activation=nn.LeakyReLU(),
          module__batchnorm=True,
          module__dropout=True,
          criterion=nn.MSELoss,
          weight_reg=True,
          w_alpha=1,
          w_l1_ratio=1,   
          jacob_reg=True, 
          jacob_type='element',
          j_alpha=1,
          j_l1_ratio=1,
          batch_size=1000,
          max_epochs=200,
          callbacks=[es],
          optimizer=torch.optim.Adam,
          optimizer__lr=1e-4,
          optimizer__weight_decay=0,
          iterator_train__drop_last=True,
          verbose=False
)
# get hyperparameters for given input
def get_tune_params(X):
        nf = X.shape[1]
        # hidden layers
        HL = list(map(int, list(
            np.array(range(int(np.log(nf / 2) / np.log(2)))) + 1)))
        # number of nodes
        U = np.random.uniform(low=np.log(0.5 * nf), high=np.log(1.1 * nf),
                              size=15)
        N = sorted(list(map(int, list(np.exp(U)))))
        # learning rate
        LR = list(10 ** np.random.uniform(low=-5, high=-3, size=15))
        LR = sorted(list(map(lambda x: round(x, 7), LR)))
        # dropout probability
        D = list(np.random.uniform(low=0.05, high=0.25, size=15))
        D = sorted(list(map(lambda x: round(x, 3), D)))
        # Penalty
        P = list(10 ** np.random.uniform(low=-8, high=-4, size=15))
        P = sorted(list(map(lambda x: round(x, 10), P)))
        return HL, N, LR, D, P

# hyperparameters for given X
HL, N, LR, D, P = get_tune_params(X=X)

# setup grid
grid_dict = {'module__hidden_layers': HL,
             'module__const_arch': [True, False],
             'module__nodes': N,
             'module__d': D,
             'j_alpha': P,
             'j_l1_ratio': [.1, .3, .5, .7, .9, .95, .99],
             'jacob_type': ['element', 'mean'],
             'w_alpha': P,
             'w_l1_ratio': [.1, .3, .5, .7, .9, .95, .99],
             'optimizer__lr': LR}
             
# setup grid search – Example
gs = RandomizedSearchCV(net, 
                        grid_dict, 
                        n_jobs=-1,
                        refit=True, 
                        random_state=123,
                        cv=10)
                        
# fit with skorch, where we assume X_train and y_train are torch tensors
gs.fit(X_train, y_train)
# get best model
gs.best_estimator_
# perform predictions
gs.predict(X_test_)

About

Jacobian regularisation for neural networks (PyTorch) and hyperparameter tuning with Skorch

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages