Converting Spacy project to Kubeflow pipeline

[ryuzakinho]

Hi, I have been using Spacy for a while for text categorization. I am currently using Spacy v3.5 and would like to convert my textcat project into a Kubeflow pipeline, for now, for education purposes.

My first attempt has been to create a Kubeflow component that just uses python's subprocess to run python -m spacy project run all .. However, this seems not to take the "scripts" directory when creating the pipeline.

Any tips on how achieve this?

[shadeMe]

Could you expand on what you mean with "when creating the pipeline"? Are you referring to the Kubeflow pipeline here? If so, the first step would be to ascertain if your spaCy project is packaged correctly.

[ryuzakinho]

Thanks, well, I am trying to write a Kubeflow component this way:

def run_all_train():
    import subprocess
    print('Test print 1')
    subprocess.run(['python', '-m', 'spacy', 'project', 'run', 'all', '.'])
    print('Test print 2')

Which does not feel very elegant as I am just trying to run a subprocess and which, for me is generating a "Killed" message on the Kubeflow run log.

I guess my question would be more: what would be the pythonic/Spacy way of running all the steps in project.yml so that I could use my project in a Kubeflow pipeline?

[shadeMe]

Perhaps the current working directory is not being resolved correctly? You can try replacing the . argument with the actual path string, or use subprocess.Popen with its cwd argument.

Alternatively, you can call project_run directly.

[ryuzakinho]

Thanks. I am now using the project_run

It turns out the Killed error was due to the fact that my Pipeline Runs were getting too small compute resources.