Skip to content

importing tedana in python seems to lock my multiprocessing to one core #473

Closed
@Aaronearlerichardson

Description

@Aaronearlerichardson

Summary

I am writing a preprocessing script in python which mostly involves using nipype to wrap afni terminal commands. Normally afni will either automatically use all available cpus or take a manually given amount to process certain commands, however when I import tendana, nipype will only use one. Could this have to do with #188 ?

Additional Detail

Here is a sample code for a wrapper in my script:

import nipype
from multiprocessing import cpu_count
from nipype.interfaces import afni as afni
from tedana.workflows import t2smap_workflow

def warp(self,fileobj1,fileobj2,out_file,transformation=None,args=None,saved_mat_file=None):

        ThreeDWarp = afni.Warp(in_file=fileobj1,out_file=out_file)
        if args is not None:
            ThreeDWarp.inputs.args=args
        if transformation == 'card2oblique':
            ThreeDWarp.inputs.oblique_parent = fileobj2
        elif transformation == 'deoblique':
            ThreeDWarp.inputs.deoblique = True
        elif transformation == 'mni2tta':
            ThreeDWarp.inputs.mni2tta = True
        elif transformation == 'tta2mni':
            ThreeDWarp.inputs.tta2mni = True
        elif transformation == 'matrix':
            ThreeDWarp.inputs.matparent = fileobj2
        elif transformation == None:
            print("Warning: no transformation input given")
        else:
            print("Warning: none of the transformation options given match the possible arguments. Matching arguments are card2oblique,"+
             " deoblique, mni2tta, tta2mni, and matrix")
        ThreeDWarp.inputs.num_threads = cpu_count()

        if saved_mat_file: #this is for if the pipline requires saving the 1D matrix tranformation information
            print('saving matrix')
            ThreeDWarp.inputs.verbose = True
            ThreeDWarp.inputs.save_warp = True
            
        ThreeDWarp.run()

Next Steps

printing os.environ gives me ({ 'OMP_NUM_THREADS': '1', 'LESSOPEN': '| /usr/bin/lesspipe %s', '_': '/usr/bin/python3', 'MKL_NUM_THREADS': '1', 'NUMEXPR_NUM_THREADS': '1', 'VECLIB_MAXIMUM_THREADS': '1', 'KMP_DUPLICATE_LIB_OK': 'True', 'KMP_INIT_AT_FORK': 'FALSE'})

I changed OMP_NUM_THREADS and fixed my issue, but wanted to post this so that the devs here would know how this can affect nipype.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugissues describing a bug or error found in the projectpriority: highissues that would be really helpful if they were fixed already

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions