Uncaught exception in lib/galaxy/tools/parameters/meta.py

[abretaud]

Hey,
After updating to 18.09, when running biomaj2galaxy to run some data managers, I get this exception:

Traceback (most recent call last):
  File "lib/galaxy/web/framework/decorators.py", line 283, in decorator
    rval = func(self, trans, *args, **kwargs)
  File "lib/galaxy/webapps/galaxy/api/tools.py", line 418, in create
    return self._create(trans, payload, **kwd)
  File "lib/galaxy/webapps/galaxy/api/tools.py", line 476, in _create
    vars = tool.handle_input(trans, incoming, history=target_history, use_cached_job=use_cached_job)
  File "lib/galaxy/tools/__init__.py", line 1385, in handle_input
    all_params, all_errors, rerun_remap_job_id, collection_info = self.expand_incoming(trans=trans, incoming=incoming, request_context=request_context)
  File "lib/galaxy/tools/__init__.py", line 1340, in expand_incoming
    expanded_incomings, collection_info = expand_meta_parameters(trans, self, incoming)
  File "lib/galaxy/tools/parameters/meta.py", line 118, in expand_meta_parameters
    process_key(incoming_key, d=nested_dict)
  File "lib/galaxy/tools/parameters/meta.py", line 87, in process_key
    d[input_name].append(subdict)
AttributeError: 'object' object has no attribute 'append'

It looks like this code comes from bf206c8, but I'm not sure how to fix it

[mvdbeek]

Is there a specific data manager where this occurs ?

[abretaud]

It's the manual one: https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_manual
I think it happened when trying to add something to the bwa data table, but not 100% sure

[mvdbeek]

I think I see the issue, this is the nested repeat that we're not dealing with properly in process_key

[mvdbeek]

do you have a chance to run this again and log the incoming parameter dict as in mvdbeek@8ee895e ?

I noticed a potential issue in the process_key code, but I'm not sure how this could have led to the above traceback using the data_manager_manual tool. I can reproduce the traceback if I simulate a parameter and a repeat having the same name, but that doesn't happen in the data_manager_manual code and I don't think that would be a valid galaxy tool ... so having the actual value would help a lot.

[abretaud]

No problem, here's the log!

Reordering incoming: {u'data_tables_0|columns_0|data_table_column_value': u'Amel_HAv3.1', u'data_tables_4|columns_2|data_table_column_value': u'Amel_HAv3.1', u'data_tables_1|columns_2|data_table_column_value': u'/db/genomes/Apis_mellifera-ncbi/Apis_mellifera-ncbi_Amel_HAv3.1/blast/Apis_mellifera-ncbi', u'data_tables_2|columns_1|data_table_column_value': u'Apis mellifera NCBI proteins (Amel_HAv3.1)', u'data_tables_1|columns_0|data_table_column_value': u'Amel_HAv3.1', u'data_tables_0|data_table_name': u'bowtie2_indexes', u'data_tables_2|data_table_name': u'blastdb_p', u'data_tables_1|columns_0|data_table_column_name': u'value', u'data_tables_4|columns_2|data_table_column_name': u'name', u'data_tables_4|columns_3|data_table_column_name': u'path', u'data_tables_3|columns_0|is_path|is_path_selector': u'no', u'data_tables_4|columns_0|data_table_column_value': u'Amel_HAv3.1', u'data_tables_1|data_table_name': u'blastdb', u'data_tables_2|columns_2|is_path|is_path_selector': u'no', u'data_tables_1|columns_0|is_path|is_path_selector': u'no', u'data_tables_0|columns_2|is_path|is_path_selector': u'no', u'data_tables_3|data_table_name': u'twobit', u'data_tables_2|columns_0|data_table_column_value': u'Amel_HAv3.1', u'data_tables_4|columns_3|data_table_column_value': u'/db/genomes/Apis_mellifera-ncbi/Apis_mellifera-ncbi_Amel_HAv3.1/bwa/all', u'data_tables_1|columns_2|is_path|is_path_selector': u'no', u'data_tables_4|data_table_name': u'bwa_indexes', u'data_tables_2|columns_1|is_path|is_path_selector': u'no', u'data_tables_2|columns_0|data_table_column_name': u'value', u'data_tables_0|columns_1|is_path|is_path_selector': u'no', u'data_tables_3|columns_1|data_table_column_name': u'path', u'data_tables_2|columns_1|data_table_column_name': u'name', u'data_tables_0|columns_3|data_table_column_value': u'/db/genomes/Apis_mellifera-ncbi/Apis_mellifera-ncbi_Amel_HAv3.1/bowtie2/all', u'data_tables_2|columns_2|data_table_column_value': u'/db/genomes/Apis_mellifera-ncbi/Apis_mellifera-ncbi_Amel_HAv3.1/blast/protein/Apis_mellifera-ncbi_protein', u'data_tables_3|columns_0|data_table_column_value': u'Amel_HAv3.1', u'data_tables_1|columns_1|data_table_column_name': u'name', u'data_tables_0|columns_3|is_path|is_path_selector': u'no', u'data_tables_0|columns_1|data_table_column_value': u'Amel_HAv3.1', u'data_tables_4|columns_2|is_path|is_path_selector': u'no', u'data_tables_0|columns_2|data_table_column_value': u'Amel_HAv3.1', u'data_tables_4|columns_3|is_path|is_path_selector': u'no', u'data_tables_1|columns_1|is_path|is_path_selector': u'no', u'data_tables_0|columns_0|data_table_column_name': u'value', u'data_tables_4|columns_1|data_table_column_value': u'Amel_HAv3.1', u'data_tables_1|columns_1|data_table_column_value': u'Amel_HAv3.1', u'data_tables_3|columns_1|is_path|is_path_selector': u'no', u'data_tables_0|columns_0|is_path|is_path_selector': u'no', u'data_tables_3|columns_1|data_table_column_value': u'/db/genomes/Apis_mellifera-ncbi/Apis_mellifera-ncbi_Amel_HAv3.1/2bit/all.2bit', u'data_tables_0|columns_2|data_table_column_name': u'name', u'data_tables_4|columns_0|data_table_column_name': u'value', u'data_tables_0|columns_1|data_table_column_name': u'dbkey', u'data_tables_1|columns_2|data_table_column_name': u'path', u'data_tables_3|columns_0|data_table_column_name': u'value', u'data_tables_4|columns_1|is_path|is_path_selector': u'no', u'data_tables': [], u'directory_content': [], u'data_tables_0|columns_3|data_table_column_name': u'path', u'data_tables_4|columns_1|data_table_column_name': u'dbkey', u'data_tables_4|columns_0|is_path|is_path_selector': u'no', u'data_tables_2|columns_2|data_table_column_name': u'path', u'data_tables_2|columns_0|is_path|is_path_selector': u'no'}

[mvdbeek]

Hmm, so the formatted view of this is this:

{'data_tables_0|columns_0|data_table_column_value': 'Amel_HAv3.1',
 'data_tables_4|columns_2|data_table_column_value': 'Amel_HAv3.1',
 'data_tables_1|columns_2|data_table_column_value': '/db/genomes/Apis_mellifera-ncbi/Apis_mellifera-ncbi_Amel_HAv3.1/blast/Apis_mellifera-ncbi',
 'data_tables_2|columns_1|data_table_column_value': 'Apis mellifera NCBI proteins (Amel_HAv3.1)',
 'data_tables_1|columns_0|data_table_column_value': 'Amel_HAv3.1',
 'data_tables_0|data_table_name': 'bowtie2_indexes',
 'data_tables_2|data_table_name': 'blastdb_p',
 'data_tables_1|columns_0|data_table_column_name': 'value',
 'data_tables_4|columns_2|data_table_column_name': 'name',
 'data_tables_4|columns_3|data_table_column_name': 'path',
 'data_tables_3|columns_0|is_path|is_path_selector': 'no',
 'data_tables_4|columns_0|data_table_column_value': 'Amel_HAv3.1',
 'data_tables_1|data_table_name': 'blastdb',
 'data_tables_2|columns_2|is_path|is_path_selector': 'no',
 'data_tables_1|columns_0|is_path|is_path_selector': 'no',
 'data_tables_0|columns_2|is_path|is_path_selector': 'no',
 'data_tables_3|data_table_name': 'twobit',
 'data_tables_2|columns_0|data_table_column_value': 'Amel_HAv3.1',
 'data_tables_4|columns_3|data_table_column_value': '/db/genomes/Apis_mellifera-ncbi/Apis_mellifera-ncbi_Amel_HAv3.1/bwa/all',
 'data_tables_1|columns_2|is_path|is_path_selector': 'no',
 'data_tables_4|data_table_name': 'bwa_indexes',
 'data_tables_2|columns_1|is_path|is_path_selector': 'no',
 'data_tables_2|columns_0|data_table_column_name': 'value',
 'data_tables_0|columns_1|is_path|is_path_selector': 'no',
 'data_tables_3|columns_1|data_table_column_name': 'path',
 'data_tables_2|columns_1|data_table_column_name': 'name',
 'data_tables_0|columns_3|data_table_column_value': '/db/genomes/Apis_mellifera-ncbi/Apis_mellifera-ncbi_Amel_HAv3.1/bowtie2/all',
 'data_tables_2|columns_2|data_table_column_value': '/db/genomes/Apis_mellifera-ncbi/Apis_mellifera-ncbi_Amel_HAv3.1/blast/protein/Apis_mellifera-ncbi_protein',
 'data_tables_3|columns_0|data_table_column_value': 'Amel_HAv3.1',
 'data_tables_1|columns_1|data_table_column_name': 'name',
 'data_tables_0|columns_3|is_path|is_path_selector': 'no',
 'data_tables_0|columns_1|data_table_column_value': 'Amel_HAv3.1',
 'data_tables_4|columns_2|is_path|is_path_selector': 'no',
 'data_tables_0|columns_2|data_table_column_value': 'Amel_HAv3.1',
 'data_tables_4|columns_3|is_path|is_path_selector': 'no',
 'data_tables_1|columns_1|is_path|is_path_selector': 'no',
 'data_tables_0|columns_0|data_table_column_name': 'value',
 'data_tables_4|columns_1|data_table_column_value': 'Amel_HAv3.1',
 'data_tables_1|columns_1|data_table_column_value': 'Amel_HAv3.1',
 'data_tables_3|columns_1|is_path|is_path_selector': 'no',
 'data_tables_0|columns_0|is_path|is_path_selector': 'no',
 'data_tables_3|columns_1|data_table_column_value': '/db/genomes/Apis_mellifera-ncbi/Apis_mellifera-ncbi_Amel_HAv3.1/2bit/all.2bit',
 'data_tables_0|columns_2|data_table_column_name': 'name',
 'data_tables_4|columns_0|data_table_column_name': 'value',
 'data_tables_0|columns_1|data_table_column_name': 'dbkey',
 'data_tables_1|columns_2|data_table_column_name': 'path',
 'data_tables_3|columns_0|data_table_column_name': 'value',
 'data_tables_4|columns_1|is_path|is_path_selector': 'no',
 'data_tables': [],
 'directory_content': [],
 'data_tables_0|columns_3|data_table_column_name': 'path',
 'data_tables_4|columns_1|data_table_column_name': 'dbkey',
 'data_tables_4|columns_0|is_path|is_path_selector': 'no',
 'data_tables_2|columns_2|data_table_column_name': 'path',
 'data_tables_2|columns_0|is_path|is_path_selector': 'no'}

The issue here is 'data_tables': [], and 'directory_content': [],, we can deal with those to make this more resilient, but also this isn't how the regular Galaxy client would submit this, I think.

[abretaud]

You mean biomaj2galaxy is not submitting standard input data? I'll have a look at the code to check if there's something to fix

[mvdbeek]

That is how it looks like, but I might be wrong and it's perfectly valid ... in any case #7071 should fix this.

[mvdbeek]

Alright, this should be fixed now, thanks for the report @abretaud!