Ansible dependencies issues with st2 ansible pack

[mab27]

• Ansible modules for network uses APIs, which means "connection: local" must be set in the playbooks, which means all the python library dependencies has be installed locally.
• This isn't working using st2 ansible pack despite I pip installed the dependencies locally.
• Broadly, how the manage the st2 virtual environnement (python libraries, ansible versions ...)

Install Ansible Roles from galaxy

mab@mab-infra:~/automation/ansible$ sudo ansible-galaxy list
mab@mab-infra:~/automation/ansible$ sudo ansible-galaxy install Juniper.junos
- downloading role 'junos', owned by Juniper
- downloading role from https://github.com/Juniper/ansible-junos-stdlib/archive/1.4.2.tar.gz
- extracting Juniper.junos to /etc/ansible/roles/Juniper.junos
- Juniper.junos (1.4.2) was installed successfully
mab@mab-infra:~/automation/ansible$ sudo ansible-galaxy list
- Juniper.junos, 1.4.2

Install python libraries required for the role:

mab@mab-infra:~/automation/ansible$ pip list | grep junos
junos-eznc (2.1.1)
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

mab@mab-infra:~/automation/ansible$ python 
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from jnpr.junos import Device
>>> dir()
['Device', '__builtins__', '__doc__', '__name__', '__package__']
>>> exit()

Ansible configuration file:

mab@mab-infra:~/automation/ansible$ more /home/mab/automation/ansible/ansible.cfg
[defaults]
inventory = hosts
roles_path = /etc/ansible/roles:./
deprecation_warnings=False

Ansible version:

mab@mab-infra:~/automation/ansible$ ansible --version
ansible 2.3.0.0
  config file = /home/mab/automation/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]

Ansible playbook details:

• The junos_get_facts module requires jnpr.junos python library as shown here: https://github.com/Juniper/ansible-junos-stdlib/blob/master/library/junos_get_facts#L163
• The playbook uses "connection: local". so the jnpr.junos python library has to be installed locally (hence the above pip and python commands)

mab@mab-infra:~/automation/ansible$ cat junos_get_facts/pb.yml 
---
 - name: create inventory directory
   hosts: localhost
   gather_facts: no
   
   tasks:
    
   - name: create inventory directory
     file: path={{playbook_dir}}/inventory state=directory
     
 - name: Get Facts
   hosts: vmx
   roles:       
    - Juniper.junos
   connection: local
   gather_facts: no
   
   tasks:
   
   - name: remove host from inventory directory
     file: path={{playbook_dir}}/inventory/{{inventory_hostname}}.conf state=absent
     
   - name: Retrieve information from devices running Junos
     junos_get_facts:
      host={{ credentials.host }}
      user={{ credentials.username }}
      passwd={{ credentials.password }}
      savedir={{playbook_dir}}/inventory
     register: junos

   - name: Print some facts
     debug: msg="device {{junos["facts"]["hostname"]}} runs version {{junos.facts.version}}" 
     when: junos.facts.version != "12.3R11.2"  

Ansible variables:

mab@mab-infra:~/automation/ansible$ cat group_vars/vmx/credentials.yml 
---
credentials:
  host: "{{ junos_host }}"
  ssh_key: /home/mab/.ssh/id_rsa
  username: mab
  password: mab123

Ansible playbook execution using ansible-playbook command (OK):

mab@mab-infra:~/automation/ansible$ sudo ansible-playbook junos_get_facts/pb.yml 

PLAY [create inventory directory] **************************************************************************************************************************************************************************

TASK [create inventory directory] **************************************************************************************************************************************************************************
ok: [localhost]

PLAY [Get Facts] *******************************************************************************************************************************************************************************************

TASK [remove host from inventory directory] ****************************************************************************************************************************************************************
ok: [vmx1]
ok: [vmx2]

TASK [Retrieve information from devices running Junos] *****************************************************************************************************************************************************
ok: [vmx2]
ok: [vmx1]

TASK [Print some facts] ************************************************************************************************************************************************************************************
ok: [vmx2] => {
    "changed": false, 
    "msg": "device vmx2 runs version 14.1R4.9"
}
ok: [vmx1] => {
    "changed": false, 
    "msg": "device vmx1 runs version 14.1R4.9"
}

PLAY RECAP *************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0   
vmx1                       : ok=3    changed=0    unreachable=0    failed=0   
vmx2                       : ok=3    changed=0    unreachable=0    failed=0   

Ansible pack for st2 installed:

mab@mab-infra:~/automation/ansible$ sudo st2 pack get ansible
+-------------+--------------------------------------------------+
| Property    | Value                                            |
+-------------+--------------------------------------------------+
| name        | ansible                                          |
| version     | 0.4.0                                            |
| author      | StackStorm, Inc.                                 |
| email       | info@stackstorm.com                              |
| keywords    | [                                                |
|             |     "ansible",                                   |
|             |     "cfg management",                            |
|             |     "configuration management"                   |
|             | ]                                                |
| description | st2 content pack containing ansible integrations |
+-------------+--------------------------------------------------+

Verify the details st2 ansible pack is going to use:

mab@mab-infra:~/automation/ansible$ sudo st2 run ansible.playbook playbook=/home/mab/automation/ansible/junos_get_facts/pb.yml cwd=/home/mab/automation/ansible/ version=true
.
id: 590f2f8e7cae220956aeb02d
status: succeeded
parameters: 
  cwd: /home/mab/automation/ansible/
  playbook: /home/mab/automation/ansible/junos_get_facts/pb.yml
  version: true
result: 
  failed: false
  return_code: 0
  stderr: ''
  stdout: "ansible-playbook 2.3.0.0
  config file = /home/mab/automation/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]"
  succeeded: true

Ansible playbook execution using st2 Ansible pack (KO):

• See error displayed: ImportError: No module named jnpr.junos
• Despite jnpr.junos python library correctly installed as proved above

mab@mab-infra:~/automation/ansible$ sudo st2 run ansible.playbook playbook=/home/mab/automation/ansible/junos_get_facts/pb.yml cwd=/home/mab/automation/ansible/ 
.
id: 590f2fd47cae220956aeb030
status: failed
parameters: 
  cwd: /home/mab/automation/ansible/
  playbook: /home/mab/automation/ansible/junos_get_facts/pb.yml
result: 
  failed: true
  return_code: 2
  stderr: Executed command "/opt/stackstorm/virtualenvs/ansible/bin/ansible-playbook /home/mab/automation/ansible/junos_get_facts/pb.yml"
  stdout: "
PLAY [create inventory directory] **********************************************

TASK [create inventory directory] **********************************************
ok: [localhost]

PLAY [Get Facts] ***************************************************************

TASK [remove host from inventory directory] ************************************
ok: [vmx1]
ok: [vmx2]

TASK [Retrieve information from devices running Junos] *************************
fatal: [vmx2]: FAILED! => {"changed": false, "failed": true, "msg": "ImportError: No module named jnpr.junos"}
fatal: [vmx1]: FAILED! => {"changed": false, "failed": true, "msg": "ImportError: No module named jnpr.junos"}
  to retry, use: --limit @/home/mab/automation/ansible/junos_get_facts/pb.retry

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0   
vmx1                       : ok=1    changed=0    unreachable=0    failed=1   
vmx2                       : ok=1    changed=0    unreachable=0    failed=1   
"
  succeeded: false

Same command in verbose mode:

mab@mab-infra:~/automation/ansible$ sudo st2 run ansible.playbook playbook=/home/mab/automation/ansible/junos_get_facts/pb.yml cwd=/home/mab/automation/ansible/ verbose=vvvv
.
id: 590f2fe17cae220956aeb033
status: failed
parameters: 
  cwd: /home/mab/automation/ansible/
  playbook: /home/mab/automation/ansible/junos_get_facts/pb.yml
  verbose: vvvv
result: 
  failed: true
  return_code: 2
  stderr: Executed command "/opt/stackstorm/virtualenvs/ansible/bin/ansible-playbook -vvvv /home/mab/automation/ansible/junos_get_facts/pb.yml"
  stdout: "Using /home/mab/automation/ansible/ansible.cfg as config file
Loading callback plugin default of type stdout, v2.0 from /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc
Loading callback plugin jsnapy of type aggregate, v2.0 from /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: pb.yml ***************************************************************
2 plays in /home/mab/automation/ansible/junos_get_facts/pb.yml

PLAY [create inventory directory] **********************************************
META: ran handlers

TASK [create inventory directory] **********************************************
task path: /home/mab/automation/ansible/junos_get_facts/pb.yml:8
Using module file /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible/modules/files/file.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1494167521.98-137850814807689 `" && echo ansible-tmp-1494167521.98-137850814807689="` echo /root/.ansible/tmp/ansible-tmp-1494167521.98-137850814807689 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmpDz9pYO TO /root/.ansible/tmp/ansible-tmp-1494167521.98-137850814807689/file.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1494167521.98-137850814807689/ /root/.ansible/tmp/ansible-tmp-1494167521.98-137850814807689/file.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/opt/stackstorm/virtualenvs/ansible/bin/python /root/.ansible/tmp/ansible-tmp-1494167521.98-137850814807689/file.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1494167521.98-137850814807689/" > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "changed": false, 
    "diff": {
        "after": {
            "path": "/home/mab/automation/ansible/junos_get_facts/inventory"
        }, 
        "before": {
            "path": "/home/mab/automation/ansible/junos_get_facts/inventory"
        }
    }, 
    "gid": 0, 
    "group": "root", 
    "invocation": {
        "module_args": {
            "attributes": null, 
            "backup": null, 
            "content": null, 
            "delimiter": null, 
            "diff_peek": null, 
            "directory_mode": null, 
            "follow": false, 
            "force": false, 
            "group": null, 
            "mode": null, 
            "original_basename": null, 
            "owner": null, 
            "path": "/home/mab/automation/ansible/junos_get_facts/inventory", 
            "recurse": false, 
            "regexp": null, 
            "remote_src": null, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": null, 
            "state": "directory", 
            "unsafe_writes": null, 
            "validate": null
        }
    }, 
    "mode": "0755", 
    "owner": "root", 
    "path": "/home/mab/automation/ansible/junos_get_facts/inventory", 
    "size": 4096, 
    "state": "directory", 
    "uid": 0
}
META: ran handlers
META: ran handlers

PLAY [Get Facts] ***************************************************************
META: ran handlers

TASK [remove host from inventory directory] ************************************
task path: /home/mab/automation/ansible/junos_get_facts/pb.yml:20
Using module file /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible/modules/files/file.py
<vmx1> ESTABLISH LOCAL CONNECTION FOR USER: root
<vmx1> EXEC /bin/sh -c 'echo ~ && sleep 0'
Using module file /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible/modules/files/file.py
<vmx2> ESTABLISH LOCAL CONNECTION FOR USER: root
<vmx2> EXEC /bin/sh -c 'echo ~ && sleep 0'
<vmx1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1494167522.14-179148203693750 `" && echo ansible-tmp-1494167522.14-179148203693750="` echo /root/.ansible/tmp/ansible-tmp-1494167522.14-179148203693750 `" ) && sleep 0'
<vmx2> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1494167522.14-144286707166271 `" && echo ansible-tmp-1494167522.14-144286707166271="` echo /root/.ansible/tmp/ansible-tmp-1494167522.14-144286707166271 `" ) && sleep 0'
<vmx1> PUT /tmp/tmpDZeWox TO /root/.ansible/tmp/ansible-tmp-1494167522.14-179148203693750/file.py
<vmx1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1494167522.14-179148203693750/ /root/.ansible/tmp/ansible-tmp-1494167522.14-179148203693750/file.py && sleep 0'
<vmx2> PUT /tmp/tmpdozB1Q TO /root/.ansible/tmp/ansible-tmp-1494167522.14-144286707166271/file.py
<vmx2> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1494167522.14-144286707166271/ /root/.ansible/tmp/ansible-tmp-1494167522.14-144286707166271/file.py && sleep 0'
<vmx1> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1494167522.14-179148203693750/file.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1494167522.14-179148203693750/" > /dev/null 2>&1 && sleep 0'
<vmx2> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1494167522.14-144286707166271/file.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1494167522.14-144286707166271/" > /dev/null 2>&1 && sleep 0'
ok: [vmx1] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "attributes": null, 
            "backup": null, 
            "content": null, 
            "delimiter": null, 
            "diff_peek": null, 
            "directory_mode": null, 
            "follow": false, 
            "force": false, 
            "group": null, 
            "mode": null, 
            "original_basename": null, 
            "owner": null, 
            "path": "/home/mab/automation/ansible/junos_get_facts/inventory/vmx1.conf", 
            "recurse": false, 
            "regexp": null, 
            "remote_src": null, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": null, 
            "state": "absent", 
            "unsafe_writes": null, 
            "validate": null
        }
    }, 
    "path": "/home/mab/automation/ansible/junos_get_facts/inventory/vmx1.conf", 
    "state": "absent"
}
ok: [vmx2] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "attributes": null, 
            "backup": null, 
            "content": null, 
            "delimiter": null, 
            "diff_peek": null, 
            "directory_mode": null, 
            "follow": false, 
            "force": false, 
            "group": null, 
            "mode": null, 
            "original_basename": null, 
            "owner": null, 
            "path": "/home/mab/automation/ansible/junos_get_facts/inventory/vmx2.conf", 
            "recurse": false, 
            "regexp": null, 
            "remote_src": null, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": null, 
            "state": "absent", 
            "unsafe_writes": null, 
            "validate": null
        }
    }, 
    "path": "/home/mab/automation/ansible/junos_get_facts/inventory/vmx2.conf", 
    "state": "absent"
}

TASK [Retrieve information from devices running Junos] *************************
task path: /home/mab/automation/ansible/junos_get_facts/pb.yml:23
Using module file /etc/ansible/roles/Juniper.junos/library/junos_get_facts
Using module file /etc/ansible/roles/Juniper.junos/library/junos_get_facts
<vmx1> ESTABLISH LOCAL CONNECTION FOR USER: root
<vmx1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<vmx2> ESTABLISH LOCAL CONNECTION FOR USER: root
<vmx2> EXEC /bin/sh -c 'echo ~ && sleep 0'
<vmx1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1494167522.41-211162944277752 `" && echo ansible-tmp-1494167522.41-211162944277752="` echo /root/.ansible/tmp/ansible-tmp-1494167522.41-211162944277752 `" ) && sleep 0'
<vmx2> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1494167522.41-231247062974027 `" && echo ansible-tmp-1494167522.41-231247062974027="` echo /root/.ansible/tmp/ansible-tmp-1494167522.41-231247062974027 `" ) && sleep 0'
<vmx1> PUT /tmp/tmpHmAo7T TO /root/.ansible/tmp/ansible-tmp-1494167522.41-211162944277752/junos_get_facts
<vmx1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1494167522.41-211162944277752/ /root/.ansible/tmp/ansible-tmp-1494167522.41-211162944277752/junos_get_facts && sleep 0'
<vmx2> PUT /tmp/tmpvj5ql9 TO /root/.ansible/tmp/ansible-tmp-1494167522.41-231247062974027/junos_get_facts
<vmx2> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1494167522.41-231247062974027/ /root/.ansible/tmp/ansible-tmp-1494167522.41-231247062974027/junos_get_facts && sleep 0'
<vmx1> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1494167522.41-211162944277752/junos_get_facts; rm -rf "/root/.ansible/tmp/ansible-tmp-1494167522.41-211162944277752/" > /dev/null 2>&1 && sleep 0'
<vmx2> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp-1494167522.41-231247062974027/junos_get_facts; rm -rf "/root/.ansible/tmp/ansible-tmp-1494167522.41-231247062974027/" > /dev/null 2>&1 && sleep 0'
fatal: [vmx1]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "console": null, 
            "host": "192.168.0.30", 
            "logfile": null, 
            "mode": null, 
            "passwd": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": "830", 
            "savedir": "/home/mab/automation/ansible/junos_get_facts/inventory", 
            "ssh_private_key_file": null, 
            "user": "mab"
        }
    }, 
    "msg": "ImportError: No module named jnpr.junos"
}
fatal: [vmx2]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "console": null, 
            "host": "192.168.0.40", 
            "logfile": null, 
            "mode": null, 
            "passwd": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": "830", 
            "savedir": "/home/mab/automation/ansible/junos_get_facts/inventory", 
            "ssh_private_key_file": null, 
            "user": "mab"
        }
    }, 
    "msg": "ImportError: No module named jnpr.junos"
}
  to retry, use: --limit @/home/mab/automation/ansible/junos_get_facts/pb.retry

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0   
vmx1                       : ok=1    changed=0    unreachable=0    failed=1   
vmx2                       : ok=1    changed=0    unreachable=0    failed=1   
"
  succeeded: false

[mab27]

Following arma suggestion:
Try to install missing pip module in Ansible pack virtualenv:
/opt/stackstorm/virtualenvs/ansible/bin/pip install

• pip install dependencies in the st2 ansible virtualenv:

mab@mab-infra:~/automation/ansible$ sudo /opt/stackstorm/virtualenvs/ansible/bin/pip install jxmlease 
[sudo] password for mab: 
The directory '/home/mab/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/mab/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting jxmlease
  Downloading jxmlease-1.0.1-py2.py3-none-any.whl
Installing collected packages: jxmlease
Successfully installed jxmlease-1.0.1
mab@mab-infra:~/automation/ansible$ sudo /opt/stackstorm/virtualenvs/ansible/bin/pip install junos-eznc
The directory '/home/mab/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/mab/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting junos-eznc
  Downloading junos-eznc-2.1.2.tar.gz (103kB)
    100% |████████████████████████████████| 112kB 861kB/s 
Collecting lxml>=3.2.4 (from junos-eznc)
  Downloading lxml-3.7.3-cp27-cp27mu-manylinux1_x86_64.whl (6.8MB)
    100% |████████████████████████████████| 6.8MB 99kB/s 
Collecting ncclient>=0.5.3 (from junos-eznc)
Requirement already satisfied: paramiko>=1.15.2 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from junos-eznc)
Collecting scp>=0.7.0 (from junos-eznc)
  Downloading scp-0.10.2-py2.py3-none-any.whl
Requirement already satisfied: jinja2>=2.7.1 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from junos-eznc)
Requirement already satisfied: PyYAML>=3.10 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from junos-eznc)
Collecting netaddr (from junos-eznc)
  Downloading netaddr-0.7.19-py2.py3-none-any.whl (1.6MB)
    100% |████████████████████████████████| 1.6MB 453kB/s 
Requirement already satisfied: six in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from junos-eznc)
Collecting pyserial (from junos-eznc)
  Downloading pyserial-3.3-py2.py3-none-any.whl (189kB)
    100% |████████████████████████████████| 194kB 612kB/s 
Requirement already satisfied: jxmlease in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from junos-eznc)
Requirement already satisfied: setuptools>0.6 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from ncclient>=0.5.3->junos-eznc)
Requirement already satisfied: pyasn1>=0.1.7 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from paramiko>=1.15.2->junos-eznc)
Requirement already satisfied: cryptography>=1.1 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from paramiko>=1.15.2->junos-eznc)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from jinja2>=2.7.1->junos-eznc)
Requirement already satisfied: appdirs>=1.4.0 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from setuptools>0.6->ncclient>=0.5.3->junos-eznc)
Requirement already satisfied: packaging>=16.8 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from setuptools>0.6->ncclient>=0.5.3->junos-eznc)
Requirement already satisfied: ipaddress in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from cryptography>=1.1->paramiko>=1.15.2->junos-eznc)
Requirement already satisfied: idna>=2.1 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from cryptography>=1.1->paramiko>=1.15.2->junos-eznc)
Requirement already satisfied: asn1crypto>=0.21.0 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from cryptography>=1.1->paramiko>=1.15.2->junos-eznc)
Requirement already satisfied: enum34 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from cryptography>=1.1->paramiko>=1.15.2->junos-eznc)
Requirement already satisfied: cffi>=1.4.1 in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from cryptography>=1.1->paramiko>=1.15.2->junos-eznc)
Requirement already satisfied: pyparsing in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from packaging>=16.8->setuptools>0.6->ncclient>=0.5.3->junos-eznc)
Requirement already satisfied: pycparser in /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography>=1.1->paramiko>=1.15.2->junos-eznc)
Installing collected packages: lxml, ncclient, scp, netaddr, pyserial, junos-eznc
  Running setup.py install for junos-eznc ... done
Successfully installed junos-eznc-2.1.2 lxml-3.7.3 ncclient-0.5.3 netaddr-0.7.19 pyserial-3.3 scp-0.10.2

• pip list for the st2 ansible virtualenv (junos-eznc is now in):

mab@mab-infra:~/automation/ansible$ sudo /opt/stackstorm/virtualenvs/ansible/bin/pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
ansible (2.2.0.0)
appdirs (1.4.3)
asn1crypto (0.22.0)
cffi (1.10.0)
cryptography (1.8.1)
enum34 (1.1.6)
idna (2.5)
ipaddress (1.0.18)
Jinja2 (2.9.6)
junos-eznc (2.1.2)
jxmlease (1.0.1)
lxml (3.7.3)
MarkupSafe (1.0)
ncclient (0.5.3)
netaddr (0.7.19)
packaging (16.8)
paramiko (2.1.2)
pip (9.0.1)
pyasn1 (0.2.3)
pycparser (2.17)
pycrypto (2.6.1)
pyparsing (2.2.0)
pyserial (3.3)
PyYAML (3.12)
scp (0.10.2)
setuptools (35.0.2)
six (1.10.0)
wheel (0.29.0)
The directory '/home/mab/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
mab@mab-infra:~/automation/ansible$ 

• Still the playbook fails with the exact same error message which is: junos-eznc >= 1.2.2 is required but does not appear to be installed.
• Am I missing something with permissions ? users ? cache ?

sudo st2 run ansible.playbook playbook=/home/mab/automation/ansible/junos_template/pb.bgp.2.yml cwd=/home/mab/automation/ansible 
.
id: 590f417d7cae220956aeb04e
status: failed
parameters: 
  cwd: /home/mab/automation/ansible
  playbook: /home/mab/automation/ansible/junos_template/pb.bgp.2.yml
result: 
  failed: true
  return_code: 2
  stderr: Executed command "/opt/stackstorm/virtualenvs/ansible/bin/ansible-playbook /home/mab/automation/ansible/junos_template/pb.bgp.2.yml"
  stdout: "
PLAY [Create BGP junos configuration] ******************************************

TASK [Render BGP configuration for junos devices] ******************************
ok: [vmx1]
ok: [vmx2]

TASK [Push bgp configuration on devices] ***************************************
fatal: [vmx1]: FAILED! => {"changed": false, "failed": true, "msg": "junos-eznc >= 1.2.2 is required but does not appear to be installed.  It can be installed using `pip install junos-eznc`"}
fatal: [vmx2]: FAILED! => {"changed": false, "failed": true, "msg": "junos-eznc >= 1.2.2 is required but does not appear to be installed.  It can be installed using `pip install junos-eznc`"}
  to retry, use: --limit @/home/mab/automation/ansible/junos_template/pb.bgp.2.retry

PLAY RECAP *********************************************************************
vmx1                       : ok=1    changed=0    unreachable=0    failed=1   
vmx2                       : ok=1    changed=0    unreachable=0    failed=1   
"
  succeeded: false

[arm4b]

As said in Slack, the first step is to install missing pip modules (jnpr.junos) in Ansible pack virtualenv and prepare environment. That's because StackStorm packs operate in their own isolated Python virtualenv.

Here is my start:

# install pip dependency
sudo /opt/stackstorm/virtualenvs/ansible/bin/pip install junos-eznc

# install custom Ansible module
st2 run ansible.galaxy.install roles=Juniper.junos

After that I'd suggest to debug your specific failed task with Ansible ad-hoc command manually. StackStorm ansible pack is just an abstraction around the Ansible binaries like ansible, ansible-playbook, ansible-galaxy.
For example this failed for me:

/opt/stackstorm/virtualenvs/ansible/bin/ansible all -i 'localhost,' -c local -vvvv --module-name=junos_get_facts

No config file found; using defaults
Set default localhost to localhost
Loading callback plugin minimal of type stdout, v2.0 from /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc
META: ran handlers
The full traceback is:
Traceback (most recent call last):
  File "/opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 125, in run
    res = self._execute()
  File "/opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 521, in _execute
    result = self._handler.run(task_vars=variables)
  File "/opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible/plugins/action/junos.py", line 50, in run
    module = module_loader._load_module_source(self._task.action, module_loader.find_plugin(self._task.action))
  File "/opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible/plugins/__init__.py", line 337, in _load_module_source
    with open(path, 'rb') as module_file:
TypeError: coercing to Unicode: need string or buffer, NoneType found

localhost | FAILED! => {
    "failed": true, 
    "msg": "Unexpected failure during module execution.", 
    "stdout": ""
}

From the above error looks like it couldn't find path to the module.
My second try with --module-path=/etc/ansible/roles/Juniper.junos/library:

/opt/stackstorm/virtualenvs/ansible/bin/ansible all -i 'localhost,' -c local -vvvv --module-name=junos_get_facts --module-path=/etc/ansible/roles/Juniper.junos/library

No config file found; using defaults
Set default localhost to localhost
Loading callback plugin minimal of type stdout, v2.0 from /opt/stackstorm/virtualenvs/ansible/lib/python2.7/site-packages/ansible/plugins/callback/__init__.pyc
META: ran handlers
Using module file /etc/ansible/roles/Juniper.junos/library/junos_get_facts
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: ubuntu
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/ubuntu/.ansible/tmp/ansible-tmp-1494175104.31-241507766167253 `" && echo ansible-tmp-1494175104.31-241507766167253="` echo /home/ubuntu/.ansible/tmp/ansible-tmp-1494175104.31-241507766167253 `" ) && sleep 0'
<localhost> PUT /tmp/tmpezir6u TO /home/ubuntu/.ansible/tmp/ansible-tmp-1494175104.31-241507766167253/junos_get_facts
<localhost> EXEC /bin/sh -c 'chmod u+x /home/ubuntu/.ansible/tmp/ansible-tmp-1494175104.31-241507766167253/ /home/ubuntu/.ansible/tmp/ansible-tmp-1494175104.31-241507766167253/junos_get_facts && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/ubuntu/.ansible/tmp/ansible-tmp-1494175104.31-241507766167253/junos_get_facts; rm -rf "/home/ubuntu/.ansible/tmp/ansible-tmp-1494175104.31-241507766167253/" > /dev/null 2>&1 && sleep 0'
localhost | FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "port": 830, 
            "user": "ubuntu"
        }
    }, 
    "msg": "missing required arguments: host"
}

^^ that looks like success. In my example adding option in ansible.cfg or using --module-path= equivalent in Ansile pack so it could find the path to custom Ansible modules should work.

So like in my case, you probably need a bit deeper debugging.
Hope that helps. At least try the logic above and provide logs what you see.

[cognifloyd]

It sounds like we need per-pack virtualenv_opts. Does such a thing exist?

/etc/st2/st2.conf has:

[actionrunner]
virtualenv_opts = --always-copy

and st2.conf.sample shows:

virtualenv_opts = --system-site-packages

That would allow ansible to access modules installed in the system, but might have unintended reprecussions for other packs. It would be excellent if the ansible pack could say "I need --enable-site-packages"

[cognifloyd]

As a wrapper around the CLI, the ansible pack shouldn't really care which python or virtualenv is used to run ansible. What if, instead of installing ansible in the pack virtualenv, we used the system's ansible binary. A pre-requisite, then, would be to install ansible on the stackstorm system. Then, whoever is using a playbook that requires additional python packages, or other software, can just install them on the system.

[cognifloyd]

Other possible ways to make the ansible pack behave better in the face of additional dependencies:

Add an action paramater to specify which ansible to use (system, or some path, other than the one in the pack):

why not make it an optional parameter on the action?
use_system_ansible or maybe ansible_path
then you get best of both worlds
you get the "just works" functionality out of the box, then the option to customize/extend later
-- @nmaludy in slack

Or add pack configuration to install additional python packages in the pack virtualenv:

you could also maybe list the additional python packages you need in the pack's config and have a "setup" action that initializes the ansible pack's virtualenv with the additional things defined in the config
-- @nmaludy in slack