Creating clusters in Vagrant with chef-metal yields a Net::SSH::Exception

[feoh]

Others on the chef mailing list have encountered this, and the only work around people have found is a rather complex implementation around using private networks.

Here's my chef-metal script:

require 'chef_metal'
require 'chef_metal_vagrant'

vagrant_cluster '/Users/cpatti/Vagrants'

vagrant_box 'chef/ubuntu-14.04' do
  url 'https://vagrantcloud.com/chef/ubuntu-14.04/version/1/provider/virtualbox.box'
end

machine 'skycrank-gearbox' do
  tag 'master'
  recipe "skycrank-gearbox::default"
  converge true
end

machine 'skycrank01' do
  tag 'gear'
  converge true
end

machine 'skycrank02' do
  tag 'gear'
  converge true
end

and this is the output I get:

cpatti@Michelles-MacBook-Pro: ~/Dropbox/src/chef/skycrank-cluster
$ chef-client -z vagrant_skycrank_cluster.rb                         [12:08:08]
Starting Chef Client, version 11.14.2
resolving cookbooks for run list: []
Synchronizing Cookbooks:
Compiling Cookbooks...
[2014-08-18T12:08:29-04:00] WARN: Node Michelles-MacBook-Pro.local has an empty run list.
Converging 3 resources
Recipe: @recipe_files::/Users/cpatti/Dropbox/src/chef/skycrank-cluster/vagrant_skycrank_cluster.rb
  * vagrant_cluster[/Users/cpatti/Vagrants] action create (up to date)
  * vagrant_box[chef/ubuntu-14.04] action create (up to date)
  * machine_batch[default] action converge
    - [skycrank02] update content in file /Users/cpatti/Vagrants/skycrank02.vm from 57a688 to 89acaf
    - [skycrank02] --- /Users/cpatti/Vagrants/skycrank02.vm 2014-08-17 18:53:59.000000000 -0400
    - [skycrank02] +++ /var/folders/9f/jvlzk0m50yng0jstkzfgf0540000gp/T/.skycrank02.vm20140818-6293-j1wtxu  2014-08-18 12:08:30.000000000 -0400
    - [skycrank01] update content in file /Users/cpatti/Vagrants/skycrank01.vm from fb2fd2 to cd3fd2
    - [skycrank01] --- /Users/cpatti/Vagrants/skycrank01.vm 2014-08-17 18:53:59.000000000 -0400
    - [skycrank01] +++ /var/folders/9f/jvlzk0m50yng0jstkzfgf0540000gp/T/.skycrank01.vm20140818-6293-1dybdo0 2014-08-18 12:08:30.000000000 -0400
    - [skycrank01] @@ -1,6 +1,7 @@
    - [skycrank01]  Vagrant.configure('2') do |outer_config|
    - [skycrank01]    outer_config.vm.define "skycrank01" do |config|
    - [skycrank01] -    config.vm.box = "precise64"
    - [skycrank01] +    config.vm.box = "chef/ubuntu-14.04"
    - [skycrank01] +    config.vm.box_url = "https://vagrantcloud.com/chef/ubuntu-14.04/version/1/provider/virtualbox.box"
    - [skycrank01]      config.vm.hostname = "skycrank01"
    - [skycrank01]    end
    - [skycrank01]  end
    - [skycrank02] @@ -1,6 +1,7 @@
    - [skycrank02]  Vagrant.configure('2') do |outer_config|
    - [skycrank02]    outer_config.vm.define "skycrank02" do |config|
    - [skycrank02] -    config.vm.box = "precise64"
    - [skycrank02] +    config.vm.box = "chef/ubuntu-14.04"
    - [skycrank-gearbox] update content in file /Users/cpatti/Vagrants/skycrank-gearbox.vm from 07b1d9 to ff8749
    - [skycrank-gearbox] --- /Users/cpatti/Vagrants/skycrank-gearbox.vm 2014-08-17 18:53:59.000000000 -0400
    - [skycrank-gearbox] +++ /var/folders/9f/jvlzk0m50yng0jstkzfgf0540000gp/T/.skycrank-gearbox.vm20140818-6293-k4ypza  2014-08-18 12:08:30.000000000 -0400
    - [skycrank-gearbox] @@ -1,6 +1,7 @@
    - [skycrank-gearbox]  Vagrant.configure('2') do |outer_config|
    - [skycrank-gearbox]    outer_config.vm.define "skycrank-gearbox" do |config|
    - [skycrank-gearbox] -    config.vm.box = "precise64"
    - [skycrank-gearbox] +    config.vm.box = "chef/ubuntu-14.04"
    - [skycrank-gearbox] +    config.vm.box_url = "https://vagrantcloud.com/chef/ubuntu-14.04/version/1/provider/virtualbox.box"
    - [skycrank-gearbox]      config.vm.hostname = "skycrank-gearbox"
    - [skycrank-gearbox]    end
    - [skycrank-gearbox]  end
    - [skycrank02] +    config.vm.box_url = "https://vagrantcloud.com/chef/ubuntu-14.04/version/1/provider/virtualbox.box"
    - [skycrank02]      config.vm.hostname = "skycrank02"
    - [skycrank02]    end
    - [skycrank02]  end
    - [skycrank02] update node skycrank02 at http://localhost:8889
    - [skycrank02]   update normal.metal.location.allocated_at from "2014-08-17 22:53:59 UTC" to "2014-08-18 16:08:30 UTC"
    - [skycrank01] update node skycrank01 at http://localhost:8889
    - [skycrank01]   update normal.metal.location.allocated_at from "2014-08-17 22:53:59 UTC" to "2014-08-18 16:08:30 UTC"
    - [skycrank-gearbox] update node skycrank-gearbox at http://localhost:8889
    - [skycrank-gearbox]   update normal.metal.location.allocated_at from "2014-08-17 22:53:59 UTC" to "2014-08-18 16:08:30 UTC"
    - [skycrank-gearbox]   update normal.metal.location.needs_reload from false to true
    - [skycrank-gearbox]   remove normal.metal.location.forwarded_ports
    - [skycrank-gearbox]   update run_list from ["recipe[skycrank-gearbox::default]"] to ["recipe[skycrank-gearbox::default]", "recipe[skycrank-gearbox::default]"]
    - run vagrant reload skycrank-gearbox skycrank01 skycrank02
    - update node skycrank-gearbox at http://localhost:8889
    -   update normal.metal.location.needs_reload from true to false
    -   add normal.metal.location.forwarded_ports = {}
    -   update run_list from ["recipe[skycrank-gearbox::default]"] to ["recipe[skycrank-gearbox::default]", "recipe[skycrank-gearbox::default]"]
    - [skycrank-gearbox] update node skycrank-gearbox at http://localhost:8889
    - [skycrank-gearbox]   update run_list from ["recipe[skycrank-gearbox::default]"] to ["recipe[skycrank-gearbox::default]", "recipe[skycrank-gearbox::default]"]
    - update node skycrank-gearbox at http://localhost:8889
    -   update run_list from ["recipe[skycrank-gearbox::default]"] to ["recipe[skycrank-gearbox::default]", "recipe[skycrank-gearbox::default]"]
    [skycrank-gearbox] [2014-08-18T16:09:52+00:00] WARN:
                       * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
                       SSL validation of HTTPS requests is disabled. HTTPS connections are still
                       encrypted, but chef is not able to detect forged replies or man in the middle
                       attacks.

                       To fix this issue add an entry like this to your configuration file:

                       ```
                         # Verify all HTTPS connections (recommended)
                         ssl_verify_mode :verify_peer

                         # OR, Verify only connections to chef-server
                         verify_api_cert true
                       ```

                       To check your SSL configuration, or troubleshoot errors, you can use the
                       `knife ssl check` command like so:

                       ```
                         knife ssl check -c /etc/chef/client.rb
                       ```

                       * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

                       Starting Chef Client, version 11.14.2
                       resolving cookbooks for run list: ["skycrank-gearbox::default"]
                       Synchronizing Cookbooks:
                         - skycrank-gearbox
                       Compiling Cookbooks...
                       Converging 0 resources

                       Running handlers:
                       Running handlers complete
                       Chef Client finished, 0/0 resources updated in 7.138812233 seconds

    ================================================================================
    Error executing action `converge` on resource 'machine_batch[default]'
    ================================================================================

    Net::SSH::Exception
    -------------------
    remote forwarding request failed

    Resource Declaration:
    ---------------------
    # In /Users/cpatti/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/chef-metal-0.13/lib/chef_metal/recipe_dsl.rb

     82:             batch = build_resource(:machine_batch, machine_batch_default_name) do
     83:               action resource.action
     84:               machines [ previous, resource ]
     85:             end
     86:             batch.from_recipe self

    Compiled Resource:
    ------------------
    # Declared in /Users/cpatti/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/chef-metal-0.13/lib/chef_metal/recipe_dsl.rb:82:in `machine'

    machine_batch("default") do
      action [:converge]
      retries 0
      retry_delay 2
      guard_interpreter :default
      driver "vagrant:/Users/cpatti/Vagrants"
      chef_server {:chef_server_url=>"http://localhost:8889", :options=>{:client_name=>"Michelles-MacBook-Pro.local", :signing_key_filename=>nil}}
      machine_options {:vagrant_options=>{"vm.box"=>"chef/ubuntu-14.04", "vm.box_url"=>"https://vagrantcloud.com/chef/ubuntu-14.04/version/1/provider/virtualbox.box"}}
      cookbook_name "@recipe_files"
      recipe_name "/Users/cpatti/Dropbox/src/chef/skycrank-cluster/vagrant_skycrank_cluster.rb"
      machines ["skycrank-gearbox", "skycrank01", "skycrank02"]
    end


Running handlers:
[2014-08-18T12:10:00-04:00] ERROR: Running exception handlers
Running handlers complete
[2014-08-18T12:10:00-04:00] ERROR: Exception handlers complete
[2014-08-18T12:10:00-04:00] FATAL: Stacktrace dumped to /Users/cpatti/.chef/local-mode-cache/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 92.292032 seconds
[2014-08-18T12:10:00-04:00] ERROR: machine_batch[default] (@recipe_files::/Users/cpatti/Dropbox/src/chef/skycrank-cluster/vagrant_skycrank_cluster.rb line 82) had an error: Net::SSH::Exception: remote forwarding request failed
[2014-08-18T12:10:00-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
FAIL: 1

[lamont-granquist]

I'm also seeing this against digital ocean droplets as well.

[jkeiser]

This is fixed on master; I'll get a release up ASAP (which may take a day or two as I crunch through everything that piled up on vacation.)

[lamont-granquist]

validated that works for me, too.