Local automated upgrade testing #3075

msheiny · 2018-02-27T22:02:13Z

Status

Ready for review

Description of Changes

Changes proposed in this pull request:

Fixes first check-box in #3018

Add two scenarios:

vagrant_packager -- builds two vagrant boxes ready for redistribution
upgrade --- fires up boxes from the first scenario (pulled from s3), and chucks all locally built deps to a local apt server.

Testing

How should the reviewer test this PR?

Caveat - this PR testing will only work under a system with libvirt/kvm. There is another ticket to also add support for virtualbox.

Packager testing:

Run make vagrant-package - be prepared to enter a sudo password about 20 min in .
you'll find the vagrant box files in molecule/vagrant_packager/build/*.box - you can manually import those if you want to mess around with em
Peek at molecule/vagrant_packager/push.yml dont run that now... but glance it over :)

Upgrade testing (local deb packages):

make build-debs - make you sure have a version of debian packages that are higher than 0.6 if the build/ directory. If you don't, running this command should give you 0.7.0~rc1
Run molecule converge -s upgrade to get .0.6 SD servers up. You'll get passed an onion address at the end. Navigate to that in a web-browser and note the version at the bottom.
Run molecule side-effect -s upgrade - Navigate back to the onion address and confirm version has bumped.

Upgrade testing (apt-test proxy):

Run molecule converge -s upgrade to get .0.6 SD servers up.
Run molecule login -s upgrade --host app-staging - To get a shell onto the app server
In the app server, run sudo apt-get update (confirm no ERRORS , you'll see a warning about duplicate repos but thats not an error and is a diff issue)
Run apt-cache policy securedrop-config (same terminal ^^) and you should see all the packages pulled in from above
Now back at the securedrop root dir run QA_APTTEST=yes molecule converge -s upgrade -- --diff -t apt
Follow step 2 -> 4 again. This time you should see versions of securedrop-config pulled in from apt-test (this should have a few versions back and look like below):

vagrant@app-staging:~$ apt-cache policy securedrop-config                                                
securedrop-config:
  Installed: 0.1.0+0.6
  Candidate: 0.1.1+0.7.0~rc3
  Version table:
     0.1.1+0.7.0~rc3 0
        500 https://apt.freedom.press/ trusty/main amd64 Packages
     0.1.1+0.7.0~rc2 0
        500 https://apt.freedom.press/ trusty/main amd64 Packages
     0.1.1+0.7.0~rc1 0
        500 https://apt.freedom.press/ trusty/main amd64 Packages
 *** 0.1.0+0.6 0
        500 https://apt.freedom.press/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     0.1.0+0.5.2 0
        500 https://apt.freedom.press/ trusty/main amd64 Packages
     0.1.0+0.5.1 0
        500 https://apt.freedom.press/ trusty/main amd64 Packages

Deployment

Any special considerations for deployment? Consider both:

None - This only affects developer upgrade testing environment.

Checklist

If you made changes to the app code:

None

If you made changes to the system configuration:

None

If you made changes to documentation:

None

codecov-io · 2018-03-01T04:16:43Z

Codecov Report

Merging #3075 into develop will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #3075   +/-   ##
========================================
  Coverage    85.79%   85.79%           
========================================
  Files           34       34           
  Lines         2154     2154           
  Branches       238      238           
========================================
  Hits          1848     1848           
  Misses         250      250           
  Partials        56       56

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb3772f...4800743. Read the comment docs.

ghost · 2018-03-01T21:31:33Z

In the test instructions there should also be pip install -r securedrop/requirements/develop-requirements.txt to get python-vagrant

ghost · 2018-03-01T21:35:10Z

@msheiny at this stage I'm not doing a proper review, just discovering the great work you did 💯. And dumping a few inconsequential remarks on the way ;-)

Both */library/molecule_vagrant.py are mostly the same and could be unified.

msheiny · 2018-03-01T21:36:36Z

Both */library/molecule_vagrant.py are mostly the same and could be unified.

That is a very good point. I managed to get some of that upstreamed to molecule... I need to do the rest of the tweaks.

ghost · 2018-03-01T21:37:46Z

some of that upstreamed to molecule

Nice ! URL ?

ghost · 2018-03-01T21:39:18Z

molecule/upgrade_test/playbook.yml

+        state: running
+      tags: always
+
+    - name: Ensure tor is running


s/tor is running/supervisor is running/

msheiny · 2018-03-01T21:39:57Z

Nice ! URL ?

ansible/molecule#1134

conorsch · 2018-03-05T19:49:02Z

The upgrade_test scenario works well for me. I've got the 0.6~rc3 debs built locally, and the scenario completes without incident. On the packaging task, however, I get a reliable failure:

    TASK [postfix : Create procmail log file.] *************************************
    fatal: [mon-staging]: FAILED! => {"changed": false, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "gid": 0, "group": "root", "mode": "0644", "msg": "chown failed: failed to look up user ossec", "owner": "root", "path": "/var/log/procmail.log", "size": 0, "state": "file", "uid": 0}

Retrying with 0.5.2 debs, rebuilding now...

conorsch · 2018-03-05T20:44:57Z

Even with 0.5.2 debs, still seeing:

TASK [postfix : Create procmail log file.] *************************************
    fatal: [mon-staging]: FAILED! => {"changed": false, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "gid": 0, "group": "root", "mode": "0644", "msg": "chown failed: failed to look up user ossec", "owner": "root", "path": "/var/log/procmail.log", "size": 0, "state": "file", "uid": 0}

msheiny · 2018-03-06T22:22:57Z

Rebased on develop

msheiny · 2018-03-06T22:28:37Z

Even with 0.5.2 debs, still seeing:

Trying to reproduce now... Just to clarify, this is on the make vagrant_package ? Thats installing straight from 0.5.2 out of the SD apt repo. Hrmmmmm

msheiny · 2018-03-06T22:35:02Z

@conorsch can you get me more data on that error? I'm not able to recreate locally...

msheiny · 2018-03-07T18:47:30Z

@kushaldas can you take a look at molecule/vagrant_packager/package.py here specifically for the mypy syntax I started using?

kushaldas · 2018-03-08T14:16:53Z

molecule/vagrant_packager/package.py

+        subprocess.check_call(sysprep_cmd.split())
+
+    def vagrant_metadata(self, img_location):
+        # type: (str) -> dict


This is actually wrong. We will have to install mypy_extensions and use TypedDict for the return type of the function.

Good to know, @kushaldas. Currently the typelint check isn't running against this file. Even adding it locally to the Makefile target, I don't see any errors reported.

ghost · 2018-03-12T16:31:54Z

Is this what I should do to get packages for both versions under test?

git checkout origin/master
make build-debs # so 0.5.2 packages are in the build directory
git checkout origin/develop
make build-debs # so 0.6~rc2 packages are in the build directory

msheiny · 2018-03-14T15:18:58Z

Is this what I should do to get packages for both versions under test?

@dachary can you clarify your question ? I don't quite understand. I'm assuming you are referring to the upgrade_test scenario .. in that case you don't really need 0.5.2 packages present - that's not a good upgrade test since the base vagrant image is also SD 0.5.2 . But you got the gist of the testing strategy to get different packages in place.

ghost · 2018-03-14T16:20:09Z

can you clarify your question ?

I'm confused about which version I should run make build-debs for. And also how I specify the origin version of the upgrade and the destination version of the upgrade?

conorsch

Took another pass through. We need yet another rebase here, @msheiny, so if you'd be so kind....

Left some in-line comments requesting clarification and some tidying up of the names. Still struggling to run the package scenario; the upgrade scenario is comparatively much more reliable for me.

conorsch · 2018-03-29T00:15:11Z

Makefile

@@ -138,6 +138,15 @@ libvirt-share: ## Configure ACLs to allow RWX for libvirt VM (e.g. Admin Worksta
 	@find "$(PWD)" -type d -and -user $$USER -exec setfacl -m u:libvirt-qemu:rwx {} +
 	@find "$(PWD)" -type f -and -user $$USER -exec setfacl -m u:libvirt-qemu:rw {} +

+.PHONY: vagrant_package
+vagrant_package: ## Package up a vagrant box of the last stable SD release


Nit: can we call this vagrant-package? It's more consistent with the existing names for makefile targets:

$ make help | perl -lanE 'print $F[0]' | grep '[-_]' -o | sort | uniq -c 1 _ 19 -

I'll ask for similar stylistic changes throughout.

conorsch · 2018-03-29T00:18:50Z

devops/scripts/vagrant_package.sh

+# Unfortunately since we need to prompt the user for sudo creds..
+# I had to break the actual vagrant package logic outside of molecule
+molecule/vagrant_packager/package.py && \
+molecule destroy -s vagrant_packager


Ha, I ran into a similar problem over in freedomofpress/ansible-role-grsecurity-build#31, with vars_prompt failing to run in non-interactive mode. I wouldn't hesitate to put logic this simple directly in the Makefile, but no objections here.

conorsch · 2018-03-29T00:21:17Z

molecule/vagrant_packager/box_files/app_metadata.json

+      ]
+    }
+  ]
+}


We should rebuild for 0.6 final now that it's released, no?

yeppp going to have to rebuild for that

conorsch · 2018-03-29T00:23:38Z

molecule/vagrant_packager/create.yml

+        #               https://github.com/ansible/ansible/issues/20885
+        content: "{{ instance_conf | to_json | from_json | molecule_to_yaml | molecule_header }}"
+        dest: "{{ molecule_instance_config }}"
+      when: server.changed | bool


Comment indicates the workaround is necessary for Ansible v2.2, but we're on v2.4.2.0 now. Still necessary?

that code is part of the skeleton from molecule ... theres a lot of refactoring that can be done upstream to be honest in the skeleton files. ive been too lazy to make a PR against it for that though ... so we can only blame this @msheiny guy

conorsch · 2018-03-29T00:26:02Z

molecule/vagrant_packager/library/molecule_vagrant.py

+#  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+#  FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+#  DEALINGS IN THE SOFTWARE.
+


This is a gigantic file, and it was introduced as part of a beefy commit with no explanation about why it's here:

$ git show 542c2cbd2e6d27e0bcd0f58cb472cb7dda9ce3f1 --name-status commit 542c2cbd2e6d27e0bcd0f58cb472cb7dda9ce3f1 Author: Michael Sheinberg <[email protected]> Date: Mon Feb 26 16:42:28 2018 -0500 Add scenario for vagrant builder M .gitignore A molecule/vagrant_packager/ansible-override-vars.yml A molecule/vagrant_packager/box_files/Vagrantfile.app-staging A molecule/vagrant_packager/box_files/Vagrantfile.mon-staging A molecule/vagrant_packager/box_files/vagrant_app-staging.yml A molecule/vagrant_packager/box_files/vagrant_mon-staging.yml A molecule/vagrant_packager/build/.gitignore A molecule/vagrant_packager/create.yml A molecule/vagrant_packager/destroy.yml A molecule/vagrant_packager/library/molecule_vagrant.py A molecule/vagrant_packager/molecule.yml A molecule/vagrant_packager/package.py A molecule/vagrant_packager/playbook.yml A molecule/vagrant_packager/prepare.yml A molecule/vagrant_packager/reboot_and_wait.yml A molecule/vagrant_packager/sd_clone.yml A molecule/vagrant_packager/side_effect.yml

I assume it contains a patch of some kind—perhaps already included in upstream?—but at the very least the commit message should explain the motivation to include, and ideally also parameters that would suffice for removal later.

got an incoming commit to address this -- you are right i should have been more explicit though about why it was needed. long story short, upstream was missing functionality so i was waiting for my changes and others to get pulled in

conorsch · 2018-03-29T00:30:11Z

molecule/vagrant_packager/package.py

+        subprocess.check_call(sysprep_cmd.split())
+
+    def vagrant_metadata(self, img_location):
+        # type: (str) -> dict


Good to know, @kushaldas. Currently the typelint check isn't running against this file. Even adding it locally to the Makefile target, I don't see any errors reported.

conorsch · 2018-03-29T00:34:09Z

Makefile

+.PHONY: translate
+translate: ## Update POT translation files from sources
+	@cd securedrop ; ./manage.py translate-messages --extract-update
+	@cd securedrop ; ./manage.py translate-desktop --extract-update


Nit: should probably use && rather than ; here.

ummmmm i dont remember touching this section 😕 maybe it was a merge-conflict that i fixed while rebasing

Good read, thanks for clarifying. Looks like the translate Makefile target was introduced in #2347, but removed in #3087. The translate target is currently absent from develop, and should remain so after merge of this PR. Can you amend, @msheiny?

conorsch · 2018-03-29T00:39:31Z

molecule/upgrade_test/library/molecule_vagrant.py

+#  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+#  FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+#  DEALINGS IN THE SOFTWARE.
+


This PR introduces two nearly identical libs with no explanation. Run diff -y --suppress-common-lines molecule/vagrant_packager/library/molecule_vagrant.py molecule/upgrade_test/library/molecule_vagrant.py for a comparison.

Can we remove one of them?

Can we remove both of them?

If not, can we get comments explaining the necessity of this duplication?

conorsch · 2018-03-29T00:41:05Z

molecule/upgrade_test/molecule.yml

+    side_effect: side_effect.yml
+
+scenario:
+  name: upgrade_test


Nit: can we call this scenario "upgrade"? It'd be more consistent with the naming strategy implicit in Molecule scenarios; e.g. I want to be able to run molecule test -s upgrade, rather than molecule test -s upgrade_test.

conorsch · 2018-03-29T00:46:37Z

Trying to reproduce now... Just to clarify, this is on the make vagrant_package ?

Yes, @msheiny—and I'm still able to reproduce:

TASK [postfix : Create procmail log file.] *************************************
fatal: [mon-staging]: FAILED! => {"changed": false, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "gid": 0, "group": "root", "mode": "0644", "msg": "chown failed: failed to look up user ossec", "owner": "root", "path": "/var/log/procmail.log", "size": 0, "state": "file", "uid": 0}

This is super annoying that I have to keep messing with this option. I suspect its a weird KVM + vagrant issue 🤷 Don't have time to dig deeper and this fixes it along with my fork of upstream molecule vagrant module. Both changes of which, I have open PRs to try to get merged in so we dont have to maintain this divergence.

Was only set to 30 days. Had to recreate and resign fake apt.freedom.press test cert.

Our unique changes that were added to support suppressing NFS were recently merged upstream in molecule 2.13: * ansible/molecule#1235 * ansible/molecule#1233

msheiny · 2018-05-07T21:20:31Z

heyyyyy @conorsch this PR is getting really huge and unwieldy ... everytime I rebase I'm hitting more weird conflicts that I then have to go in and fix ... discovering some really weird issues that are out of scope for this PR (see this last commit 8a298dd for example.. no idea why that was only coming up now for files I didnt touch).

Sorry to nudge ya here but can you give another go through when you have some downtime? Keep in mind this PR really only affects CI and a local dev environment edge-case for specific Linux users so I kindly ask that you keep review pegged to the the upgrade scenario and if thats good we can try to solve the issues you were seeing in your environment in another PR related to the vagrant_packager?

The sooner I can get this in I can also start working on the server-side story to try and get this running in CI ❤️

conorsch · 2018-05-08T17:12:27Z

Taking another look, @msheiny. Might ping you for some real-time collab if I run into any snags during re-review.

msheiny · 2018-05-08T20:38:49Z

ping @conorsch i slightly updated the testing steps here. There is another scenario to test proxy thru to apt-test

conorsch · 2018-05-08T22:25:45Z

Ran through the entire workflow again today. The upgrade flow is quite sound. Built local debs from the release/0.7 branch, then provisioned the base machines for upgrade. Confirmed the VMs were running 0.6, as expected from the prod repo. After applying the side-effect, the machines were running 0.7.0~rc3, as expected. So far so good!

Still could not run the make vagrant-package target without error. First, I had to apply this diff for the scenario to run successfully:

diff --git a/molecule/vagrant_packager/molecule.yml b/molecule/vagrant_packager/molecule.yml
index 21f530716..f1ece2ec0 100644
--- a/molecule/vagrant_packager/molecule.yml
+++ b/molecule/vagrant_packager/molecule.yml
@@ -39,8 +39,8 @@ provisioner:
     name: ansible-lint
   inventory:
     links:
-      group_vars: ../../../install_files/ansible-base/group_vars
-      host_vars: ../../../install_files/ansible-base/host_vars
+      group_vars: ../../install_files/ansible-base/group_vars
+      host_vars: ../../install_files/ansible-base/host_vars
   env:
     ANSIBLE_ROLES_PATH: ".molecule/sd-orig/install_files/ansible-base/roles"
   options:

After doing so, encountered the same error, reported above during prior reviews:

TASK [postfix : Configure Postfix service.] ************************************
    changed: [mon-staging]
    
    TASK [postfix : Create procmail log file.] *************************************
    fatal: [mon-staging]: FAILED! => {"changed": false, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "gid": 0, "group": "root", "mode": "0644", "msg": "chown failed: failed to look up user ossec", "owner": "root", "path": "/var/log/procmail.log", "size": 0, "state": "file", "uid": 0}
    
    NO MORE HOSTS LEFT *************************************************************
    
    NO MORE HOSTS LEFT *************************************************************
    
    PLAY RECAP *********************************************************************
    app-staging                : ok=64   changed=38   unreachable=0    failed=0
    mon-staging                : ok=70   changed=43   unreachable=0    failed=1

@msheiny, let's sync on the error above tomorrow in real time to debug. The effort should be timeboxed, though: if we can't solve it to our mutual satisfaction within an hour, I propose we separate out the package logic and work on it separately. The upgrade logic is highly valuable, and should be merged soon to unblock others on the QA workflow. The package logic is less critical, given that it need only be run once per release.

msheiny · 2018-05-09T15:12:33Z

I propose we separate out the package logic and work on it separately

I really don't want to do this. The package logic only affects you and me at the moment. No one else is going to be running it (not even CI). This is very similar to the condition when we initially merged the docker logic when it was undocumented and had failing tests.

Sometimes its easier to keep merging undocumented and isolated components early as possible and continue to iterate and improve. I believe this is one of those scenarios. It's been really painful to continue to rebase this PR. I keep finding unrelated bugs and having to fix them and it has really snow-balled into a huge ordeal. I'd strongly prefer to merge as is, assuming you do not find any issues with the upgrade testing component.

Seems that with the latest molecule bump, the relative pathing calculation has changed in respect to the ansible host/group vars directories. Previously it was checking from the ephemeral dir, now its from the scenario dir.

Not sure why this wasnt needed before :| This will soon be obsolete as soon as we are testing against post 0.7 (which includes the ssh over local net features).

Especially want to make sure we avoid vagrant images and additional build folders

Also added `.python3` temp folder to the ignore list

These are out of scope on this PR for me to address and for the sake of brevity I'm going to exclude them for now since they are not recent changes.

This applies strictly to the upgrade test scenario, if you have an environment variable `QA_APTTEST` set to yes/true then your upgrade test will be getting packages from apt-test instead of from local apt packages.

In the bump of molecule we also brought up the version of testinfra which introduced a bug in detecting open udp ports. They switched to using `ss` instead of `netstat` when its available. I took the path of least resistance, opened a bug report, and made a hacky work-around using `lsof`. Bug report -> pytest-dev/pytest-testinfra#311

msheiny · 2018-05-09T15:29:41Z

First, I had to apply this diff for the scenario to run successfully:

Rebased and added a fix for that issue you saw ;)

With the bump to molecule, the ephemeral directory spot has changed. For now, lets utilize the old directory space since its already git/docker ignored. We just need to ensure that we intentionally create that directory. There was also a change in how the vagrant boxes are named. The `.molecule` prefix is gone and replaced with the scenario name.

Without this SSHd will not be listening on all interfaces upon reboot

conorsch · 2018-05-10T00:44:31Z

Spent some time with the one-and-only @msheiny debugging the vagrant-package workflow today. The failure related to chowning to the ossec user, reported several times above, turned out to be caused by the entire ossec role being skipped. Was not able to reproduce the problem on a fresh clone of the repo, which indicates fairly clearly it's a problem with my local env. Perhaps adding a clean action as proposed in #2395 would help.

Once we clarified that issue, I was able to confirm working end-to-end scenario flows for both make vagrant-package and molecule converge -s upgrade. There's still a substantial shortcoming in that none of these additions are documented, but I'm willing to follow up on that work as we knock out the related issues tracked in #3018.

Thanks for your patient assistance on this, @msheiny. Let's get it in!

msheiny added the PR: Pending additional work label Feb 27, 2018

msheiny requested a review from conorsch as a code owner February 27, 2018 22:02

msheiny requested a review from a user February 27, 2018 22:02

msheiny requested review from heartsucker and redshiftzero as code owners February 27, 2018 22:02

msheiny changed the title ~~Local automated build~~ Local automated upgrade testing Feb 27, 2018

msheiny removed the PR: Pending additional work label Mar 1, 2018

ghost added feature development env goals: improve developer workflow labels Mar 1, 2018

ghost reviewed Mar 1, 2018

View reviewed changes

molecule/upgrade_test/playbook.yml Outdated

state: running

tags: always

- name: Ensure tor is running

Copy link

ghost Mar 1, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/tor is running/supervisor is running/

msheiny force-pushed the LocalAutomatedBuild branch from ca952d6 to f250640 Compare March 6, 2018 22:15

kushaldas reviewed Mar 8, 2018

View reviewed changes

conorsch suggested changes Mar 29, 2018

View reviewed changes

msheiny added 4 commits May 7, 2018 10:59

Whoopss - Bump expiration for testing CA

3e63b64

Was only set to 30 days. Had to recreate and resign fake apt.freedom.press test cert.

Symlink vagrant molecule into upgrade scenario

eb43e00

Make upgrade side-effect explicitly upgrade deps

f75c9cd

msheiny force-pushed the LocalAutomatedBuild branch from 37edb45 to 472aa4c Compare May 7, 2018 15:05

Remove fork of molecule_vagrant ansible library

180826c

Our unique changes that were added to support suppressing NFS were recently merged upstream in molecule 2.13: * ansible/molecule#1235 * ansible/molecule#1233

msheiny force-pushed the LocalAutomatedBuild branch 2 times, most recently from 0ab9386 to 8a298dd Compare May 7, 2018 21:14

msheiny added 7 commits May 9, 2018 11:28

Address pathing issue with latest molecule update

a013de4

Seems that with the latest molecule bump, the relative pathing calculation has changed in respect to the ansible host/group vars directories. Previously it was checking from the ephemeral dir, now its from the scenario dir.

Ensure pre 0.7 SD, SSH works localnet after reboot

e9c95f4

Not sure why this wasnt needed before :| This will soon be obsolete as soon as we are testing against post 0.7 (which includes the ssh over local net features).

Add additional ignore items to docker build

5bd3170

Especially want to make sure we avoid vagrant images and additional build folders

Move flake8 configuration to separate file

a3fdcab

Also added `.python3` temp folder to the ignore list

Exclude flake8 NITs

e2e3650

These are out of scope on this PR for me to address and for the sake of brevity I'm going to exclude them for now since they are not recent changes.

Optional - apt as a reverse-proxy to apt-test

094dd0f

This applies strictly to the upgrade test scenario, if you have an environment variable `QA_APTTEST` set to yes/true then your upgrade test will be getting packages from apt-test instead of from local apt packages.

msheiny force-pushed the LocalAutomatedBuild branch from 3f7eb02 to aa396fb Compare May 9, 2018 15:29

msheiny added 2 commits May 9, 2018 15:23

Fix timeout on SD 0.6 vagrant_packager on reboot

4800743

Without this SSHd will not be listening on all interfaces upon reboot

msheiny force-pushed the LocalAutomatedBuild branch from fd03c60 to 4800743 Compare May 9, 2018 19:25

conorsch approved these changes May 10, 2018

View reviewed changes

msheiny merged commit b1ead0d into develop May 10, 2018

msheiny deleted the LocalAutomatedBuild branch May 10, 2018 13:25

conorsch mentioned this pull request Jul 30, 2018

Document the local upgrade testing scenario #3669

Closed

Local automated upgrade testing #3075

Local automated upgrade testing #3075

Uh oh!

Conversation

msheiny commented Feb 27, 2018 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Status

Description of Changes

Testing

Deployment

Checklist

If you made changes to the app code:

If you made changes to the system configuration:

If you made changes to documentation:

Uh oh!

codecov-io commented Mar 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ghost commented Mar 1, 2018

Uh oh!

ghost commented Mar 1, 2018

Uh oh!

msheiny commented Mar 1, 2018

Uh oh!

ghost commented Mar 1, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msheiny commented Mar 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

conorsch commented Mar 5, 2018

Uh oh!

conorsch commented Mar 5, 2018

Uh oh!

msheiny commented Mar 6, 2018

Uh oh!

msheiny commented Mar 6, 2018

Uh oh!

msheiny commented Mar 6, 2018

Uh oh!

msheiny commented Mar 7, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ghost commented Mar 12, 2018

Uh oh!

msheiny commented Mar 14, 2018

Uh oh!

ghost commented Mar 14, 2018

Uh oh!

conorsch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

msheiny commented Feb 27, 2018 •

edited by ghost

Loading

codecov-io commented Mar 1, 2018 •

edited

Loading

msheiny commented Mar 1, 2018 •

edited

Loading

conorsch commented May 10, 2018 •

edited

Loading