all: change package manager from dep to go mod (#1542)

Converts the mono repo from using the `dep` package manager to using the [go mod](https://github.com/golang/go/wiki/Modules) that has become the standard dependency manager in recent releases of Go.

Make the monorepo more accessible to Go developers using standard Go tooling. Modules will be enabled by default in Go 1.13, and were enabled by default when not using a GOPATH in Go 1.11+.

Summary of changes:
* [x] Add a `go.mod` file that contains the same dependencies and dependency versions/hashes/sources.
* [x] Add a `go.sum` which is the smallest set of hashes required to validate that the dependencies at those versions remain consistent.
* [x] Add a `go.list` which is a way for us to visibly see when our dependencies change. Note: This is not a standard file, but is something we can use to validate reproducibility until the `go mod` commands provide something similar.
* [x] Update CircleCI and TravisCI builds so that they succeed
* [x] Update CircleCI and TravisCI builds so that they fail if any of the above `go.*` files are out of date.
* [x] Update documentation within the repository
* [x] Test that other projects can import this repo. (This will be much easier, unfortunately, to test after merging, but I am currently attempting to test this before merging.)
* [x] Remove the existing `dep` files, `Gopkg.toml` and `Gopkg.lock`. (This will occur just before merging.)

Known limitations & issues:
This change bumps our minimum supported version of Go to 1.11.4. Versions prior to that version contained a bug when Modules are in use that prevented the importing of some packages. There are more details about the issue here: golang/go#30446 (comment).

All changes:
* all: update ci config to use modules instead of dep
* all: go mod init
* all: go mod tidy -v
* all: go get github.com/gobuffalo/[email protected]
* all: go get github.com/spf13/cobra@9c28e4bbd74e
* all: go mod tidy -v
* all: go get github.com/BurntSushi/toml@99064174e013
* all: go get github.com/spf13/viper@db7ff930a189, go mod tidy
* all: go get github.com/mitchellh/mapstructure@2caf8efc9366, go mod tidy
* all: go get github.com/kr/pretty@e6ac2fc51e89, go mod tidy
* all: go get github.com/kr/text@e373e137fafd, go mod tidy
* all: go get github.com/sirupsen/logrus@070c81def33f, go mod tidy
* all: go get github.com/spf13/pflag@4bd69631f475, go mod tidy
* all: go get github.com/spf13/jwalterweatherman@3d60171a6431, go mod tidy
* all: go get github.com/magiconair/[email protected], go mod tidy
* all: go get github.com/spf13/cast@4d07383ffe94, go mod tidy
* all: go get github.com/davecgh/go-spew@5215b55f46b2, go mod tidy
* all: go get github.com/stretchr/testify@976c720a22c8, go mod tidy
* all: go get github.com/stretchr/objx@1a9d0bb9f541, go mod tidy
* all: commit go.sum file
* all: commit go.list file generated with go list -m all > go.list
* all: download dependencies in an earlier step
* all: change the install process for staticcheck to leverage modules
* all: add check of dependencies
* all: add checks and tests running with go1.13rc1
* all: allow install of a newer versions of some packages for buster images
* all: do a check of deps before building for publishing
* doc: update readmes with new building instructions
* all: update travis.yml with modules and go1.13rc1
* all: update dockerfiles with modules
* all: set go.mod's go directive to go 1.11
* all: run travis builds on the latest minor release of each
* doc: increase min version of Go to 1.11.4
* all: cache go mod download
* all: use new git repo for gocheck
* all: skip go.sum checks for go 1.11.* because it is inconsistent with other versions of go
* all: remove reference to dep in dockerfile
* all: remove reference to dep in horizon dockerfile
* all: remove curl in horizon dockerfile because was only needed for dep
* doc: remove mention of go-dep
* doc: remove mention of dep in developing documentation
* doc: update min Go required to go1.11.4
* doc: improve developer building instructions
* all: install postgres-client-9.6 via postgres apt
* all: go get github.com/rcrowley/go-metrics@51425a2415d2, go mod tidy
* all: remove duplicate steps from poor copy and paste
* all: go get golang.org/x/net@ba9fcec4b297b415637633c5a6e8fa592e4a16c3, go mod tidy
* all: use a cache key that references go.mod too
* all: remove dep files: Gopkg.*

Author: leighmcculloch

.circleci/config.yml

@@ -64,20 +64,25 @@ commands:
   install_go_deps:
     steps:
       - checkout
-      - run:
-          name: Install dep
-          command: curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
       - restore_cache:
           keys:
-            - v1-pkg-cache
+            - go-mod-v1-{{ checksum "go.mod" }}-{{ checksum "go.sum" }}
       - run:
-          name: Install project dependencies using dep
-          command: dep ensure -v
+          name: Download dependencies
+          command: |
+            go mod download
       - save_cache:
-          key: v1-pkg-cache
+          key: go-mod-v1-{{ checksum "go.mod" }}-{{ checksum "go.sum" }}
           paths:
-            - "/go/src/github.com/stellar/go/vendor"
-            - "/go/src/pkg/dep"
+            - /go/pkg/mod
+
+  # check_go_deps validates that the dependencies are expected.
+  check_go_deps:
+    steps:
+      - checkout
+      - run:
+          name: Check dependencies
+          command: ./gomod.sh
 
   check_ingest_state:
     steps:
@@ -108,7 +113,17 @@ commands:
     steps:
       - run:
           name: Install dependencies
-          command: sudo apt-get update && sudo apt-get install postgresql-client-9.6 mariadb-client-10.1
+          command: |
+            sudo apt-get update
+            (
+              sudo apt-get install postgresql-client-9.6 || (
+                echo "deb http://apt.postgresql.org/pub/repos/apt/ $(env -i bash -c '. /etc/os-release; echo $VERSION_CODENAME')-pgdg main" | sudo tee /etc/apt/sources.list.d/pgdg.list &&
+                wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - &&
+                sudo apt-get update &&
+                sudo apt-get install postgresql-client-9.6
+              )
+            )
+            (sudo apt-get install mariadb-client-10.1 || sudo apt-get install mariadb-client-10.3)
       - run:
           name: Install dockerize
           command: wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && sudo tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz
@@ -149,16 +164,35 @@ jobs:
     working_directory: /go/src/github.com/stellar/go
     docker:
       - image: circleci/golang:1.11-stretch
+        environment:
+          GO111MODULE: "on"
     steps:
+      - install_go_deps
+      - check_go_deps
       - govet
 
   # check_code_1_12 performs code checks using Go 1.12.
   check_code_1_12:
     working_directory: /go/src/github.com/stellar/go
     docker:
       - image: circleci/golang:1.12-stretch
+        environment:
+          GO111MODULE: "on"
+    steps:
+      - install_go_deps
+      - check_go_deps
+      - gofmt
+      - govet
+      - staticcheck
+
+  # check_code_1_13 performs code checks using Go 1.13.
+  check_code_1_13:
+    working_directory: /go/src/github.com/stellar/go
+    docker:
+      - image: circleci/golang:1.13rc1-buster
     steps:
       - install_go_deps
+      - check_go_deps
       - gofmt
       - govet
       - staticcheck
@@ -169,6 +203,7 @@ jobs:
     docker:
       - image: circleci/golang:1.11-stretch
         environment:
+          GO111MODULE: "on"
           PGHOST: localhost
           PGPORT: 5432
           PGUSER: circleci
@@ -189,6 +224,28 @@ jobs:
     docker:
       - image: circleci/golang:1.12-stretch
         environment:
+          GO111MODULE: "on"
+          PGHOST: localhost
+          PGPORT: 5432
+          PGUSER: circleci
+          MYSQL_HOST: 127.0.0.1
+          MYSQL_PORT: 3306
+      - image: circleci/postgres:9.6.5-alpine-ram
+        environment:
+          POSTGRES_USER: circleci
+      - image: circleci/mysql:5.7
+      - image: circleci/redis:5.0-alpine
+    steps:
+      - install_go_deps
+      - test_packages
+
+  # test_code_1_13 performs all package tests using Go 1.13.
+  test_code_1_13:
+    working_directory: /go/src/github.com/stellar/go
+    docker:
+      - image: circleci/golang:1.13rc1-buster
+        environment:
+          GO111MODULE: "on"
           PGHOST: localhost
           PGPORT: 5432
           PGUSER: circleci
@@ -214,6 +271,7 @@ jobs:
     steps:
       - check_deprecations
       - install_go_deps
+      - check_go_deps
       - build_packages
       - attach_workspace:
           at: ./dist
@@ -257,8 +315,10 @@ workflows:
     jobs:
       - check_code_1_11
       - check_code_1_12
+      - check_code_1_13
       - test_code_1_11
       - test_code_1_12
+      - test_code_1_13
       - ingest_state_diff:
           filters:
               branches:

Dockerfile

@@ -1,8 +1,7 @@
 FROM golang:1.12
-RUN curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
 WORKDIR /go/src/github.com/stellar/go
 
 COPY . .
-RUN dep ensure -v
+ENV GO111MODULE=on
 RUN go install github.com/stellar/go/tools/...
 RUN go install github.com/stellar/go/services/...