Python 3 upgrade (#3522)

* Initial Python 3 upgrade effort

* Fixes towards python3 support

 * update heronpy release scripts for python3
 * update dist Dockerfiles to only use python3
 * remove python2 from docker images
 * upgrade pylint for python3 support
 * upgrade PEX so transative dependencies are captured

Additionally:
 * fix Ubuntu 16.04 images
 * fix linting issues found by newer pylint

There is an issue with encapsulation in the builds where the global python3 environment is used
while PEX installs a nested transitive dependency of pylint: `pylint>astroid>wrapt`. This seems
to be because of logic in its setup.py which can be disabled with `WRAPT_INSTALL_EXTENSIONS=false`

* Fix new pylint issues

* update setuptools

* Make pex_pytest non-zip-safe

* Rough proto_library fix

The issue encountered was protocolbuffers/protobuf#1491 which
may be fixed by a pending PR to protoc, or with a switch to the official protobuf rules
and the import_prefix parameter to proto_library.

* WIP: Fix python3 incompatibilities

 * bytes vs str issues
 * update kazoo
 * order of processes in executor test changed due to dict ordering?
 * some places needed / switched to // - may be more not caught by tests
 * add travis_wait as some stages going over 10 minutes without output in CI

TODO:
 * make sure the kazoo upgrade is correct, it was done only by updating package versoin

* Try fixing build time issue in travis

* Upgrade docker rules

* Upgrade to python3 in CI

* Fix python integration tests

* Fix more bytes vs str errors + update vagrant

* Update Travis to Python3.7 + fix Vagrant on mac

* Reduce requirement to python3.6 + py3 fixes

 * use universal_newline in popen instead of text in Popen for py3.6
 * fix bytes/str issues in deserialisation
 * fix file open modes
 * use set instead of sets.Set
 * fix __import__(level) default

* Update cloudpickle

* Fix python addressing of release.yaml

* Additions to get docker image builds working and tested

 * use new external pkg_* rules
 * add python to compile docker images until pkg_*
 * add --host_force_python=PY3 to other bazel.rc files

* WIP: Add CI for docker images/releases

 * use kind to create ephemeral clusters
 * start consolidating scripts with python

* Fix helm chart

* bytes vs str fix

* Mention Python 3.6 requirement in README.md

* updatedockerfile

Co-authored-by: Neng Lu <[email protected]>
Co-authored-by: Nicholas Nezis <[email protected]>
Co-authored-by: bed debug <[email protected]>
Co-authored-by: huijunwu <[email protected]>

Author: 4 people

.travis.yml

@@ -16,9 +16,10 @@ addons:
     packages:
       - libtool-bin
       - libcppunit-dev
+      - python3
       - pkg-config
-      - python-dev
-      - python-wheel
+      - python3-dev
+      - python3-wheel
       - wget
       - zip
       - zlib1g-dev
@@ -34,13 +35,17 @@ before_install:
   - chmod +x bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh
   - ./bazel-${BAZEL_VERSION}-installer-linux-x86_64.sh --user
 
+install:
+  - sudo apt-get install python3-pip python3-setuptools
+  - pip3 install travis-wait-improved
+
 script:
   - which gcc
   - gcc --version
   - which g++
   - g++ --version
   - which python
   - python -V
-  - which python2.7
-  - python2.7 -V
-  - scripts/travis/ci.sh
+  - which python3
+  - python3 -V
+  - travis-wait-improved --timeout=180m scripts/travis/ci.sh

README.md

@@ -31,7 +31,7 @@ Confluence: https://cwiki.apache.org/confluence/display/HERON
 
 #### Heron Requirements:
  * Java 11
- * Python 2.7
+ * Python 3.6
  * Bazel 3.0.0
 
 ## Contact

WORKSPACE

@@ -171,15 +171,15 @@ py_repositories()
 # pip_repositories()
 
 # for pex repos
-PEX_SRC = "https://pypi.python.org/packages/3a/1d/cd41cd3765b78a4353bbf27d18b099f7afbcd13e7f2dc9520f304ec8981c/pex-1.2.15.tar.gz"
+PEX_WHEEL = "https://pypi.python.org/packages/18/92/99270775cfc5ddb60c19588de1c475f9ff2837a6e0bbd5eaa5286a6a472b/pex-2.1.9-py2.py3-none-any.whl"
 
 PY_WHEEL = "https://pypi.python.org/packages/53/67/9620edf7803ab867b175e4fd23c7b8bd8eba11cb761514dcd2e726ef07da/py-1.4.34-py2.py3-none-any.whl"
 
 PYTEST_WHEEL = "https://pypi.python.org/packages/fd/3e/d326a05d083481746a769fc051ae8d25f574ef140ad4fe7f809a2b63c0f0/pytest-3.1.3-py2.py3-none-any.whl"
 
 REQUESTS_SRC = "https://pypi.python.org/packages/d9/03/155b3e67fe35fe5b6f4227a8d9e96a14fda828b18199800d161bcefc1359/requests-2.12.3.tar.gz"
 
-SETUPTOOLS_SRC = "https://pypi.python.org/packages/68/13/1bfbfbd86560e61fa9803d241084fff41a775bf56ee8b3ad72fc9e550dad/setuptools-31.0.0.tar.gz"
+SETUPTOOLS_WHEEL = "https://pypi.python.org/packages/a0/df/635cdb901ee4a8a42ec68e480c49f85f4c59e8816effbf57d9e6ee8b3588/setuptools-46.1.3-py3-none-any.whl"
 
 VIRTUALENV_SRC = "https://pypi.python.org/packages/d4/0c/9840c08189e030873387a73b90ada981885010dd9aea134d6de30cd24cb8/virtualenv-15.1.0.tar.gz"
 
@@ -210,9 +210,9 @@ http_file(
 
 http_file(
     name = "pex_src",
-    downloaded_file_path = "pex-1.2.15.tar.gz",
-    sha256 = "0147d19123340677b9793b00ec86fe65b6697db3ec99afb796da2300ae5fec14",
-    urls = [PEX_SRC],
+    downloaded_file_path = "pex-2.1.9-py2.py3-none-any.whl",
+    sha256 = "5cad8d960c187541f71682fc938a843ef9092aab46f27b33ace7e570325e2626",
+    urls = [PEX_WHEEL],
 )
 
 http_file(
@@ -223,10 +223,10 @@ http_file(
 )
 
 http_file(
-    name = "setuptools_src",
-    downloaded_file_path = "setuptools-31.0.0.tar.gz",
-    sha256 = "0818cc0de692c3a5c83ca83aa7ec7ba6bc206f278735f1e0267b8d0e095cfe7a",
-    urls = [SETUPTOOLS_SRC],
+    name = "setuptools_wheel",
+    downloaded_file_path = "setuptools-46.1.3-py3-none-any.whl",
+    sha256 = "4fe404eec2738c20ab5841fa2d791902d2a645f32318a7850ef26f8d7215a8ee",
+    urls = [SETUPTOOLS_WHEEL],
 )
 
 http_archive(
@@ -367,11 +367,12 @@ http_archive(
 # end helm
 
 # for docker image building
+DOCKER_RULES_VERSION = "0.14.1"
 http_archive(
     name = "io_bazel_rules_docker",
-    sha256 = "aed1c249d4ec8f703edddf35cbe9dfaca0b5f5ea6e4cd9e83e99f3b0d1136c3d",
-    strip_prefix = "rules_docker-0.7.0",
-    urls = ["https://github.com/bazelbuild/rules_docker/archive/v0.7.0.tar.gz"],
+    sha256 = "dc97fccceacd4c6be14e800b2a00693d5e8d07f69ee187babfd04a80a9f8e250",
+    strip_prefix = "rules_docker-%s" % DOCKER_RULES_VERSION,
+    urls = ["https://github.com/bazelbuild/rules_docker/archive/v%s.tar.gz" % DOCKER_RULES_VERSION],
 )
 
 load(
@@ -381,6 +382,10 @@ load(
 
 container_repositories()
 
+load("@io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")
+
+container_deps()
+
 load(
     "@io_bazel_rules_docker//container:container.bzl",
     "container_pull",
@@ -391,11 +396,19 @@ container_pull(
     digest = "sha256:495800e9eb001dfd2fb41d1941155203bb9be06b716b0f8b1b0133eb12ea813c",
     registry = "index.docker.io",
     repository = "heron/base",
-    tag = "0.4.0",
+    tag = "0.5.0",
 )
 
 # end docker image building
 
+http_archive(
+    name = "rules_pkg",
+    url = "https://github.com/bazelbuild/rules_pkg/releases/download/0.2.5/rules_pkg-0.2.5.tar.gz",
+    sha256 = "352c090cc3d3f9a6b4e676cf42a6047c16824959b438895a76c2989c6d7c246a",
+)
+load("@rules_pkg//:deps.bzl", "rules_pkg_dependencies")
+rules_pkg_dependencies()
+
 # for nomad repear
 http_archive(
     name = "nomad_mac",

bazel_configure.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -413,7 +413,7 @@ def main():
   env_map['AUTOMAKE'] = discover_tool('automake', 'Automake', 'AUTOMAKE', '1.9.6')
   env_map['AUTOCONF'] = discover_tool('autoconf', 'Autoconf', 'AUTOCONF', '2.6.3')
   env_map['MAKE'] = discover_tool('make', 'Make', 'MAKE', '3.81')
-  env_map['PYTHON'] = discover_tool('python', 'Python', 'PYTHON', '2.7')
+  env_map['PYTHON3'] = discover_tool('python3', 'Python3', 'PYTHON3', '3.4')
 
   if platform == 'Darwin':
     env_map['LIBTOOL'] = discover_tool('glibtool', 'Libtool', 'LIBTOOL', '2.4.2')

config/configure.ac

@@ -100,8 +100,8 @@ AM_CONDITIONAL([SYSLINUX], [test x$sys_os = xlinux])
 ACX_PTHREAD
 
 # Check the python version required
-AM_PATH_PYTHON([2.4.3])
-AC_PATH_PROG([PYTHON], [python], [],[])
+AM_PATH_PYTHON([3.4])
+AC_PATH_PROG([PYTHON3], [python3], [],[])
 
 abs_top_builddir=`pwd`
 AC_SUBST(abs_top_builddir)

deploy/kubernetes/minikube/apiserver.yaml

@@ -64,7 +64,7 @@ spec:
       initContainers:
         - name: init-heron-apiserver
           image: apache/bookkeeper:4.7.3
-          command: ['sh', '-c', '/opt/bookkeeper/bin/dlog admin bind -l /ledgers -s zookeeper:2181 -c distributedlog://zookeeper:2181/heron']
+          command: ['sh', '-c', '/opt/bookkeeper/bin/dlog admin bind -l /ledgers -s zookeeper:2181 -c distributedlog://zookeeper:2181/heronbkdl']
       containers:
         - name: heron-apiserver
           image: heron/heron:latest
@@ -79,9 +79,9 @@ spec:
               -D heron.executor.docker.image=heron/heron:latest
               -D heron.class.uploader=org.apache.heron.uploader.dlog.DLUploader
               -D heron.uploader.dlog.topologies.num.replicas=1
-              -D heron.uploader.dlog.topologies.namespace.uri=distributedlog://zookeeper:2181/heron
+              -D heron.uploader.dlog.topologies.namespace.uri=distributedlog://zookeeper:2181/heronbkdl
               -D heron.statefulstorage.classname=org.apache.heron.statefulstorage.dlog.DlogStorage
-              -D heron.statefulstorage.dlog.namespace.uri=distributedlog://zookeeper:2181/heron
+              -D heron.statefulstorage.dlog.namespace.uri=distributedlog://zookeeper:2181/heronbkdl
 
 ---
 apiVersion: v1

deploy/kubernetes/minikube/bookkeeper.yaml

@@ -65,6 +65,11 @@ spec:
           envFrom:
             - configMapRef:
                 name: bookie-config
+          volumeMounts:
+            - name: journal-disk
+              mountPath: /bookkeeper/data/journal
+            - name: ledgers-disk
+              mountPath: /bookkeeper/data/ledgers
       containers:
         - name: bookie
           image: apache/bookkeeper:4.7.3
@@ -91,13 +96,11 @@ spec:
               valueFrom:
                 fieldRef:
                   fieldPath: status.hostIP
-    
           volumeMounts:
             - name: journal-disk
               mountPath: /bookkeeper/data/journal
             - name: ledgers-disk
               mountPath: /bookkeeper/data/ledgers
-
       volumes:
           # Mount local disks
         - name: journal-disk

docker/base/Dockerfile.base.debian9

@@ -19,7 +19,7 @@ FROM openjdk:11-jdk-slim-stretch
 
 RUN apt-get -y update && apt-get -y install \
     netcat-openbsd \
-    python \
+    python3 \
     unzip \
     curl \
     supervisor && \

docker/compile/Dockerfile.centos7

@@ -35,8 +35,8 @@ RUN yum -y install \
       libtool \
       make \
       patch \
-      python-devel \
-      cppunit-devel \
+      python \
+      python3-devel \
       zip \
       unzip \
       wget \

docker/compile/Dockerfile.debian10

@@ -33,9 +33,10 @@ RUN apt-get update && apt-get -y install \
       libcppunit-dev \
       pkg-config \
       python \
-      python-dev \
+      python3 \
+      python3-dev \
       software-properties-common \
-      python-setuptools \
+      python3-setuptools \
       tree \
       zip \
       unzip \

docker/compile/Dockerfile.debian9

@@ -32,9 +32,9 @@ RUN apt-get update && apt-get -y install \
       libtool-bin \
       libcppunit-dev \
       pkg-config \
-      python-dev \
-      python3-dev \
       software-properties-common \
+      python \
+      python3-dev \
       python3-setuptools \
       tree \
       zip \

docker/compile/Dockerfile.ubuntu14.04

@@ -33,7 +33,7 @@ RUN apt-get update && apt-get -y install \
       libssl-dev \
       git \
       libtool \
-      python-dev \
+      python3-dev \
       pkg-config \
       libcppunit-dev \
       zip \

docker/compile/Dockerfile.ubuntu16.04

@@ -30,17 +30,18 @@ RUN apt-get update && apt-get -y install \
       build-essential \
       cmake \
       curl \
-      libssl-dev \
       git \
+      libssl-dev \
       libtool-bin \
-      libunwind8 \
       libunwind-setjmp0-dev \
+      python \
+      python3-dev \
       pkg-config \
-      python-dev \
       libcppunit-dev \
+      software-properties-common \
       tree \
-      zip \
       unzip \
+      zip \
       wget
 
 RUN apt-get update && apt-get -y install \

docker/compile/Dockerfile.ubuntu18.04

@@ -29,7 +29,8 @@ RUN apt-get update && apt-get -y install \
       libunwind8 \
       libcppunit-dev \
       patch \
-      python-dev \
+      python \
+      python3-dev \
       pkg-config \
       wget \
       zip \

docker/compile/Dockerfile.ubuntu20.04

@@ -31,7 +31,8 @@ RUN apt-get update && apt-get -y install \
       libunwind8 \
       libcppunit-dev \
       patch \
-      python-dev \
+      python3-dev \
+      python \
       pkg-config \
       wget \
       zip \

docker/dist/Dockerfile.dist.centos7

@@ -24,7 +24,8 @@ RUN yum -y install epel-release \
     java-11-openjdk-headless \
     supervisor \
     nmap-ncat \
-    python \
+    python3 \
+    python3-setuptools \
     unzip \
     which \
     && yum clean all

docker/dist/Dockerfile.dist.debian10

@@ -21,7 +21,8 @@ RUN apt-get update \
     && apt-get -y install \
     curl \
     netcat-openbsd \
-    python \
+    python3 \
+    python3-dev \
     supervisor \
     unzip \
     && apt-get clean

docker/dist/Dockerfile.dist.debian9

@@ -21,7 +21,8 @@ RUN apt-get -y update \
     && apt-get -y install \
     curl \
     netcat-openbsd \
-    python \
+    python3 \
+    python3-dev \
     supervisor \
     unzip \
     && apt-get clean all \

docker/dist/Dockerfile.dist.ubuntu14.04

@@ -21,7 +21,8 @@ RUN apt-get -y update \
     && apt-get -y install \
     curl \
     netcat-openbsd \
-    python \
+    python3 \
+    python3-distutils \
     software-properties-common \
     supervisor \
     unzip \

docker/dist/Dockerfile.dist.ubuntu16.04

@@ -21,7 +21,8 @@ RUN apt-get -y update \
     && apt-get install -y \
     curl \
     netcat-openbsd \
-    python \
+    python3 \
+    python3-distutils \
     software-properties-common \
     supervisor \
     unzip \

docker/dist/Dockerfile.dist.ubuntu18.04

@@ -22,7 +22,8 @@ RUN apt-get -y update \
     curl \
     netcat-openbsd \
     openjdk-11-jre-headless \
-    python \
+    python3 \
+    python3-distutils \
     supervisor \
     unzip \
     && apt-get clean

docker/dist/Dockerfile.dist.ubuntu20.04

@@ -24,7 +24,8 @@ RUN apt-get -y update \
     curl \
     openjdk-11-jre-headless \
     netcat-openbsd \
-    python \
+    python3 \
+    python3-distutils \
     supervisor \
     unzip \
     && apt-get clean

docker/test/Dockerfile.centos7

@@ -36,7 +36,7 @@ RUN yum -y install \
       cppunit-devel \
       make \
       patch \
-      python-devel \
+      python3-devel \
       python3-devel \
       python3-setuptools \
       zip \

docker/test/Dockerfile.ubuntu18.04

@@ -29,7 +29,6 @@ RUN apt-get update && apt-get -y install \
       libunwind8 \
       libcppunit-dev \
       patch \
-      python-dev \
       python3-dev \
       wget \
       zip \