zookeeper to 3.5.8 #3555

huijunwu · 2020-07-03T03:37:19Z

No description provided.

nicknezis · 2020-07-03T05:45:35Z

We will need to also update the Zookeeper included in the Dockerfiles. Also, do we know if the version of Bookkeeper used in the Helm chart will work with this version of Zookeeper?

joshfischer1108 · 2020-07-03T11:20:20Z

I wonder if Bookkeeper is something we should even keep around to work with Heron. It seems to cause more problems than it's worth in our context.

…or-heron into huijunwu/20200702

huijunwu · 2020-07-03T23:18:26Z

updated dockerfile

nicknezis · 2020-07-04T04:32:15Z

The Helm chart is challenging because Heron and Bookkeeper both use a shared Zookeeper. This makes updates like this challenging. I think it would be preferable we can update the Helm chart to use S3 instead of Bookkeeper for Upload/Download of the analytic artifacts.
Other options would be to update the Helm chart to include another Zookeeper instance dedicated for Bookkeeper. Ideally with the official Bookkeeper Helm chart and not the current copy of yamls we are maintaining.

huijunwu · 2020-07-05T06:19:02Z

Tests with 1) minikube, 2) helm

0] environment:

$ hostnamectl
         Icon name: computer-desktop
           Chassis: desktop
  Operating System: Ubuntu 20.04 LTS
            Kernel: Linux 5.4.0-40-generic
      Architecture: x86-64
$ git status
On branch huijunwu/20200702
$ docker --version
Docker version 19.03.8, build afacb8b7f0

./docker/scripts/build-artifacts.sh ubuntu20.04 0.0.0 ~/heron-release
~/heron-release/heron-install-0.0.0-ubuntu20.04.sh --user

$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
$ minikube version
minikube version: v1.11.0
commit: 57e2f55f47effe9ce396cea42a1e0eb4f611ebbd

eval $(minikube -p minikube docker-env)
./docker/scripts/build-docker.sh ubuntu20.04 0.0.0 ~/heron-release

$ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.13.12"}

kubectl proxy -p 8001 &

1] minikube:

DIR=./deploy/kubernetes/minikube
sed 's!heron/heron:latest!heron/heron:0.0.0!g' ${DIR}/zookeeper.yaml > /tmp/zookeeper.yaml
sed 's!heron/heron:latest!heron/heron:0.0.0!g' ${DIR}/tools.yaml > /tmp/tools.yaml
sed 's!heron/heron:latest!heron/heron:0.0.0!g' ${DIR}/apiserver.yaml > /tmp/apiserver.yaml

kubectl create -f /tmp/zookeeper.yaml
kubectl get pods
kubectl create -f ${DIR}/bookkeeper.yaml
kubectl create -f /tmp/tools.yaml
kubectl create -f /tmp/apiserver.yaml

# submit a job
curl http://localhost:8001/api/v1/namespaces/default/services/heron-apiserver:9000/proxy/api/v1/version

heron config kubernetes set service_url http://localhost:8001/api/v1/namespaces/default/services/heron-apiserver:9000/proxy
heron submit kubernetes ~/.heron/examples/heron-api-examples.jar org.apache.heron.examples.api.AckingTopology acking
heron kill kubernetes acking

# clean
kubectl delete -f /tmp/zookeeper.yaml
kubectl delete -f ${DIR}/bookkeeper.yaml
kubectl delete -f /tmp/tools.yaml
kubectl delete -f /tmp/apiserver.yaml

2] helm(minikube):

bazel build --config=ubuntu scripts/packages:generate-helm-charts

helm install mytest bazel-bin/scripts/packages/heron-0.0.0.tgz --set platform=minikube

# submit a job
curl http://localhost:8001/api/v1/namespaces/default/services/mytest-apiserver:9000/proxy/api/v1/version

heron config mytest set service_url http://localhost:8001/api/v1/namespaces/default/services/mytest-apiserver:9000/proxy
heron submit mytest ~/.heron/examples/heron-api-examples.jar org.apache.heron.examples.api.AckingTopology acking
heron kill mytest acking

# clean
helm uninstall mytest

nicknezis · 2020-07-07T03:11:33Z

@huijunwu I just realized that you still need to run bazel run @unpinned_maven//:pin to update the maven_install.json file.

Maybe we should move the comment listed here to right above the dependencies list.

incubator-heron/WORKSPACE

Line 153 in f7715f3

    
           # https://github.com/bazelbuild/rules_jvm_external#updating-maven_installjson

nicknezis · 2020-07-07T03:13:52Z

WORKSPACE

@@ -66,7 +66,7 @@ maven_install(
    name = "maven",
    artifacts = [
        "antlr:antlr:2.7.7",
-        "org.apache.zookeeper:zookeeper:3.4.14",
+        "org.apache.zookeeper:zookeeper:3.5.8",


The maven_install.json pin file needs to be updated.

We should probably add these instructions to the documentation somewhere.

Yup. And maybe above the list of dependencies instead of below them.

nicknezis · 2020-07-07T03:30:45Z

I'm having issues compiling this on OSX with this command:
bazel build --config=darwin_nostyle heron/...

❯ ./bazel_configure.py
Platform Darwin
Using C compiler          :	/usr/bin/gcc (4.2.1)
Using C++ compiler        :	/usr/bin/g++ (4.2.1)
Using C preprocessor      :	/usr/bin/cpp (11.0.0)
Using C++ preprocessor    :	/usr/bin/cpp (11.0.0)
Using linker              :	/usr/bin/ld
Using JDK                 :	/System/Library/Frameworks/JavaVM.framework/Versions/A
Using Automake            :	/usr/local/Cellar/automake/1.16.2/bin/automake (1.16.2)
Using Autoconf            :	/usr/local/Cellar/autoconf/2.69/bin/autoconf (2.69)
Using Make                :	/usr/bin/make (3.81)
Using Python              :	/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 (2.7.16)
Using Libtool             :	/usr/local/Cellar/libtool/2.4.6_2/bin/glibtool (2.4.6)
Using archiver            :	/usr/bin/ar
Using coverage tool       :	/usr/bin/gcov
dwp                       :	not found, but ok
Using nm                  :	/usr/bin/nm
objcopy                   :	not found, but ok
Using objdump             :	/usr/bin/objdump
Using strip               :	/usr/bin/strip

The error I get:

INFO: From Linking heron/proto/libproto-cxx.a:
/Library/Developer/CommandLineTools/usr/bin/libtool: file: bazel-out/darwin-fastbuild/bin/heron/proto/_objs/proto-cxx/empty.pic.o has no symbols
warning: /Library/Developer/CommandLineTools/usr/bin/libtool: warning for library: bazel-out/darwin-fastbuild/bin/heron/proto/libproto-cxx.a the table of contents is empty (no object file members in the library define global symbols)
ERROR: /private/var/tmp/_bazel_nnezis/42af5204dd5936eb8287d40b2ccac539/external/org_apache_zookeeper/BUILD.bazel:7:8: Executing genrule @org_apache_zookeeper//:zookeeper-srcs failed (Exit 1)
/bin/bash: line 7: ./configure: No such file or directory
INFO: Elapsed time: 80.026s, Critical Path: 28.42s
INFO: 443 processes: 443 local.
FAILED: Build did NOT complete successfully

…to huijunwu/20200702

huijunwu · 2020-07-08T07:57:18Z

tested on darwin

git checkout huijunwu/20200702

python3 -m venv .
source ./bin/activate
$ python -V
Python 3.7.8

# only for occasional scenario: multiple pex rc configs
PY3=$(which python3)
echo "PEX_PYTHON_PATH=$(dirname ${PY3})" > ~/.pexrc
echo "PEX_PYTHON=${PY3}" >> ~/.pexrc
cat ~/.pexrc

$ ./bazel_configure.py
Platform Darwin
Using C compiler          : /usr/bin/clang (11.0.0)
Using C++ compiler        : /usr/bin/clang++ (11.0.0)
Using C preprocessor      : /usr/bin/cpp (11.0.0)
Using C++ preprocessor    : /usr/bin/cpp (11.0.0)
Using linker              : /opt/local/bin/ld-latest
Using JDK                 : /Library/Java/JavaVirtualMachines/...jdk11
Using Automake            : /usr/local/Cellar/automake/1.16.1_1/bin/automake (1.16.1)
Using Autoconf            : /.../autoconf (2.69)
Using Make                : /usr/bin/make (3.81)
Using Python3             : /usr/local/Cellar/python/3.7.8/Frameworks/Python.framework/Versions/3.7/bin/python3.7 (3.7.8)
Using Libtool             : /.../glibtool (2.4.6)
Using archiver            : /opt/local/bin/ar
Using coverage tool       : /usr/bin/gcov
dwp                       : not found, but ok
Using nm                  : /opt/local/bin/llvm-nm-mp-8.0
objcopy                   : not found, but ok
Using objdump             : /usr/bin/objdump
Using strip               : /opt/local/bin/strip

bazel build --config=darwin_nostyle heron/...

bazel build --config=darwin_nostyle scripts/packages:binpkgs && \
bazel build --config=darwin_nostyle scripts/packages:tarpkgs

huijunwu · 2020-07-08T08:01:16Z

tested on centos

sudo yum install -y ant cppunit-devel

python3 -m venv .
source ./bin/activate

$ ./bazel_configure.py
Platform Linux
Using C compiler          :     /usr/bin/gcc (4.8.5)
Using C++ compiler        :     /usr/bin/g++ (4.8.5)
Using C preprocessor      :     /usr/bin/cpp (4.8.5)
Using C++ preprocessor    :     /usr/bin/cpp (4.8.5)
Using linker              :     /usr/bin/ld.bfd
Using JDK                 :     /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64
Using Automake            :     /usr/bin/automake (1.13.4)
Using Autoconf            :     /usr/bin/autoconf (2.69)
Using Make                :     /usr/bin/make (3.82)
Using Python3             :     /usr/bin/python3.6 (3.6.8)
Using Libtool             :     /usr/bin/libtool (2.4.2)
Using archiver            :     /usr/bin/ar
Using coverage tool       :     /usr/bin/gcov
Using dwp                 :     /usr/bin/dwp
Using nm                  :     /usr/bin/nm
Using objcopy             :     /usr/bin/objcopy
Using objdump             :     /usr/bin/objdump
Using strip               :     /usr/bin/strip

bazel build --config=centos_nostyle heron/...
bazel build --config=centos_nostyle scripts/packages:binpkgs

./docker/scripts/build-artifacts.sh centos7 0.0.0 ~/heron-release

eval $(minikube -p minikube docker-env)
./docker/scripts/build-docker.sh centos7 0.0.0 ~/heron-release

nicknezis · 2020-07-09T14:54:55Z

I'm getting this error when trying to build on darwin (OSX):

ERROR: /private/var/tmp/_bazel_nnezis/42af5204dd5936eb8287d40b2ccac539/external/org_apache_zookeeper/BUILD.bazel:7:8: Executing genrule @org_apache_zookeeper//:zookeeper-srcs failed (Exit 1)
acinclude.m4:315: warning: macro 'AM_PATH_CPPUNIT' not found in library
configure.ac:37: error: Missing AM_PATH_CPPUNIT or PKG_CHECK_MODULES m4 macro.
acinclude.m4:327: CHECK_CPPUNIT is expanded from...
configure.ac:37: the top level
autom4te: /usr/bin/m4 failed with exit status: 1
aclocal: error: echo failed with exit status: 1
autoreconf: aclocal failed with exit status: 1
Buildfile: /private/var/folders/nv/s1d1vxld0c51bfkmxltp0zk80000gn/T/zookeeper.XXXXX.urp0mbJd/build.xml

I even tried adding --without-cppunit to the ./configure line, but without success.

I also tried brew install javacc and brew install cppunit to work through issues I found. Not sure what else I might need.

❯ ./bazel_configure.py
Platform Darwin
Using C compiler          :	/usr/bin/gcc (4.2.1)
Using C++ compiler        :	/usr/bin/g++ (4.2.1)
Using C preprocessor      :	/usr/bin/cpp (11.0.0)
Using C++ preprocessor    :	/usr/bin/cpp (11.0.0)
Using linker              :	/usr/bin/ld
Using JDK                 :	/System/Library/Frameworks/JavaVM.framework/Versions/A
Using Automake            :	/usr/local/Cellar/automake/1.16.2/bin/automake (1.16.2)
Using Autoconf            :	/usr/local/Cellar/autoconf/2.69/bin/autoconf (2.69)
Using Make                :	/usr/bin/make (3.81)
Using Python3             :	/usr/local/Cellar/[email protected]/3.8.3_1/Frameworks/Python.framework/Versions/3.8/bin/python3.8 (3.8.3)
Using Libtool             :	/usr/local/Cellar/libtool/2.4.6_2/bin/glibtool (2.4.6)
Using archiver            :	/usr/bin/ar
Using coverage tool       :	/usr/bin/gcov
dwp                       :	not found, but ok
Using nm                  :	/usr/bin/nm
objcopy                   :	not found, but ok
Using objdump             :	/usr/bin/objdump
Using strip               :	/usr/bin/strip

joshfischer1108 · 2020-07-10T02:07:15Z

the build is failing on my mac as well.

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.5
BuildVersion:	19F101

$ ./bazel_configure.py 
Platform Darwin
Using C compiler          :	/usr/bin/clang (11.0.3)
Using C++ compiler        :	/usr/bin/clang++ (11.0.3)
Using C preprocessor      :	/usr/bin/cpp (11.0.3)
Using C++ preprocessor    :	/usr/bin/cpp (11.0.3)
Using linker              :	/usr/bin/ld
Using JDK                 :	/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home
Using Automake            :	/usr/local/Cellar/automake/1.15.1/bin/automake (1.15.1)
Using Autoconf            :	/usr/local/Cellar/autoconf/2.69/bin/autoconf (2.69)
Using Make                :	/usr/bin/make (3.81)
Using Python3             :	/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/bin/python3.7 (3.7.4)
Using Libtool             :	/usr/local/Cellar/libtool/2.4.6_1/bin/glibtool (2.4.6)
Using archiver            :	/usr/bin/ar
Using coverage tool       :	/usr/bin/gcov
dwp                       :	not found, but ok
Using nm                  :	/usr/bin/nm
objcopy                   :	not found, but ok
Using objdump             :	/usr/bin/objdump
Using strip               :	/usr/bin/strip

Command ran

$ bazel build --config=darwin scripts/packages:binpkgs

Sample error output below:

INFO: From Building external/com_google_protobuf/libprotobuf_java.jar (122 source files, 1 source jar) [for host]:
warning: -parameters is not supported for target value 1.7. Use 1.8 or later.
ERROR: /private/var/tmp/_bazel_joshfischer/70e9bd6aaead2621044e8d790ac25401/external/org_apache_zookeeper/BUILD.bazel:7:1: Executing genrule @org_apache_zookeeper//:zookeeper-srcs failed (Exit 127)
/bin/bash: line 7: ant: command not found
Target //scripts/packages:binpkgs failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /private/var/tmp/_bazel_joshfischer/70e9bd6aaead2621044e8d790ac25401/external/org_apache_zookeeper/BUILD.bazel:75:1 Executing genrule @org_apache_zookeeper//:zookeeper-srcs failed (Exit 127)
INFO: Elapsed time: 485.550s, Critical Path: 197.04s
INFO: 397 processes: 397 local.
FAILED: Build did NOT complete successfull

huijunwu · 2020-07-10T02:22:47Z

looks like it needs some packages
brew install ant cppunit pkg-config

joshfischer1108 · 2020-07-10T02:42:37Z

Shouldn't the build be hermetic?

huijunwu · 2020-07-10T03:16:16Z

Good question.
We may add check in bazel_configure.py.
The docker build is hermetic at present.

joshfischer1108 · 2020-07-10T04:28:17Z

@huijunw After adding those packages the build was successful.

Target //scripts/packages:binpkgs up-to-date:
  bazel-bin/scripts/packages/heron-install.sh
INFO: Elapsed time: 2056.177s, Critical Path: 483.67s
INFO: 1744 processes: 1744 local.
INFO: Build completed successfully, 2843 total actions

nicknezis · 2020-07-10T11:25:58Z

It worked for me after adding the missing package.
Updating bazel_configure.py is a great idea. For this pull request, would it be sufficient to update the website documentation with the updated packages that are needed? It feels like that should happen in this PR.

joshfischer1108 · 2020-07-10T12:26:33Z

Agreed @nicknezis .

@huijunwu could you make that change the documentation please?

huijunwu · 2020-07-10T18:31:57Z

updated the doc

website2/docs/compiling-linux.md

* Upgrade zookeeper3.5.8 Co-authored-by: Huijun Wu <[email protected]> Co-authored-by: Nicholas Nezis <[email protected]>

huijunwu and others added 2 commits July 3, 2020 02:15

upgradezookeeper3.5.8

1e0f8ee

change tab to space

512cc05

nicknezis self-requested a review July 3, 2020 05:44

huijunwu added 6 commits July 3, 2020 12:34

updatedocker

25317ac

Merge branch 'huijunwu/20200702' of https://github.com/apache/incubat…

6dd745f

…or-heron into huijunwu/20200702

updateurl

1807c0d

updatedockerant

b140b4f

updatedockerfile

ba3ede5

updateinstallzk

995cadc

huijunwu requested a review from nwangtw July 3, 2020 14:45

huijunwu mentioned this pull request Jul 3, 2020

Python 3 upgrade #3522

Merged

huijunwu added 3 commits July 4, 2020 21:49

fixzkinstall

af88c10

removespace

33fa281

updatehelmminikube

f7715f3

nicknezis approved these changes Jul 5, 2020

View reviewed changes

nicknezis reviewed Jul 7, 2020

View reviewed changes

huijunw added 2 commits July 6, 2020 22:51

fixdarwin-mavenjson

86936ba

Merge branch 'master' of https://github.com/apache/incubator-heron in…

2458c45

…to huijunwu/20200702

huijunwu marked this pull request as draft July 7, 2020 17:00

updatemaven_install.json

050c095

fixcentos

789cd56

huijunwu marked this pull request as ready for review July 8, 2020 08:01

huijunwu requested a review from nicknezis July 9, 2020 05:05

joshfischer1108 approved these changes Jul 10, 2020

View reviewed changes

addantcheck

9dc87bb

huijunwu mentioned this pull request Jul 10, 2020

bazel_configure.py needs update to reflect the latest changes #3564

Open

updatedoc

ea4ca14

nicknezis reviewed Jul 10, 2020

View reviewed changes

website2/docs/compiling-linux.md Outdated Show resolved Hide resolved

Updated website install steps to reference Python 3

d9e70f6

nicknezis merged commit 278c05a into master Jul 10, 2020

nicknezis deleted the huijunwu/20200702 branch July 10, 2020 21:11

nicknezis added a commit that referenced this pull request Sep 14, 2020

zookeeper to 3.5.8 (#3555)

b89800f

* Upgrade zookeeper3.5.8 Co-authored-by: Huijun Wu <[email protected]> Co-authored-by: Nicholas Nezis <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

zookeeper to 3.5.8 #3555

zookeeper to 3.5.8 #3555

huijunwu commented Jul 3, 2020

nicknezis commented Jul 3, 2020

joshfischer1108 commented Jul 3, 2020 •

edited

Loading

huijunwu commented Jul 3, 2020

nicknezis commented Jul 4, 2020

huijunwu commented Jul 5, 2020

nicknezis commented Jul 7, 2020

nicknezis Jul 7, 2020

joshfischer1108 Jul 7, 2020

nicknezis Jul 7, 2020

nicknezis commented Jul 7, 2020

huijunwu commented Jul 8, 2020

huijunwu commented Jul 8, 2020

nicknezis commented Jul 9, 2020 •

edited

Loading

joshfischer1108 commented Jul 10, 2020 •

edited

Loading

huijunwu commented Jul 10, 2020

joshfischer1108 commented Jul 10, 2020

huijunwu commented Jul 10, 2020

joshfischer1108 commented Jul 10, 2020

nicknezis commented Jul 10, 2020

joshfischer1108 commented Jul 10, 2020

huijunwu commented Jul 10, 2020

zookeeper to 3.5.8 #3555

zookeeper to 3.5.8 #3555

Conversation

huijunwu commented Jul 3, 2020

nicknezis commented Jul 3, 2020

joshfischer1108 commented Jul 3, 2020 • edited Loading

huijunwu commented Jul 3, 2020

nicknezis commented Jul 4, 2020

huijunwu commented Jul 5, 2020

nicknezis commented Jul 7, 2020

nicknezis Jul 7, 2020

Choose a reason for hiding this comment

joshfischer1108 Jul 7, 2020

Choose a reason for hiding this comment

nicknezis Jul 7, 2020

Choose a reason for hiding this comment

nicknezis commented Jul 7, 2020

huijunwu commented Jul 8, 2020

huijunwu commented Jul 8, 2020

nicknezis commented Jul 9, 2020 • edited Loading

joshfischer1108 commented Jul 10, 2020 • edited Loading

huijunwu commented Jul 10, 2020

joshfischer1108 commented Jul 10, 2020

huijunwu commented Jul 10, 2020

joshfischer1108 commented Jul 10, 2020

nicknezis commented Jul 10, 2020

joshfischer1108 commented Jul 10, 2020

huijunwu commented Jul 10, 2020

joshfischer1108 commented Jul 3, 2020 •

edited

Loading

nicknezis commented Jul 9, 2020 •

edited

Loading

joshfischer1108 commented Jul 10, 2020 •

edited

Loading