Fixes #2348 - Converts the source code to python 3 #2825

karlcow · 2019-03-20T06:59:35Z

This PR fixes issue #2348

Proposed PR background

Python 2.* is reaching the EOL. We need to move to python 3.

This is a first PR as a draft to explore a bit more things. This is not yet merged with the latest version of master.

miketaylr · 2019-03-25T15:48:30Z

Thanks @karlcow, I will start reviewing this week. I assume failing tests are WIP?

miketaylr · 2019-03-25T15:49:03Z

edit: spoke too soon, just some style nits:

webcompat/helpers.py:652:45: E261 at least two spaces before inline comment
webcompat/helpers.py:655:17: E261 at least two spaces before inline comment
webcompat/helpers.py:661:45: E261 at least two spaces before inline comment
webcompat/helpers.py:664:17: E261 at least two spaces before inline comment

😄

miketaylr

LGTM!

webcompat/api/uploads.py

webcompat/form.py

webcompat/views.py

config/__init__.py

karlcow · 2019-03-27T02:02:52Z

@miketaylr

ok if I didn't mess up, I just pushed a rebase of this branch based on the latest master with tests running.

git checkout master
git fetch origin
git merger origin/master
git checkout 2348/1
git rebase master
(fix a couple of merge conflicts. Surprisingly not that much.)
git add .
nosetests
(All tests passed)
git push -f origin 2348/1

Let's see the logs git log --oneline

27e1437 Issue #2348 - Converts webcompat/ to python 3
e5c0bcf Issue #2348 - Converts webcompat/ to python 3
d3d1621 Merge pull request #2814 from webcompat/greenkeeper/eslint-config-prettier-4.1.0
2a443c7 Merge pull request #2798 from webcompat/greenkeeper/yargs-13.1.0
63b8303 v18.0.0 changelog update
9c4af5c Merge pull request #2820 from webcompat/issues/2819
ed57e44 Issue #2819 - Add browser-fenix to EXTRA_LABELS

https://github.com/webcompat/webcompat.com/blob/master/CHANGELOG.md#1800---2019-03-14

karlcow · 2019-03-28T00:26:19Z

Just updated the circleCi config to a different image for this branch so we run with python 3.
Let's see if it fails…

🥁 and it does.

karlcow · 2019-03-28T00:30:38Z

Ah interesting…

#!/bin/bash -eo pipefail
. venv/bin/activate
pep8 --ignore=E402 webcompat/ tests/ config/secrets.py.example
npm run lint
npm run build
nosetests
python run.py -t &
sleep 5
node --max-old-space-size=8192 ./tests/functional/_intern.js --reporters="runner" --firefoxBinary=`which firefox`
firefox --version

/home/circleci/repo/venv/lib/python3.7/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1
  EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]')
/home/circleci/repo/venv/lib/python3.7/site-packages/pep8.py:2124: UserWarning: 

pep8 has been renamed to pycodestyle (GitHub issue #466)
Use of the pep8 tool will be removed in a future release.
Please install and use `pycodestyle` instead.

$ pip install pycodestyle
$ pycodestyle ...

  '\n\n'
webcompat/helpers.py:652:45: E261 at least two spaces before inline comment
webcompat/helpers.py:655:17: E261 at least two spaces before inline comment
webcompat/helpers.py:661:45: E261 at least two spaces before inline comment
webcompat/helpers.py:664:17: E261 at least two spaces before inline comment
Exited with code 1

I will open an issue on the main project
PyCQA/pycodestyle#466

They renamed the tool.
https://twitter.com/gvanrossum/status/737438918799282178?lang=en

karlcow · 2019-03-28T00:35:01Z

Just created #2828

karlcow · 2019-04-04T05:15:44Z

pushed another rebase to be in sync with current master.

karlcow · 2019-04-11T01:12:31Z

Updated

0f12e4b Issue #2348 - Updates circleci configuration to python 3
95543d8 Issue #2348 - Converts webcompat/ to python 3
fa11bee Issue #2348 - Converts webcompat/ to python 3
eff4017 Merge pull request #2838 from marimeireles/issues/2837/1
f18b73e Fixes #2837 - Lowercase the format before checking if it's valid
94980e4 v19.0.1 changelog update
59bdaf7 Merge pull request #2835 from karlcow/2080/1
4bcbb9a Merge pull request #2834 from karlcow/1838/1
e773656 Merge branch 'master' into 1838/1
96931da Issue #1838 - Adjusts package.json which is used by circleci
54d485a Issue #1838 - Adjusts the documentation for new py dependencies
022ee4c Merge pull request #2829 from karlcow/2828/1
1a66f72 Issue #2080 - Aligns navigation menu in issues list
06569de Issue #1838 - Adjusts python dependencies config
18598f8 v19.0.0 changelog update

→ nosetests
.............................................................................................................
----------------------------------------------------------------------
Ran 109 tests in 2.014s

OK

but I get a fail on circleci. Let's see what is wrong.

karlcow · 2019-04-11T01:13:43Z

ah!

#!/bin/bash -eo pipefail
. venv/bin/activate
pycodestyle --ignore=E402,W504 webcompat/ tests/ config/secrets.py.example
npm run lint
npm run build
nosetests
python run.py -t &
sleep 5
node --max-old-space-size=8192 ./tests/functional/_intern.js --reporters="runner" --firefoxBinary=`which firefox`
firefox --version

webcompat/helpers.py:653:45: E261 at least two spaces before inline comment
webcompat/helpers.py:656:17: E261 at least two spaces before inline comment
webcompat/helpers.py:662:45: E261 at least two spaces before inline comment
webcompat/helpers.py:665:17: E261 at least two spaces before inline comment
Exited with code 1

I probably need to adjust and create a pycodestyle pre-commit hook.

karlcow · 2019-04-11T02:16:50Z

Linting is ok.
hmmm more errors this time. Let's see what went wrong.

karlcow · 2019-04-11T02:31:40Z

locally I don't have any errors.
Ah interesting, We are hitting this error.

https://circleci.com/gh/webcompat/webcompat.com/461

https://stackoverflow.com/questions/39142928/cant-load-flask-config-from-parent-directory?rq=1

karlcow · 2019-04-11T04:08:09Z

In this error message, the interesting line is maybe:

======================================================================
ERROR: Failure: ModuleNotFoundError (No module named 'environment')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/circleci/repo/venv/lib/python3.7/imp.py", line 244, in load_module
    return load_package(name, filename)
  File "/home/circleci/repo/venv/lib/python3.7/imp.py", line 216, in load_package
    return _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/circleci/repo/config/__init__.py", line 21, in <module>
    from config.secrets import *  # noqa
  File "/home/circleci/repo/config/secrets.py", line 39, in <module>
    import environment as env
ModuleNotFoundError: No module named 'environment'

======================================================================
ERROR: Failure: ImportStringError (import_string() failed for 'config'. Possible reasons are:

- missing __init__.py in a package;
- package or module path not included in sys.path;
- duplicated package or module name taking precedence in sys.path;
- missing module, class, function or variable;

Debugged import:

- 'config' not found.

Original exception:

ModuleNotFoundError: No module named 'environment')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/werkzeug/utils.py", line 547, in import_string
    __import__(import_name)
  File "/home/circleci/repo/config/__init__.py", line 21, in <module>
    from config.secrets import *  # noqa
  File "/home/circleci/repo/config/secrets.py", line 39, in <module>
    import environment as env
ModuleNotFoundError: No module named 'environment'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/circleci/repo/venv/lib/python3.7/imp.py", line 244, in load_module
    return load_package(name, filename)
  File "/home/circleci/repo/venv/lib/python3.7/imp.py", line 216, in load_package
    return _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/circleci/repo/webcompat/__init__.py", line 25, in <module>
    app.config.from_object('config')
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/flask/config.py", line 170, in from_object
    obj = import_string(obj)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/werkzeug/utils.py", line 564, in import_string
    ImportStringError, ImportStringError(import_name, e), sys.exc_info()[2]
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/werkzeug/_compat.py", line 147, in reraise
    raise value.with_traceback(tb)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/werkzeug/utils.py", line 547, in import_string
    __import__(import_name)
  File "/home/circleci/repo/config/__init__.py", line 21, in <module>
    from config.secrets import *  # noqa
  File "/home/circleci/repo/config/secrets.py", line 39, in <module>
    import environment as env
werkzeug.utils.ImportStringError: import_string() failed for 'config'. Possible reasons are:

- missing __init__.py in a package;
- package or module path not included in sys.path;
- duplicated package or module name taking precedence in sys.path;
- missing module, class, function or variable;

Debugged import:

- 'config' not found.

Original exception:

ModuleNotFoundError: No module named 'environment'

Maybe this line.

  File "/home/circleci/repo/webcompat/__init__.py", line 25, in <module>
    app.config.from_object('config')

karlcow · 2019-04-11T04:10:26Z

I would love to be able to reproduce this locally.

karlcow · 2019-04-11T05:06:37Z

I noticed that circleci is using packages.json to install stuff. Not sure that's good to do that.
But anyway we were using pip2 in packages.json, I wonder if it will have an impact on circleci. :)

karlcow · 2019-04-11T05:08:33Z

ok it didn't solve anything for the specific circleci issue.

karlcow · 2019-04-11T07:27:05Z

oh holy cow… so much time spent on something which should have been so simple.
OK this convince me that we should make our dev local deployment a mirror of what we do online.

the error is coming from

cp config/secrets.py.example config/secrets.py

and the fact that secrets.py.example was not python 3 ready for the import.
Or at least I think it's the origin of the mistake.

Let's see that now. doh! Angry gods of cows.

karlcow · 2019-04-11T07:40:52Z

so at least now the tests are running it seems we have some things to fix.
It could be in an untested areas of the python code from what I can see in the current log.

127.0.0.1 - - [11/Apr/2019 07:39:03] "GET /api/issues?page=2&per_page=50&state=open&stage=all&sort=created&direction=desc HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/circleci/repo/venv/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/circleci/repo/webcompat/helpers.py", line 396, in wrapped_func
    checksum = hashlib.md5(json.dumps(get_args)).hexdigest()
TypeError: Unicode-objects must be encoded before hashing
Error fetching http://localhost:5000/issues?page=2&per_page=50&state=open&stage=all&sort=created&direction=desc
�[31m× chrome 71.0.3578.98 on Linux - Issue-list - Loading partial params results in merge with defaults (10.112s)
    NoSuchElement: [POST http://localhost:4444/wd/hub/session/d968d8737ad1f803ebaf70b3c55db52b/element / {"using":"css selector","value":".js-IssueList:nth-of-type(1)"}] no such element: Unable to locate element: {"method":"css selector","selector":".js-IssueList:nth-of-type(1)"}
      (Session info: headless chrome=71.0.3578.98)
      (Driver info: chromedriver=2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb),platform=Linux 4.4.0-144-generic x86_64)
      at Object.openPage  <tests/functional/lib/helpers.js:29:8>
      at Test.Loading partial params results in merge with defaults [as test]  <tests/functional/issue-list-non-auth.js:243:32>
      at <src/lib/Test.ts:263:51>
      at Command.<anonymous>  <tests/functional/lib/helpers.js:38:12>
      at Object.openPage  <tests/functional/lib/helpers.js:31:8>
      at Test.Loading partial params results in merge with defaults [as test]  <tests/functional/issue-list-non-auth.js:243:32>
      at <src/lib/Test.ts:263:51>�[0m

This is an easy one.

karlcow · 2019-05-08T06:22:52Z

ok more things to fix.

karlcow · 2019-05-09T02:27:23Z

ok back on track! yooohoo.

karlcow · 2019-08-15T05:18:56Z

@miketaylr time for the merge pull request.

miketaylr · 2019-08-15T15:44:04Z

@miketaylr time for the merge pull request.

wowowow! i'll leave you the honor.

karlcow added 2 commits February 8, 2019 16:22

Issue #2348 - Converts webcompat/ to python 3

005f39a

Issue #2348 - Converts webcompat/ to python 3

99ac5db

karlcow requested a review from miketaylr March 20, 2019 06:59

miketaylr approved these changes Mar 26, 2019

View reviewed changes

webcompat/api/uploads.py Show resolved Hide resolved

webcompat/form.py Show resolved Hide resolved

webcompat/views.py Show resolved Hide resolved

config/__init__.py Show resolved Hide resolved

karlcow force-pushed the 2348/1 branch from 99ac5db to 27e1437 Compare March 27, 2019 01:53

karlcow mentioned this pull request Mar 27, 2019

Python 3 Migration #1332

Closed

6 tasks

karlcow force-pushed the 2348/1 branch from 27e1437 to 3703597 Compare March 28, 2019 00:24

karlcow mentioned this pull request Mar 28, 2019

Update from pep8 to pycodestyle #2828

Closed

karlcow mentioned this pull request Mar 28, 2019

Fixes #2828 - Changes from pep8 to pycodestyle #2829

Merged

karlcow force-pushed the 2348/1 branch from 3703597 to fec5137 Compare April 4, 2019 05:14

karlcow force-pushed the 2348/1 branch from fec5137 to fde101c Compare April 5, 2019 01:02

karlcow mentioned this pull request Apr 8, 2019

2019Q2 - 4.1 Python 3 compatibility mozilla/webcompat-team-okrs#83

Closed

karlcow force-pushed the 2348/1 branch from fde101c to 0f12e4b Compare April 11, 2019 01:09

karlcow added 5 commits April 24, 2019 13:02

Issue #2348 - Adjusts CircleCi for virtualenv

9807af5

Issue #2348 - fixing keys

d242a2b

Issue #2348 - Adds secrets.py in circleCI

3f61944

Issue #2348 - Fixes the import in secret.py.example

4071e4d

Issue #2348 - Fixes bytes issue on md5

454b68e

karlcow force-pushed the 2348/1 branch from 7099bea to 454b68e Compare April 24, 2019 04:05

karlcow added 4 commits May 8, 2019 13:11

Merge branch 'master' into py3-merge

eb5ac5d

Adds venv/ to .gitignore

b820562

Merge branch 'master' into 2348/1

6179d4d

Merge branch '2348/1' into py3-merge

6901eaf

Issue #2348 - Fixes md5 encoding issue

8910a95

karlcow added 7 commits May 14, 2019 15:44

Merge branch 'master' into 2348/1

eb69b2c

Merge branch 'master' into 2348/1

fece63c

Issue #2348 - Removes dashboard tests

e46ef27

Issue #2348 - Aligns circleci with soon to be prod.

6bb6ab1

Issue #2348 - Upgrade mock module

c2a534c

Issue #2348 - Fixing typo

f3910db

Issue #2348 - Upgrade python modules

a6dfec5

karlcow marked this pull request as ready for review August 15, 2019 04:42

Karl Dubost and others added 3 commits August 15, 2019 13:53

Merge branch 'master' into 2348/1

47f7b65

Issue #2348 - merge master

3a5343e

Issue #2348 - merge master

254982a

karlcow merged commit bb2d07a into master Aug 16, 2019

karlcow added the status: add to changelog label Aug 16, 2019

karlcow removed the status: add to changelog label Sep 16, 2019

karlcow deleted the 2348/1 branch September 16, 2019 02:43

Fixes #2348 - Converts the source code to python 3 #2825

Fixes #2348 - Converts the source code to python 3 #2825

Uh oh!

Conversation

karlcow commented Mar 20, 2019

Proposed PR background

Uh oh!

miketaylr commented Mar 25, 2019

Uh oh!

miketaylr commented Mar 25, 2019

Uh oh!

miketaylr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

karlcow commented Mar 27, 2019

Uh oh!

karlcow commented Mar 28, 2019

Uh oh!

karlcow commented Mar 28, 2019

Uh oh!

karlcow commented Mar 28, 2019

Uh oh!

karlcow commented Apr 4, 2019

Uh oh!

karlcow commented Apr 11, 2019

Uh oh!

karlcow commented Apr 11, 2019

Uh oh!

karlcow commented Apr 11, 2019

Uh oh!

karlcow commented Apr 11, 2019

Uh oh!

karlcow commented Apr 11, 2019

Uh oh!

karlcow commented Apr 11, 2019

Uh oh!

karlcow commented Apr 11, 2019

Uh oh!

karlcow commented Apr 11, 2019

Uh oh!

karlcow commented Apr 11, 2019

Uh oh!

karlcow commented Apr 11, 2019

Uh oh!

karlcow commented May 8, 2019

Uh oh!

karlcow commented May 9, 2019

Uh oh!

karlcow commented Aug 15, 2019

Uh oh!

miketaylr commented Aug 15, 2019

Uh oh!

Uh oh!