Source Code from Python 2 to Python 3

[karlcow]

This issue has a dependency on #1332

[miketaylr]

@karlcow what's the difference between this bug and #1332? I guess this is a single task of that bug?

[karlcow]

Yup. The #1332 describes all the tasks to do for moving to python 3.
This one is just about the code. I plan also to document a bit more in here certain things about the code specifically.

• such as python2/python3
• or python3 only
• tools to do it.

[karlcow]

Documenting what I have done.

git clone git@github.com:webcompat/webcompat.com.git webcompat-py3
cd webcompat-py3/
git checkout -b 2348/1
virtualenv env
# Using base prefix '/Library/Frameworks/Python.framework/Versions/3.7'
# New python executable in /Users/karl/code/webcompat-py3/env/bin/python3.7
# Also creating executable in /Users/karl/code/webcompat-py3/env/bin/python
# Installing setuptools, pip, wheel...
# done.
source env/bin/activate
# (env) 16:35:51 ~/code/webcompat-py3
pip3 install -r config/requirements.txt 

which gives

Collecting blinker==1.4 (from -r config/requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/1b/51/e2a9f3b757eb802f61dc1f2b09c8c99f6eb01cf06416c0671253536517b6/blinker-1.4.tar.gz
Collecting Flask==1.0.2 (from -r config/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl
Collecting Flask-Limiter==0.9.3 (from -r config/requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/9b/1b/23841342194bdd55e33d85a6f2e023c69d837fb69cee173352b7edba3c1f/Flask-Limiter-0.9.3.tar.gz
Collecting Flask-SQLAlchemy==2.1 (from -r config/requirements.txt (line 4))
  Using cached https://files.pythonhosted.org/packages/b3/52/227aaf4e8cebb153e239c518a9e916590b2fe0e4350e6b02d92b546b69b7/Flask-SQLAlchemy-2.1.tar.gz
Collecting Flask-WTF==0.14.2 (from -r config/requirements.txt (line 5))
  Using cached https://files.pythonhosted.org/packages/60/3a/58c629472d10539ae5167dc7c1fecfa95dd7d0b7864623931e3776438a24/Flask_WTF-0.14.2-py2.py3-none-any.whl
Collecting GitHub-Flask==3.1.5 (from -r config/requirements.txt (line 6))
  Using cached https://files.pythonhosted.org/packages/b0/8f/baf44b1b93fe2ee55fd66fb47f6ff5c3b21751cf3efc7786979ce968ad56/GitHub-Flask-3.1.5.tar.gz
Collecting mock==2.0.0 (from -r config/requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl
Collecting nose==1.3.7 (from -r config/requirements.txt (line 8))
  Using cached https://files.pythonhosted.org/packages/15/d8/dd071918c040f50fa1cf80da16423af51ff8ce4a0f2399b7bf8de45ac3d9/nose-1.3.7-py3-none-any.whl
Collecting pep8==1.7.0 (from -r config/requirements.txt (line 9))
  Using cached https://files.pythonhosted.org/packages/8a/cb/7d0fdca7e03f997945fb1bd60a8ddfea5c51229b865c470b4f7a64619d20/pep8-1.7.0-py2.py3-none-any.whl
Collecting Pillow==5.2.0 (from -r config/requirements.txt (line 10))
  Using cached https://files.pythonhosted.org/packages/8d/50/362dc2580ae2539331768c0c3295730c791d79216c9a271114022e8c9895/Pillow-5.2.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting requests==2.20.0 (from -r config/requirements.txt (line 11))
  Using cached https://files.pythonhosted.org/packages/f1/ca/10332a30cb25b627192b4ea272c351bce3ca1091e541245cccbace6051d8/requests-2.20.0-py2.py3-none-any.whl
Collecting ua-parser==0.8 (from -r config/requirements.txt (line 12))
  Using cached https://files.pythonhosted.org/packages/10/24/b13d43fbdc69b324be35058ee67e7c570ca728737bc37eadf92fa4668812/ua_parser-0.8.0-py2.py3-none-any.whl
Collecting WTForms==2.1 (from -r config/requirements.txt (line 13))
Collecting Werkzeug>=0.14 (from Flask==1.0.2->-r config/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl
Collecting Jinja2>=2.10 (from Flask==1.0.2->-r config/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Collecting click>=5.1 (from Flask==1.0.2->-r config/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Collecting itsdangerous>=0.24 (from Flask==1.0.2->-r config/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting limits (from Flask-Limiter==0.9.3->-r config/requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/62/97/27fbc02dcae873d2e12995d882a02ceab1cb8f7726f64e7f290351e1a13b/limits-1.3.tar.gz
Collecting six>=1.4.1 (from Flask-Limiter==0.9.3->-r config/requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting SQLAlchemy>=0.7 (from Flask-SQLAlchemy==2.1->-r config/requirements.txt (line 4))
Collecting pbr>=0.11 (from mock==2.0.0->-r config/requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/f3/04/fddc1c2dd75b256eda4d360024692231a2c19a0c61ad7f4a162407c1ab58/pbr-5.1.1-py2.py3-none-any.whl
Collecting idna<2.8,>=2.5 (from requests==2.20.0->-r config/requirements.txt (line 11))
  Using cached https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests==2.20.0->-r config/requirements.txt (line 11))
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests==2.20.0->-r config/requirements.txt (line 11))
  Using cached https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests==2.20.0->-r config/requirements.txt (line 11))
  Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->Flask==1.0.2->-r config/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/96/52/eef455862764cb6d6c136aa52c7f9fc4e7c1c644790a7107b1244a2b8a53/MarkupSafe-1.1.0-cp37-cp37m-macosx_10_6_intel.whl
Building wheels for collected packages: blinker, Flask-Limiter, Flask-SQLAlchemy, GitHub-Flask, limits
  Building wheel for blinker (setup.py) ... done
  Stored in directory: /Users/karl/Library/Caches/pip/wheels/92/a0/00/8690a57883956a301d91cf4ec999cc0b258b01e3f548f86e89
  Building wheel for Flask-Limiter (setup.py) ... done
  Stored in directory: /Users/karl/Library/Caches/pip/wheels/fa/3d/a0/8c30f3af3805184ac3834126cc4fc419370f3aa1eb0c380a48
  Building wheel for Flask-SQLAlchemy (setup.py) ... done
  Stored in directory: /Users/karl/Library/Caches/pip/wheels/bd/18/c4/5b1ebaec15e2bb933089576ebf905ea29976b2f37ed629e1c3
  Building wheel for GitHub-Flask (setup.py) ... done
  Stored in directory: /Users/karl/Library/Caches/pip/wheels/e9/db/0b/6a746701ebef1519458099246a190aa04226eaa78719419ac3
  Building wheel for limits (setup.py) ... done
  Stored in directory: /Users/karl/Library/Caches/pip/wheels/f1/aa/a0/27ef2911592b62d3365a6c853a23728b0e7b01c3d8e67607d3
Successfully built blinker Flask-Limiter Flask-SQLAlchemy GitHub-Flask limits
Installing collected packages: blinker, Werkzeug, MarkupSafe, Jinja2, click, itsdangerous, Flask, six, limits, Flask-Limiter, SQLAlchemy, Flask-SQLAlchemy, WTForms, Flask-WTF, idna, chardet, certifi, urllib3, requests, GitHub-Flask, pbr, mock, nose, pep8, Pillow, ua-parser
Successfully installed Flask-1.0.2 Flask-Limiter-0.9.3 Flask-SQLAlchemy-2.1 Flask-WTF-0.14.2 GitHub-Flask-3.1.5 Jinja2-2.10 MarkupSafe-1.1.0 Pillow-5.2.0 SQLAlchemy-1.2.16 WTForms-2.1 Werkzeug-0.14.1 blinker-1.4 certifi-2018.11.29 chardet-3.0.4 click-7.0 idna-2.7 itsdangerous-1.1.0 limits-1.3 mock-2.0.0 nose-1.3.7 pbr-5.1.1 pep8-1.7.0 requests-2.20.0 six-1.12.0 ua-parser-0.8.0 urllib3-1.24.1

[karlcow]

npm install
npm WARN deprecated postcss-cssnext@3.1.0: 'postcss-cssnext' has been deprecated in favor of 'postcss-preset-env'. Read more at https://moox.io/blog/deprecating-cssnext/

> husky@1.3.1 install /Users/karl/code/webcompat-py3/node_modules/husky
> node husky install

husky > setting up git hooks
husky > done

> gifsicle@4.0.1 postinstall /Users/karl/code/webcompat-py3/node_modules/gifsicle
> node lib/install.js

  ✔ gifsicle pre-build test passed successfully

> jpegtran-bin@4.0.0 postinstall /Users/karl/code/webcompat-py3/node_modules/jpegtran-bin
> node lib/install.js

  ✔ jpegtran pre-build test passed successfully

> optipng-bin@5.1.0 postinstall /Users/karl/code/webcompat-py3/node_modules/optipng-bin
> node lib/install.js

  ✔ optipng pre-build test passed successfully
npm notice created a lockfile as package-lock.json. You should commit this file.
added 1248 packages from 743 contributors and audited 10810 packages in 31.173s
found 2 low severity vulnerabilities
  run `npm audit fix` to fix them, or `npm audit` for details

[karlcow]

→ npm run build

> webcompat@ build /Users/karl/code/webcompat-py3
> grunt

Running "checkDependencies:default" (checkDependencies) task

Running "jst:compile" (jst) task
File webcompat/static/js/dist/templates.js created.

Running "concat:dist" (concat) task

Running "concat:untriaged" (concat) task

Running "concat:issues" (concat) task

Running "concat:issueList" (concat) task

Running "concat:userActivity" (concat) task

Running "uglify:dist" (uglify) task
>> 1 file created 253.1 kB → 234.61 kB

Running "uglify:ga" (uglify) task
>> 1 file created 780 B → 585 B

Running "uglify:issues" (uglify) task
>> 1 file created 370.93 kB → 185.87 kB

Running "uglify:issueList" (uglify) task
>> 1 file created 38.09 kB → 20.32 kB

Running "uglify:userActivity" (uglify) task
>> 1 file created 22.1 kB → 11.52 kB

Running "uglify:untriaged" (uglify) task
>> 1 file created 13.96 kB → 6.97 kB

Running "uglify:contributors" (uglify) task
>> 1 file created 1.1 kB → 733 B

Running "postcss:dist" (postcss) task
>> 1 processed stylesheet created.

Running "cssmin:combine" (cssmin) task
>> 1 file created. 1.87 kB → 38.47 kB

Done.

[karlcow]

Created the secrets.py with the tokens. Then started the project.

python -V
# Python 3.7.1
python run.py

Let the fun begin.

Traceback (most recent call last):
  File "/Users/karl/code/webcompat-py3/env/lib/python3.7/site-packages/werkzeug/utils.py", line 418, in import_string
    __import__(import_name)
  File "/Users/karl/code/webcompat-py3/config/__init__.py", line 16, in <module>
    import urlparse
ModuleNotFoundError: No module named 'urlparse'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run.py", line 35, in <module>
    from webcompat import app
  File "/Users/karl/code/webcompat-py3/webcompat/__init__.py", line 20, in <module>
    app.config.from_object('config')
  File "/Users/karl/code/webcompat-py3/env/lib/python3.7/site-packages/flask/config.py", line 170, in from_object
    obj = import_string(obj)
  File "/Users/karl/code/webcompat-py3/env/lib/python3.7/site-packages/werkzeug/utils.py", line 443, in import_string
    sys.exc_info()[2])
  File "/Users/karl/code/webcompat-py3/env/lib/python3.7/site-packages/werkzeug/_compat.py", line 137, in reraise
    raise value.with_traceback(tb)
  File "/Users/karl/code/webcompat-py3/env/lib/python3.7/site-packages/werkzeug/utils.py", line 418, in import_string
    __import__(import_name)
  File "/Users/karl/code/webcompat-py3/config/__init__.py", line 16, in <module>
    import urlparse
werkzeug.utils.ImportStringError: import_string() failed for 'config'. Possible reasons are:

- missing __init__.py in a package;
- package or module path not included in sys.path;
- duplicated package or module name taking precedence in sys.path;
- missing module, class, function or variable;

Debugged import:

- 'config' not found.

Original exception:

ModuleNotFoundError: No module named 'urlparse'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run.py", line 37, in <module>
    if 'import_name' in e and e.import_name == 'config':
TypeError: argument of type 'ImportStringError' is not iterable

[karlcow]

Also to encourage us. Pip starts to announce the deprecation.

[webcompat.com] out: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.

[karlcow]

That's called progress. :)
Yup first version working for unittests.

→ nosetests 
.............................................................................................................
----------------------------------------------------------------------
Ran 109 tests in 1.182s

OK

I need to run the functional tests too.