`get_checksum` should not be able to break the app

[miketaylr]

The root cause of #3352 is that get_checksum throws because it's trying to find a file that doesn't exist (due to a simple typo).

We try to handle a KeyError but don't handle another big pile of exceptions:

Jun 16 19:27:59 staging-server webcompat-staging: Traceback (most recent call last):
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/templates/__init__.py", line 33, in get_checksum
Jun 16 19:27:59 staging-server webcompat-staging:     checksum = cache_dict[file_path]
Jun 16 19:27:59 staging-server webcompat-staging: KeyError: '/[snip]/staging.webcompat.com/webcompat/static/dist/flashed-messages.e5.js'
Jun 16 19:27:59 staging-server webcompat-staging: #012During handling of the above exception, another exception occurred:
Jun 16 19:27:59 staging-server webcompat-staging: Traceback (most recent call last):
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2292, in wsgi_app
Jun 16 19:27:59 staging-server webcompat-staging:     response = self.full_dispatch_request()
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1815, in full_dispatch_request
Jun 16 19:27:59 staging-server webcompat-staging:     rv = self.handle_user_exception(e)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1718, in handle_user_exception
Jun 16 19:27:59 staging-server webcompat-staging:     reraise(exc_type, exc_value, tb)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/_compat.py", line 35, in reraise
Jun 16 19:27:59 staging-server webcompat-staging:     raise value
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1813, in full_dispatch_request
Jun 16 19:27:59 staging-server webcompat-staging:     rv = self.dispatch_request()
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1799, in dispatch_request
Jun 16 19:27:59 staging-server webcompat-staging:     return self.view_functions[rule.endpoint](**req.view_args)
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/helpers.py", line 568, in policy
Jun 16 19:27:59 staging-server webcompat-staging:     response = make_response(view(*args, **kwargs))
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/views.py", line 365, in show_issue
Jun 16 19:27:59 staging-server webcompat-staging:     json_data=to_str(issue_data[0]))
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/templating.py", line 135, in render_template
Jun 16 19:27:59 staging-server webcompat-staging:     context, ctx.app)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/templating.py", line 117, in _render
Jun 16 19:27:59 staging-server webcompat-staging:     rv = template.render(context)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
Jun 16 19:27:59 staging-server webcompat-staging:     return original_render(self, *args, **kwargs)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
Jun 16 19:27:59 staging-server webcompat-staging:     return self.environment.handle_exception(exc_info, True)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
Jun 16 19:27:59 staging-server webcompat-staging:     reraise(exc_type, exc_value, tb)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
Jun 16 19:27:59 staging-server webcompat-staging:     raise value.with_traceback(tb)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/[snip]/staging.webcompat.com/webcompat/templates/issue.html", line 1, in top-level template code
Jun 16 19:27:59 staging-server webcompat-staging:     {% extends "layout.html" %}
Jun 16 19:27:59 staging-server webcompat-staging:   File "/[snip]/staging.webcompat.com/webcompat/templates/layout.html", line 36, in top-level template code
Jun 16 19:27:59 staging-server webcompat-staging:     {% include "shared/flashed-messages.html" %}
Jun 16 19:27:59 staging-server webcompat-staging:   File "/[snip]/staging.webcompat.com/webcompat/templates/shared/flashed-messages.html", line 11, in top-level template code
Jun 16 19:27:59 staging-server webcompat-staging:     <script nomodule src="{{ url_for('static', filename='dist/flashed-messages.e5.js')|bust_cache }}"></script>
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/templates/__init__.py", line 42, in bust_cache
Jun 16 19:27:59 staging-server webcompat-staging:     return file_path + '?' + get_checksum(STATIC_PATH + file_path)
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/templates/__init__.py", line 35, in get_checksum
Jun 16 19:27:59 staging-server webcompat-staging:     checksum = md5_checksum(file_path)
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/templates/__init__.py", line 47, in md5_checksum
Jun 16 19:27:59 staging-server webcompat-staging:     with open(file_path, 'rb') as fh:
Jun 16 19:27:59 staging-server webcompat-staging: FileNotFoundError: [Errno 2] No such file or directory: '/[snip]/staging.webcompat.com/webcompat/static/dist/flashed-messages.e5.js'
Jun 16 19:27:59 staging-server webcompat-staging: #012During handling of the above exception, another exception occurred:
Jun 16 19:27:59 staging-server webcompat-staging: Traceback (most recent call last):
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/error_handlers/__init__.py", line 61, in custom_error_handler
Jun 16 19:27:59 staging-server webcompat-staging:     return error_response(request, err.code)
Jun 16 19:27:59 staging-server webcompat-staging: AttributeError: 'FileNotFoundError' object has no attribute 'code'
Jun 16 19:27:59 staging-server webcompat-staging: #012During handling of the above exception, another exception occurred:
Jun 16 19:27:59 staging-server webcompat-staging: Traceback (most recent call last):
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2309, in __call__
Jun 16 19:27:59 staging-server webcompat-staging:     return self.wsgi_app(environ, start_response)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2295, in wsgi_app
Jun 16 19:27:59 staging-server webcompat-staging:     response = self.handle_exception(e)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1748, in handle_exception
Jun 16 19:27:59 staging-server webcompat-staging:     return self.finalize_request(handler(e), from_error_handler=True)
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/error_handlers/__init__.py", line 64, in custom_error_handler
Jun 16 19:27:59 staging-server webcompat-staging:     return error_response(request, 500)
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/error_handlers/__init__.py", line 36, in error_response
Jun 16 19:27:59 staging-server webcompat-staging:     error_message=ERROR_DICT[code]), code
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/templating.py", line 135, in render_template
Jun 16 19:27:59 staging-server webcompat-staging:     context, ctx.app)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/local/lib/python3.7/dist-packages/flask/templating.py", line 117, in _render
Jun 16 19:27:59 staging-server webcompat-staging:     rv = template.render(context)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/lib/python3/dist-packages/jinja2/asyncsupport.py", line 76, in render
Jun 16 19:27:59 staging-server webcompat-staging:     return original_render(self, *args, **kwargs)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 1008, in render
Jun 16 19:27:59 staging-server webcompat-staging:     return self.environment.handle_exception(exc_info, True)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 780, in handle_exception
Jun 16 19:27:59 staging-server webcompat-staging:     reraise(exc_type, exc_value, tb)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/usr/lib/python3/dist-packages/jinja2/_compat.py", line 37, in reraise
Jun 16 19:27:59 staging-server webcompat-staging:     raise value.with_traceback(tb)
Jun 16 19:27:59 staging-server webcompat-staging:   File "/[snip]/staging.webcompat.com/webcompat/templates/error.html", line 1, in top-level template code
Jun 16 19:27:59 staging-server webcompat-staging:     {% extends "layout.html" %}
Jun 16 19:27:59 staging-server webcompat-staging:   File "/[snip]/staging.webcompat.com/webcompat/templates/layout.html", line 36, in top-level template code
Jun 16 19:27:59 staging-server webcompat-staging:     {% include "shared/flashed-messages.html" %}
Jun 16 19:27:59 staging-server webcompat-staging:   File "/[snip]/staging.webcompat.com/webcompat/templates/shared/flashed-messages.html", line 11, in top-level template code
Jun 16 19:27:59 staging-server webcompat-staging:     <script nomodule src="{{ url_for('static', filename='dist/flashed-messages.e5.js')|bust_cache }}"></script>
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/templates/__init__.py", line 42, in bust_cache
Jun 16 19:27:59 staging-server webcompat-staging:     return file_path + '?' + get_checksum(STATIC_PATH + file_path)
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/templates/__init__.py", line 35, in get_checksum
Jun 16 19:27:59 staging-server webcompat-staging:     checksum = md5_checksum(file_path)
Jun 16 19:27:59 staging-server webcompat-staging:   File "./webcompat/templates/__init__.py", line 47, in md5_checksum
Jun 16 19:27:59 staging-server webcompat-staging:     with open(file_path, 'rb') as fh:
Jun 16 19:27:59 staging-server webcompat-staging: FileNotFoundError: [Errno 2] No such file or directory: '/[snip]/staging.webcompat.com/webcompat/static/dist/flashed-messages.e5.js'

[miketaylr]

I feel like this is a "good-first- @karlcow " bug

[karlcow]

good-first as our pants is on fire, hurry up we need to fix it :p
So this is also related to #3337

[karlcow]

The get_checksum:

webcompat.com/webcompat/templates/__init__.py

Lines 21 to 42 in d6781a0

	@app.template_filter('bust_cache')
	def bust_cache(file_path):
	"""Jinja2 filter to add a cache busting param based on md5 checksum.
	Uses a simple cache_dict to we don't have to hash each file for every
	request. This is kept in-memory so it will be blown away when the app
	is restarted (which is when file changes would have been deployed).
	Doesn't return hash on development
	"""
	def get_checksum(file_path):
	try:
	checksum = cache_dict[file_path]
	except KeyError:
	checksum = md5_checksum(file_path)
	cache_dict[file_path] = checksum
	return checksum
	if app.config['LOCALHOST']:
	return file_path
	return file_path + '?' + get_checksum(STATIC_PATH + file_path)

So this is basically checking if the key is contained in a cache dictionary, aka a KeyError
if not it creates the new key by computing md5_checksum.

webcompat.com/webcompat/templates/__init__.py

Lines 45 to 55 in d6781a0

	def md5_checksum(file_path):
	"""Return the md5 checksum for a given file path."""
	with open(file_path, 'rb') as fh:
	m = hashlib.md5()
	while True:
	# only read in 8k of the file at a time
	data = fh.read(8192)
	if not data:
	break
	m.update(data)
	return m.hexdigest()

and because the file is not existent, we receive a FileNotFoundError

but it doesn't stop here. The request is then considered to be in error and the error_handler is trying to handle it and then it fails with

AttributeError: 'FileNotFoundError' object has no attribute 'code'

because

webcompat.com/webcompat/error_handlers/__init__.py

Lines 58 to 64 in 8656f17

	if not app.config['LOCALHOST']:
	app.logger.exception('Exception thrown:')
	try:
	return error_response(request, err.code)
	except AttributeError:
	# Somethign bad happened, we're not dealing with an HTTPError
	return error_response(request, 500)

[karlcow]

to simulate the error locally it is better to put in comments the localhost directive.

that will help.

[karlcow]

Debugging in Visual Studio Code

Then launch.json in .vscode

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python : Flask",
            "type": "python",
            "request": "launch",
            "module": "flask",
            "env": {
                "FLASK_APP": "webcompat:app",
                "FLASK_ENV": "development",
                "FLASK_DEBUG": "0"
            },
            "args": [
                "run",
                "--no-debugger",
                "--no-reload"
            ],
            "jinja": true
        }
    ]
}

Then set a breaking point into the code.

see for more details.
https://code.visualstudio.com/docs/python/tutorial-flask

[karlcow]

so after going through the code. The responsibility of checking the file existence is indeed in md5_checksum, but we probably need to also get_checksum so that we handle a failed checksum properly with something that would make sense for the person looking in the devtools why the js or css file is not there.

I have an idea for this.

the get_checksum could return a string which is missing_file and that would then be visible in the uri such as https://webcompat.com/dist/vendor.js?missing_file and we would tell us right away the origin of the issue. This is better than a silent fail.

[karlcow]

The FileNotFoundError bubbling up to the error_handler will not happen anymore so I guess we could ignore for now the error_handler code error for now.