Skip to content
This repository was archived by the owner on Mar 3, 2023. It is now read-only.

Clean heron-ui backend #3597

Merged
merged 2 commits into from
Nov 11, 2020
Merged

Clean heron-ui backend #3597

merged 2 commits into from
Nov 11, 2020

Conversation

Code0x58
Copy link
Contributor

@Code0x58 Code0x58 commented Aug 3, 2020

  • replace tornado server with FastAPI+Jinja2+Uvicorn
  • factor out torado from heron.tools.common
  • add type annotations
  • expose port 8889 of Vagrant VM for heron-ui

This should only be an internal refactor.

Async querying of the tracker was dropped, but could be reintroduced
using aiohttp if speed becomes
an issue for requests that aggregate tracker data.

This will have a conflict with #3588 which I'll fix after the first merge. A follow on may be to add a heron-ui smoke test, maybe with something like headless chromium and grepping the logs. update: the PR has now been rebased and the tracker client updates have been applied to explorer

@Code0x58 Code0x58 force-pushed the clean-ui-backend branch 5 times, most recently from c5ed819 to a954a23 Compare August 4, 2020 08:00
@huijunwu huijunwu requested a review from nwangtw August 6, 2020 15:27
@huijunwu huijunwu added the ui label Aug 6, 2020
@huijunwu
Copy link
Member

huijunwu commented Aug 6, 2020

Tested on MacOs
Some observations (not sure if they are is related to this PR, or they already existed before the PR)

[2020-08-06 08:38:20 -0700] [INFO]: 127.0.0.1:62051 - "GET /topologies/local/default/helloworld/1/filedata?offset=-1&length=-1&path=./log-files/container_1_exclaim1_3.log.0 HTTP/1.1" 500
[2020-08-06 08:38:20 -0700] [ERROR]: Exception in ASGI application
Traceback (most recent call last):
  File "/Users/huijunw/.pex/installed_wheels/6cd94f24fe549f5f278fef0989ec1818c8c82eb7/uvicorn-0.11.7-py3-none-any.whl/uvicorn/protocols/http/httptools_impl.py", line 390, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/Users/huijunw/.pex/installed_wheels/6cd94f24fe549f5f278fef0989ec1818c8c82eb7/uvicorn-0.11.7-py3-none-any.whl/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/Users/huijunw/.pex/installed_wheels/6cd94f24fe549f5f278fef0989ec1818c8c82eb7/uvicorn-0.11.7-py3-none-any.whl/uvicorn/middleware/message_logger.py", line 65, in __call__
    raise exc from None
  File "/Users/huijunw/.pex/installed_wheels/6cd94f24fe549f5f278fef0989ec1818c8c82eb7/uvicorn-0.11.7-py3-none-any.whl/uvicorn/middleware/message_logger.py", line 61, in __call__
    await self.app(scope, inner_receive, inner_send)
  File "/Users/huijunw/.pex/installed_wheels/18b4a830bde6aa705858d31555dbe1aea1e917c3/fastapi-0.60.1-py3-none-any.whl/fastapi/applications.py", line 181, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/Users/huijunw/.pex/installed_wheels/18b4a830bde6aa705858d31555dbe1aea1e917c3/fastapi-0.60.1-py3-none-any.whl/fastapi/routing.py", line 196, in app
    raw_response = await run_endpoint_function(
  File "/Users/huijunw/.pex/installed_wheels/18b4a830bde6aa705858d31555dbe1aea1e917c3/fastapi-0.60.1-py3-none-any.whl/fastapi/routing.py", line 149, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/concurrency.py", line 34, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "/usr/local/Cellar/[email protected]/3.8.4/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/huijunw/.pex/code/f640e2d8a896287a3702fc6b6996d7aca3c3d65e/heron/tools/ui/src/python/main.py", line 290, in file_data
    return Response(content=data, media_type="application/binary")
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/responses.py", line 49, in __init__
    self.body = self.render(content)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/responses.py", line 57, in render
    return content.encode(self.charset)
AttributeError: 'dict' object has no attribute 'encode'

Screen Shot 2020-08-06 at 8 40 22 AM

Screen Shot 2020-08-06 at 8 41 53 AM

[2020-08-06 08:43:49 -0700] [ERROR]: Unable to get response from http://127.0.0.1:8888/topologies/pid: 404 Client Error: Not Found for url: http://127.0.0.1:8888/topologies/pid?cluster=local&environ=default&topology=helloworld&instance=container_2_word_6
[2020-08-06 08:43:49 -0700] [INFO]: 127.0.0.1:62832 - "GET /topologies/local/default/helloworld/container_2_word_6/pid HTTP/1.1" 500
[2020-08-06 08:43:49 -0700] [ERROR]: Exception in ASGI application
Traceback (most recent call last):
  File "/Users/huijunw/.pex/installed_wheels/6cd94f24fe549f5f278fef0989ec1818c8c82eb7/uvicorn-0.11.7-py3-none-any.whl/uvicorn/protocols/http/httptools_impl.py", line 390, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/Users/huijunw/.pex/installed_wheels/6cd94f24fe549f5f278fef0989ec1818c8c82eb7/uvicorn-0.11.7-py3-none-any.whl/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/Users/huijunw/.pex/installed_wheels/6cd94f24fe549f5f278fef0989ec1818c8c82eb7/uvicorn-0.11.7-py3-none-any.whl/uvicorn/middleware/message_logger.py", line 65, in __call__
    raise exc from None
  File "/Users/huijunw/.pex/installed_wheels/6cd94f24fe549f5f278fef0989ec1818c8c82eb7/uvicorn-0.11.7-py3-none-any.whl/uvicorn/middleware/message_logger.py", line 61, in __call__
    await self.app(scope, inner_receive, inner_send)
  File "/Users/huijunw/.pex/installed_wheels/18b4a830bde6aa705858d31555dbe1aea1e917c3/fastapi-0.60.1-py3-none-any.whl/fastapi/applications.py", line 181, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/Users/huijunw/.pex/installed_wheels/18b4a830bde6aa705858d31555dbe1aea1e917c3/fastapi-0.60.1-py3-none-any.whl/fastapi/routing.py", line 196, in app
    raw_response = await run_endpoint_function(
  File "/Users/huijunw/.pex/installed_wheels/18b4a830bde6aa705858d31555dbe1aea1e917c3/fastapi-0.60.1-py3-none-any.whl/fastapi/routing.py", line 149, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/Users/huijunw/.pex/installed_wheels/5d659e4c2dba112a90305109923229b0159343b9/starlette-0.13.6-py3-none-any.whl/starlette/concurrency.py", line 34, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "/usr/local/Cellar/[email protected]/3.8.4/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/huijunw/.pex/code/f640e2d8a896287a3702fc6b6996d7aca3c3d65e/heron/tools/ui/src/python/main.py", line 464, in pid_snippet
    command = info["command"]
TypeError: 'NoneType' object is not subscriptable

Screen Shot 2020-08-06 at 8 43 42 AM

Screen Shot 2020-08-06 at 8 43 55 AM

@nicknezis nicknezis requested a review from huijunwu August 6, 2020 16:12
@Code0x58 Code0x58 force-pushed the clean-ui-backend branch 2 times, most recently from acc615b to 1a5e07a Compare August 7, 2020 00:49
@Code0x58
Copy link
Contributor Author

Code0x58 commented Aug 7, 2020

Thanks @huijunw - nice logs+reproduction examples! The logs + download file were bugs introduced in the PR, but fixed fixed in the updated branch now.

The pid/jmap/mem*/stack links just seem to not work, the tracker is returning 404 (which it does for any exception), and it seems to be doing that on master as well when testing with the local cluster

@Code0x58 Code0x58 force-pushed the clean-ui-backend branch 3 times, most recently from 5f26afa to eb610da Compare August 15, 2020 23:58
@Code0x58
Copy link
Contributor Author

The reported issues that were introduced by this PR have been fixed, and merge conflicts with the explorer have just been resolved.

Affected components:

  • explorer: this already has happy path smoke test coverage in CI - it looked ok when scanning through CI's output
  • UI:
    • tables and links all seem to be functioning now after fixes were made for the issues @huijunw found
    • Topology Metrics charts may have issues, but I can't tell if the changes after rebasing have introduced issues, or if it was like that before (probably not helped by my testing being to use a random topology pulled from the CI logs)

@nwangtw
Copy link
Contributor

nwangtw commented Aug 16, 2020

Good work!

I am a little concerned about: "Async querying of the tracker was dropped". Does it mean when UI is requesting data from tracker, the UI will not response?

"Topology Metrics" what is missing now?

@@ -7,7 +7,7 @@ pex_library(
),
reqs = [
"requests==2.12.3",
"tornado==4.0.2",
"tornado==4.5.3",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tornado is still needed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

both heron-shell and heron-tracker use tornado still, this PR just removes the use of it in explorer+UI. I forgot to remove tornado from the reqs of the explorer after rebasing though

@Code0x58
Copy link
Contributor Author

Thanks!

The UI server continues to be able to serve multiple HTTP requests in parallel, it just may take longer to respond as it make requests to the tracker sequentially now. Using async requests would be a nice change again, but that may be nicer/almost free in the future after changing the tracker to use FastAPI, as the generated OpenAPI spec may be usable to auto-generate client libs.

The "Topology Metrics" didn't always render to some of timelines. The logs made it look like multiplication was failing as it expected the 2nd timeseries to be univariate but [""] wasn't present, and there was another exception that looked related where the timeseries used in backpressure calculation didn't contain even [{"data": {}}].

@nwangtw
Copy link
Contributor

nwangtw commented Aug 16, 2020

Sounds good to me.

I will leave it to @huijunw to approve since he is running tests.

@joshfischer1108
Copy link
Member

@huijunw anything else we need to check on this pr?

@Code0x58 Code0x58 force-pushed the clean-ui-backend branch 2 times, most recently from c8ca580 to a2afb1e Compare November 3, 2020 19:39
 * replace tornado server with FastAPI+Jinja2+Uvicorn
 * factor out torado from heron.tools.common
 * add type annotations
 * expose port 8889 of Vagrant VM for heron-ui
 * pin to latest 4.x version of Tornado
 * visual changes to container pages

This should only be an internal refactor.

Async querying of the tracker was dropped, but could be reintroduced
using [aiohttp](https://docs.aiohttp.org/en/stable/) if speed becomes
an issue for requests that aggregate tracker data.
@huijunwu
Copy link
Member

looks good to me

@nicknezis nicknezis merged commit 97a9168 into apache:master Nov 11, 2020
@Code0x58 Code0x58 deleted the clean-ui-backend branch November 18, 2020 14:40
@Code0x58 Code0x58 mentioned this pull request Dec 5, 2020
4 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants