Skip to content
This repository was archived by the owner on Mar 3, 2023. It is now read-only.

clean up site docs #3626

Merged
merged 8 commits into from
Oct 31, 2020
Merged

clean up site docs #3626

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
36b2f50
clean up
joshfischer1108 Oct 30, 2020
3c4bbaf
Update website2/docs/topology-development-streamlet-api.md
joshfischer1108 Oct 30, 2020
b5eb44d
clean up duplicates
joshfischer1108 Oct 30, 2020
08122d8
clean up
joshfischer1108 Oct 30, 2020
9580b8e
Merge branch 'joshfischer/clean-up-site' of github.com:joshfischer110…
joshfischer1108 Oct 30, 2020
19faec0
last version clean up
joshfischer1108 Oct 30, 2020
99484b1
commit before reset
joshfischer1108 Oct 30, 2020
2df7043
adding version to link
joshfischer1108 Oct 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 26 additions & 26 deletions website2/docs/compiling-code-organization.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ sidebar_label: Code Organization

This document contains information about the Heron codebase intended primarily
for developers who want to contribute to Heron. The Heron codebase lives on
[github]({{% githubMaster %}}).
[github](https://github.com/apache/incubator-heron/tree/master).

If you're looking for documentation about developing topologies for a Heron
cluster, see [Building Topologies](topology-development-topology-api-java) instead.
Expand Down Expand Up @@ -51,7 +51,7 @@ Information on setting up and using Bazel for Heron can be found in [Compiling H
* **Inter-component communication** --- Heron uses [Protocol
Buffers](https://developers.google.com/protocol-buffers/?hl=en) for
communication between components. Most `.proto` definition files can be found in
[`heron/proto`]({{% githubMaster %}}/heron/proto).
[`heron/proto`](https://github.com/apache/incubator-heron/tree/master/heron/proto).

* **Cluster coordination** --- Heron relies heavily on ZooKeeper for cluster
coordination for distributed deployment, be it for [Aurora](schedulers-aurora-cluster) or for a [custom
Expand All @@ -61,7 +61,7 @@ Management](#state-management) section below.

## Common Utilities

The [`heron/common`]({{% githubMaster %}}/heron/common) contains a variety of
The [`heron/common`](https://github.com/apache/incubator-heron/tree/master/heron/common) contains a variety of
utilities for each of Heron's languages, including useful constants, file
utilities, networking interfaces, and more.

Expand All @@ -70,7 +70,7 @@ utilities, networking interfaces, and more.
Heron supports two cluster schedulers out of the box:
[Aurora](schedulers-aurora-cluster) and a [local
scheduler](schedulers-local). The Java code for each of those
schedulers can be found in [`heron/schedulers`]({{% githubMaster %}}/heron/schedulers)
schedulers can be found in [`heron/schedulers`](https://github.com/apache/incubator-heron/tree/master/heron/schedulers)
, while the underlying scheduler API can be found [here](/api/org/apache/heron/spi/scheduler/package-summary.html)

Info on custom schedulers can be found in [Implementing a Custom
Expand All @@ -83,10 +83,10 @@ Deployment](schedulers-local).

The parts of Heron's codebase related to
[ZooKeeper](http://zookeeper.apache.org/) are mostly contained in
[`heron/state`]({{% githubMaster %}}/heron/state). There are ZooKeeper-facing
interfaces for [C++]({{% githubMaster %}}/heron/state/src/cpp),
[Java]({{% githubMaster %}}/heron/state/src/java), and
[Python]({{% githubMaster %}}/heron/state/src/python) that are used in a variety of
[`heron/state`](https://github.com/apache/incubator-heron/tree/master/heron/state). There are ZooKeeper-facing
interfaces for [C++](https://github.com/apache/incubator-heron/tree/master/heron/state/src/cpp),
[Java](https://github.com/apache/incubator-heron/tree/master/heron/state/src/java), and
[Python](https://github.com/apache/incubator-heron/tree/master/heron/state/src/python) that are used in a variety of
Heron components.

## Topology Components
Expand All @@ -95,25 +95,25 @@ Heron components.

The C++ code for Heron's [Topology
Master](heron-architecture#topology-master) is written in C++ can be
found in [`heron/tmaster`]({{% githubMaster %}}/heron/tmaster).
found in [`heron/tmaster`](https://github.com/apache/incubator-heron/tree/master/heron/tmaster).

### Stream Manager

The C++ code for Heron's [Stream
Manager](heron-architecture#stream-manager) can be found in
[`heron/stmgr`]({{% githubMaster %}}/heron/stmgr).
[`heron/stmgr`](https://github.com/apache/incubator-heron/tree/master/heron/stmgr).

### Heron Instance

The Java code for [Heron
instances](heron-architecture#heron-instance) can be found in
[`heron/instance`]({{% githubMaster %}}/heron/instance).
[`heron/instance`](https://github.com/apache/incubator-heron/tree/master/heron/instance).

### Metrics Manager

The Java code for Heron's [Metrics
Manager](heron-architecture#metrics-manager) can be found in
[`heron/metricsmgr`]({{% githubMaster %}}/heron/metricsmgr).
[`heron/metricsmgr`](https://github.com/apache/incubator-heron/tree/master/heron/metricsmgr).

If you'd like to implement your own custom metrics handler (known as a **metrics
sink**), see [Implementing a Custom Metrics Sink](extending-heron-metric-sink).
Expand All @@ -123,7 +123,7 @@ sink**), see [Implementing a Custom Metrics Sink](extending-heron-metric-sink).
### Topology API

Heron's API for writing topologies is written in Java. The code for this API can
be found in [`heron/api`]({{% githubMaster %}}/heron/api).
be found in [`heron/api`](https://github.com/apache/incubator-heron/tree/master/heron/api).

Documentation for writing topologies can be found in [Building
Topologies](topology-development-topology-api-java), while API documentation can be found
Expand All @@ -142,7 +142,7 @@ The Java API for simulator can be found in
Heron's codebase includes a wide variety of example
[topologies](heron-topology-concepts) built using Heron's topology API for
Java. Those examples can be found in
[`heron/examples`]({{% githubMaster %}}/heron/examples).
[`heron/examples`](https://github.com/apache/incubator-heron/tree/master/heron/examples).

## User Interface Components

Expand All @@ -152,45 +152,45 @@ Heron has a tool called `heron` that is used to both provide a CLI interface
for [managing topologies](user-manuals-heron-cli) and to perform much of
the heavy lifting behind assembling physical topologies in your cluster.
The Python code for `heron` can be found in
[`heron/tools/cli`]({{% githubMaster %}}/heron/tools/cli).
[`heron/tools/cli`](https://github.com/apache/incubator-heron/tree/master/heron/tools/cli).

Sample configurations for different Heron schedulers

* [Local scheduler](schedulers-local) config can be found in [`heron/config/src/yaml/conf/local`]({{% githubMaster %}}/heron/config/src/yaml/conf/local),
* [Aurora scheduler](schedulers-aurora-cluster) config can be found [`heron/config/src/yaml/conf/aurora`]({{% githubMaster %}}/heron/config/src/yaml/conf/aurora).
* [Local scheduler](schedulers-local) config can be found in [`heron/config/src/yaml/conf/local`](https://github.com/apache/incubator-heron/tree/master/heron/config/src/yaml/conf/local),
* [Aurora scheduler](schedulers-aurora-cluster) config can be found [`heron/config/src/yaml/conf/aurora`]({https://github.com/apache/incubator-heron/tree/master/heron/config/src/yaml/conf/aurora).

### Heron Tracker

The Python code for the [Heron Tracker](user-manuals-heron-tracker-runbook) can be
found in [`heron/tools/tracker`]({{% githubMaster %}}/heron/tools/tracker).
found in [`heron/tools/tracker`](https://github.com/apache/incubator-heron/tree/master/heron/tools/tracker).

The Tracker is a web server written in Python. It relies on the
[Tornado](http://www.tornadoweb.org/en/stable/) framework. You can add new HTTP
routes to the Tracker in
[`main.py`]({{% githubMaster %}}/heron/tools/tracker/src/python/main.py) and
[`main.py`](https://github.com/apache/incubator-heron/tree/master/heron/tools/tracker/src/python/main.py) and
corresponding handlers in the
[`handlers`]({{% githubMaster %}}/heron/tools/tracker/src/python/handlers) directory.
[`handlers`](https://github.com/apache/incubator-heron/tree/master/heron/tools/tracker/src/python/handlers) directory.

### Heron UI

The Python code for the [Heron UI](user-manuals-heron-ui) can be found in
[`heron/tools/ui`]({{% githubMaster %}}/heron/tools/ui).
[`heron/tools/ui`](https://github.com/apache/incubator-heron/tree/master/heron/tools/ui).

Like Heron Tracker, Heron UI is a web server written in Python that relies on
the [Tornado](http://www.tornadoweb.org/en/stable/) framework. You can add new
HTTP routes to Heron UI in
[`main.py`]({{% githubMaster %}}/heron/web/source/python/main.py) and corresponding
handlers in the [`handlers`]({{% githubMaster %}}/heron/web/source/python/handlers)
[`main.py`](https://github.com/apache/incubator-heron/tree/master/heron/web/source/python/main.py) and corresponding
handlers in the [`handlers`](https://github.com/apache/incubator-heron/tree/master/heron/web/source/python/handlers)
directory.

### Heron Shell

The Python code for the [Heron Shell](user-manuals-heron-shell) can be
found in [`heron/shell`]({{% githubMaster %}}/heron/shell). The HTTP handlers and
found in [`heron/shell`](https://github.com/apache/incubator-heron/tree/master/heron/shell). The HTTP handlers and
web server are defined in
[`main.py`]({{% githubMaster %}}/heron/shell/src/python/main.py) while the HTML,
[`main.py`](https://github.com/apache/incubator-heron/tree/master/heron/shell/src/python/main.py) while the HTML,
JavaScript, CSS, and images for the web UI can be found in the
[`assets`]({{% githubMaster %}}/heron/shell/assets) directory.
[`assets`](https://github.com/apache/incubator-heron/tree/master/heron/shell/assets) directory.

## Tests

Expand Down
2 changes: 1 addition & 1 deletion website2/docs/guides-effectively-once-java-topologies.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ public class EffectivelyOnceTopology {

### Submitting the topology

The code for this topology can be found in [this GitHub repository](https://github.com/streamlio/heron-java-effectively-once-example). You can clone the repo locally like this:
The code for this topology can be found in. You can clone the repo locally like this:

```bash
$ git clone https://github.com/streamlio/heron-java-effectively-once-example
Expand Down
28 changes: 10 additions & 18 deletions website2/docs/topology-development-streamlet-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,34 +113,26 @@ $ heron submit local \

### Java Streamlet API starter project

If you'd like to up and running quickly with the Heron Streamlet API for Java, you can clone [this repository](https://github.com/streamlio/heron-java-streamlet-api-example), which includes an example topology built using the Streamlet API as well as the necessary Maven configuration. To build a JAR with dependencies of this example topology:

```bash
$ git clone https://github.com/streamlio/heron-java-streamlet-api-example
$ cd heron-java-streamlet-api-example
$ mvn assembly:assembly
$ ls target/*.jar
target/heron-java-streamlet-api-example-latest-jar-with-dependencies.jar
target/heron-java-streamlet-api-example-latest.jar
```
If you'd like to up and running quickly with the Heron Streamlet API for Java, you can view the example topologies [here](https://github.com/apache/incubator-heron/tree/{{ heron:version }}/examples/src/java/org/apache/heron/examples/streamlet)

If you're running a [local Heron cluster](getting-started-local-single-node), you can submit the built example topology like this:

```bash
$ heron submit local target/heron-java-streamlet-api-example-latest-jar-with-dependencies.jar \
io.streaml.heron.streamlet.WordCountStreamletTopology \
WordCountStreamletTopology
$ heron submit local \
~/.heron/examples/heron-streamlet-examples.jar \
org.apache.heron.examples.streamlet.WindowedWordCountTopology \
streamletWindowedWordCount
```

#### Selecting delivery semantics

Heron enables you to apply one of three [delivery semantics](heron-delivery-semantics) to any Heron topology. For the [example topology](#java-streamlet-api-starter-project) above, you can select the delivery semantics when you submit the topology with the topology's second argument. This command, for example, would apply [effectively-once](heron-delivery-semantics) to the example topology:
Heron enables you to apply one of three [delivery semantics](heron-delivery-semantics) to any Heron topology. For the example topology above, you can select the delivery semantics when you submit the topology with the topology's second argument. This command, for example, would apply [effectively-once](heron-delivery-semantics) to the example topology:

```bash
$ heron submit local target/heron-java-streamlet-api-example-latest-jar-with-dependencies.jar \
io.streaml.heron.streamlet.WordCountStreamletTopology \
WordCountStreamletTopology \
effectively-once
$ heron submit local \
~/.heron/examples/heron-streamlet-examples.jar \
org.apache.heron.examples.streamlet.WireRequestsTopology \
wireRequestsTopology
```

The other options are `at-most-once` and `at-least-once`. If you don't explicitly select the delivery semantics, at-least-once semantics will be applied.
Expand Down
Loading