Adds Elasticsearch 6.x support using Span2 model

[codefromthecrypt]

This adds Elasticsearch 6.x support via single-type indexes:

• zipkin:span-2017-08-05 - span2 (single endpoint) format
• zipkin:dependency-2017-08-05 - dependency links in existing format

This indexing model will be available in the next minor release of
Zipkin, particularly for Elasticsearch 2.4+. If you aren't running
Elasticsearch 2.4+, yet. Please upgrade.

Those wishing to experiment with this format before the next minor
release can set ES_EXPERIMENTAL_SPAN2=true to use this style now.
When set, writes will use the above scheme, but both the former and new
indexes will be read.

Fixes #1676
See #1644 for the new span2 model
See #1679 for the dual-read approach, which this is similar to

Details

This uses the span2 model from #1499 to implement Elasticsearch without using nested queries. This works with Elasticsearch 2.4+, 5.x and 6.0 alpha2.
2.4 was released Aug 2016 and works if run with ES_JAVA_OPTS=-Dmapper.allow_dots_in_name=true

The key changes are the following:

• there's only one type per index, and this is handled by a :type suffix. ex zipkin:span-2017-08-01 and zipkin:dependency-2017-08-01
• this uses the span2 model as yeah it is more effective. currently there is no "servicespan" type/index anymore
• the index naming above intentionally dodges existing collectors and indexes. we don't want to write a new type to an old index template
• for this reason, the index templates themselves are also at type-specific locations

all of the above should support running old and new collectors simultaneously eventhough the current code doesn't yet read the old stuff

cc @openzipkin/elasticsearch this resumes #1651 which I can't get to re-open

[jcarres-mdsol]

So then new spans will be stored in the span2 format? How are you planning to roll this out?

[codefromthecrypt]

we started discussing this in gitter. One of the issues we have is that ES 6 does not support multiple types. One way to roll this out is to store the new span (and dependency link) in indexes that are not the same as the old (ex with a delimited -span in the index name).

This means that an old server can still read the old data, but a new server will store and read the new format. IOTW, you'd cut traffic over, maybe leaving an old server running for a couple days with special instructions to go there.

For those who want to migrate old data to the new index space, we can write a job, maybe spark, to do that. Thoughts?

[codefromthecrypt]

ps this branch needs a bit of polish wrt ES 6

[codefromthecrypt]

another way (more work) would be to make this code support querying the old format (by issuing redundant commands for each query). This could be removed at some point in the future.

[jcarres-mdsol]

I like that the second item, maybe a configuration item so you can opt in this new format at some point when you are ready? Or things are getting too complicated?

[codefromthecrypt]

I like that the second item, maybe a configuration item so you can opt in this new format at some point when you are ready? Or things are getting too complicated?

opting into writes delays an inevitable which is that the index template will have to change else ES 6 will break. I suppose we can try to detect versions accordingly if we keep both indexing strategies..

[codefromthecrypt]

@jcarres-mdsol I don't think it will be feasible to keep the code writing both versions concurrently (even if we could maintain code to read both). It is really a lot of work and we don't have that many hands participating on this sort of thing. ES 6 will soon start hitting us, so doing this at the same time makes most sense.

If you want to remain writing the old version, you could leave a node running on a prior release and keep POST traffic going there until you are ready to cutover. Reads could use the new version regardless (once dual reads are woven back in). Is this acceptable?

[codefromthecrypt]

NEXT STEP: make this work with ES 6 w/o breaking ES 2 and 5

[codefromthecrypt]

OK the code now works with Elasticsearch 2.4+, 5.x and 6.0 alpha

2.4 was released Aug 2016 and works if run with ES_JAVA_OPTS=-Dmapper.allow_dots_in_name=true

[codefromthecrypt]

NEXT STEP: have @garyd203 or any other willing user give this branch a spin. Meanwhile see how hard it is to simultaneously support the old format (tomorrow)

[codefromthecrypt]

@jcarres-mdsol PS I think you are right about the next version using this format opt-in (via an experimental prop) and a following release default. I think I can find a way to do this neatly enough now that the index templates no longer clash with eachother. With luck can-do over the weekend.

[codefromthecrypt]

FYI I have code that works portably with the old and new format (enabled by a flag). just testing last bits of it.

[codefromthecrypt]

The new single-type indexing is setup by default when using ES 6.x. For other versions, we need to set ES_EXPERIMENTAL_SPAN2=true. When set and <6.x fan-out reads occur.

I'll be polishing this up a bit more and adding more tests.

[codefromthecrypt]

OK the work here is complete. I'll do some local ad-hoc testing prior to merge

[codefromthecrypt]

Verified all works as expected:

First, I ran with no experimental flag:

$ STORAGE_TYPE=elasticsearch java -jar ./zipkin-server/target/zipkin-server-*exec.jar

I ran the brave demo two times and verified that the old elasticsearch index format was created

[2017-08-06T12:44:17,076][INFO ][o.e.c.m.MetaDataCreateIndexService] [dZwDerT] [zipkin-2017-08-06] creating index, cause [auto(bulk api)], templates [zipkin_template], shards [5]/[1], mappings [_default_, servicespan, dependencylink, span]
[2017-08-06T12:44:17,176][INFO ][o.e.c.m.MetaDataMappingService] [dZwDerT] [zipkin-2017-08-06/VV22hH8nRqiSLnDBGE6mrQ] update_mapping [span]

Then, I started the server with the experimental flag:

$ ES_EXPERIMENTAL_SPAN2=true STORAGE_TYPE=elasticsearch java -jar ./zipkin-server/target/zipkin-server-*exec.jar

I ran the brave demo another time and verified that the new elasticsearch index format was created

[2017-08-06T12:45:10,671][INFO ][o.e.c.m.MetaDataCreateIndexService] [dZwDerT] [zipkin:span-2017-08-06] creating index, cause [auto(bulk api)], templates [zipkin:span_template], shards [5]/[1], mappings [_default_, span]
[2017-08-06T12:45:10,755][INFO ][o.e.c.m.MetaDataMappingService] [dZwDerT] [zipkin:span-2017-08-06/XnMg6KryRwK46QxZgFzALw] update_mapping [span]

I looked in the UI and verified there were three traces. Then, I stopped the one with the flag and saw two (because the new format is ignored). Next, I ran with an old version (1.24.0) and found it could read the two spans without any problem. Finally, I ran the brave demo with the old version (1.24.0) to make sure it could write properly.

@jcarres-mdsol everything's now ready for opt-in. Unless you are running 6.x or set ES_EXPERIMENTAL_SPAN2=true, there's no difference in behavior between old code and new.

[codefromthecrypt]

Travis is failing to start one of the docker test containers, which leads to a 10m timeout. It might be a resource issue. I will try to figure it out.

[codefromthecrypt]

fixed the build. I lowered the heap to 512m and also added some extra stuff when ES_DEBUG=true to help the next person

[codefromthecrypt]

I don't expect anyone to go through the 2100 lines needed to get this to work as opt-in (albeit mostly copy/paste tests).

I'm going to merge this for a patch release, so that folks can experiment with it.

[jcarres-mdsol]

Woah, that's great, thanks!

[codefromthecrypt]

PS I will be updating zipkin-dependencies after I finish testing that it can write both index types properly.

[xeraa]

Nice work 👍

[codefromthecrypt]

Using the docker-zipkin commands with a slightly revised file like so:

diff --git a/docker-compose-elasticsearch.yml b/docker-compose-elasticsearch.yml
index 0bbd62b..02d5e83 100644
--- a/docker-compose-elasticsearch.yml
+++ b/docker-compose-elasticsearch.yml
@@ -9,17 +9,18 @@ version: '2'
 services:
   # Run Elasticsearch instead of MySQL
   storage:
-    image: openzipkin/zipkin-elasticsearch
+    image: openzipkin/zipkin-elasticsearch5
     container_name: elasticsearch
     # Uncomment to expose the storage port for testing
-    # ports:
-    #   - 9200:9200
+    ports:
+      - 9200:9200

   # Switch storage type to Elasticsearch
   zipkin:
     image: openzipkin/zipkin
     environment:
       - STORAGE_TYPE=elasticsearch
+      - ES_EXPERIMENTAL_SPAN2=true
       # Point the zipkin at the storage backend
       - ES_HOSTS=http://elasticsearch:9200
   dependencies:

A query directly to ES for errors is ... super easy now! No more nested queries!! cc @openzipkin/elasticsearch @kellabyte

$ curl -s '192.168.99.100:9200/zipkin:span-*/_search?q=_exists_:tags.error'

[codefromthecrypt]

Here's an example with duration.. ex finding spans that are error and over a 100ms (remember zipkin time units are microseconds):

curl -s '192.168.99.100:9200/zipkin:*/_search?default_operator=AND&q=_exists_:tags.error+duration:>100000'|jq '.hits.hits[]._source'
{
  "timestamp_millis": 1502101460678,
  "traceId": "9032b04972e475c5",
  "id": "9032b04972e475c5",
  "kind": "SERVER",
  "name": "get",
  "timestamp": 1502101460678880,
  "duration": 612898,
  "localEndpoint": {
    "serviceName": "brave-webmvc-example",
    "ipv4": "192.168.1.113"
  },
  "remoteEndpoint": {
    "serviceName": "",
    "ipv4": "127.0.0.1",
    "port": 60149
  },
  "tags": {
    "error": "500 Internal Server Error",
    "http.path": "/a"
  }
}

[codefromthecrypt]

@openzipkin/elasticsearch @SirTyro For lack of feedback, I didn't switch single-type indexing to default in 1.30 as I said I would. Current release is 1.30.3

I plan to set this on by default in the next release, which will be 1.31. The dual-prong reads will remain, meaning you can read both index formats, but writes will change to the new format as planned. Writes would act the same as if you set ES_EXPERIMENTAL_SPAN2=true today.

This has to happen because we cannot afford dual maintenance, and I need to complete other work to finish the span2 migration. The last mile work is technically much harder if we don't know which format elasticsearch will write until runtime.

[codefromthecrypt]

switching in 1.31 via #1698