Skip to content

[JEP-210] Log handling rewrite #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 71 commits into from
Oct 4, 2018
Merged
Show file tree
Hide file tree
Changes from 69 commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
c229421
[JENKINS-38381] Prototype of log handling rewrite.
jglick Sep 23, 2016
754ab9e
Pick up https://github.com/jenkinsci/maven-hpi-plugin/pull/25.
jglick Sep 26, 2016
e3ccb7b
Updated dependency versions.
jglick Sep 26, 2016
721bc75
Obsolete Javadoc comment.
jglick Sep 26, 2016
30d64c7
Reproduced AssociatedNodeNote error in a test.
jglick Sep 26, 2016
78c5fbf
Another snapshot version forgotten.
jglick Sep 26, 2016
875b3ed
Lighter-weight way of marking which node printed a line.
jglick Sep 26, 2016
f2f1296
FindBugs
jglick Sep 27, 2016
141d748
Updated TODOs.
jglick Sep 27, 2016
19530bd
Extracting createBuildListener.
jglick Sep 30, 2016
4579f77
2.16 parent
jglick Oct 3, 2016
6f1ab49
Prototype SPI for replacing the log file of a build.
jglick Oct 5, 2016
e57d7c6
Deleting comment implemented as https://github.com/jenkinsci/jenkins-…
jglick Oct 6, 2016
7db5d9d
Faulty _getLogInputStream caused StackOverflowError’s in the normal c…
jglick Oct 6, 2016
e4968ef
Pending https://github.com/jenkinsci/jenkins-test-harness/pull/38 avo…
jglick Oct 7, 2016
e6a58e8
Reimplemented HTML display of Pipeline console to include some featur…
jglick Oct 10, 2016
f46b98c
FindBugs
jglick Oct 11, 2016
62ccf49
Hide the show/hide link except during hover.
jglick Oct 12, 2016
51fdc25
Do not hide a “show” link.
jglick Oct 12, 2016
d34d03e
[JENKINS-27394] Better implementation of collapsible sections, finall…
jglick Oct 12, 2016
aa12940
Discovered a problem with NewNodeConsoleNote and LabelAction.
jglick Oct 12, 2016
08268a3
Found a fix for the LabelAction timing problem.
jglick Oct 12, 2016
d2a39dd
Fixed a bug in enclosing logic.
jglick Oct 12, 2016
89559d5
Noting LessAbstractTaskListener.
jglick Oct 14, 2016
211bd14
Merge branch 'master' into logs-JENKINS-38381
jglick Dec 7, 2016
72d29d0
hpi-plugin.version=1.121
jglick Dec 16, 2016
2de9add
getLog should take a long start argument.
jglick Dec 21, 2016
b0c6369
Need https://github.com/jenkinsci/maven-hpi-plugin/pull/48 to get hpi…
jglick Dec 21, 2016
0f511fa
Giving up on jitpack for now.
jglick Jan 3, 2017
c88e980
Added a crude scalability test.
jglick Jan 3, 2017
a0e766b
Stronger assertions.
jglick Jan 4, 2017
c7b8610
Adapted test to annotateHtml coalescing change.
jglick Jan 6, 2017
33e9151
Switched to enclosingId to simplify JavaScript.
jglick Jan 7, 2017
a8ea094
Adapted test to new behavior of NewNodeConsoleNote.
jglick Jan 7, 2017
24266f8
Merge branch 'master' into logs-JENKINS-38381
jglick Feb 5, 2018
59ba822
Minor build errors.
jglick Feb 5, 2018
2ecc99f
Merge branch 'UTF-8-JENKINS-31096' into logs-JENKINS-38381
jglick Feb 13, 2018
1c75a1c
Merge branch 'UTF-8-JENKINS-31096' into logs-JENKINS-38381
jglick Jun 11, 2018
4c8c8ab
Skipping an expensive test on Windows which currently fails anyway.
jglick Jun 11, 2018
a01addc
Use the latest LTS as a baseline.
jglick Jun 11, 2018
a86881f
TestUnit2 unnecessarily restricted.
jglick Jun 11, 2018
18d78f3
Race condition finishing build caught by WorkflowRunRestartTest.lazyL…
jglick Jun 11, 2018
ab259f8
Merge branch 'UTF-8-JENKINS-31096' into logs-JENKINS-38381
jglick Jun 15, 2018
976c775
Apparent race condition in NewNodeConsoleNote sometimes resulting in …
jglick Jun 15, 2018
216223d
Removed reference to obsolete LessAbstractTaskListener.
jglick Jun 18, 2018
c2f2a8f
Passing complete flag to getLog.
jglick Jun 18, 2018
46bf528
Refactored to LogStorage.
jglick Jun 20, 2018
39779fc
Bumps.
jglick Jun 20, 2018
ae2b07e
Javadoc fix.
jglick Jun 21, 2018
5616213
FindBugs
jglick Jun 21, 2018
d41f02f
Fixing deprecation warnings in test.
jglick Jun 25, 2018
9c48541
Merge branch 'master' into logs-JENKINS-38381
jglick Jul 24, 2018
4c76b65
Merge branch 'UTF-8-JENKINS-31096' into logs-JENKINS-38381
jglick Aug 6, 2018
49c91cd
Merge branch 'master' into logs-JENKINS-38381
jglick Aug 7, 2018
096aeab
Adapted to FileLogStorage.
jglick Aug 13, 2018
d450172
Replacing deprecated FlowScanningUtils.fetchEnclosingBlocks with Flow…
jglick Aug 15, 2018
8873326
Found a couple of places where WorkflowRun.listener was being imprope…
jglick Aug 15, 2018
acd07c0
Updated comments.
jglick Aug 15, 2018
3474475
Nulling out listener after close, and expanding upon an earlier comme…
jglick Aug 15, 2018
17a0d2a
Logging a condition which is probably impossible.
jglick Aug 17, 2018
0a39b1c
Showing that it is quick (0ms for me) to display short node log text …
jglick Aug 23, 2018
6eac61d
Merge branch 'master' into logs-JENKINS-38381
jglick Sep 4, 2018
6f5d227
Apparently flaky tests.
jglick Sep 4, 2018
2aa0348
Obsolete comment.
jglick Sep 5, 2018
ae21cee
Demonstrating that a JEP-210 upgrade preserves both whole-build and s…
jglick Sep 6, 2018
5854f54
Avoid duplicated labels on blocks.
jglick Sep 7, 2018
e830112
Merge branch 'master' into logs-JENKINS-38381
jglick Sep 10, 2018
f9325b5
Bumps.
jglick Sep 24, 2018
46f8333
Under certain conditions behaviors can be repeatedly applied, so they…
jglick Sep 24, 2018
9e07b63
Call AsynchronousExecution.completed even if saving build.xml fails. …
jglick Oct 1, 2018
5623f82
compatibleSinceVersion=2.26
jglick Oct 4, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
<parent>
<groupId>org.jenkins-ci.plugins</groupId>
<artifactId>plugin</artifactId>
<version>3.19</version>
<version>3.23</version>
<relativePath />
</parent>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
Expand Down Expand Up @@ -64,10 +64,11 @@
<properties>
<revision>2.26</revision>
<changelist>-SNAPSHOT</changelist>
<jenkins.version>2.73.3</jenkins.version>
<jenkins.version>2.121.1</jenkins.version>
<java.level>8</java.level>
<no-test-jar>false</no-test-jar>
<workflow-support-plugin.version>2.20</workflow-support-plugin.version>
<useBeta>true</useBeta>
<workflow-support-plugin.version>2.21-rc629.b249afdd9a58</workflow-support-plugin.version> <!-- TODO https://github.com/jenkinsci/workflow-support-plugin/pull/15 -->
<scm-api-plugin.version>2.2.6</scm-api-plugin.version>
<git-plugin.version>3.2.0</git-plugin.version>
<jenkins-test-harness.version>2.33</jenkins-test-harness.version>
Expand All @@ -81,7 +82,7 @@
<dependency>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
<artifactId>workflow-api</artifactId>
<version>2.27</version>
<version>2.30-rc820.ba604ce51277</version> <!-- TODO https://github.com/jenkinsci/workflow-api-plugin/pull/17 -->
</dependency>
<dependency>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
Expand Down Expand Up @@ -120,6 +121,12 @@
<classifier>tests</classifier>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jenkins-ci.plugins</groupId>
<artifactId>pipeline-stage-step</artifactId>
<version>2.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jenkins-ci.plugins.workflow</groupId>
<artifactId>workflow-scm-step</artifactId>
Expand Down
376 changes: 109 additions & 267 deletions src/main/java/org/jenkinsci/plugins/workflow/job/WorkflowRun.java

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
/*
* The MIT License
*
* Copyright (c) 2015, CloudBees, Inc.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/

package org.jenkinsci.plugins.workflow.job.console;

import hudson.Extension;
import hudson.MarkupText;
import hudson.Util;
import hudson.console.ConsoleAnnotationDescriptor;
import hudson.console.ConsoleAnnotator;
import hudson.console.ConsoleNote;
import hudson.model.TaskListener;
import java.io.IOException;
import java.io.PrintStream;
import java.util.Iterator;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.annotation.CheckForNull;
import javax.annotation.Nonnull;
import org.jenkinsci.plugins.workflow.actions.LabelAction;
import org.jenkinsci.plugins.workflow.flow.FlowExecution;
import org.jenkinsci.plugins.workflow.graph.BlockEndNode;
import org.jenkinsci.plugins.workflow.graph.BlockStartNode;
import org.jenkinsci.plugins.workflow.graph.FlowNode;
import org.jenkinsci.plugins.workflow.job.WorkflowRun;
import org.jenkinsci.plugins.workflow.log.LogStorage;
import org.kohsuke.accmod.Restricted;
import org.kohsuke.accmod.restrictions.NoExternalUse;

/**
* Console line with note printed when a new {@link FlowNode} is added to the graph.
* Defines the {@code pipeline-new-node} CSS class and several attributes which may be used to control subsequent behavior:
* <ul>
* <li>{@code nodeId} for {@link FlowNode#getId}
* <li>{@code startId} {@link FlowNode#getId} for {@link BlockStartNode}, else {@link BlockEndNode#getStartNode}, else absent
* <li>{@code enclosingId} the immediately enclosing {@link BlockStartNode}, if any
* <li>{@code label} for {@link LabelAction} if present
* </ul>
* @see LogStorage#startStep
*/
@Restricted(NoExternalUse.class)
public class NewNodeConsoleNote extends ConsoleNote<WorkflowRun> {

private static final Logger LOGGER = Logger.getLogger(NewNodeConsoleNote.class.getName());

/**
* Prefix used in metadata lines.
*/
private static final String CONSOLE_NOTE_PREFIX = "[Pipeline] ";

public static void print(FlowNode node, TaskListener listener) {
PrintStream logger = listener.getLogger();
synchronized (logger) {
try {
listener.annotate(new NewNodeConsoleNote(node));
} catch (IOException x) {
LOGGER.log(Level.WARNING, null, x);
}
logger.println(CONSOLE_NOTE_PREFIX + node.getDisplayFunctionName()); // note that StepAtomNode will never have a LabelAction at this point
}
}

private final @Nonnull String id;
private final @CheckForNull String enclosing;
private final @CheckForNull String start;

private NewNodeConsoleNote(FlowNode node) {
id = node.getId();
if (node instanceof BlockEndNode) {
enclosing = null;
start = ((BlockEndNode) node).getStartNode().getId();
} else {
Iterator<BlockStartNode> it = node.iterateEnclosingBlocks().iterator();
enclosing = it.hasNext() ? it.next().getId() : null;
start = node instanceof BlockStartNode ? node.getId() : null;
}
}

@Override
public ConsoleAnnotator<?> annotate(WorkflowRun context, MarkupText text, int charPos) {
StringBuilder startTag = new StringBuilder("<span class=\"pipeline-new-node\" nodeId=\"").append(id);
if (start != null) {
startTag.append("\" startId=\"").append(start);
}
if (enclosing != null) {
startTag.append("\" enclosingId=\"").append(enclosing);
}
FlowExecution execution = context.getExecution();
if (execution != null) {
try {
FlowNode node = execution.getNode(id);
if (node != null) {
LabelAction a = node.getAction(LabelAction.class);
if (a != null) {
String displayName = a.getDisplayName();
assert displayName != null;
startTag.append("\" label=\"").append(Util.escape(displayName)); // TODO is there some better way to escape for attribute values?
Copy link
Member

@svanoort svanoort Feb 27, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noting here for future review/integration that this appears to be how we propose to deal with rendering of branch labels -- however it requires rich rendering of annotation content by the UI to be useful (i.e. not visible in plaintext logs).

Is that the right decision? Not sure, the mechanism seems more straightforward in some ways.

Edit: note that the other approach (prefixes) also required some rendering but didn't rely on the browser JS as much. I think this one seems somewhat more sane though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, this keeps the HTML leaner and does more of the fanciness on the client side.

}
}
} catch (IOException x) {
Logger.getLogger(NewNodeConsoleNote.class.getName()).log(Level.WARNING, null, x);
}
}
startTag.append("\">");
text.addMarkup(0, text.length(), startTag.toString(), "</span>");
return null;
}

private static final long serialVersionUID = 1L;

@Extension public static final class DescriptorImpl extends ConsoleAnnotationDescriptor {}

}

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,10 @@
import hudson.model.Run;

/**
* Console note for Workflow metadata specific messages.
* See {@link WorkflowConsoleLogger} for more information.
* @deprecated No longer used, but retained for serial-form compatibility of old build logs.
* @see NewNodeConsoleNote
*/
@Deprecated
public class WorkflowRunConsoleNote extends ConsoleNote<Run<?, ?>> {

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,17 @@
<l:task icon="images/24x24/up.png" href="${rootURL}/${it.parent.url}" title="${%Back to Project}" contextMenu="false"/>
<l:task icon="images/24x24/search.png" href="${buildUrl.baseUrl}/" title="${%Status}" contextMenu="false"/>
<l:task icon="images/24x24/notepad.png" href="${buildUrl.baseUrl}/changes" title="${%Changes}"/>
<p:console-link/>
<j:choose> <!-- TODO <p:console-link/> may not currently be used as it calls getLogFile -->
<j:when test="${it.logText.length() > 200000}">
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps LogStorage needs a new API like

default int approximateLogLines(@Nonnull FlowExecutionOwner.Executable build) {
    return overallLog(build, false).length() / 80;
}

which could be overridden by external implementations to make an API call.

Of course there is the broader goal of JENKINS-17406 to use paginated display.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specifically:

  • console-link.jelly uses it.logFile.length(), which for WorkflowRun now prints a warning—very bad. Thus we specifically avoid this control until core is fixed. Fortunately it is not much to copy.
  • console.jelly uses it.logText.length(), which is fine for the built-in FileLogStorage: boils down to a single File.length() call after verifying that the first few bytes of the file are not a gzip magic header.
  • For pipeline-log-fluentd-cloudwatch and the like, this is very inefficient, and we would prefer an API for an approximation. (On reflection this need not be in LogStorage; could be in LargeText or AnnotatedLargeText.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ick, well okay, I'll let you deal with what that does to the fluentd-cloudwatch logging.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs to be revisited when resuming work on external log sinks. I am no longer working on that.

<l:task href="${buildUrl.baseUrl}/console" icon="icon-terminal icon-md" title="${%Console Output}"/>
<l:task href="${buildUrl.baseUrl}/consoleText" icon="icon-document icon-md" title="${%View as plain text}"/>
</j:when>
<j:otherwise>
<l:task icon="images/24x24/terminal.png" href="${buildUrl.baseUrl}/console" title="${%Console Output}">
<l:task href="${buildUrl.baseUrl}/consoleText" icon="icon-document icon-md" title="${%View as plain text}"/>
</l:task>
</j:otherwise>
</j:choose>
<l:task icon="images/24x24/notepad.png" href="${buildUrl.baseUrl}/configure" title="${h.hasPermission(it,it.UPDATE)?'%Edit Build Information':'%View Build Information'}"/>
<st:include page="delete.jelly"/>
<j:if test="${it.hasAllowTerm()}">
Expand Down
Loading