NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpFields$Mutable.contains(String)" because "this._fields" is null #5908
Comments
|
as I'm looking into it, Jersey 2.x is not declared to support Jetty 12.x. The version of Jetty which is used in the Jersey 2.x is |
|
@senivam I tried to look for it, but couldn't really find it, where is the official info on which jersey goes with which jetty? regarding jetty 9, it's EOL: jetty/jetty.project#7958 for context: the attached project is a minimal reproducer for a problem that we have in our huge production system, which is complicated to migrate to other versions anyways, so would prefer moving forwards recent jetty in the long run. |
|
OK, I've investigated deeper, and it looks, the server just behaves accordingly to the reproducer. In my investigation, the reproducer actually does not work on the 9.4.x Jetty either. It does not visibly throw any exception because it's hidden behind the otherwise it would fail either. The client sends the request using and the So, the server attempts to read from the stream and drops into sleep for the |
|
Agreed, the reproducer is set up to simulate a EOF/connection close from the client, while a server thread is reading it (through the network socket). The closing of the connection by the client triggers the JettyEofExceptionMapper trying to write to the response. However this only triggers the actual problem (from how I understand the jetty team's response) that then jersey in multiple threads accesses the same reponse object without the required coordination - or completes the request and then access it again (eg reponse writing task for the EOF mapper?) - which gives the NPE because the HTTP reponse is being re-used for the next unrelated request in the queue. So the way I read it, the NPE is only a symptom from missing synchronization (or order) in accessing org.eclipse.jetty.ee8.nested.Response objects in this situation. Sorry that I cannot be more specific / have more details, I lack enough understanding of the jersey async task framework here I guess to put this into the right context |
|
So, having your explanation, I've figured out, that prior to Jetty 12.0.19 (like 12.0.18 and below) the exception is: and after - 12.0.19 (+) the exception is: Jersey is 2.46 all the way. Is there any exception you are expecting here or it should work as in Jetty 9.4.x without throwing any exception? |
|
I think any of the two exceptions is just a symptom of what joakime mentioned here, where in this jersey situation the servlet spec is violated I presume. So when jersey calls Response.setContentLength (on jetty), I guess the section "5.8 Lifetime of the Response Object" of the spec applies. I'm assuming those parts:
in the reproducer, we have started an AsyncContext, via org.glassfish.jersey.server.model.ResourceMethod.Builder#managedAsync - so this section applies. I think the complete is called already on the AsyncContext (or it's response) but we are still writing it, leading to the non-deterministic behavior = NPE or UnsupportedOperationException. So I would guess to comply with the spec, jersey would need to ensure that Response.setContentLength is not called at all if already completed. |
|
Or in other words, the reason for this could be a race condition between multiple concurrent calls to the methods mentioned servlet spec 5.7: sendError / complete / setContentLength / ... |
|
the issue is that Jersey is not trying to set content legth on some unknown response. It actually processes reponce provided by the thus it tries to set content-legth of |
|
Ah! That's where the 11 comes from in
|
|
As I'm seeing this from Jersey's point of view, it's not failure of asynchronous request processing. It's one single request / response sequence which does not handle errors properly, but it looks like that this case is on Jetty's side. Jersey just tries to use API provided by Jetty. But as I was saying from the very beginning - the 2.x version of Jersey does not support Jetty 12.0 yet. And this issue shows that it would not be an easy task to upgrade Jersey to support the latest Jetty. |
|
Just to avoid having unsupported combos, where should we be looking to stay within supported combinations? In the areas of the user guide I'm only seeing JDK compabilities. (as we were about to release actually in this combination) |
|
because it's only in the 2.x works on the 9.4.x Jetty, If the 12.0 Jetty is meant to be supporting all Jakarta's versions and other Jettys are after the EOL, we should consider updating to it in all versions of Jersey (instead of using 9.4.x and 11.0). But currently the status is that. if you are migrating to Jetty 12.0, consider migrating to Jersey 3.1 also, this is the most current version of Jersey we can provide. |
|
Thanks for that info and input! |
Under contention, when using jetty + jersey and having a resource configured with org.glassfish.jersey.server.model.ResourceMethod.Builder#managedAsync and an EOF exception mapping filter and closing many client connections this NPE can be observed regularly.
Per analysis from the jetty team, the response object seems to wrongly re-used after completion (not in line with servlet spec) of the request triggering this NPE.
version(s)
jetty 12.0.19 + jersey-bom 2.46 (same behavior in 2.34 as in attached reproducer project)
Jetty Environment
ee8
Java version/vendor
(use: java -version)openjdk 21.0.7 2025-04-15 LTS
OpenJDK Runtime Environment Temurin-21.0.7+6 (build 21.0.7+6-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.7+6 (build 21.0.7+6-LTS, mixed mode, sharing)
OS type/version
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy
Description
2025-04-30 13:36:16 GRAVE org.glassfish.jersey.server.ServerRuntime$Responder writeResponse - Error while closing the output stream in order to commit response. java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpFields$Mutable.contains(String)" because "this._fields" is null at org.eclipse.jetty.ee8.nested.Response.containsHeader(Response.java:326) at org.glassfish.jersey.servlet.internal.ResponseWriter.writeResponseStatusAndHeaders(ResponseWriter.java:135) at org.glassfish.jersey.server.ServerRuntime$Responder$1.getOutputStream(ServerRuntime.java:625) at org.glassfish.jersey.message.internal.CommittingOutputStream.commitStream(CommittingOutputStream.java:171) at org.glassfish.jersey.message.internal.CommittingOutputStream.flushBuffer(CommittingOutputStream.java:276) at org.glassfish.jersey.message.internal.CommittingOutputStream.commit(CommittingOutputStream.java:232) at org.glassfish.jersey.message.internal.CommittingOutputStream.close(CommittingOutputStream.java:247) at org.glassfish.jersey.message.internal.OutboundMessageContext.close(OutboundMessageContext.java:842) at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:389) at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:707) at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373) at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:419) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$4.run(ServerRuntime.java:872) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:889) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:867) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2$1.run(ServerRuntime.java:821) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:811) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583)in specific setup with async thread pool, jersey, and EOFMapper
How to reproduce?
see attached project. running it a few times prodcues the NPE for me, sometimes also just issue 2.
npe-report.zip
Also seen issue 2
java.lang.IllegalStateException: AsyncContext completed and/or Request lifecycle recycled
at org.eclipse.jetty.ee8.nested.AsyncContextState.state(AsyncContextState.java:42)
Also seen issue 3
java.io.IOException: content-length 11 != 0 written
at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.succeeded(HttpChannelState.java:1548)
** Also seen*
In this similar setup in our more complex application we also saw threads of the secondary thread pool (in the repro called "CUSTOM") hanging forever, that we think is related, but will try to get a separate reproducer for that. But we think it might be related / caused originally by this NPE
see attached files.
npe-report.zip
originally reported at jetty in this ticket, but redirected here
The text was updated successfully, but these errors were encountered: