401 error when network hiccups

[GRRedWings]

I'm trying to track down why occasionally I get a 401 error .

Here is the configuration in Apache, it has a

<Location /someLocation>
Require valid-user
AuthType oauth2
OAuth2TokenVerify jwk "..." verify.iat=skip&expiry=5

This is the apache error I get
172.24.0.1 - FLCm7KqMmt-CnMOKmf1EpJYd5wbPUdJ8ILQPO_TT09Q [23/Aug/2021:16:20:40 +0000] "POST /someLocation/resources/vx.y/doIt HTTP/1.1" 200 186
double free or corruption (fasttop)
172.24.0.1 - - [23/Aug/2021:16:20:43 +0000] "POST /someLocation/resources/vx.y/doIt HTTP/1.1" 401 381
[Mon Aug 23 16:20:43.903993 2021] [core:notice] [pid 1:tid 139697330451584] AH00052: child pid 93 exit signal Aborted (6)

Am I reading this right that some sort of partial request is being processed and causing a memory error?

[zandbelt]

that looks like a memory corruption indeed, can you confirm you're using the latest version of module+library and are you able to produce a coredump+stacktrace?

[GRRedWings]

I admit that the version is a few months old. Probably Jan/Feb vintage. Do you know offhand how to produce those coredump/stacktarce? I can try to figure it out. I'm running this in a docker image.

[zandbelt]

ok, just send me the Dockerfile and I'll figure it out

[GRRedWings]

I'm going to try to update the versions of your libraries and Apache. but here is the dockerfile I'm running with.

FROM httpd:2.4

# Change this from a copy, to me having the deb files for both
RUN mkdir -p /tmp

COPY libhiredis0.13_0.13.3-2.2_amd64.deb /tmp/libhiredis0.13_0.13.3-2.2_amd64.deb
COPY libcjose0_0.6.1.5-1_bionic+1_amd64.deb /tmp/libcjose0_0.6.1.5-1_bionic+1_amd64.deb
COPY liboauth2_1.4.0.1-1.bionic+1_amd64.deb /tmp/liboauth2_1.4.0.1-1.bionic+1_amd64.deb
COPY liboauth2-apache_1.4.0.1-1.bionic+1_amd64.deb /tmp/liboauth2-apache_1.4.0.1-1.bionic+1_amd64.deb
COPY libapache2-mod-oauth2_3.2.0-1.bionic+1_amd64.deb /tmp/libapache2-mod-oauth2_3.2.0-1.bionic+1_amd64.deb
COPY libapache2-mod-auth-openidc_2.4.5-1.bionic+1_amd64.deb /tmp/libapache2-mod-auth-openidc_2.4.5-1.bionic+1_amd64.deb

RUN apt-get update && \
#    apt-get install -y ca-certificates libssl-dev check libjansson-dev libz-dev apache2 libmemcached-dev
    apt-get install -y ca-certificates apache2 libmemcached-dev

# Install the packages we just copied in
run dpkg -i /tmp/libhiredis0.13_0.13.3-2.2_amd64.deb
run dpkg -i /tmp/libcjose0_0.6.1.5-1_bionic+1_amd64.deb
run dpkg -i /tmp/liboauth2_1.4.0.1-1.bionic+1_amd64.deb
run dpkg -i /tmp/liboauth2-apache_1.4.0.1-1.bionic+1_amd64.deb
run dpkg -i /tmp/libapache2-mod-oauth2_3.2.0-1.bionic+1_amd64.deb
run dpkg -i /tmp/libapache2-mod-auth-openidc_2.4.5-1.bionic+1_amd64.deb

# Modify the httpd.conf to load modules we use, and tell it to use our custom.conf
RUN echo "LoadModule proxy_module modules/mod_proxy.so" >> /usr/local/apache2/conf/httpd.conf
RUN echo "LoadModule proxy_http_module modules/mod_proxy_http.so" >> /usr/local/apache2/conf/httpd.conf
RUN echo "LoadModule ssl_module modules/mod_ssl.so" >> /usr/local/apache2/conf/httpd.conf
RUN echo "LoadModule oauth2_module /usr/lib/apache2/modules/mod_oauth2.so" >> /usr/local/apache2/conf/httpd.conf
RUN echo "LoadModule auth_openidc_module /usr/lib/apache2/modules/mod_auth_openidc.so" >> /usr/local/apache2/conf/httpd.conf


RUN mkdir /usr/local/apache2/conf-available
RUN touch /usr/local/apache2/conf-available/custom.conf
RUN echo "Include conf-available/custom.conf" >> /usr/local/apache2/conf/httpd.conf

[GRRedWings]

172.27.0.1 - - [23/Aug/2021:18:16:15 +0000] "POST /someLocation/resources/x.y/doIt HTTP/1.1" 401 381
double free or corruption (fasttop)
172.27.0.1 - FLCm7KqMmt-CnMOKmf1EpJYd5wbPUdJ8ILQPO_TT09Q [23/Aug/2021:18:16:15 +0000] "POST /someLocation/resources/x.y/alsoDoIt/F443EE26 HTTP/1.1" 200 186
[Mon Aug 23 18:16:16.117599 2021] [core:notice] [pid 1:tid 140301462574208] AH00052: child pid 93 exit signal Aborted (6)

I think I upgraded to the lastest version of your libraries. And still got the error. I seem to get it in conjunctions with my Android device having a network blip.

FROM httpd:2.4

RUN mkdir -p /tmp

COPY libhiredis0.13_0.13.3-2.2_amd64.deb /tmp/libhiredis0.13_0.13.3-2.2_amd64.deb
COPY libcjose0_0.6.1.5-1_bionic+1_amd64.deb /tmp/libcjose0_0.6.1.5-1_bionic+1_amd64.deb
COPY liboauth2_1.4.3-1.bionic+1_amd64.deb /tmp/liboauth2_1.4.3-1.bionic+1_amd64.deb
COPY liboauth2-apache_1.4.3-1.bionic+1_amd64.deb /tmp/liboauth2-apache_1.4.3-1.bionic+1_amd64.deb
COPY libapache2-mod-oauth2_3.2.0-1.bionic+1_amd64.deb /tmp/libapache2-mod-oauth2_3.2.0-1.bionic+1_amd64.deb
COPY libapache2-mod-auth-openidc_2.4.9.2-1.bionic+1_amd64.deb /tmp/libapache2-mod-auth-openidc_2.4.9.2-1.bionic+1_amd64.deb

RUN apt-get update &&
apt-get install -y ca-certificates apache2 libmemcached-dev

run dpkg -i /tmp/libhiredis0.13_0.13.3-2.2_amd64.deb
run dpkg -i /tmp/libcjose0_0.6.1.5-1_bionic+1_amd64.deb
run dpkg -i /tmp/liboauth2_1.4.3-1.bionic+1_amd64.deb
run dpkg -i /tmp/liboauth2-apache_1.4.3-1.bionic+1_amd64.deb
run dpkg -i /tmp/libapache2-mod-oauth2_3.2.0-1.bionic+1_amd64.deb
run dpkg -i /tmp/libapache2-mod-auth-openidc_2.4.9.2-1.bionic+1_amd64.deb

RUN echo "LoadModule proxy_module modules/mod_proxy.so" >> /usr/local/apache2/conf/httpd.conf
RUN echo "LoadModule proxy_http_module modules/mod_proxy_http.so" >> /usr/local/apache2/conf/httpd.conf
RUN echo "LoadModule ssl_module modules/mod_ssl.so" >> /usr/local/apache2/conf/httpd.conf
RUN echo "LoadModule oauth2_module /usr/lib/apache2/modules/mod_oauth2.so" >> /usr/local/apache2/conf/httpd.conf
RUN echo "LoadModule auth_openidc_module /usr/lib/apache2/modules/mod_auth_openidc.so" >> /usr/local/apache2/conf/httpd.conf

RUN mkdir /usr/local/apache2/conf-available
RUN touch /usr/local/apache2/conf-available/custom.conf
RUN echo "Include conf-available/custom.conf" >> /usr/local/apache2/conf/httpd.conf

[zandbelt]

I'm not able to reproduce: can you set the log level to debug and send me the error_log?

[GRRedWings]

What is the best way to send you the error log? with a link? Can I email it to you anywhere?

[zandbelt]

you can send it to support@zmartzone.eu

[GRRedWings]

Just checking in. Did you receive my email and logs?

[zandbelt]

I did but unfortunately it did not help me to reproduce so I have nothing to work from :-(

[GRRedWings]

Any chance there are newer versions of these libraries and I'm just out of date?

COPY libhiredis0.13_0.13.3-2.2_amd64.deb /tmp/libhiredis0.13_0.13.3-2.2_amd64.deb
COPY libcjose0_0.6.1.5-1_bionic+1_amd64.deb /tmp/libcjose0_0.6.1.5-1_bionic+1_amd64.deb
COPY libapache2-mod-oauth2_3.2.0-1.bionic+1_amd64.deb /tmp/libapache2-mod-oauth2_3.2.0-1.bionic+1_amd64.deb

[zandbelt]

I tried to reproduce with your Dockerfile+versions, but no luck

[GRRedWings]

Also to be sure. I feel that the issue is with our Android application (using OkHttp), and if the network blips on the Android device, that's when I get the 401. It also seems like for some reason that there is some sort of timing. I'm going to write some other tests to see if I can repeat this easier.

[GRRedWings]

I've spent some more time trying to re-produce or find out what is going on. Based on your chance to look at the logs, do you have any clues on what is going on? Is it bad or partial data coming in? Is there something in the module that might be a problem? I'm kinda at a loss of where to dig.

[zandbelt]

I'm lost too; I would assume that partial data would lead to other, Apache-level errors