v4.0.2-mocha API returns errors on some requests

[itrocket-team]

Summary of Bug

In the new version, we can't see transaction details via the API.
For example: https://celestia-testnet-api.itrocket.net/cosmos/tx/v1beta1/txs/8715C3C7FC1583908AA40D60B84DB3C9021AD8E2F11A301D1C461849857D4042

{
  "code": 3,
  "message": "error in json rpc client, with http response metadata: (Status: 200 OK, Protocol HTTP/1.1). error unmarshalling result: illegal base64 data at input byte 4: invalid request",
  "details": [
  ]
}

Version

cosmos_sdk_version: v1.29.1-sdk-v0.50.12
go: go version go1.24.1 linux/amd64
name: celestia-app
server_name: celestia-appd
version: 4.0.2-mocha

Steps to Reproduce

https://celestia-testnet-api.itrocket.net/cosmos/tx/v1beta1/txs/8715C3C7FC1583908AA40D60B84DB3C9021AD8E2F11A301D1C461849857D4042

[mindstyle85]

@evan-forbes @rootulp please take a look at this

[rootulp]

Also reported by BitGo

curl --location 'https://api-mocha.pops.one/cosmos/tx/v1beta1/txs/13E065E855FE36DB2E6E36F67BC14C308F5DC8D751B929090305121E7186886F'

{
    "code": 3,
    "message": "error in json rpc client, with http response metadata: (Status: 200 OK, Protocol HTTP/1.1). error unmarshalling result: illegal base64 data at input byte 4: invalid request",
    "details": []
}

[evan-forbes]

interesting, I can't replicate this in testnode or when running a node locally when running v4, so that might mean this issue has to do with running the v3 binary via the multiplexer or that we're missing some config, since the multiplexer doesn't start the web api/servers there

it leaves that to the embedded application

celestia-app/multiplexer/abci/multiplexer.go

Line 155 in 5d4d979

m.logger.Debug("using embedded app, not continuing with grpc or api servers")

[rootulp]

so that might mean this issue has to do with running the v3 binary via the multiplexer

100% agree. It may help debugging to run the v3 binary via the multiplexer. We have two scripts that may help:

./scripts/block-sync.sh --network mocha

./scripts/mocha.sh