Requester pay bucket cannot be mounted even with billing project provided

[shengqh]

Describe the issue
I created a GCP VM, installed most recent gcsfuse, ran gcloud init to set up my google account, my project and time zone. I can use gsutil to access my requester pay bucket. However, gcsfuse didn't work even with billing project provided. I saw "GcsConnection":{"BillingProject":"vangard-workflow-data"} in log but don't know why the gcsfuse didn't use that information. The requester pay bucket is in another project, not in vangard-workflow-data.

System & Version (please complete the following information):

• OS: Debian GNU/Linux 12
• Platform: GCE VM
• Version: gcsfuse version 2.11.1 (Go version go1.24.0)

Steps to reproduce the behavior with following information:

quanhu_sheng_1@sqh1:~$ gcsfuse --log-severity=TRACE --foreground --billing-project=vangard-workflow-data --implicit-dirs working-set-redeposit ~/mnt/biovu
{"timestamp":{"seconds":1742790945,"nanos":175878608},"severity":"INFO","message":"Start gcsfuse/2.11.1 (Go version go1.24.0) for app "" using mount point: /home/quanhu_sheng_1/mnt/biovu\n"}
{"timestamp":{"seconds":1742790945,"nanos":175936242},"severity":"INFO","message":"GCSFuse config","config":{"AppName":"","CacheDir":"","Debug":{"ExitOnInvariantViolation":false,"Fuse":false,"Gcs":false,"LogMutex":false},"EnableAtomicRenameObject":false,"EnableHns":true,"FileCache":{"CacheFileForRangeRead":false,"DownloadChunkSizeMb":50,"EnableCrc":false,"EnableODirect":false,"EnableParallelDownloads":false,"ExperimentalParallelDownloadsDefaultOn":false,"MaxParallelDownloads":16,"MaxSizeMb":-1,"ParallelDownloadsPerFile":16,"WriteBufferSize":4194304},"FileSystem":{"DirMode":"755","DisableParallelDirops":false,"FileMode":"644","FuseOptions":[],"Gid":-1,"HandleSigterm":true,"IgnoreInterrupts":true,"KernelListCacheTtlSecs":0,"PreconditionErrors":true,"RenameDirLimit":0,"TempDir":"","Uid":-1},"Foreground":true,"GcsAuth":{"AnonymousAccess":false,"KeyFile":"","ReuseTokenFromUrl":true,"TokenUrl":""},"GcsConnection":{"BillingProject":"vangard-workflow-data","ClientProtocol":"http1","CustomEndpoint":"","ExperimentalEnableJsonRead":false,"GrpcConnPoolSize":1,"HttpClientTimeout":0,"LimitBytesPerSec":-1,"LimitOpsPerSec":-1,"MaxConnsPerHost":0,"MaxIdleConnsPerHost":100,"SequentialReadSizeMb":200},"GcsRetries":{"ChunkTransferTimeoutSecs":10,"MaxRetryAttempts":0,"MaxRetrySleep":30000000000,"Multiplier":2,"ReadStall":{"Enable":false,"InitialReqTimeout":20000000000,"MaxReqTimeout":1200000000000,"MinReqTimeout":1500000000,"ReqIncreaseRate":15,"ReqTargetPercentile":0.99}},"ImplicitDirs":true,"List":{"EnableEmptyManagedFolders":false},"Logging":{"FilePath":"","Format":"json","LogRotate":{"BackupFileCount":10,"Compress":true,"MaxFileSizeMb":512},"Severity":"TRACE"},"MetadataCache":{"DeprecatedStatCacheCapacity":20460,"DeprecatedStatCacheTtl":60000000000,"DeprecatedTypeCacheTtl":60000000000,"EnableNonexistentTypeCache":false,"ExperimentalMetadataPrefetchOnMount":"disabled","NegativeTtlSecs":5,"StatCacheMaxSizeMb":32,"TtlSecs":60,"TypeCacheMaxSizeMb":4},"Metrics":{"CloudMetricsExportIntervalSecs":0,"EnableOtel":true,"PrometheusPort":0,"StackdriverExportInterval":0},"Monitoring":{"ExperimentalOpentelemetryCollectorAddress":"","ExperimentalTracingMode":"","ExperimentalTracingSamplingRatio":0},"OnlyDir":"","Write":{"BlockSizeMb":33554432,"CreateEmptyFile":false,"EnableStreamingWrites":false,"GlobalMaxBlocks":9223372036854775807,"MaxBlocksPerFile":1}}}
{"timestamp":{"seconds":1742790945,"nanos":176558766},"severity":"INFO","message":"Creating Storage handle..."}
{"timestamp":{"seconds":1742790945,"nanos":176583002},"severity":"INFO","message":"UserAgent = gcsfuse/2.11.1 (Go version go1.24.0) (GPN:gcsfuse) (Cfg:0:0:0:0)\n"}
{"timestamp":{"seconds":1742790945,"nanos":177724704},"severity":"INFO","message":"Creating a mount at "/home/quanhu_sheng_1/mnt/biovu"\n"}
{"timestamp":{"seconds":1742790945,"nanos":177778660},"severity":"INFO","message":"Creating a new server...\n"}
{"timestamp":{"seconds":1742790945,"nanos":177809460},"severity":"INFO","message":"Set up root directory for bucket working-set-redeposit"}
{"timestamp":{"seconds":1742790945,"nanos":177833562},"severity":"INFO","message":"GetStorageLayout <- (working-set-redeposit)"}
{"timestamp":{"seconds":1742790945,"nanos":419037245},"severity":"ERROR","message":"Error while mounting gcsfuse: mountWithStorageHandle: fs.NewServer: create file system: SetUpBucket: BucketHandle: storageLayout call failed: rpc error: code = InvalidArgument desc = Bucket is a requester pays bucket but no user project provided.\nerror details: name = BadRequest field = desc = Bucket is a requester pays bucket but no user project provided.\n"}
Error: mountWithStorageHandle: fs.NewServer: create file system: SetUpBucket: BucketHandle: storageLayout call failed: rpc error: code = InvalidArgument desc = Bucket is a requester pays bucket but no user project provided.
error details: name = BadRequest field = desc = Bucket is a requester pays bucket but no user project provided.
{"timestamp":{"seconds":1742790945,"nanos":419166712},"severity":"INFO","message":"Error occurred during command execution: mountWithStorageHandle: fs.NewServer: create file system: SetUpBucket: BucketHandle: storageLayout call failed: rpc error: code = InvalidArgument desc = Bucket is a requester pays bucket but no user project provided.\nerror details: name = BadRequest field = desc = Bucket is a requester pays bucket but no user project provided."}

[abhishek10004]

Hi @shengqh,
Thanks for reaching out to us. It seems like we are not setting the billing project for the storage control client calls which results in this failure and mounting fails due to that. Will look into it and update the thread.

[Type-Here]

Hi, I'm in the same situation, providing --billing-project with billing activated but got still the error: "Bucket is a requester pays bucket but no user project provided." Any updates or workaround available? Thanks.

[charith87]

Hi @Type-Here and @shengqh
This was a break in behavior from 2.9.0 version onwards where the initial storageLayout API call is a blocking call. hence the mount fails due to it.
So any version prior like 2.8.0 to that would still work provided the bucket mount is a flat bucket. ( For Heirarchical Namespace buckets, it gets mounted as a flat bucket as well due to the underlying issue of not setting the billing project.).

It needs a code change to pass "x-goog-user-project" in the storageLayout call. We'll try to expedite this fix.

cc @marcoa6 @mustvicky

[gargnitingoogle]

I have a fix for this open in PR #3256 . Though, even with that PR merged, atomic renames of folders won't work for hierarchical buckets as GCS has a bug in RenameFolder API for requester-pays buckets (b/414790551). So, even after that PR is merged, customer will have to live with that limitation until that GCS bug is fixed.