Extended Attributes: Requested buffer size 33554432 is above the limit of 16777216

[rupertpaulson]

Not sure if this is a real issue, so I'll open a new discussion first.

I'm running borg 1.2.0 on macOS 11.6.5 and while running a backup I noticed the following issue:

borg create

1.png: extended stat: Requested buffer size 33554432 is above the limit of 16777216.
3.png: extended stat: Requested buffer size 33554432 is above the limit of 16777216.
2.png: extended stat: Requested buffer size 33554432 is above the limit of 16777216.

borg check

32 orphaned objects found!
Archive consistency check complete, problems found.

So I had a closer look at the extended attributes, as the file size was nothing special.

24M Nov 27  2017 1.png
com.apple.metadata:kMDItemWhereFroms	  32M
com.apple.quarantine	  62B

20M Nov 28  2017 2.png
com.apple.metadata:kMDItemWhereFroms	  27M
com.apple.quarantine	  62B

16M Nov 27  2017 3.png
com.apple.metadata:kMDItemWhereFroms	  22M
com.apple.quarantine	  62B

Please don't ask me why there are these huge attributes included in these files, they are screenshoots I created a few years ago with :screenshot --fullpage within Firefox if I remember correctly.

After removing the com.apple.metadata:kMDItemWhereFroms attribute with xattr, obviously, the problem was solved, but should borg not be able to handle the files regardless of weird attribute sizes?

[ThomasWaldmann]

https://developer.apple.com/documentation/coreservices/kmditemwherefroms

according to this, the expected size is way less than what you have there.

can you carefully look what's actually in there for these files?

[rupertpaulson]

I haven't had a look on how borg handles the xattr of a file yet, for example if dedub is also applied on this data and if it's written separately to the main file into the backup data structure. But apparently the MemoryLimitExceeded issue results in those files missing in the backup, which probably leads to the orphaned objects, I'd assume.
Wouldn't it be possible to check the xattr size instead of always allocating 2GiB? And if the size exceeds a defined limit the xattr could be deleted before writing the file into the repo and the user could be informed via a warning. At least the main data would be present in the backup.
I know, the whole scenario is a very rare exception, but still, it's a bit annoying that a stupid behavior of Firefox (or another app writing bad xattr) can cause the repo ending up in a bad state.

original folder that is backed up with borg create:

-rw-r--r--@  1 paulson  staff  25172558 Nov 27  2017 1.png
	com.apple.metadata:kMDItemWhereFroms	33563496
	com.apple.quarantine	      62
-rw-r--r--@  1 paulson  staff  21372980 Nov 28  2017 2.png
	com.apple.metadata:kMDItemWhereFroms	28497392
	com.apple.quarantine	      62
-rw-r--r--@  1 paulson  staff  17125604 Nov 27  2017 3.png
	com.apple.metadata:kMDItemWhereFroms	22834224
	com.apple.quarantine	      62
-rw-r--r--@  1 paulson  staff   1823786 Aug  6  2019 4.png
	com.apple.metadata:kMDItemWhereFroms	 2431800
	com.apple.quarantine	      62

extracted folder (three files missing):

-rw-r--r--@  1 paulson  staff  1823786 Aug  6  2019 4.png
	com.apple.metadata:kMDItemWhereFroms	2431800
	com.apple.quarantine	     62

[ThomasWaldmann]

a single xattr value is not cut into chunks. but all metadata values (including xattrs) are written into the archive metadata stream and that stream is chunked and deduped.

[ThomasWaldmann]

please open issues about:

• the file being skipped completely just because of this issue with a single xattr value
• the xattr value max. size being too low in borg

if you can provide a very simple reproducer script that shows the issue, that would be great (macOS based script is ok for me).

[rupertpaulson]

Okay, will do.

I'll upload the png file together with a small script as I can't think of another easy way to add a large extended attribute in a script. xattr -w xattr1 "$(base64 /dev/urandom | head -c 20000000)" testfile doesn't work.

xattr doesn't let me create attributes > 1MB: zsh: argument list too long: xattr

[rupertpaulson]

Mmh I just tried to create a GitHub repo with the files and a script but it seems that git removes all xattr. If I create a ZIP or TAR archive the large extended attribute is also removed after extraction:

-rw-r--r--  1 paulson  staff    42B Apr 22 14:06 file-without-xattr
-rw-r--r--@ 1 paulson  staff    19M Apr 22 14:42 larger-file-with-xattr
	com.apple.metadata:kMDItemWhereFroms	   0B
	com.apple.quarantine	  62B
-rw-r--r--@ 1 paulson  staff    85B Apr 22 14:08 small-file-with-xattr
	com.apple.metadata:kMDItemWhereFroms	   0B
	com.apple.quarantine	  62B

I guess, that could be a desirable behavior of Borg as well, to handle the file but to just drop the weirdly large xattr value.

So I put everything in a APFS dmg image if you like to check out the issue. Please let me know if you can reproduce it, and if you still think it's worth opening the issues.