Publish go-ipfs as a gx package

[whyrusleeping]

In order for other applications to reliably depend on the go-ipfs source code, we should publish go-ipfs as a gx package, once per release (or maybe more frequently depending on the need).

[geraintguan]

Can I help with this? I have experience with go but not the gx package manager, but I'm happy to do some research to implement this. 😄

[wigy-opensource-developer]

@g-corneu Hi! Do you have some progress on this? To fix DeCentral-Budapest/ipns-gen#1 we would need this to happen. Are you planning to finish fixing this issue?

[Kubuxu]

Publishing gx package is very simple even out of the tree, here is gx package of the master right now: QmYTzqEnC31PnxQoA9WhtDsJnVTeDkHaUnh33X3LmKsKLf

I will make sure that during next release it is done too.

[wigy-opensource-developer]

Wow, what a prompt response. Thanks a lot. I thought that the original idea of this issue was to add a target to the Makefile that does the release to gx, so a make release could do the trick when the time comes.

[Kubuxu]

gx packages usually have .gx/lastpubver file that contains hash and version of last "officially" published release, I will make sure that it is introduced with next release of go-ipfs.

[ghost]

Yes we should publish it as a proper package. Need it for ipns-pub and other similar tools too.

[wigy-opensource-developer]

@Kubuxu I have just tried to publish my own version of the ipfs/go-ipfs source code onto an ipfs daemon running on localhost.

$ gx publish
publishing: Post http://127.0.0.1:5001/api/v0/pin/add?arg=QmVVc5QXsnLKNgdZbU5SdADuWc9cYjiBpsKbib3Sz1QH7r&encoding=json&r=true&stream-channels=true: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
$ ipfs object links QmVVc5QXsnLKNgdZbU5SdADuWc9cYjiBpsKbib3Sz1QH7r
Qmdd3Qjr8L6biDfNyMWT9U6PhxXGzMsivYsym1oj5BWNXq 57836143 go-ipfs

So the publishing took too long, I guess. Is it a known problem with gx + a complex enough source code? Do you see any workarounds? Should I report this problem on gx or gx-go, too?

[Kubuxu]

Can you try running ipfs refs -r Qmdd3Qjr8L6biDfNyMWT9U6PhxXGzMsivYsym1oj5BWNXq?
My suspect right now if GC running while the go-ipfs was being published. Can you try disabling GC by increasing StorageMax to some big value (1000GB) for example.

[wigy-opensource-developer]

I did increase the StorageMax to 1000GB, although I was quite far from the 90% of the 10GB that is the default trigger. Well, I thought that gx would ignore files that match rules in .gitignore. Actually, it did not exclude those that were defined in subfolders. I wondered why the sources of something would be 58MBs and then got rid of the ignored files before publishing.

$ git status --ignored
On branch feat/name_upload
nothing to commit, working directory clean

After getting to this state, the sources went down to 2.6 MB as you can see on https://ipfs.io/ipfs/QmeQ9Ktc14W93VvghbktL5yjAFf4UErPgjb1mBK5pX5kEu but the error message while pinning the root was still shown.

$ time gx publish
publishing: Post http://127.0.0.1:5001/api/v0/pin/add?arg=QmeQ9Ktc14W93VvghbktL5yjAFf4UErPgjb1mBK5pX5kEu&encoding=json&r=true&stream-channels=true: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

real    8m49.876s
user    0m0.632s
sys     0m0.336s

So I went and checked if pinning was done:

$ ipfs pin ls | grep QmeQ9Ktc14W93VvghbktL5yjAFf4UErPgjb1mBK5pX5kEu

No, it was not. So I have started

$ time ipfs pin add QmeQ9Ktc14W93VvghbktL5yjAFf4UErPgjb1mBK5pX5kEu
^C
Error: request canceled

real    195m37.676s
user    0m0.416s
sys     0m0.208s

and waited 3 hours without any results. Is my system screwed up, or is this a regression with the 0.4.4 fixes of pinning? Funny is that traversing all the refs locally takes a fraction of a second:

$ time ipfs refs -r QmeQ9Ktc14W93VvghbktL5yjAFf4UErPgjb1mBK5pX5kEu
QmPfBnfQLJE1ofbcSbUtEqikCmHdfJnUkHerummmGpitzN
QmShi6TbwKNRrdFTs4hUEyFKW6vdhdTsocYJrn2a8pzSUo
...
QmbkEdm5SCtMvNMhJevkpfDBbHGkCyrp7s876PXGjWuimH
Qma5gTNVnTJvNV8jVSJ8jnEd6haxmJARXVuwv8rtF5AFLe

real	0m0.092s
user	0m0.020s
sys	0m0.012s

[whyrusleeping]

This is complete, we publish new gx packages for every go-ipfs release