Stripping issue with nekotools

[delahee]

Something discussed with Todd Kulick, he said that neko was uber BUT there was a "big fracking" issue for neko to get much more leverage and it was that some tools and neko binaries had an incorrect Elf markup.

This is dued to the concatenation of binaries, it seems they are not modifying the elf table and then the packet managers and scanners from various unix flavors are stripping the not declared elf binary section from any nekotools made binary.

Thus rendering nekotools produced tools and binaries totally bogus...
So i guess nekotools should append executable section in a more careful way...

I know the theory about that but I think @ncannasse would go much faster taking action for this and this should go in a roadmap as very prioritized.

@ncannasse feel free to pester Todd as he promised me, he should file an issue :-D

[ncannasse]

We could do something ELF specific in nekotools boot (I have to check the ELF header format), but then we might to fix it for PE32 (Windows) as well. I guess/hope OSX uses ELF as well...

[delahee]

I very pedantically proclaim this issue very important. :)

[hughsando]

The 'objcopy' utility can do with very easily (linux - not sure about mac).
See https://github.com/foxx/elfdataembed - the makefile for writing and the small bit of code for reading.

[kulick]

It would be cool to fix this since when it hits someone, they probably don't understand what happened to them. :)

I understand the sucky part of this in terms of how it requires not just platform specific, but ABI (application binary interface) specific logic in neko and nekoboot.

That elfdataembed library looks very appropos. 👍

[delahee]

After a quick tweeter poll, many of the power user (massive,kha,ofl,etc) haxe community use Neko for production and team tools, it just confirms that neko has become a strategic asset.

[ncannasse]

BTW OSX uses Mach-O executable format. Do we have similar problems here as well ?

[ncannasse]

Thanks for the link @hughsando , sadly the code can't be ported to neko/haxe since it uses libELF, unless we package it as well. I'll check first what's possible to do with ELF hacking

[kulick]

Assign to me to have a try?

[ncannasse]

Sure, I just pushed some ELF code there (https://github.com/HaxeFoundation/format/tree/elf), keep on from here, it's already midnight there :)
ELF doc http://man7.org/linux/man-pages/man5/elf.5.html
I think just inserting a new section might work, not sure which name we should attribute it (I've just starting understanding what's inside ELF)

[ncannasse]

@kulick any update on this or should I continue working on it ?

[kulick]

Hey, sorry I haven't gotten back to ya. I have been playing with it some. I have a few questions for you about the strategy for fixing this.

I see that you started writing ELF parsing code in Haxe. That surprised me a little since I believe that we really want a solution that is part of neko without any external dependencies. I suppose we could implement a solution in Haxe, compile it to neko bytecode, and then check that bytecode into the neko source, but I generally eschew such solutions that create circular dependencies. Is that the path you were imagining?

Maybe you were imagining a second, Haxe-based program that people would run on the output of nekotools -boot that tweaked the neko program to apply an ELF section header to the bytecode? This could work and I definitely like the idea of writing the ELF parsing in Haxe if we aren't going use some library, but I'd prefer a strategy where nekotools simply always does the right thing automatically without people knowing that they need to run a second program after nekotools -boot...

I suppose the standard solution that I imagine would be most simple on Linux alone is to use objcopy to attach the bytecode (somehow from nekotools or alternatively kind of replacing nekotools altogether) and to use libbfd or libelf (plus maybe the dataembed stuff Hugh references above) in the neko interpreter to find the bytecode at execution time. This seems to be what people (on Linux) normally do (including the dataembed stuff). This strategy has weaknesses: dependency on objcopy at nekotools execution time and dependency on libelf or libbfd (not sure how big those are at the moment). Due to these dependencies, I started investigating a different path...

When I look at the problem, I initially think that the solution should be to write some C code that is conditionally included in nekotools when USE_ELF_BYTECODE is defined or some such. Maybe this code would use libelf if it is small enough and only if the conditional is set. Then I reminded myself how neko and nekotools are actually built and I see a problem. Nekotools is actually built in neko script, not C. This isn't a deal breaker, but it does make things a bit trickier. I have started down this path. Why isn't nekotools just more C code, by the way? I understand that this is really a style thing and either strategy could work. ;)

I'm interested in your thoughts/ideas on these comments. I welcome other people's thoughts, too. ;)

[ncannasse]

hey todd,

My original idea was to experiment with Haxe, in order to build a small piece of code that allows to patch ELF by appending the neko bytecode, and once well tested, rewrite a very short version of this code in Neko for inclusion in nekotools "boot" command. That was also a good occasion to learn a bit about ELF. I expect the final "elf patch" code to be around 30-100 lines of code.

[hughsando]

I think if you ship a solution that is a ".n" file you are not adding any
dependencies, since you need neko to build a neko-tool. Not sure if
objcopy is present on every practical linux box, if so this seems the
easiest solution.
As for introspection, I'm thinking you don't want a libelf dependency for
the generated tool - and since we have reasonable control over how the
bytes are inserted, hacking something to do the reading might work - even
if involves scanning for a magic byte sequence.

On Sun, Jul 5, 2015 at 9:19 PM, Nicolas Cannasse notifications@github.com
wrote:

hey todd,

My original idea was to experiment with Haxe, in order to build a small
piece of code that allows to patch ELF by appending the neko bytecode, and
once well tested, rewrite a very short version of this code in Neko for
inclusion in nekotools "boot" command. That was also a good occasion to
learn a bit about ELF. I expect the final "elf patch" code to be around
30-100 lines of code.

—
Reply to this email directly or view it on GitHub
#75 (comment).

[kulick]

Ok, I finally have a working solution for this. Yeah! :)

I'll send a pull request RSN...