Face detection don't work on AWS Lambda due to node-canvas not working on node 8.10

[bobmoff]

Hi there!

I am trying to get this up and running on AWS Lambda but are having trouble as it seem the 'canvas' package is broken on node 8.10. I am trying to get some face detection going.

Automattic/node-canvas#1252 (comment)

It looks like its kind of a pain to try and run any other version of node on AWS Lambda, so I started trying to use it without the canvas package. As you mention in the README and I also noticed that the faceapi.locateFaces() function also takes a Tensor4D class as input. I have never used Tensorflow and I am a little confused as how to turn a ArrayBuffer from axios into a correctly shaped Tensor4D object.

I am fetching a jpeg image using axios.

I found the tf.tensor4d function but not sure what shape and dtype it should be.

Do you have any idea?

My code so far:

const { data: imageBuffer } = await axios.get(url, {
	responseType: 'arraybuffer'
})
const imageTensor = tf.tensor4d(imageBuffer, [?, ?, ? ,?])
const faces = await detector.locateFaces(imageTensor)

Error messages look similar to this one:

Error: Based on the provided shape, [1,2,3,4], and dtype float32, the tensor should have 24 values but has 68695

Any help is greatly appreciated!

[justadudewhohacks]

Hey man, hows it going?

Ohh that's a pity, node-canvas is obviously the easiest way to use face-api.js with node.

Well you can either pass in tf.Tensor3Ds or tf.Tensor4Ds with a batch size of 1:
const imageTensor = tf.tensor3d(imageBuffer, [height, width, channels]), where channels should be 3 and in RGB order.

However, the imageBuffer you receive from axios is most likely jpeg or png encoded right? You have to create the image tensor from the raw image data, so you will have to decode the data first. There are probably npm packages, which can do that for you. For example I have seen people using ffmpeg for this.

[bobmoff]

Thanks!

Tried the get-pixels npm package and it works!

get-pixels return a 4 channel array all the time, even for jpegs 🤷 , so had to remove the alpha channel.

const pixels = await new Promise<Pixels>(resolve => {
	getPixels(url, (err, pixels: Pixels) => {
		if (err) {
			throw err
		}
		console.log('pixels:', pixels)
		resolve(pixels)
	})
})

// remove alpha channel
const RGBValues = []
pixels.data.forEach((px, i) => {
	if ((i + 1) % 4 != 0) {
		RGBValues.push(px)
	}
})
const imageTensor = tf.tensor3d(RGBValues, [pixels.shape[1], pixels.shape[0], 3]) as any
const faces = await detector.locateFaces(imageTensor)

Now to the task of getting it up on Lambda without breaking their code size limit, wish me luck!

[wangsijie]

Thanks!

Tried the get-pixels npm package and it works!

get-pixels return a 4 channel array all the time, even for jpegs 🤷 , so had to remove the alpha channel.

const pixels = await new Promise<Pixels>(resolve => {
	getPixels(url, (err, pixels: Pixels) => {
		if (err) {
			throw err
		}
		console.log('pixels:', pixels)
		resolve(pixels)
	})
})

// remove alpha channel
const RGBValues = []
pixels.data.forEach((px, i) => {
	if ((i + 1) % 4 != 0) {
		RGBValues.push(px)
	}
})
const imageTensor = tf.tensor3d(RGBValues, [pixels.shape[1], pixels.shape[0], 3]) as any
const faces = await detector.locateFaces(imageTensor)

Now to the task of getting it up on Lambda without breaking their code size limit, wish me luck!

Thanks, that helped me. But it's quite slowly (1000ms) compared to canvas ( < 10ms ).

[bobmoff]

tensorflow team is working on a solution: tensorflow/tfjs#298 (comment)

[justadudewhohacks]

Nice!

[bobmoff]

It wasn't easy, but I finally got it running in a Lambda on AWS.

The tfjs-node package includes a native library and has to be built using the correct environment. I achieved this using Docker and the lambci/lambda:nodejs8.10 image from the https://github.com/lambci/docker-lambda project. This image seem to be one the best images to simulate the Lambda environment.

The built tfjs-node module includes a 110mb binary file.

@tensorflow/tfjs-node/deps/lib/libtensorflow.so

The Lambda size limit is 250mb (uncompressed).

Well, with 140mb left for other modules it should be fine right?

I tried deploying the package using serverless deploy resulting in this cryptic error message.

ENOENT: no such file or directory, open '/Users/bobmoff/Projects/picular/serverless-face-api/node_modules/@tensorflow/tfjs-node/build/Release/libtensorflow.so'

The file exists alright, but its a symlink that points to

libtensorflow.so -> /var/task/node_modules/@tensorflow/tfjs-node/deps/lib/libtensorflow.so

Ahh, the symlink was created inside the docker container and it is a not a relative link. Ok, so i delete the symlink and created a new one that is relative.

libtensorflow.so -> ../../deps/lib/libtensorflow.so

But zipping up the package, using serverless deploy results in a zip that when uncompressed is 343mb ?? Using Disk Inventory X on the uncompressed folder I saw this:

Why are there 2 large binaries? Shouldn't one of them just be a symlink ? The symlink is gone, and replaced with a copy of the binary. Hmm.

After a bit of research I learned that the default behaviour when zipping symlinks is that they get "resolved" (link is followed and original content is copied). Ok. But there is a --symlinks flag for the zip program that can modify this behaviour to keep symlinks as they are, instead of resolving them. This solved the size issue, nut now I realised that I depend on a local project that is linked that actually need to be resolved when packaged. I couldn't find any way to specify what folder/file should be resolved or not when packaging through serverless. There is a way to exclude files/folders from package though. I excluded the symlink.

package:
    exclude:
        - node_modules/@tensorflow/tfjs-node/build/Release/libtensorflow.so

So using the following commands I first package the project (excluding the symlink) and then manually include the symlink again into the zip archive. (-y is the shorthand for --symlinks)

sls package --package my-artifacts
zip -y my-artifacts/serverless-face-api.zip node_modules/@tensorflow/tfjs-node/build/Release/libtensorflow.so

Now I have a package that is 233mb! Yeah, 17mb to spare! Hehe. Ofc there are more size reduction that can be made be excluding more stuff, like excluding unused weights/models from face-api etc.. But I was just happy to get below the limit.

Happy as a unicorn on christmas I deployed my neat "little" package to AWS.

sls deploy --package my-artifacts

.. and lived happily ever after.

[MatthewAlner]

libtensorflow.so appears to be 183mb now! don't suppose you could share your package.json for version numbers?

[bobmoff]

Sure, here are my deps, haven upgraded in a while :)

{
    "@tensorflow/tfjs-node": "^0.2.1",
    "axios": "^0.18.0",
    "face-api.js": "^0.17.1",
    "get-pixels": "^3.3.2",
    "lodash": "^4.17.11",
    "module-alias": "^2.1.0",
    "moment": "^2.23.0",
    "monk": "^6.0.6",
    "url-join": "^4.0.0"
}

[henrydawson]

The tensorflow package (version 1.2.3) is now 294MB! libtensorflow.so.1.14.0 is now 216MB, and libtensorflow_framework.so.1.14.0 is another 35MB. Pretty sure there is no way to get this package into Lambda.

@MatthewAlner did you manage to get it working with a newer version? Or do I need to roll all the way back to the versions used by @bobmoff

[bobmoff]

Perhaps you could try to deploy using https://www.openfaas.com/

[MatthewAlner]

I didn't in the end 😞

[henrydawson]

Thanks for all the helpful info in this issue, and of course thanks again to @justadudewhohacks . I did eventually manage to get it working on Lambda, but it wasn't easy. I ran into all the same problems that @bobmoff did, and if I'd read his post more carefully at the start I would have saved myself a lot of time (in particular the libtensorflow.so linking issue). I'm using the Node 10.x runtime.

The newer versions of tfjs-node just get bigger and bigger, especially their bundled .so files. There is no way I could find to make the newer versions fit within Lambda's 250MB limit, so I had to roll way back to v0.2.1. You also need to roll face-api.js back to a similarly old version, as it seems face-api.js is coupled to a specific version of tfjs-node. Below is my deps:

  "dependencies": {
    "@tensorflow/tfjs-node": "^0.2.1",
    "face-api.js": "^0.17.1",
    "get-pixels": "^3.3.2"
  }

If you really wanted to use the latest version of Tensorflow and Face-api, I suspect you could use a technique like this https://github.com/lucleray/tensorflow-lambda, which basically uploads a zipped copy of the dependencies, and then unzips them into /tmp before running them. There is obviously overhead for unzipping the files, but if your lambda instances are often reused then that initial overhead fades away.

[henrydawson]

Hey @bobmoff, just wondering if you had any issues with memory leaks in your setup? My face detection lambdas get reused most of the time because they get called so often, and I've noticed the memory used slowly grows with each invocation, until it finally breaks the Lambda memory limit and starts again. No idea if it's get-pixels, tfjs-node or face-api.js that is the issue.

[bobmoff]

Howdy @henrydawson. No, sorry. I havent experienced that, but that is probably due to the fact that I havent had the pressure like you seem to have, so my containers/lambdas have probably been recycled before that happenes.

Sorry to not be of any more help :/

[truff77]

Hi @henrydawson and all,

just some feedback on canvas running on lambda and also a question.
I've managed to make canvas run on lambda by using https://github.com/lambci/docker-lambda to make some tests. Thanks to the "build" image I had to run a bash shell on the docker build image to:
1/ run npm install
2/ copy 3 libraries from the docker build image /lib64 to the root of my project: libblkid.so.1, libmount.so.1, libuuid.so.1

I'm now trying to use https://github.com/lucleray/tensorflow-lambda for tensorflow but with no luck.
My little project is based on one of the examples from the repository. It uses the common directory as in the examples directory and I've tryed to subsitute the import '@tensorflow/tfjs-node'; from the env.ts file with import "tensorflow-lambda" but it does not work.
It looks like tfjs-node have a mechanism to supersede the tfjs-core functions but this is not happening with tensorflow-lambda. Any hint on how I could make this happen ?

Thanks in advance !

[bettysteger]

@bobmoff i am new to this, and wanted to ask if you've got an example code to get started with face-api.js and AWS lambda?

[bobmoff]

@bobmoff i am new to this, and wanted to ask if you've got an example code to get started with face-api.js and AWS lambda?

Sorry. Not using this any more. But Lambda have upped the limit to 10GB for each function so size should not be a problem any longer.

[nasr18]

@bobmoff I really wanna move this facial recognition to lambda. Any help would be highly appreciated. I tried with this version also

"dependencies": {
    "@tensorflow/tfjs-node": "^0.2.1",
    "face-api.js": "^0.17.1",
    "get-pixels": "^3.3.2"
  }

but still limit exceeding.

[bobmoff]

Are you reaching the max limit of 10gb ?

The issue I had was getting below 250mb.

[nasr18]

Are you reaching the max limit of 10gb ?

The issue I had was getting below 250mb.

Going above deployment package unzipped limit size (250mb)

[bobmoff]

If you switch using containers you can use 10 gb.

https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/

[henrydawson]

Hi, I did get this working on Lambda a long time ago, but that was with older versions, and I ran into a lot of issues (especially the hard to resolve memory issues mentioned above). Eventually, I gave up on Lambda and just run it on EC2 using spot instances, which was cheaper and better for my workload. It was also simpler as I didn't have mess around with the file size issues.

That said, I recently used EFS with Lambda for some image processing that required extra space, and it was really easy to use. I imagine you could use EFS to get around all the size limits.

https://aws.amazon.com/blogs/compute/using-amazon-efs-for-aws-lambda-in-your-serverless-applications/

[nasr18]

@bobmoff @henrydawson thank you both of you for pointing me in the direction. I will check both aws lambda container and EFS.

[nasr18]

@bobmoff I have successfully deployed in Lambda using containers. But I'm facing one issue. I'm just doing faceapi.detectSingleFace().withFaceLandmarks().withFaceDescriptor() in lambda to detect single face and returning that output as a response via API Gateway. I can't able to return without doing JSON.stringify(result) and If I do stringify the result, that result not working in final comparison. I dont know what I'm doing wrong.

update: I managed to fix it. And its working fine now.