by Liz Rice

How to claw your way out of AWS Lambda function hell using the power of Docker

When you’re building Lambda functions, it’s easy to get trapped in an “Invalid ELF header” nightmare. The problem is that your binaries are built for the wrong platform. Here’s what’s going on, and how you can fix it easily using Docker.

When most people get started with Lambda functions, they’ll use the online editor in the console to input their code. This is fine for your first example or two, but pretty soon you’ll want to do something wild and crazy like, y’know, import a library.

Over the past few weekends I’ve been working on my first Alexa skill for an Amazon Echo and I’ve reached that point where I want to use some third party code to add some additional functionality.

The online editor simply lets you edit a single file. If you want to refer to other files — including imported libraries — you can upload them in a ZIP file (Amazon call this a deployment package). But if you‘re working on a Mac or a Windows computer, there’s a catch.

When you use pip to install Python libraries on your laptop, it gives you binaries (.so files) that are built to run on your machine. But when the Lambda function runs in the AWS cloud it is going to be running on Linux — and binaries built for Mac (these are often called ‘darwin’ builds) or Windows won’t run on Linux (and vice versa).

If you upload the Mac version, you’ll see “invalid ELF header” logs when you try to test your Lambda function.

So you’re going to need Linux versions of those library files. But what if you don’t have a Linux box to hand?

You could grab yourself an EC2 instance from Amazon (or a droplet on Digital Ocean, or any Linux VM of your preference) but to my mind that’s quite a performance, and could even cost you a little bit of money (especially if you forget to take the EC2 box down again when you don’t need it).

I think the easiest solution is to use Docker.

The Docker approach

With Docker you can very easily can run a Linux container locally on your Mac, install the Python libraries within the container so they are automatically in the right Linux format, and zip up the files ready to upload to AWS. You’ll need Docker for Mac (or Windows) installed first.

Spin up an Ubuntu container that can see the Lambda code you want to upload.

docker run -v <directory with your code>:/working -it --rm ubuntu

• The -vflag makes your code directory available inside the container in a directory called “working”.
• The -itflag means you get to interact with this container.
• The --rm flag means Docker will remove the container when you’re finished.
• ubuntu is the name of an official container image containing, you guessed it, Ubuntu. If this container image isn’t already on your machine, Docker will download it for you.

You should now be inside the container at a shell prompt looking something like this:

root@c1996f32a397:/#

Install pip and zip:

$ apt-get update$ apt-get install python-pip$ apt-get install zip

Move into the working directory (you should be able to see your Lambda function code here):

$ cd working

Use pip to get the library/ies you’re interested in. You can use the -t flag to tell pip to put the libraries here in the current directory, which will be more convenient later as it’s where the AWS deployment package wants them to be:

$ pip install -t . <library>

If you’re very curious, you can take a look to see what this installs. In my own case I installed the editdistance library, which gave me the following additional directories and files.

editdistance:__init__.py __init__.pyc _editdistance.h bycython.so def.h

editdistance-0.3.1.dist-info:DESCRIPTION.rst INSTALLER METADATA RECORD WHEEL metadata.json top_level.txt

You can see that bycython.so file? This is the correct, Linux version of the binary that AWS was objecting to when I hit the Invalid ELF header (shown in the error log screenshot above).

Create the ZIP file with your Lambda code (in my case, a single file called lambda_function.py) and the libraries (for me, the two editdistance directories and their contents.

$ zip linux-lambda.zip lambda_function.py

$ zip -r linux-lambda.zip edit*

The -r flag on zip tells it to recursively add the contents of directories.

Now you have an archive file called linux-lambda.zip which is ready to upload to AWS. And because the directory is mounted from the host (your Mac) into the container, you can simply upload the file into the console.

Back in the terminal type exit to quit the container, and it will be as if it never existed, except for the existence of the linux-lambda.zip file, which is still available on the host.

Upload the ZIP file in the console, save it and try running a test. Invalid ELF header error no more!

If this article helps you out, please hit the ? button to make it easier for other people to find it. If you really like it, why not go bananas and share it too?

I’ve written a few other posts about what I’m learning as I write my first Alexa skill, like this one where I add database storage capabilities to my Lambda function. If you find them useful, you might be interested in a book I’m writing called Adventures with Alexa. Pick your own price!