Fast Deployments to Multiple Golang Lambda Functions
- 1. Build Process: Fast Deployments To Multiple Lambda Functions
- 2. Problem Description • Multiple Microservices • Each service has ~50 lambda functions • Each function handling very specific task • Build process uses CircleCI, running within containers • Golang binary to be compiled for each function – Golang uses all available CPU in build environment – Compiling in parallel within build environment slows things down – Build and test was taking ~25 minutes per app 2
- 3. Potential Solutions • CircleCI caching – Caching multiple binaries etc. takes up a lot of space – More than 5 minutes devoted to save and restore cache – Does not lower compile times sufficiently • Parallel compilation – Each “go build” appears to use all available CPU – Parallel compile times actually appear slower than sequential • Build using a build lambda – Each build runs within an independent environment – Horizontal scaling is automatic – Helps to separate concerns during the build process – DRY build code that can be re-used across all our apps 3
- 4. Inside the lambda environment • Based on container technology • Runs Amazon Linux • CPU resources tied to configured memory size • Maximum execution time of 5 minutes – AWS might be increasing this limit in the future – talk to Support • Only writable area is /tmp – Ephemeral • Files written to /tmp are persistent if the invocation uses an existing “warm” container – Limited to 512 MB • AWS might be increasing this limit in the future – talk to Support • Lambda handler runs as root user 4
- 5. Setting up the lambda env for builds • Set GOPATH and GOROOT to live within /tmp • Check if Golang is installed, install if required – It will not be installed in a “cold” lambda container – If not installed, download and install within /tmp/go • Check if Apex is required, install if not already done – We use Apex for older apps developed before native AWS Go support, using a NodeJS shim around the Golang binary – Apex is installed only if specified in the lambda’s event JSON • Checkout application code – Uses a Github “machine user” token set as an env variable – Checks out into /tmp 5
- 6. Testing • We use “go test” to run unit tests • Coveralls.io for test/coverage reporting • Test output is returned as part of the lambda’s response JSON – Output is base64-encoded – CircleCI build script collates outputs from all build-lambda invocations within a build, then uploads them to Coveralls • If a test fails, lambda invocation fails – This failure is detected by the CircleCI build scripts, which take appropriate actions 6
- 7. Building • We use “go build” to build the binary • Code dependencies are vendored and stored in the repository • Built binary is uploaded to an S3 bucket specified in the event JSON, using the Python AWS SDK 7
- 8. Invoking the build lambda from the CI • Depend on convention to separate function code and shared code • Iterate through all directories containing code: – Invoke build-lambda in test-only mode for shared code – Invoke build-lambda in test+build mode for function code • Use GNU Parallel to invoke build-lambda simultaneously – Build AWS CLI commands for each invocation, save to file – Pipe saved file through GNU Parallel to run commands in parallel – Halt build if any build-lambda invocation fails • Use JQ to parse lambda’s invocation result from AWS, and take appropriate actions: – Build error if .FunctionError is not null – Base64-decode test coverage output and append to Coveralls – Output .LogResult to STDOUT if .FunctionError is not null 8
- 9. Logging • The build-lambda logs to Cloudwatch • The CircleCI script invokes the build-lambda using the “--log-type Tail” AWS CLI option – This returns a field in the result named ”.LogResult” – Contains the last 4K of logs, base64-encoded • If the invocation failed, then the CircleCI script outputs the base64-decoded contents of “.LogResult” to STDOUT • Build errors can be easily viewed on the CircleCI build output view • Further log processing can be done using Cloudwatch 9
- 10. Deploying built binaries to functions • Use AWS CLI’s “update-function-code” command • Directly deploy from S3 without need for file transfers • Integration tests performed by downloading binaries from S3 into the CircleCI environment, and running tests against them – This ensures that the actual binary that was tested is deployed to the lambda function 10
- 11. Q & A Q & A 11