Lambda Invoke: The Minimal Loveable API

Today we discuss how API Gateways might be overkill for your application. The Lambda Invoke API is an alternate integration pattern teams should consider when building serverless applications.

This post is the first in a series that will explore the technical foundations of an automation platform that supports Proactive Ops. This series will primarily focus on AWS, but the concepts can be applied to other cloud platforms.

Monolithic web apps often feature a REST API bolted onto the side. When first adopting serverless patterns, teams often implement API first micro services. Using this common pattern can reduce the cognitive load during a period of significant change. At the same time, API first isn’t always the most efficient way for services to interact. This is especially true if the services are only consumed by internal components of the application.

Before we look at alternatives, let’s compare the monolithic flow to one for serverless micro services. On the surface these two approaches appear to be very similar. While the logic is broken out differently, the key flow is a request hitting an API and getting a response. This simplification is deceiving.

Monolithic Request Flow

Let's explore the flow when a monolithic app receives a request. A monolith hosted on a single server has few moving parts handling the authenticated web API request flow. The diagram below shows the flow for a basic web application using local oAuth.

Depending on the language used, there may be a web server handling the HTTP request. It doesn't fundamentally change the flow. There are several steps in the sequence of method or function calls. The functions within the application handle authentication, authorisation, processing, and preparing the response. This sequence of calls is usually handled in a single thread. Even for less experienced developers, this flow is easy to map out and follow.

Serverless Request Flow

The flow for the same request for a serverless application looks very different. A call to an API backed by a Lambda function results in a more distributed sequence of calls. Each layer of the stack is responsible for playing its small part.

The digram shows the key calls involved in processing the request. This is somewhat of a simplification. Just as we didn’t include the details of the web server forking a process in the monolithic flow, here we skip firecracker spinning up the VM, the Lambda assuming a role etc. We need to stop before we stray into Wardley mapping and add mining the sand for the CPUs to the flow. The serverless flow is already a lot more complex compared to the monolith.

Sometimes complexity can be a good thing. In this case it allows for clean separation of responsibilities. Each component is responsible for a small part of the flow. It is optimised for performing its task. While there are a lot more steps, it all happens very quickly.

The downside is each step adds overhead, which increases latency.

Not all services need to be exposed to customers. Many services are internal only. If the consumers of the API are internal to the application why do we need a web API? Removing the API Gateway will simplify our flow, thus reducing overhead and improving latency.

Lambda Invoke as a RPC call

Amazon’s Lambda Invoke API allows directly calling, or “invoking”, a Lambda function. This makes the Invoke API a common Remote Procedure Call (RPC) interface for Lambda.

When an API Gateway receives a request, it calls the Invoke API. As shown in the serverless flow above, there are a few steps before API Gateway calls the Lambda containing our business logic.

When a client hits an endpoint of an API Gateway, the gateway needs to invoke a function to handle the authentication before passing the request payload to the target Lambda. Both of these invocations require IAM checks before they can run. If we call the Lambda directly via the Lambda Invoke API we avoid one of the IAM checks and the execution of the authentication Lambda. This improves responses times, especially when the Lambda function is warm.

This simplified flow removes the overhead of maintaining the authorisation Lambda and configuration. This overhead exists even if it is common for all Lambdas invoked by the API. It also removes the latency introduced by the authorisation function, especially when it is cold.

Let’s explore this simplified flow.

This pattern encourages code reuse, as the Lambda is now exposing the raw logic. This can simplify the client logic too, as we remove the need to create and maintain a REST client. Instead the AWS SDK is used to call the function directly. To facilitate further code reuse, several Step Functions can invoke the same Lambda.

Separating Handlers and Logic

When creating Lambda functions this way, separate the Lambda function handler from the business logic. This gives you greater flexibility. For example at this point in time only two other services need to invoke the Lambda. In the future you could hook this logic up to AppSync or a Lambda URL. The example below demonstrates how to structure the code to allow expansion in the future.

"""Example Lambda function.

This file contains an example of exposing functionality via a "pure Lambda" function
that can be called by the Lambda Invoke API. The core logic used by the Lambda is
separated from the handler to allow code reuse. The same logic called by the Lambda
URL handler.

For smaller project this can all be bundled into a single file. For large projects
each handler should be in a separate file with the logic in its own file.
"""
import json
import os

import boto3
from aws_lambda_powertools import Logger
from aws_lambda_powertools.utilities.data_classes import (
    LambdaFunctionUrlEvent,
    event_source,
)
from aws_lambda_powertools.utilities.typing import LambdaContext

DYNAMODB_TABLE = os.environ["DYNAMODB_TABLE"]

logger = Logger()

class InvalidRequestError(Exception):
    """Raised when the provided payload is invalid."""

    pass

@logger.inject_lambda_context
def lambda_handler(event: dict, _: LambdaContext) -> dict:
    """Handle direct Lambda request.

    :param event: The request payload.

    :return: The requested resource.
    """
    if "id" not in event:
        raise InvalidRequestError("Missing key 'id'.")
    if not isinstance(event["id"], int):
        raise InvalidRequestError("id must be an integer.")

    return get_record(str(event["id"]))

@logger.inject_lambda_context
@event_source(data_class=LambdaFunctionUrlEvent)
def url_handler(event: LambdaFunctionUrlEvent, _: LambdaContext) -> dict:
    """Handle Lambda URL requests.

    :param event: The request payload.

    :return: The requested resource as HTTP response.
    """
    record_id = event.query_string_parameters.get("id", "")
    if not record_id or not record_id.isdigit():
        return {
            "statusCode": 400,
            "headers": {"Content-Type": "application/json"},
            "body": '{"error": "Bad request."}',
        }

    record = get_record(record_id)
    if not record:
        return {
            "statusCode": 404,
            "headers": {"Content-Type": "application/json"},
            "body": '{"error": "Not found."}',
        }

    return {
        "statusCode": 200,
        "headers": {"Content-Type": "application/json"},
        "body": json.dumps(record),
    }

def get_record(record_id: int) -> dict:
    """Fetch a record from DB.

    :param record_id: The ID of the record to fetch.

    :return: The requested record.
    """
    dynamodb = boto3.resource("dynamodb")
    table = dynamodb.Table(DYNAMODB_TABLE)
    item = table.get_item(Key={"id": record_id}).get("Item")
    return item

This approach of separating the interface and business logic is a key component of Alistair Cockburn’s Hexagonal Architecture or “Ports and Adapters” pattern. This is a pattern I encourage in all serverless applications. Not only does it allow reuse of logic, it simplifies testing.

Each AWS service defines its own unique event schemas. This results in engineers needing to write logic to support each service’s unique request and response formats. AWS Powertools helps abstract away these differences. If you’re using a supported language such as Python, Typescript, Java, or .net using Powertools should be a no brainer. If you’re not already using Powertools, you should consider it.

Too Good to be True?

There are some challenges with using the Invoke API.

Authentication and authorisation for invoking the Lambda relies on Amazon’s native IAM permissions. While Cognito pools can be used for auth, this reintroduces complexity and latency. When adopting this pattern, planning and ongoing vigilance is needed to ensure that teams don’t end up with a complex sprawling mess of IAM permissions.

To avoid naming conflicts it is common practice to specify naming prefixes for each Lambda function. The deployment tooling generates the unique suffix. This introduces the need to maintain a mapping of human friendly names and the deployed names for functions. One solution is to store these mappings in AWS System Manager Parameter Store. The parameters can be shared with consumers.

Web APIs can help define clean boundaries between services. As the Invoke API makes it easy to access logic in another service, it is easy to turn your micro services architecture into a distributed monolith. Often the distributed monolith evolves slowly over time. When someone realises, it can be too late to untangle the mess. Consider this each time you evaluate using the the Invoke API.

Just Another API

On the surface exposing Lambda functions directly appears to be quite different to a web API or other public interface. The fundamentals are the same. The normal rules of APIs apply. The API needs to be versioned so consumers are aware of backwards compatibility breakages. It needs to be documented so engineers can build their integrations with minimal effort. Cascading failures are still a risk, so design your integrations with this in mind.

The Lambda Invoke API is the minimal loveable API for exposing Lambda functions. We should embrace this RPC interface where the consumers are internal.

There are two use cases for this pattern. The first is query actions, where the caller needs data from this service. This pattern can be used for mutations, but without proper planning and error handling it can result in cascading failures. Instead of using this approach for propagating state changes, consider using asynchronous events.

The other is when building a collection of small composable Lambdas. I’ll save that for a future post. 🌊

Need Help?

If you want to adopt Proactive Ops, but you're not sure where to start, get in touch! I am happy to help get you.

Proactive Ops is produced on the unceeded territory of the Ngunnawal people. We acknowledge the Traditional Owners and pay respect to Elders past and present.