Projen for Serverless blueprints

This is an evaluation of the tool Projen as a way to give your project’s services and repositories a blueprint.

TLDR; Projen does the trick! Also, there is no — similar — replacement, yet.

Reasons for needing a blueprint tool

A serverless landscape has a nature where lots of infrastructure components are tied together to produce great value. A potential downside is that most application and platform engineers are not (yet) used to an explosion of that many infrastructure pieces and do not take the necessary countermeasures to ease the landscaping. The big cloud vendors have not produced a meaningful tool to manage this as well.

Months into a project — or when new people are onboarded— it becomes increasingly hard to keep track of all resources. Two years ago I devoted an entire article on it where I called it Serverless Spaghetti. I believe bigger projects need some blueprint solution to streamline your configuration and infrastructure.

Reasons for using a blueprint tool for Serverless

The blueprint should provide a good developer experience (DevEx), and deal with shared non-functionals. In modern environments, teams get quite some leeway to ‘personalize’ their stack, and discover what works out for them. Still, the more mature a platform gets, the more (shared) non-functionals can be taken care of as a cross-cutting concern. Many organizations now have dedicated platform teams to take care of this.

Topics, like built pipelines, environment setup, security scans, and database backups, are not where local flavoring shines.

Some more background. In most of my projects, I take a repository-per-service approach. Where my deployment stacks are analog to a repo. After five years of doing serverless, this still feels like the only way to go. Repo-per-service offers a place for facilitating infrastructure (like message queues), and still offers a natural ‘boundary’ where people start a discussion on shared resources, exactly like they should. A smaller granularity (like a repo per lambda) introduces too much complexity for your deployment state. On a bigger scale (mono-repo) you will cross multiple domains, teams, and organizational boundaries too easily.

With dozens of services and repositories, you need some way to keep your projects up to date.

Requirements for a serverless Blueprint tool

A service blueprint should provide for several requirements. In general, it should enforce a generic way to build, test, and deploy a service. It should have reasonable defaults and preferably offer some way to extend or customize this.

My top 10 requirements:

• Offer a generic way to build, test, and deploy.
• Take care of a project’s boilerplate and be a kickstart for new initiatives.
• Facilitate general file structuring of a code repository.
• Standardize configuration files, with your organization’s ‘defaults’.
• Contains facilities to produce a CICD-built pipeline.
• Offer standardized quality control gates aligned with the organization
• Offer some standardization on infrastructure cross-cutting concerns (e.g. backup, monitoring, etc)
• Include many industry/organization best practices out of the box.
• Support the full project lifecycle. It is not just about bootstrapping a project. It needs to be capable of upgrading projects as well.
• Integrate with industry-standard tools for CI, building, and testing.

Candidate solutions

Let's start with the bad news first. The current landscape is fragmented. CICD pipelines are vendor-specific. Infrastructure tooling has a reasonably small scope — purely to create resources in your cloud account (e.g. Terraform, CDK, SAM, and CloudFormation). Built and test tools traditionally come in all shapes and sizes. They are tied to the application layer programming language and have even more diversity. All tools need some configuration, which you need to maintain in many places. In practice, this means that in many organizations you will find teams producing solutions to fix one of these shared concerns. Like a shared Github-built template, a shared Terraform module, or some ‘platform’ or ‘library’ component.

Tools like Serverless Framework have been around for a while, but only solve half of the puzzle. My main concern is, that most are optimized for the ‘getting started’ phase. Now Serverless is becoming part of Enterprise solutions it needs more maturity and capabilities for mature environments.

Now the good news. Projen is the first solution that meets my requirements. In the past year I have used it for a small use-case and to drive an entire service blueprint for a Serverless landscape. I am pleased with the results.

Projen

Projen is a type of ‘generator’ tool. Roughly similar tools that you might have heard of or used are Yeoman (NodeJS), Cookiecutter (Python), and Copier (CLI-based). These tools provide an abstraction on top of existing build tools to ease bootstrapping projects, and some help maintaining it as well.

Projen’s history started in the NodeJS, Typescript, and AWS ecosystem and branched out to other languages and offers a range of project ‘starters’. The creator was also a contributor to CDK and this is where a tight integration shows its roots. Since its inception, it has moved to a more community-driven ecosystem offering templates for a wide range of NodeJS-ecosystem starters but also support for Python and Java-based processes are included.

I recommend giving Projen a try yourself to feel the DevEx. Next, I will discuss my personal experience with Projen.

My Projen experiences on AWS Serverless

It has been over a year since my first Projen test drive, and for the past six months, I have been a daily ‘user’ and have ‘customized’ its Typescript App template to the extent that it is now at least 80% custom.

So let’s see what is likable and what takes some time to get used to. So far, I have not yet tried the tool in a multi-language project. All of my endeavors have been in the NodeJS ecosystem. However, Projen would just as easily support a Python app Projen setup as well.

Pros

• The ‘getting started’ phase was enjoyable. If you use one of its ‘templates’ called ‘project types’ you immediately feel ‘the power’.
• After hundreds of updates to the template, the template is starting to feel like ‘home’. Most configuration options have been overridden, which is a good thing.
• The management of GitHub pipelines using YAML generation offers more advanced control of a project’s pipeline than I have ever seen using templates. In most organizations, people use shared and centralized workflow definitions. Configuration or extending these build steps is usually very limited. With Projen you have complete control of the output. This is a game-changer for complicated CICD pipelines!
• Even when your template is highly customized. Projen still feels like a helping hand compared to crafting something completely yourself (and maintaining it). The abstractions made by the engineers have a good balance to them.

Cons

• The documentation is far from perfect. You will find yourselves debugging your template to find out how the different lifecycle phases produce files. The good news is, that when you build/customize your template, this disadvantage slowly fades to the background.
• Not updating the config files you are used to, like the package.json takes some getting used to. For many, this is muscle memory.
• The multi-language support comes from JSII. Which makes compilation slow, hard to debug, and limits Typescript language support for your templates (not your actual app).
• I have only used Projen with GitHub Actions. I have seen some community activity, but I expect serious downsides when using this for other CIs like Gitlab.

Would I recommend using it?

Yes, definitely. I especially like the management of the Github Action pipelines. The contributions seem active enough for now. Also, I have not found a similar tool that could replace Projen for my Serverless projects.