Software development in the 2020s
“from batteries included to batteries as service mesh”
Our stack has evolved. In this article I will provide a perspective on how we deal with abstractions, components, and composability when we are building applications. Our stack has changed in twenty years from application development frameworks with ‘batteries included’ to decentralised composable libraries ten years later and to composable service meshes now.
This article is my reflection on the transition of application development frameworks. Even the concept of an ‘application development framework’ is becoming old and is replaced with ‘stack’. I like to stop and reflect where we came from and where we are heading to every once in a while. This article will give you my personal journey, moving Java EE to Serverless over the years.
2000s — Application frameworks with batteries included
I started developing software professionally in the 00’s. After exploring lightweight runtimes like PHP, ASP.Net, Coldfusion during university. Later I went for Java EE (Enterprise edition) because at that time it seemed like the most liberating set of tooling money could ‘buy’. I can remember a day in 2007 where one of my teammates just bought an up to date reference book of Java EE 5 and it came with a huge folded poster depicting all the api’s one could use.
We hung it on the wall of our office and regularly spent time reflecting on which capabilities offered by the API would be useful on certain issues. Using just bare Java EE we could manage to create a complete backend, and using SOA manage to create insanely complicated orchestrated enterprise architectures!
These days we would call these application monoliths. Dependency management consisted of a Maven pom file with a handful of module versions. Deployment artifacts consisted often of a single ear file which were produced by build pipelines. These are the days of three-tier or n-tier architectures. Most discussions about which third party modules to incorporate where about database or middleware connectors. Most modules were shared libraries containing domain logic or utilities, scoped to the company’s domain. Even with built pipelines and artifact repo’s it already proved a daunting task to manage. People became experts in knowing every detail of the libraries they used, which often took years of investment.
The 2010s — Decentralised composable libraries
In the 2010s, people wanted more choice, and less vendor lock-in. This led to improved abstractions. Modularity, and composability became key-words in marketing application development frameworks. Low coupling and high cohesion gave us a plug and play experience (on most days if you got the versions right). In the Java ecosystem, the Spring Framework became widely accepted. It even offers a wizard (Spring Initializr) where you could a.o. select your favorite persistence connector, desired unit test suite, and programming language of choice and it will provide you a readymade software stack.
At some points in my career I felt like the virtues of composability and reuse in software engineering are too ambitious. However the composability of hundreds of libraries that work happily together which are maintained by independent agents is certainly an impressive feat of engineering! Looking for example at the Spring Framework ecosystems, thousands independent dependencies work together with low coupling and an amazing cohesion.
Architecture-wise people moved from n-tier to microservice architecture. Beautiful and effective new technologies like GraphQL finally completely decoupled our frontend-teams from the backend layer. Trends like domain driven design, and messaging middleware like Kafka are providing healthier abstractions to further divide our business logic into services.
Midway through this decade we took it one step further. Micro-frontends and component-based building further decoupled the frontend. Libraries like React don’t even promote themselves as frameworks but instead provide basic rendering primitives which lead to an entire ecosystem of 3rd party libraries which you can mix and match to create what we used to call ‘our application framework’. In the backend we see a similar movement. Polyglot programming, and polyglot persistence created a situation where our backend landscape is becoming increasingly heterogeneous. People are no longer ‘stuck on a stack’ but use the best technologies to solve the solution at hand. Design paradigms like CQRS are finally becoming more commonly adopted. Cloud and container abstractions are certainly fuelling our increasingly diverse backend and storage landscape.
Expertise also shifted in this decade. Experts transitioned from being an expert in technology to being an expert in adopting technology. The fragmentation of our technology landscape inadvertently leads to an increase in generalists. Also, both Agile and DevOps have been promoting an increase in breath of knowledge, multidisciplinarity, and the rise of the Fullstack engineer have pushed us into broadening the skillset.
The 2020s — Batteries as service mesh
An explosion of subscription based services known as XaaS (anything as a service) is leading to an ever expanding array of options. The concept of a service mesh has been around for over a decade. However, initially this concept mainly explained that our application will be part of this huge mesh of interoperating entities. However we have been slowly dissolving our application into the mesh as well!
Modern security frameworks have made interoperability across service secure and reliable. Ever expanding standardisation have made it possible to connect services providers which offer services which sounded like edge-case solutions ten years ago. It’s clear that tech companies are trying to get to the ‘long tale’ of the service offering market.
Over two decades, our architectures have been slowly transitioning to higher abstractions. Serverless is the next logical step in this transition. It offers a higher level abstraction on your run-time. It’s easy to see that Serverless is providing an abstraction in terms of infrastructure, it’s in the name! However I also see that it will push us to a next step in how we view our stack as well. From my first proof of concept on, I just fell in love with the developer experience in terms of speed, the mature infrastructure, and the way it focuses your attention to code only the stuff that really matters. However you will find out really quick that functions (like in AWS Lambda) are not offering everything you need to build complicated application landscapes. You will be forced to integrate with peripheral services inside and outside your cloud provider. In the last three projects I worked in, the amount of lines of code in terms of application logic relative to infrastructure code is close to equal.
I think we need to change how we look at infrastructure automation. It’s no longer coordinating the services ‘under’ our application code, it’s about automating the mesh that makes up our stack!
What should you do in the 2020s?
I hope you enjoyed a small trip through memory lane. Here are a couple of recommendations if you want to ride the wave and develop applications using Serverless primitives.
- Be open to cloud adoption. Generic cloud platforms are ahead in offering many of the basic primitives that make up the ‘infrastructure’ of your applications. I expect more openness and collaboration to be possible the next few years, but at this point sticking to a cloud provider for core infrastructure gives the best developer experience.
- Add peripheral mesh services. By peripheral I mean services outside of the cloud vendor’s offering. Services like Sentry can (partly) replace AWS’s Cloudwatch and X-Ray. Auth0 is a nice alternative to AWS’s Cognito. Google Cloud Platform’s Firestore is a nice example of a serverless offering which is open to 3rd party collaboration.
- Adopt a general purpose scripting languages like Javascript (or Typescript), or Python. This offers the easiest adoption for cross-service provider api’s.
- Think about developer en runtime ease of use. Multi-cloud solutions can be difficult to monitor, debug, and trace.
