Nico de Wet

Name your Event-Driven Architecture Design Patterns

Wed, 20 Mar 2019 00:00:00 +1300

Introduction

A few years ago I inherited a system that was sold as being bleeding edge because it was using all the latest cloud technologies. It emerged that the original designer had bought the book Serverless Single Page Apps and decided to run with it.

A key challenge that became apparent in this system was that it was hard to reason about. This is despite an enormous amount of effort and care having gone into unit testing at least some of the layers. I had to have a coffee with the departed original designer to establish the sytem context in the first instance.

In time I came to realise that the system had an event-driven architecture in all its layers, but this description is not good enough because the term has many meanings, I would have to be more precise to help the reader by naming the exact design patterns that were used. But why name the exact event-driven design patterns? It is important to do because it makes the system easier to reason about without going into implementation detail.

So, with this, in this post I’ll describe four key event-driven architecture design patterns, the first two of which were used in the cited system.

The four event-driven architecture design patterns are listed below and in turn described in greater detail further on.

The Event Notification Pattern
The Event-carried State Transfer Pattern
The Event Sourcing Pattern
The Command Query Responsibility Segregation (CQRS) Pattern

The Event Notification Pattern

Almost all systems use the event notification pattern somewhere, that is at some level.

When you call a system event-driven, the event notification pattern tends to be seen by senior engineers as a particularly important part of the architecture.

Example Application

Our example application is a retail loyalty scheme application. The dollar value of tax invoices issued to members using point of sale software are used to credit the loyalty scheme point balance of members using some business rule.

In the UML 2 component diagrams illustrated below, remember that the dashed arrows represent a dependency relationship. So in the diagram directly below the Invoice Sink Service depends on the Loyalty Scheme Credit Manager.

The coupling between the Invoice Sink Service and the Loyalty Scheme Credit Manager is something that we may not want. Put differently we may look for a way to get around the cited coupling in terms of the dependency relationship by making sure the Loyalty Scheme Credit Manager knows about the Invoice Sink Service but not the other way around. The reason why we may want to do this is because something that is very generic, such as collecting invoices from point of sale software is generally something we want to depend on rather then have it depend on many other components.

Now, the way in which we can reverse the dependency between the Invoice Sink Service and the Loyalty Scheme Credit Manager, as illustrated in directly below, is to use an event.

A tax invoice issued event is conceptually an example of such an event.

Events and Commands

Its worth noting that the way in which communication patterns are realised affects whether we use the term events or commands. In terms of naming tax invoice issued is an event-oriented naming style and it is not a command to any other system and this is important. The reason why it is important is because our naming style more accurately describes the system to someone that wants to understand how it works.

In terms of how events or commands are implemented, it would generally be in the same way, using say messagaes and queues. So the only difference is in naming and consequential semantics.

Additional Benefits of Using Events for Notification

An additional benefit of the reversed dependencies between systems, and using events for notification between systems, is that it becomes easier to add additional systems or components that can listen to particular events using say publish-subscribe messaging patterns. In our example application, if there had been an “Invoice Sink Service” team in principle there would be no need to even talk to the said team to consume tax invoice issued events.

There is a particularly significant trade-off when it comes to the use of the event notification pattern. While we have, as a benefit, the decoupling of the receiver from the sender (i.e. great flexibility) we have as a drawback the difficulty in reasoning about the overall system behaviour. The latter occurs because there is no single system or codebase that can be consulted to reason about the overall system behaviour. The only way to figure out what is going on system wide is to follow or trace through the flow of messages throught the overall system. This is true when it comes to GUI software and similarly it is true is larger enterprise systems.

The Event-carried State Transfer Pattern

This is simply putting so much information into the event that going back to the source system for further detail is not necessary. For example a tax invoice issued event may contain all line item details, the payment receipt information if available and a host of other data such as the merchant. Likewise an merchant address changed event would conceivably contain the old address as well as the new address.

The overarching consideration is whether the querying load on the source system can be eliminated for unantipicated queries related to events. The answer is yes but this requires tranferring the state from the source system into a suitable data store at the target system.

With this pattern contact initiated from the Invoice Sink Service to the Point of Sale Software is forbidden. It logically follows that the latter will keep a copy of all the data its ever going to need. So that is the trade-off, on the upside downstream systems no longer need to call the source system which may eliminate remote network calls and so enhance performance.

On the upside availablity may be improved in terms of downstream systems which can tolerate the upstream system being down. On the downside duplicate data needs to be stored. In terms of downsides, as per the CAP Theorem, the price of high availablity is a lack of consistency and so we have eventual consistency. So, we are trading off high availablity for consistency.

So in summary with Event-carried State Trasfer trade-off the following benefits.

Decoupling.
Reduced load on supplier (duplicating data to avoid calling back).

Against the following costs:

Eventual consistency (due to replicated data).

The Event Sourcing Pattern

With this pattern, events that contain the details of a change are stored before being processed. The event is then processed to affect the necessary change. With this we effectively have an event log and the current application state which are two representations of the same world. The event log contains all the events that ever change the application state and that lead to its current application state. Both the event log and current application state would commonly be persisted but this in itself is not necessary to use the pattern.

In order to test that we are using the event sourcing pattern we would have to test whether we can, at any time, reproduce the current application state. The reason why we would want to do this testing is because a key benefit of the event sourcing pattern is being able to discard and rebuild the application state. An additional benefit is that you don’t need to focus on preserving the application state because it can easily be rebuilt.

Metaphors or systems that use event sourcing:

Version control e.g. git
Accounting ledgers - the current application state is an account balance which is derived from every credit and debit to that account.

Key pattern benefits:

Audit
Debugging
Historic State
Alternative State - In essence storing events provides new and interesting ways to look at data when compared with a current state approach. In that regard eventstore.org says As the events represent every action the system has undertaken any possible model describing the system can be built from the events.
Memory Image - No need to store the application state in a persistent store, the entire application state can run in memory and so this leads to high performance systems. Application developers can focus on having a rich domain model as per Domain-Driven Design by Eric Evans which in itself challenging. As systems increasingly have large amounts of non-volatile main memory this pattern becomes increasingly viable. Many systems have been built with the assumption that main memory is significantly limited (and so persistant store usage becomes mandatory), but this may no longer be the case. Entire systems can fit into main memory.

Key pattern drawbacks or downsides:

Unfamiliar - and so harder to work with.
External Systems - e.g. you cannot replay a call to an external system two weeks ago, you have to save every response from an external system as an event and make it a part of the replaying mechanism.
Event Schema - the issue here is how can you store events in such a way that you are confident that you can change the replaying code that processes them. With this the event schema becomes an important consideration.
Identifiers - when identifiers are generated it must be done in such a way that we can replay. This adds complexity as this must be thought through.
Asynchrony [questionable] - This may be cited as a drawback, because it makes it harder to reason about a system, but note there is no need for asynchrony in an event sourced system. Asynchrony does not have to be used when using an event sourced approach. People may introduce asynchrony to enhance responsiveness but this introduces complexity.
Versioning [questionable] - This is a potential drawback due to the ability to replay. The code in the system would be changing over time, and it would need to be compatible with the events in the event log. Or does it need to be compatible with the events in the event log? One could keep events up to a certain point in time in particular systems (e.g. a daily event log), which one could call a snapshot, so with this it’s not necessary to be overly concerned with the versioning aspect. Naturally this would not be possible in some systems and there is advice on how to deal with this.
Rolling Snapshots - Given that it may become impractical to replay millions of events we would need to introduce storing the state when all events of a particular type up to a point have been replayed. This adds complexity.

Note [Event Store], “the stream database written from the ground up for event sourcing”, has useful Event Sourcing Basics reference material. I would recommend reading all the subsections because for example the how to deal with Rolling Snapshots is good to know about.

The Command Query Responsibility Segregation (CQRS) Pattern

With this when we have a persistent store, we have a Query Model, which deals with reads, that is distinct from a Command model, which deals with updates, each with its own distinct service interfaces. With this you seperate the components that read and write from your persistent store. To be clear these components are seperate pieces of software i.e. modules. They may or may not be seperated at run-time.

The question is, when is this pattern appropriate? What are the trade-offs? Do teams get in trouble when using this pattern (as Martin Fowler suggests) and if so why? I’ll adress these further on.

It may be worth noting what Martin Fowler has stated, which is that this pattern often gets confused when people claim CQRS should be used in all systems because of how useful it is to have the seperate models for reading things and populating different models. As Folwer states, at the persistance layer we have been using seperate operational databases (online transaction processing databases) and reporting databases for a very long time and this pattern is almost unversally pervasive. That said, the separation of the update model from the read model in the service layer is rare.

It is interesting to read the Microsoft treatment of the pattern in conjunction with my most recent experience of applying Robert C. Martin’s Clean Architecture in an ambitious content management product development effort. In my mind we faced problems similar to what they mention relating to the same POJO being used for read and write operations. An interesting question is whether CQRS would apply at the core layer, that is when it comes to the Domain Driven Designed core layer, or only at specific layers such as at the interactor layer. Robert C. Martin’s Clean Architecture is certain more complex than the traditional CRUD design mentioned in the Microsoft documentation because traditional CRUD designs don’t have a clean core layer.

Useful References

Serverless Single Page Apps - Fast, Scalable and Available - Ben Rady, The Pragmatic Bookshelf, 2016
The Many Meanings of Event-Driven Architecture - Martin Fowler, GOTO Conference, Chicago 2017
What do you mean by “Event-Driven” - Martin Fowler, 07 February 2017
Event-Driven Architecture - Mark Richards, Software Architecture Patterns, O’Reilly Media, February 2015
Event-driven architecture style
Building Event-Driven Microservices with Event Sourcing and CQRS (Lidan Hifi, Israel)
Event Sourcing Basics
Versioning in an Event Sourced System
Versioning of Events in Event Sourced Systems
Command and Query Responsibility Segregation (CQRS) pattern
CQRS

From Options Analysis to the Automated Measurement and Management of Architecture Debt

Sun, 24 Feb 2019 00:00:00 +1300

As a sofware engineer, it is a certainty that, having contributed to and reviewed technical or non-functional requirements, you’ll be called upon to provide robust solution option analysis along with costing considerations. In the interest of the robustness, or the thoroughness, of an options analysis, which one would expect in an engineering discipline, one technique you could leverage is the Architecture Tradeoff Analysis Method (ATAM). In this post I touch on ATAM, its use in industry and the rearview mirror views of one of its luminary authors, Rick Kazman, roughly 20 years later.

If a software architecture is a key business asset for an organization, then architectural analysis must also be a key practice for that organization. Why? Because architectures are complex and involve many design tradeoffs. Without undertaking a formal analysis process, the organization cannot ensure that the architectural decisions made—particularly those which affect the achievement of quality attribute such as performance, availability, security, and modifiability—are advisable ones that appropriately mitigate risks.

The Architecture Tradeoff Analysis Method (ATAM) is a technique for analyzing software architectures that was developed and refined in practice by the Software Engineering Institute (SEI) at Carnegie Mellon University. Notice that Carnegie Mellon permits unlimited distribution of the ATAM technical report, of course reading the material is the easy part.

Arguably presenting stakeholders with a thorough options analysis is the bedrock of servant leadership and the mere act of acknowledging the need to openly analyse architectural design tradeoffs, formally and in workshops, is a great step forward for many organizations. ATAM starts with the need to identify the quality attributes (often called non-functional requirements which I believe Rick Kazman describes as a dysfunctional term) of a software architecture. It then proceeds with techniques to providing insight into how quality goals interact with each other and crucially how they trade off against each other.

Its worth noting that the ATAM principles and practices can and should be applied when it comes to detailed design which may initially not be regarded as architecturally significant. Requirements and insights into them change rapidly, taking the time to analyse and review options is important at all levels. Arguably agile development practices such as XP are designed to facilitate high bandwidth design review however not necessarily at the level of enterprise or software architecture.

It’s worth considering why, if they are sensible, practices such as ATAM are, in my experience at least, not prevalent in industry. As Rick Kazman, one of the authors of the ATAM technical report from 2000, noted in his 2018 European Conference on Software Architecture keynote the reason why is because software engineering researchers have failed to factor in the practical and often difficult context(s) that software engineers experience in industry i.e. reality. It is telling that the title of his keynote is Measuring and Managing Architecture Debt and that he provides the following image to represent the state of the average software project in industry.

He goes on to state that we typically don’t measure the degree to which we are in debt with the debt occuring due to the incremental accumulation of architectural flaws. He goes on to define how he believes the incremental accumulation of architectural flaws take place.

Little tiny design mistakes that programmers make innocenty because they are busy doing their “real job” which is implementing features and fixing bugs.

The topic that he addresses is broadly speaking Architecture Health Management. Of course while there is the notion of incremental accumulation of architectural flaws there are also what I would call big bang manifestations of architectural flaws. These events could be never fit for purpose from the start or when a critical business decision gets made for example a pivot in terms of business strategy.

In any case, I highly recommend watching the cited keynote on Measuring and Managing Architecture Debt. Perhaps in time the automated identification and quantification or architectural and design issues (e.g. Decoupling Level measures) that burden projects with ever-increasing maintenance costs will become standard practice. It appears as if the notion is gaining traction with books such as Your Code as a Crime Scene being published in recent years.

It is worth noting that at the end of Kazman’s presentation a member of the audience asked about the use of libraries and the rise of functional programming and how that gets factored into automated architectural debt analysis which entails for example Decoupling Level metrics. Good questions but they still don’t take away from the benefit of giving architects measures to quantify what they often know in their gut, for example that a particular system may be hard to maintain or test. The resulting benefit being that it arms the architect with hard facts that can be taken to management in terms of why they should invest in refactoring (maintenance) rather than building new customer facing features.

Finally, Kazman provides an answer to a question posed about microservices and DevOps in terms of how relevant their work is to industry. His answer is that even though a depedency may only exist at runtime its still a dependency. So the challenge is the extract the same type of metrics (e.g. Decoupling Level) from these dynamic architectures but the principles still apply, the challenge is to figure out how to discern dependencies that exist at runtime.

A does not call B in a programming language sense, its not a method call or something, its gonna be a message, maybe its a pub-sub or through a broker, or through some other mechanisms. So, we need different extraction tools to determine the dependencies which are often only established at runtime. However, once we have that information, the analysis can proceed exactly the same. A dependency is still a dependency. And so if you have a cycle of dependencies, thats probably a bad thing, whether that was via method calls or via some runtime relationship.

His final caution on the value of automated analysis is telling with the context of students implementing design patterns incorrectly 75% of the time after a semester long course on design patterns. He’s basically saying that patterns are important and of great value but if you cannot analyse them automatically the flaws will just breed and grow.

Clean Architecture and When To Avoid Libraries

Sat, 16 Feb 2019 00:00:00 +1300

I recently had the privilige of partaking in the inception phase of a multi-year project that was totally committed to Robert C. Martin’s Clean Architecture.

It was fun, but also timeconsuming. In that regard, although Mr. Martin makes convincing arguments when it comes to motivating designing and building a system using this approach, an excellent companion is When To Avoid Libraries by Benjamin Sandofsky.

In my mind Mr. Sandofsky makes a more balanced argument because he is pragmatic when he states

If you’re building a prototype, go with magic

Magic refers to frameworks and mystical libraries. Prototyping is naturally quite important when requirements are unclear or when spiking a new engineering initiative.

Spring Boot React Maven Starter For Agile Developers

Thu, 07 Feb 2019 00:00:00 +1300

In this post I talk about a Spring Boot React Maven Starter project set up to basically get agile developers going as fast as possible. What does as fast as possible mean? Also, what’s an agile developer? In other words what overarching principles apply here?

To answer the first question, as fast as possible means you can run it all up on your local with 3 commands:

$ git clone https://github.com/nicodewet/spring-boot-react-maven-starter.git && cd spring-boot-react-maven-starter/
$ mvn clean install
$ java -jar server/target/server-0.0.1-SNAPSHOT.jar

There is no Docker (yet - it will come later) and after the third step you can open http://localhost:8080 in your browser and see the app.

To answer the second question, agile developer means that working software in production at the end of the first sprint is not negotiable. It also means we accept that our user interface and interpretation of requirements expressed in software is a mere hypothesis, so we want to do everything we can to test the hypothesis as quickly as possible. Infinite scalability when we have 0 users is not a primary concern for example.

With the context of the cited sample project, to facilitate rapid development the following has been done:

The entire stack gets served from a single Spring Boot application so that we can build a simple Docker based pipeline to production.
The entire app is in a single repo with clearly marked client and server directories.
The developer can focus on core concerns and so React (Javascript) & Spring Boot (Java 8), not infrastructure or cloud concerns.
A single Maven command packages the app.
Multi-module Maven build.
No TypeScript - we’ll start with the variant of Javascript Next features provided by Create React App out of the box.

So, if the above sounds good to you, and perhaps you are stronger when it comes to Spring Boot than React, I’d recommend doing the following:

Read through Matt Raible’s Bootiful Development with Spring Boot and React
Checkout the repo I created and satisfy yourself that it works as advertised.
Delete the React components and create them from scratch (follow Matt Raible’s tutorial but without TypeScript).

As a final note, I think TypeScript is great and the way to go, but not before you have mastered Javascript and Javascript Next.

The Only Thing Necessary for the Triumph of Faux Agile is that Good Men Do Nothing

Sun, 27 Jan 2019 00:00:00 +1300

The U.S. Department of Defence recently published an informative document on How to Detect ‘Agile BS’. The schematic they include in the document is simple and some argue it offers advice to managers on how to fine-tune their projects.

In actual fact the document offers advice, and a set of questions, to everyone on a team. This makes it clear, in my mind, that preventing agile-scrum-fall or faux agile is a shared responsibility, it does not rest squarely on management shoulders.

In terms of context, when reading the document I could not help but wonder how time and again teams (or entire programs of work) remain in the ‘Agile BS’ state.

Surely at some stage, at least some individuals must have harboured thoughts to that effect.

Sadly, in my exeprience, these same invididuals keep quiet - and this is where things start going wrong. As the prolific speaker and author Robert C. Martin (aka Uncle Bob) said in one of his presentations, when addressing programmers, “you were hired to say no”.

References:

Organizational Health and Software Architecture

Fri, 05 Oct 2018 00:00:00 +1300

I sometimes pick up a business book to broaden my horizons. The latest one happens to be The Advantage: Why Organizational Health Trumps Everything Else In Business by Patrick Lencioni.

According to Lencioni, an organization needs to be both smart and healthy in order to succeed.

Smart characteristics:

Strategy
Marketing
Finance
Technology

Healthy characteristics:

Minimal Politics
Minimal Confusion
High Morale
High Productivity
Low Turnover

As a sofware engineer, the following statement by Lencioni caught my eye as something to note as it hints at what one may suspect. Namely that one cannot expect software engineering expertise, and software architecture in particular, to make an unhealthy organization healthy.

When an organization is unhealthy, no amount of heroism or technical expertise is going to make up for the confusion and politics that take root.

It may still be worth asking the question as to whether a software architect and software architecture could have a significantly negative impact on the health of an organization. In my mind the answer is undoubtedly yes.

In other words, the actions of a software architect could lead to the following:

Increasing Levels of Politics
Increasing Confusion
Lower Morale
Lower Productivity
Higher Turnover

In my mind, the take-away is to simply bear this in mind and ideally to explicitly include the avoidance of the above as quality attributes when designing a system and engaging with stakeholders.

One could argue that most would take this for granted, but I don’t think that is prudent.

Developers Should Abandon Agile

Sat, 23 Jun 2018 00:00:00 +1200

There is no doubt in my mind that daring to be critical of agile methods is career limiting.

In fact, just recently, with the best of intentions, I inadvertently talked my way out of a job by criticizing agile methods. Putting the same thoughts into writing on my blog is probably even more career limiting. But, I’ve seen development teams run for the door too many times to say nothing.

On that note, here I’m whole-heartedly agreeing with Ron Jefferies’ recent Developers Should Abandon Agile post. Like him I’m a programmer and after 20 years I don’t have any intention of ever stopping.

At this point, if you are a professional programmer you should stop reading this post and read Ron Jefferies’ post (or this associated article). I cannot say what he said better than what he did. I will however attempt to distill his advice.

In essence, rather than focus on whatever agile method you may be asked to use, focus your attention and energy on these two aspects:

your core skills, a part of which should entail writing clean, refactorable, well tested code
releasing it in a consumable fashion with confidence multiple times per day to add real value to the business you may be serving

No matter what agile method you may be asked to use, if you achieve both of the above your stakeholders are likely to be delighted.

Update: 4 September 2018

The following article, which is based on a keynote speach by Martin Fowler at Agile Australia 2018, is strongly supportive of Ron Jeffries views and my own. In it Martin Fowler suggests that the main challenges we should focus on, in 2018, are dealing with faux-agile and fighting the Agile Industrial Complex. Indeed.

The State of Agile Software in 2018

Coding on Crack

Sun, 05 Nov 2017 00:00:00 +1300

Craic or “crack” is a term that was taught to me by some rowdy Irish lads at an engineering school in Paris. They generally used the term in the context of enjoyment.

These times were back in ‘02/’03 just as Dubya was training for the Bush shoeing incident. It was a wildly optimistic time of chanting open bar, open bar, open bar to no avail at the local student pub.

Since those days, and leaving grad school, I’ve spent a solid 13 years in the software development trenches, mostly getting my hands dirty.

One of the key lessons I’ve learned in this time is to never compromise on your professionalism and values. In the end, you’ll enjoy yourself a lot more and you’ll probably stay in business longer than the rest.

This brings me to the ACM Code of Ethics of Professional Conduct. It’s worth a read from time to time no matter what your association with the ICT industry and in particular if you call yourself a software engineer. I should read it more often.

Re-Wiring Enterprise IT With DevOps

Thu, 13 Oct 2016 00:00:00 +1300

This article chronicles an enterprise experience of introducing DevOps and continuous deployment practices into, from a seasoned DevOps practitioners perspective, a virtual Ground Zero with overhead tracer fire. In the spirit of No More Consultants this was done in-house at the kiwi enterprise Jukt Micronics by a committed team of veterans and DevOps newcomers.

Full Article [PDF 749K]