Wednesday 25 December 2019

First-Class Module

This is the second in a series looking at the Inversion of Coupling Control type system for composition.  This article discusses a more general Module type system than the previous article's First-Class Procedure Type.

Note: some functional programming languages also attempt to define First-Class Modules.  The First-Class Modules defined in this article are created from inverted functions.

First-Class Procedure

To recap the last article, the First-Class Procedure's type is defined as follows.  Note that we exclude the dependency type, as dependencies are auto-wired.

FirstClassProcedureType {
    Class<?> parameterType;
    ContinuationType[] continuations;
}

ContinuationType {
    String name;
    Class<?> argumentType;
}

This defines the First-Class Procedure to have a single input parameter and multiple continuations out for further composition of logic and handling of exceptions.

Module

Having a single input with multiple outputs is fine for methods, functions, etc wrapped in First-Class Procedures.   However, when systems grow we don't want the complexity to make the inputs/outputs suffer similar increased complexity.  We want the inputs/outputs to provide an interface to encapsulate the complexity of the Module.  Note that without encapsulation, we don't get the ability to modularise the complexity of the application.

To enable an interface to the Module, let's create the following input / output types:

InputType {
    String name;
    Class<?> parameterType;
}

OutputType {
    String name;
    Class<?> argumentType;
}

To understand why these types are created, we are going to use the visual configuration of Inversion of Coupling Control to better aid understanding what is happening.

The following Module configuration represents a single input, handled by a First-Class Procedure that sends its result to an output:


In the above configuration, the First-Class Procedure is encapsulated in the Module.  All that is exposed from the Module is the Inputs and Outputs.  The resulting type of the above Module would be the following:
  • Input named "Input" with a parameter passed to the First-Class Procedure
  • Output named "Output" with the argument provided by the result of the First-Class Procedure execution
This, however, provides little improvement on the First-Class Procedure interface.

What becomes useful is the encapsulation of multiple First-Class Procedures to undertake functionality of the Module:


While a new procedure was included within the Module, there was no change to the interface of the Module.  Other configuration using the Module would be unaware of the internal addition of another First-Class Procedure.

We also need not limit ourselves to single inputs and outputs.   We could have an arbitrarily complex Module that has multiple Inputs and Outputs:


The resulting Module encapsulated the detail to have the following interface:
  • Input "Input"
  • Input "Input-2"
  • Output "Output"
  • Output "Output-2"
  • Output "Output-3"

Module Type

The resulting type for the Module is the following:

SectionType {
    InputType[] inputs;
    OutputType[] outputs
}
Note that OfficeFloor's naming is derived from its foundation in business concepts and subsequently calls a Module a "Section".

The Module (Section) has multiple inputs and multiple outputs.  These inputs/outputs can then be connected to respective outputs/inputs of other Modules.

Furthermore, Modules may themselves contain other Modules.  As inputs/outputs are connected for composition, Modules have the same input/output connectivity as First-Class Procedures.  The following configuration demonstrates embedding the Module at the start of this article within another Module:


Whether it is the Module containing a single First-Class Procedure or two First-Class Procedures is encapsulated and not of concern within the above configuration.  The use of the Module is only on the Inputs / Outputs exposed by the Module.  The rest of the complexity of the Module is encapsulated. This allows modularising the application's complexity.

First-Class Module

So the title mentioned "First-Class Modules", yet we've only discussed visually wiring together the Modules.

To essentially be "First-Class" the Module needs to be assigned to a variable.  Yes, there are other conditions.  However, for me, this is the simplest way of thinking about something being first class.

Well the above graphical configuration is built on Sections (Modules) being configured together programmatically.  The graphical configuration is actually a layer above the First-Class Modules (Sections) to enable easier comprehension of how the application is modularised.

You can see this in OfficeFloor's implementation of the graphical configuration used above in this article.  The above graphical configuration is via an Activity. An Activity is a specific specialisation of a Section (ActivityLoaderImpl source here).  The Activity translates the XML from the graphical configuration into the creation of Sections, First-Class Procedures, Inputs, Outputs.  Each of these in the Activity implementation are assigned to variables, stored in data structures, passed to functions, returned from functions, etc.  This makes the Section (Module) essentially "First-Class".

This input / output interface based on continuations is extremely flexible.  It is so much so that First-Class Procedures themselves are also just a specialised implementation of a Section (see ProcedureEmployer).

Summary

We have seen how we can encapsulate First-Class Procedures within First-Class Modules, and even First-Class Modules within themselves.

We have shown how the graphical configuration is actually taking advantage of the "First-Class" nature.  The graphical configuration is actually a higher level composition that provides both:
  • easier to comprehend modularising of the application
  • quicker configuration of the application (effectively just draw lines for composition)
Note that it is quite possible to programmatically configure up our application.  However, this requires understanding First-Class Procedures / Modules in significantly more depth.  Much more than junior developers may want to initially.

The graphical configuration of First-Class Modules, therefore, provides the simplicity for building modularised applications.  This is without having to deal with the complexity of the underlying constructs.  Something I'm finding other composition strategies are still having trouble with.

In the next article we look at how First-Class Modules can provide composition of varying existing composition strategies.  You may find that existing composition strategies only really consider programming in the small, rather than programming in the much larger - where First-Class Modules become a lot more effective in modularising and simplifying your applications.

Thursday 21 November 2019

First-Class Procedure Type System for Composition

This is the first article in a series looking at the Inversion of Coupling Control type system for composition.  The series will demonstrate how the resulting type system allows for easy composition.  This avoids much of the complexity of Functional / Object-Oriented composition.  The resulting type system and it's resulting composition is simple enough even for junior developers to comprehend.

This series has four articles:

  1. This article discussing the First-Class Procedure type system
  2. First-Class Module
  3. Demonstration application
  4. The underlying theoretical model for composition

First-Class Procedure

This article will look at the type system and composition of First-Class Procedures.

We've previously talked about the OO Matrix and how the method (and for that matter the function) suffers from coupling. We've also discussed how this coupling can be reduced by Inversion of Coupling Control. We've also shown how these inverted methods/functions can be weaved seamlessly together. We've even seen the industry moving towards these concepts with microservices.

What we haven't yet discussed is the type system and composition available with First-Class Procedures.

To discuss this, let's start with the well known concept of Dependency Injection.

Dependency Injection

Meta-data of dependency injection describes a list of dependencies with the following three attributes:
  • Field (or constructor parameter) to inject the dependency (providing the dependency's name - e.g. field name)
  • Type of the dependency
  • Optional qualifier (distinguishes dependencies of the same type)
With this information, the matching to dependencies can be undertaken. Note, some dependency injection frameworks have improvements on this.  However, the above meta-data is typically adequate for auto-wiring dependencies of the object.

Continuation Injection

Continuation Injection reduces a function/method invocation to a similarly simple set of attributes as Dependency Injection.

As only a single parameter can be passed (remaining are dependency injected), there is only one type required.   This is the optional parameter type.

Also, the continuations from the First-Class Procedure are all named.

Note that auto-wiring continuations is deemed error prone.   Automatically wiring continuation to matching First-Class Procedures requires differing parameter types for each continuation in the system.  In practice, most of the time it is the same simple domain data types being passed around.   So you end up having to qualify every continuation.

Furthermore, qualifying continuations provides little improvement on reducing complexity.  In qualifying all continuations you end up writing configurations for each continuation. In other words, you end up providing a mapping of each continuation name to servicing First-Class Procedure.  This configuration is very error prone.  We will show how this is made less error prone and easily comprehensible shortly.

Therefore, each required continuation by the First-Class Procedure is described with the following meta-data:
  • Name of the continuation
  • Type of the parameter for the continuation (i.e. type of argument sent to continuation)
Note that exceptions from the method/function are also modeled as continuations.  The type is the exception type. The name is the name of the exception type.

First-Class Procedure Type System

Putting the Continuation Injection meta-data together with the Dependency Injection meta-data, we get the following meta-data (type) information for the First-Class Procedure:

FirstClassProcedureType {
    Class<?> parameterType;
    ContinuationType[] continuations;
    DependencyType[] dependencies;
}

ContinuationType {
    String name;
    Class<?> argumentType;
}

DependencyType {
    String name;
    Class<?> objectType;
    String qualifier;
}


The above type can describe all First-Class Procedures.  As a method/function requires parameters, these are described by the DependencyType listing.  As method/functions require calling other method/functions, this is described by the ContinuationType listing.  As there is only one parameter to the First-Class Procedure, the parameter type describes the type of argument that must be supplied by invoking ContinuationType.

Note that we have not discussed Thread Injection typing.  This is actually derived from the DependencyType listing by matching on object's type to thread pool.  See Thread Injection for more information.

As we now can type the First-Class Procedure, we can look at using this type system for composition.

First-Class Procedure Composition

The composition of First-Class Procedures is focused on wiring together the continuations.  Dependency Injection provides the objects (state) to the methods/functions.  This wiring of dependencies is done separate to the continuations.  Therefore, we need only focus on continuations being wired to their servicing First-Class Procedures.

Now it is possible to do this in code and even configuration files.  However, as the system grows in complexity this listing of continuation name to servicing First-Class Procedures becomes very unwieldy.  Trying to decipher the system behaviour from lists of linkages becomes very difficult.

Therefore, we look at the First-Class Procedure as the following:
  • a processing node with an input anchor
  • varying number of output anchors for each of it's continuations
  • linkages as lines from output anchor to input anchor
This representation makes for graphical configuration of the composition.  The First-Class Procedure is nodes in the graph with lines representing the continuations between them.  Therefore, composition of First-Class Procedures is quite literally drawing lines between them.

As the graphical configuration is visually easy to comprehend, it makes it very easy for even junior developers to understand the application.

Summary

We have demonstrated how Continuation Injection simplifies invocation of methods/functions.  This simple meta-data model for Continuation Injection is similar to Dependency Injection for objects.

This simple meta-data enables a type system for First-Class Procedures.  This type system enables graphical composition of First-Class Procedures.

Furthermore, being graphical, the composition is very easy for junior developers to comprehend.

The next article will discuss how the First-Class Procedure type system is actually made more general to enable modularising of the application.  However, these modules still maintain a similar type system for inclusion in the graphical configuration.

Tuesday 15 October 2019

OfficeFloor IntelliJ Plugin

OfficeFloor is now available to develop within IntelliJ.

While the plugin was originally developed for Eclipse, we have now isolated the functionality into IDE agnostic bundle.   This means with a bridging implementation, it is possible to port the OfficeFloor plugin to other IDEs.

However, for now we are content with Eclipse and IntelliJ for OfficeFloor development.

Though if you would like to see OfficeFloor development supported by other IDEs, please get involved!

IntelliJ OfficeFloor Plugin is available at https://plugins.jetbrains.com/plugin/13151-officefloor (and in the IntelliJ Marketplace - when using latest IntelliJ version).

Monday 9 September 2019

Inverting Functions: Effect Thread binding for Stateless Actors

Functional programming can be perceived as "hard". Yes, spend time with it and it gets simpler and the benefits make your code definitely better. However, when type errors can start spanning multiple lines, it does suggest the abstract concepts may be "hard" to see clearly.

We really need to make it easier for junior developers to assist in functional programming of larger systems.

Now as functional programming pulls heavily on mathematics, I look to a prominent mathematician's early 1800's statement regarding getting better clarity on hard problems:
"Invert, always Invert", Mathematician Carl Jacobi
(though he was German, so it was actually "man muss immer umkehren")

What I understand Carl Jacobi was saying is that invert the problem for a different perspective to try and get an answer. Therefore, maybe in inverting functional programming we can get clarity on making it perceived as "easier".

So what can we invert about the function?

We invert the coupling.

I've written about Inversion of Coupling Control in object oriented paradigms. This is even to the point of identifying the OO Matrix. In summary, there are 5 couplings of the method that restrict us:
  1. Method Name
  2. Varying number of parameters
  3. Varying number of exceptions
  4. Return type
  5. Executing thread
Yet, what has this to do with functional programming?

The function suffers similar coupling. Ok, the exceptions are not thrown but returned as values. However, changing the function's return type, name, parameters, executing thread requires changing all other functions calling that function.

Now beyond concurrency, Actors provide means to isolate the coupling impact of these changes. As long as the messaging semantics between Actors does not change, the Actor is free to change all its internal functions as it sees fit.  This isolates the impact of changing a function to just the Actor.

So I have to ask. Functional programming is appearing in many mainstream languages. Furthermore, we've now been moving to multi-threaded programming for some time now. So why aren't Actors also becoming more prevalent?

For me, the reason is the Thread binding in the Actor is to State rather than Effect.  This to me makes use of Actors difficult.

My reasoning comes from these observations:
  • Actors encapsulate changes to State to make this thread-safe
  • Effects can wrap changes to State (whether internally in the system or externally)
  • Multi-threading provides more efficient execution of blocking Effects (such as I/O) *
  • Threading is configured to the Actor
* multi-threading also enables parallel processing CPU intensive algorithms on multiple CPUs.  For the purposes of this discussion, we will model this as an Effect requiring a different Thread.

Therefore, what I see in Actor modelling is the following relationship:
Thread (pool) -> Actor -> State
When it should actually be:
Thread (pool) -> (Blocking / Long running) Effect
In attempting to get the right break down of the system into Actors, we have to simultaneously:
  • Consider an Actor an Effect to assign it the appropriate Thread (pool)
  • Decompose State of the system into these arbitrary Actors
Now we're caught in the precarious problem of trying to find the right mix of logic, threading and state decomposition.  This is a "hard" problem to get right.  Once threading becomes involved, it immediately becomes a problem only for intermediate to senior developers.  No longer can we have junior developers assist in the system (or at least they need significant supervision).

Furthermore, developer resources are scarce and we need a way for junior developers to assist in building systems.  Yes, there is a beauty and satisfaction in building a finely tuned system that fits together like clockwork.  However, businesses don't really want to keep building expensive finely tuned watches just so customers can tell the time.  We need to make things easier not harder.

Hence, that's why I believe we need to invert the function to create a stateless Actor with focus on mapping Threading to Effects.  Details of how this can be achieved is available in the following articles:
What this achieves is the following:
def Actor(m: Message, s: State, t: Thread): Array[Effect]
def Effect(r: Resource): AlteredState

Simplifying this down:
def Actor(m: Message, s: State, t: Thread, r: Array[Resource]): AlteredState

Hey! Effect was just removed!

Yes, because we don't really make the mapping decision of Thread to Effect.  We make it based on the nature of the Effect - blocking / long running.  The Effect encapsulates this so we don't really know whether we are just quickly updating internal memory or making a blocking I/O interaction.

We make the Thread mapping decision based on the resources used by the Effect.  If the Effect requires a database connection, it is likely to be making blocking SQL calls.  If the Effect requires a HTTP Client, it is likely to be making blocking HTTP calls.  If, however, the Effect only uses non-blocking resources, it is very unlikely to be blocking.

Now if we extract out the Thread from the above Actor, we get:

def ActorEffect(m: Message, s: State, r: Array[Resources]): AlteredState
def Actor(e: ActorEffect, t: Thread): AlteredState

The junior developer is now free of threading to write the ActorEffect.

Later a senior developer is able to make the correct mappings of Thread to ActorEffect.  This is because the determination can be made by the resources (whether blocking or not) used by the ActorEffect.

Furthermore, the ActorEffect can be written just as a function with State passed in.  This makes the ActorEffect stateless so the junior developer is not involved heavily in thread-safety.

The ActorEffect is now able to be written by the skill level of the junior developer.

There are further improvements on the weaving together of the ActorEffects.  The following article provides working code of how junior level functions can be weaved together with threading configured separately afterwards:

   Weaving together functions (and other paradigms)

Therefore, it is now easy for junior developers to assist in building large functional applications.  With the use of Prototype Threading by IoC, this is potentially to millions (even billions) of stateless Actors scattered across thousands of underlying physical machines. But the ease of writing small functions remains for the junior developers making it easier for them to be involved on large scale projects.

And thus I ask, whether in attempting to improve functional programming by finding even more niche and complex mathematics is the right way forward?   Or do we follow Carl Jacobi's advice and invert, always invert!

Thursday 8 August 2019

What is The OO Matrix?

Morpheus: I've seen an agent punch holes in designs with only a single dependency. Developers have fought against them with APIs that resulted in nothing but limitations. Yet their strength and their speed are still based in a world that is built on coupling. Because of that, they will never be able to write as simple or as fast code as you can.

Neo: What are you trying to tell me, that I can design objects for easy refactoring within applications?

Morpheus: No, Neo. I'm trying to tell you that when you're ready, you won't have to.

Take the red pill. Read on and I show you how deep The OO Matrix goes...


In my opinion, The OO Matrix is what happened to Object Orientation when the interfacing of objects devolved to the method.

As per my other article Inversion of (Coupling) Control, there are 5 caller coupling aspects to the method.
  1. Return type
  2. Method name
  3. Variable number of parameters
  4. Variable number of exceptions
  5. Executing thread
This method based object interaction is a long way away from the intended message passing vision by Alan Kay (as I understand the history).  Typically, we have envisioned Object Orientation to look as follows, with nice round objects linked by lines:



However, objects interacting via methods have very different shapes.  The variability of the method coupling has objects look a lot more like jigsaw pieces.  The connectors are not simple lines, but complex methods that shape the object differently:



Piecing these method interfacing objects together forms a rigid jigsaw.  This rigid jigsaw is difficult to refactor due to all the coupling.  The result is The OO Matrix:

 

With the birth of Java in the 90's and the ratification of the C++ standard in the same decade, "it is not without a sense of irony" that the Matrix film came out about the same time we locked ourselves into programming within The OO Matrix.  Some of us may sense it's existence with refactoring frustrations, while others grew up in it knowing no difference.

The OO Matrix exists all around our programming coupling us to the machine.  If we consider:
  • the method represents pushing/popping state on/off the thread stack
  • the thread stack represents execution by machines
  • then designing interfaces based in methods is binding us to the machines
  • and as such we perpetuate The OO Matrix

And really, where in the real world do we find a thread stack? Object Orientation was derived from chemical reactions. Businesses operate in message passing (e.g. emails and documents). Even my understanding of our brains is built on message passing. In other words, thread stacks are a machine based construct. Using the thread stack derived method for object interaction binds us into the machine - The OO Matrix.

See my other articles on First-Class Procedures and Local Microservices about how we can break free from The OO Matrix.  These articles demonstrate using messages to interface between objects (i.e. single parameter unnamed methods). This breaks us free from the method, the thread stack, and subsequently freeing our code from machines - The OO Matrix.

Well, this is how I like to describe it - as it's a pretty dry topic otherwise!

But I certainly cry about all those hours (if not days/weeks) of refactoring code in my life due to the method coupling problem.  Especially, as if we followed Alan Kay's vision of focusing on message passing, this should never have been a problem.  And to me, this may actually be a more expensive mistake than null.