I have somehow managed to revive my blog without having to write any code, change any CSS or do any busy work at all thanks to the miracle of Github pages and the fact that my website was always just a pile of static pages generated by some terribly written JS that hasn't changed in nearly a decade - who says code rots?

Where the hell have I been?

I'm honestly quite surprised at how many people have reached out over the last few years to ask this question - I can either consider this a testament to how much impact I've made over those years, or simply because I've always been a noisy person and any silence from over in my corner is deafening.

The answer is that it turns out that when I'm actively engaged in writing software that people are actually going to use and I'm being fulfilled by that task, that I really don't feel the urge to try and talk about software development with people. The fact is I've spent nearly a decade now helping our customers with their bespoke media workflows and that job has been all the software development chat that I really need in my life.

The last few years we've been heads down productising (is this a word?) the knowledge and code we've built up over the years so that our potential customers (who often already have software teams of their own) can start to build out some fairly advanced capabilities without having to become media technology companies themselves and without them having to teach us everything about their business.

Those were definitely some words

There are a lot of off the shelf media technology solutions out there for performing a wide variety of tasks, and they are typically (because of the complexity involved) fire and forget "set it up and forget about it" systems. Media is messy and there are a lot of media products out there who are responsible for taking content from a variety of sources, packaging up that content and then distributing that content to end users so it can be consumed.

A starter solution for fledgling companies who reach the limits of these black boxes might be to write some scripts to glue these tools together (often using ffmpeg), and while this might get you so far, sooner or later this approach runs out of steam and somebody needs to write some actual code.

At this point you've got two options - you can perform this task in-house and take on the complexities that this involves (hint: this is a dangerous path!), or you can take on the services of one of the many systems integrators out there and get them to do this for you with all the in-house expertise they have, exchanging the cost of technical ownership for a simple license or service fee.

We (id3as) have always taken on the more challenging of these jobs, the jobs that others would maybe consider to be technically expensive and the jobs that have maybe even already been attempted by others to little success. We have been unafraid to use new (and old) technologies to solve problems and because of our experience and existing codebase we often find that the bulk of the effort is in the understanding of the business requirements of our customers rather than the technical challenges presented by the media itself. And isn't that the rub? The people with the business knowledge that know what they want are the best people to be building software around their own requirements, but the people who know how to deal with the complexities of fairly involved media workflows are external to that. For our own sins this often means we end up with customer logic inside our shared code out of necessity because everybody has a different idea about how fairly basic operations should take place (quality of service controls, pid mapping, source failover, etc).

We have repeatedly for our own benefit attempted to separate these concerns in order to better be able to respond to the needs of our customers without burying ourselves in complicated configuration and while we have always managed to get to somewhere better than we were, we haven't (until recently) come up with a platform that really delivers this.

Enter Norsk

I'm not entirely sure how it happened - we were taking our existing stack and upgrading it to help us maintain our own velocity (that's when we started heavily using Purescript for those that have been paying attention!), and in the process of doing this we started to see the shape of a product that we could use to keep our customer's business logic and the complex media orchestration separate and wholeheartedly pursued it.

Not only did it make sense, but it started to look as though we might be able to let other people use this platform to build their own complex workflows and that was incredibly exciting to us - obviously we did the only sensible thing at this point and committed to it - using it to do a few jobs for our existing customers as well as sharing it with a few trusted partners to see what they made of it.

The result of all this is that we're now shipping a product comprised of some incredibly clever technology written in a language that helps us avoid failure (Purescript), built on a platform that makes its name out of handling failure when it happens (Erlang) and allowing business logic to be written in a universally understood language (Javascript/Typescript) whilst taking care of the nitty gritty details of workflow management and ensuring automatically that customers have all the introspection, logs, metrics, and debugging tools that companies have come to expect in modern distributed systems. As we always have done, we integrate directly with the codecs we use and implement most protocols ourself so we have the power to work with the "nuances" of other implementations and as a result of this we have a flexibility to respond to challenges that is hard for a lot of people to fathom.

In a nutshell

You spin up a Norsk server, connect to it over the Typescript SDK and say

• "I want these ingest nodes to exist",
• "I want these transforms and control nodes to exist"
• "I want these output nodes to exist"
• "These are the actions I want you take as data flows over these nodes"
  • Subscribe nodes to the nodes
  • Update internal dashboards/etc
  • Update the configuration of running nodes on the fly

That last section is probably the most interesting - instead of fire and forget configuration, every decision that Norsk makes about media flows through the client application - a new stream appears and the client application is asked "where do you want this to go?", metadata changes and the client application is asked "does this change what you want?", various control nodes can flow frame by frame information to the client and allow it to respond in realtime to events in the media and update config across any node without causing any service interruption.

Norsk takes care of the media technology problems such as orchestration, codec knowledge, protocols and conversions and the client application gets to take care of decisioning and control - this is somwhat revolutionary. I several times over a recent conference where we launched the product tongue-in-cheek simplified this to "it's like being able to run ffmpeg and change everything about anything while it's running", but even that's a gross simplification of what Norsk actually gives the customer.

I'm really excited about what this means for the streaming industry and I'm really excited to see where we end up taking it. We have a lot of capability in our core platform that isn't yet exposed in Norsk so we're only really starting out in our task to place that power in the hands of our clients.

Anyway - I'll be doing various technical blogs about Norsk and related subjects in the coming months over at the Norsk Blog, and I'd be delighted if anybody were to join me over there for that.

• Purerl Updates - Processes and Self
• Purerl Updates - Subscriptions
• Purerl Updates - Untagged Unions
• Purerl Updates - Maxing out Pinto OTP
• Purerl Updates - GenStatem

One of the big advantages of using an existing language like Purescript when building out typed support for Erlang, is the presence of existing tooling and libraries for the language (such as language servers, package management, syntax highlighting, etc - as well as libraries for all the normal maths, string manipulation, date times, testing and all that).

That's not to say you get those things for free however, in a perfect world we could add libraries to our package set directly from the original repos - the litmus test for whether this is possible is "Is there any Javascript in the original repo?". If there isn't then the library will work out of the box but if it has other dependencies that aren't yet in the package set then they will need adding to the package set too.

On top of that, one of the things we 'lack' in Purerl is the Aff monad. We (as far as we can tell so far) don't need it because we're just not writing code that way - or rather with the presence of various means of performing async actions in Erlang it simply isn't necessary. This does mean that any library that brings in Aff (such as some of the popular testing frameworks) will need porting to use sometihng else unless somebody feels like porting Aff? No I thought not.

Consider the library purescript-parsing which was needed recently to parse RFC8216.

On the face of it, there is nothing but Purescript in the repo and the tests are incredibly simple, however it has a dependency on purescript-unicode which whilst being entirely Purescript also, not only has a dependency on purescript-quickcheck which absolutely does have some Erlang in it, but also generates Erlang that breaks the Erlang compiler because of 'complexity'. (hard coded lists of unicode characters meant nested function calls all the way down).

Quickcheck needs the JS re-writing as Erlang, and then if you want to build unicode/parser independently and run their tests then they need their packages updating so they can pull the fork of quickcheck. Thankfully these libraries all (sensibly) have their tests written as a main :: Effect Unit rather than using a fancy testing library so the changes are very fast and easy to implement. As a result of running the test suite of course, some missing features were found in purescript-strings for which, thankfully built-in functions exist in Erlang so no hard work was required.

We also needed purescript-datetime which has a lot of the underlying implementation built off of Javascript's date system and needed rewriting to use Erlangs. Since then this has become embedded in a lot of other packages because we consider things like Seconds and Milliseconds to be important primitives worth representing as something other than Int and defining our own newtypes everywhere doesn't make a lot of sense. If you're going to be using Datetime heavily then you probably also want to format dates and times so that means bringing in purescript-formatters which has a dependency on parsing (cool, already ported!), but tests written against Aff for no good reason (That's fine, replaced them with good ol' purescript-erl-test-eunit.

I guess what I'm trying to say is that there are yaks all the way down, and that all of these libraries kindly written by other people don't come for free. On the bright side because we've been actively writing Purescript for three years we've brought across a lot of them ourselves already. The complete list of packages ported to the id3as organisation at this time of writing is:

• purescript-these
• purescript-parsing
• purescript-pathy
• purescript-formatters
• purescript-unicode
• purescript-datetime
• purescript-quickcheck
• purescript-uri
• purescript-sequences
• purescript-dagre
• purescript-longs

These will end up in the default package set over time so they can be referred to without needing to know where they are - but if you're venturing down the route of using Purescript and needing a package that doesn't exist, then check the Purerl org and the Id3as org to see if somebody hasn't already done the leg work!

• Purerl Updates - Processes and Self
• Purerl Updates - Subscriptions
• Purerl Updates - Untagged Unions
• Purerl Updates - Maxing out Pinto OTP

One of the big motivations for the big push on purescript-erl-pinto was to facillitate the addition of other OTP behaviours than GenServer.

One of our "go-to" servers is gen_statem. When writing any sort of implementation of a stateful network protocol for example, it is nice to model various steps of the protocol as 'states' on which only certain types of message can be enacted. A gen_statem has a list of these possible states (as an atom in this case), as well as 'Data' which is an arbitrary term.

The simplest example outside of this however, is the locked door. A door which is locked cannot be opened, and a door that is opened cannot be locked. If the door is open we can take the treasure behind the door and to unlock the door we need a key. This could be modelled with three states here, an

• Locked
• Unlocked
• Open

Some of a plain ol' gen_statem to represent this would look like this.

init([]) ->
    {ok, locked , #{ key => <<"open sesame">>, treasure => <<"The moon is made of cheese">>}}.

handle_event({call,From}, { unlock, K1 }, locked, #{ key := K2, treasure := Secret }) 
  when K1 =:= K2 ->
  {next_state, unlocked, #{ treasure => Secret }, [{reply, From, ok}]};

handle_event({call,From}, { unlock, _K1 }, locked, Data) 
  {keep_state, Data, [{reply, From, bad_key}]};

handle_event({call,From}, { unlock, _K1 }, unlocked, Data) 
  {keep_state, Data, [{reply, From, ok}]};

handle_event({call,From}, open, unlocked, Data) 
  {next_state, open, Data, [{reply, From, ok}]};

And so on and so forth.

A locked door has a key and a treasure, a call to 'unlock' with the right key will give you an open door with just the 'treasure', a call to unlock with the wrong key will result in an error, a call to unlock when already unlocked is just fine thank-you and a door that is unlocked can be opened. A few things of note that we're not a huge fan of here.

• There is nothing to tie the current 'state' with the 'data', thus there is nothing stopping a locked door from not having a key (oh dear!)
• There are a lot of tuples with arbitrary atoms floating around and the possibility for typos is endless
• States aren't defined up front, any arbitrary atom can be returned and we'll now be in that state

We could be generous and say that two of these are actually a feature and not a bug, but beauty is in the eye of the beholder and if you're a fan of type systems then this has all the beauty of my macaroni art from pre-school.

From a Purescript perspective, it'd be nice if we could tie the current state with "state specific data" using an ADT, as well as have some state that is common to all.

data State 
  = Locked { key :: String, treasure :: String }
  | Unlocked { treasure :: String }
  | Open

startLink :: Effect (StartLinkResult DoorLockPid)
startLink = do
  GenStatem.startLink $ GenStatem.defaultSpec init
  where
    init = pure $ InitOk (Locked { key: "open sesame", treasure: "the moon is made of cheese" }) {}

Now, the only problem with our new State is that the extra data bundled with it is means that it can't be used directly as a 'state' in gen_statem because each variant would be a new state!

A typeclass provided therefore be able to extract a 'stateid' to pass to the underlying gen_statem implementation

data StateId
  = StateIdLocked
  | StateIdUnlocked
  | StateIdOpen

derive instance eqStateId :: Eq StateId

instance stateHasStateId :: HasStateId StateId State where
  getStateId (Locked _) = StateIdLocked
  getStateId (Unlocked _) = StateIdUnlocked
  getStateId (Open _) = StateIdOpen

A default implementation could exist of course to simply extract an atom from the underlying representation, building that is left as an exercise for the reader..

Anyway, just like with GenServer and Supervisor, we can represent a GenStatem with a Pid and a Type in order to communicate with it and build up various callbacks and such. The type specs are quite daunting beause GenStatem has quite a few different 'message types' - most of thes can be ignored and set to Void/Unit until needed.

newtype StatemType :: Type -> Type -> Type -> Type -> Type -> Type -> Type -> Type
newtype StatemType info internal timerName timerContent commonData stateId state
  = StatemType Void

newtype StatemPid :: Type -> Type -> Type -> Type -> Type -> Type -> Type -> Type
newtype StatemPid info internal timerName timerContent commonData stateId state
  = StatemPid (Process info)

Our very simple DoorLock Genstatem which doesn't have any messages outside of direct 'calls' could be defined as

type DoorlockType = StatemType Unit Unit Unit Unit {} StateId State
type DoorlockPid = StatemPid Unit Unit Unit Unit {} StateId State

With these types defined, a semi-complete implementation of our door-lock GenStatem could therefore look like

serverName :: RegistryName DoorlockType
serverName = Local $ atom "doorlock"

startLink :: Effect (StartLinkResult DoorlockPid)
startLink = 
  Statem.startLink $ Statem.defaultSpec init { name = Just serverName } 

data UnlockResult = Unlocked | BadCode 

unlock :: String -> Effect UnlockResult
unlock k1 = GenStatem.call (ByName serverName) 
  \from (Locked { key: k2 } _ = 
    if k2 == k1 then do
      let actions = GenStatem.newActions # GenStatem.addReply (GenStatem.reply from Unlocked)
      pure EventNextStateWithActions (Unlocked { treasure }) {} actions
    else do
      let actions = GenStatem.newActions # GenStatem.addReply (GenStatem.reply from BadCode)
      pure $ EventKeepStateAndDataWithActions actions

The GenStatem functionality in Pinto is being used in a few places in our codebases now, but it is not complete. There are some hand-wavey bits around timers and messaging that will get fixed when somebody needs it, but the pattern and general types are pretty much now in place. Various callbacks can be supplied by overriding the default spec, those callbacks will receive certain kinds of messages that will need adding to the type signatures and various return values can be built up by using the functions provided in the GenStatem module.

There are other kinds of Genserver available that could do with representations in Pinto and as we saw in the last entry, now we have a generic way of representing 'pids' and 'types' that can supply and enforce their own internal messages, it should be possible to do this without making any substantial changes to pinto itself.

A note on Pinto's future

Pinto is now 'conceptually' pretty complete and hopefully shouldn't undergo any more major changes. The types that have been added can be used to describe pretty much any running proces and various functionalities that can be exposed to interact with those proceses - from the simple "This process has a state and can receive messages" to "This process has a state, can receive this type of video stream and this type of audio stream, send these messages on a timer..." and then be stuck into Supervisors in the same way as any other server.

There are probably better ways of representing this in Purescript by deviating further from OTP itself and re-imagining a more Purescript central means of modelling these things. Eventually somebody is going to take the plunge and do this and it probably won't be us. Should this happen, I don't think it'll take long to move away from Pinto and to it - so if anybody fancies the task...

• Purerl Updates - Processes and Self
• Purerl Updates - Subscriptions
• Purerl Updates - Untagged Unions

When I first threw together Pinto, it was with the aim that there simply be a usable path for building supervision trees and gen servers, and not that we would have access to the full functionality of those things straight out of the gate. This proved to be a sensible choice because as we learned more about how we would like to write code in Purescript built around various messaging mechanisms it also turned out that this code would need to evolve more than a few times before we were all truly happy with the model.

A lot of usable application code was written with this "V1" pass, but it was always known that eventually we would want to not only have access to other "gen" models (such as statem), but the bullet would need biting and the model would need expanding to allow us to use the full gamut of functionality exposed by gen_server.

Thankfully, it turns out that the general concepts that had evolved during the development and real-world use of V1 carried over, and the changes as far as the end-user is concerned are largely additive. (Note: I didn't do most of the work on V2, that was @philipstears and @adrianroe, so kudos to them on gathering the will for it).

The types

In V1 Pinto, the types around the spinning up of processes was very GenServer focused

data ServerName state msg = Local Atom
                          | Global GlobalName
                          | Via NativeModuleName Foreign

Not only that, but the result of calling the various startLinks for Supervisors/GenServers was very simplistic - losing any information about whatever the process was that was started. (Hello Raw.Pid)

data StartLinkResult
  = Ok Pid
  | Ignore
  | AlreadyStarted Pid
  | Failed Foreign

As a result, this meant things like Supervisor had their ServerName defined as

type SupervisorName = ServerName Unit Unit

Which isn't wrong as such, but is completely pointless. State and Msg are GenServer constructs and they aren't even the only ones we need for a GenServer which is about to get a revamp anway. The answer (as it often does), lies in adding type parameters - while we're at it, changing the API to use an Either because that's a good way to represent success and failure.

data NotStartedReason serverProcess
  = Ignore
  | AlreadyStarted serverProcess
  | Failed Foreign

type StartLinkResult serverProcess
  = Either (NotStartedReason serverProcess) serverProcess

We can't add constraints to the type definion itself, but we can enforce that when these things are used that HasPid serverProcess =>, which is what Pinto will do in most of its APIs. By doing this, we can have GenServers export a new type (ServerPid state msg) and Supervisors export their own type (SupervisorPid) which both have Raw.Pid underlying them, but GenServer can also (HasProcess msg (ServerPid state msg)) which Just Makes Sense.

A similar thing can be done to ServerName

data RegistryName :: Type -> Type
data RegistryName serverType
  = Local Atom
  | Global Foreign
  | Via NativeModuleName Foreign

Why would we want that? Because calls into running processes (such as GenServer.call or Supervisor.startChild) might like to know type information in order to enforce some level of correctness, and while we're at it we can support doing this either either the Pid gained from startLink, or with the registered name of that child

data RegistryReference :: Type -> Type -> Type
data RegistryReference serverPid serverType
  = ByPid serverPid
  | ByName (RegistryName serverType)

Now this has been done, we're free to re-write Supervisor (or indeed, as we will see, split it into two different modules), and re-write GenServer with all the types we want without that leaking across implementations.

GenServers

Consider the following very boring GenServer in V1 that starts up, subscribes to a bus, and receives messages via handleInfo

type BookWatchingStartArgs = {}
type State = {}

data Msg = BookMsg BookEvent 

serverName :: ServerName State Msg
serverName = Local $ atom "handle_info_example"

startLink :: BookWatchingStartArgs -> Effect StartLinkResult
startLink args =
  Gen.buildStartLink serverName (init args) $ Gen.defaultStartLink { handleInfo = handleInfo }

currentState :: Effect State
currentState = Gen.call serverName \state -> pure $ CallReply state state

init :: BookWatchingStartArgs -> Gen.Init State Msg
init args = do
  self <- Gen.self
  _ <- Gen.lift $ SimpleBus.subscribe BookLibrary.bus $ BookMsg >>> send self
  pure $ {}


handleInfo :: Msg -> State -> Gen.HandleInfo State Msg
handleInfo msg state = do
  case msg of
    BookMsg bookEvent -> 
      -- TODO: Something with bookEvent
      pure $ CastNoReply state

There are a few things of note here

• The GenServer has to have a name (not strictly true but..), this isn't really necessary for lot of simple_one_for_one for example
• How can 'init' fail?
• How does one stop this gen server from inside the gen server?
• How does one defer a reply in response to a 'call' (Usually gen_server would give you a 'from' argument, this has been dropped here)
• The use of 'Gen.self' (see also: Purerl Updates - Processes and Self
• How does one do a handle_continue, and what is its type going to be?

As we can see, this largely comes down to missing items from the surface area of the gen server rather than a fundamental design flaw, but nevertheless there are some breaking changes to get to the point where these things are supposed.

Starting a GenServer

A few things need to change here

• A GenServer can ask to 'continue' an exchange by returning the current operation and re-entering with a message sent internally - we need a type for this
• A GenServer can ask to 'shutdown', and can supply a 'reason' for that shutdown that can be intercepted elsewhere/internally and it'd be nice if that was typed
• The return result of the startLink isn't typed at all, this should be a Pid of some sort that encapsulates that types of messages this GenServer can receive
• Name should be optional

Thus, we need two extra types added to our ServerName (Cont and Stop), and StartLinkResult actually needs to be typed around some sort of Pid.

serverName :: RegistryName (GenServer.ServerType Unit Unit Msg State)
serverName = Local $ atom "handle_info_example"

startLink :: BookWatchingStartArgs -> Effect (StartLinkResult (GenServer.ServerPid Unit Unit Msg State))
startLink args = GenServer.startLink $ (GenServer.defaultSpec (init args)) { name = Just serverName, handleInfo = Just handleInfo }

init :: BookWatchingStartArgs -> GenServer.InitFn Unit Unit Msg State 
init _args = do
  _ <- SimpleBus.subscribe BookLibrary.bus BookMsg 
  pure $ InitOk {}

So

• serverName is now a RegistryName parameterised with GenServer.ServerType (Cont Stop Msg State)
• GenServer.startLink now simply takes a record full of config for which a default can be summoned with defaultSpec
• serverName is completely optional (as is handleInfo, both are Maybes to indicate this)
• StartLinkResult is now parameterised with the type of Pid we're starting with, in this case a GenServer.ServerPid (Cont Stop Msg State)
• init now returns an indication as to whether it has actually succeeded or not..

We'll also note that with the addition of HasSelf, we can now use subscribe without liftEffect, just relying on ambient context to get the messages sent to self (See also Subscriptions)

Our handleInfo has changed as well (as has every other callback, but in exactly the same way) as it needs to also include all the additional type information for this GenServer

handleInfo :: GenServer.InfoFn Unit Unit Msg State
handleInfo msg state = 
  case msg of
   BookMsg bookEvent -> 
      -- TODO: Something with bookEvent
      pure $ GenServer.return state

Conceptually, nothing has really changed in our basic use cases except more types have been added to various signatures to make additional functionality possible (which we can default to Unit until we need them), as well as all of the callbacks now operating inside a HasSelf msg => (actually a ResultT cont stop msg state), so that all of the type information for this GenServer is available at all times.

A heap of methods now hang off GenServer to help return the appropriate responses from the various callbacks (CallFn, CastFn, InfoFn, ContFn, TerminateFn) which are all typed safely around whatever we've declared in our signature.

It's certainly a bit more verbose because of all of these extra types, but typically what ends up happening in our code is that we have business-process specific monads written which sit inside a GenServer and only expose the bits we need so that we don't have to write Unit Unit Unit State repeatedly. A lesson from the last couple of years with typed FP - with flexibility comes verbosity...

Supervisors

Supervisors in V1 were okay for most cases except simple_one_for_one.

• A reading of the documentation for Erlang OTP reveals that in most cases, both the arguments and the return value of API calls into supervisor:** are special-cased for *simple_one_for_one
• There was no way in V1 of tying the construction of the simple_one_for_one supervisor and the subsequent calls to start_child together, thus it was very hand-wavey with regards to type safety

The solution? The solution is to have two modules - one for "simple_one_for_one" (Pinto.Supervisor.SimpleOneForOne) and one for everything else (Pinto.Supervisor). This makes the API much easier because the return values of functions stop being dependent on the type of supervisor being ran (in Purescript at least) and we can disregard the special cases in both modules because the code doesn't allow for us to end up in situations where we're calling one type of supervisors with another's pid.

Just like with GenServers, we'll see that these are constructed around RegistryName and StartLinkResult pid, this is once again so that we have typed pids available for invocation into the various APIs in a standardised manner.

Normal supervisors (from demo-ps).

startLink :: Effect (StartLinkResult SupervisorPid)
startLink = do
  Supervisor.startLink (Just $ Local $ atom "example_sup") init

init :: Effect SupervisorSpec
init = do
  connectionString <- BookConfig.connectionString
  webPort <- BookConfig.webPort
  pure
    { flags:
        { strategy: OneForOne
        , intensity: 1
        , period: Seconds 5.0
        }
    , childSpecs:
        (worker "book_web" $ BookWeb.startLink { webPort })
        : (worker "empty_server" $ EmptyGenServer.startLink {})
        : (worker "book_library" $ BookLibrary.startLink { connectionString })
        : (worker "handle_info_example" $ HandleInfoExample.startLink {})
        : (worker "monitor_example" $ MonitorExample.startLink {})
        : nil
    }
worker ::
  forall childProcess.
  HasPid childProcess =>
  String -> Effect (StartLinkResult childProcess) -> ErlChildSpec
worker id start =
  spec
    { id
    , childType: Worker
    , start
    , restartStrategy: RestartTransient
    , shutdownStrategy: ShutdownTimeout $ Milliseconds 5000.0
    }

You'll note that the server has a name (Local $ atom "example_sup"), calls to startChild/terminateChild/deleteChild can be made with ByName $ Local $ atom "example_sup" if we so needed. In the case of simple_one_for_one (below), this server name has the start args and pid type of the children encoded in it so that calls to startChild/etc are typed.

simple_one_for_one:

serverName :: RegistryName (OneForOne.SupervisorType OneForOneGenServerStartArgs OneForOneGenPid)
serverName = Local $ atom $ "one_for_one_example"

startLink :: Effect (StartLinkResult (OneForOne.SupervisorPid OneForOneGenServerStartArgs OneForOneGenPid))
startLink = OneForOne.startLink (Just serverName) init

init :: Effect (ChildSpec OneForOneGenServerStartArgs OneForOneGenPid)
init =
  pure { intensity: 100
    , period: Seconds 60.0
    , childType: Worker
    , start: OneForOneGen.startLink
    , restartStrategy: RestartTransient
    , shutdownStrategy: ShutdownTimeout $ Milliseconds 5000.0
  } 

startClient :: OneForOneGenServerStartArgs -> Effect OneForOneGenPid
startClient args = do
  crashIfChildNotStarted <$> Sup.startChild (ByName serverName) args

Monitors

With untagged unions, it is entirely possible now to use monitors directly from erl-kernel, if our process is either single-purpose or is already using untagged unions with say, gun. However, in a lot of cases it still makes sense to use the message router in Pinto for this because we want our code to maintain simplicity with a simple ADT. For this convenience an implementation of Monitor still exists inside Pinto which implements this for us.

For anything that has a Pid, and assuming we're inside something that HasSelf, we can call

ref <- Monitor.monitor pid ProcessDown

where ProcessDown is a constructor that takes a Pinto.Monitor.MonitorMsg

data MyMsg 
  = Tick
  | ProcessDown Monitor.MonitorMsg

A MonitorMsg is presently defined as

type MonitorObject
  = Foreign
type MonitorInfo
  = Foreign
data MonitorType
  = Process
  | Port
data MonitorMsg
  = Down (MR.RouterRef MonitorRef) MonitorType MonitorObject MonitorInfo

Which will need expanding should anybody actually need that information. Because we have a router sat in the way, the usual usage of monitor is to pass all of the information we need into the message that will be sent to us and ignore the MonitorMsg entirely

ref <- Monitor.monitor pid (const $ ProcessDown pid)

where

data MyMsg 
  = Tick
  | ProcessDown Pid

Pull requests are accepted should somebody want to make the effort to expand those types into reality for some reason.

Timers

Timers have also had a slight change in line with the new HasSelf typeclass and thoughts around subscriptions, as well as importing the recently ported DateTime libraries so that we have sensible duration types available to us.

So instead of

me <- self
liftEffect $ Timer.sendEvery 1000 me Tick

We have the far simpler

Timer.sendEvery (Milliseconds 1000.0) Tick

This will probably be moved to erl-kernel in time, because it no longer does anything special and looks exactly like the underlying API.

Summary

Conceptually not a lot has changed and yet we've ended up in a place where we can use the full range of functionalities exposed by GenServers and Supervisors in a type-safe manner, we can chalk that up as a win - especially as it means this Github issue from March 2019 can finally be closed.

• Purerl Updates - Processes and Self
• Purerl Updates - Subscriptions

We are now here to talk about untagged unions and how they can be used when dealing with legacy messages being sent to our process and why we might want to do something differently at all.

Consider a dependency to a legacy Erlang library that has a subscribe call that sends us messages

legacy_api:subscribe(),

receive 
  { data, Bin } -> ..
  { err, Reason } -> ..
  eof -> ..

It isn't possible to subcribe to this in Purescript without doing something to unpack those Erlang types. We could have

workerLoop :: ProcessM Foreign Unit
workerLoop = do
  msg <- receive
  case (LegacyApi.unpack msg) of 
    Err err -> ...
    Data bin -> ...
    Fin -> ...

startWorker :: ProcessM Foreign Unit
startWorker = do
  LegacyApi.subscribe
  workerLoop

Simply leaving it to the caller to know that the Foreign needs running through the API again in order to decipher it.

We could also use the MessageRouter in Pinto to spin up another process that knows how to do this for us so this translation is already done once we're in user code

import Pinto.MessageRouter as MR

workerLoop :: ProcessM LegacyApi.Msg Unit
workerLoop = do
  msg <- receive
  case msg of 
    Err err -> ...
    Data bin -> ...
    Fin -> ...

startWorker :: ProcessM LegacyApi.Msg Unit
startWorker = do
  me <- self
  MR.startRouter LegacyApi.subscribe LegacyApi.unsubscribe (self self <<< LegacyApi.unpack)
  workerLoop

This is a viable solution for most scenarios because it is easy to understand, reasonably lightweight, allows us to accept more than one kind of message, etc. It incur the "cost" of spinning up another process, and it does incur the cost of doubling the number of messages being sent for a single interaction.

Usually this isn't a problem (or more - if doubling the messages or processes is going to cause you issues, then it's possible that the codebase already has issues anyway!).

Having said all of that, for FFI around existing Erlang where there are there are numerous variants, writing mapping code in Erlang could be rather error prone (and passing in constructors from Purescript gets tiresome after half a dozen) - it starts to make sense to instead try to describe the data types "in place" in Purescript using something formal.

For this, there is the package purescript-untagged-union.

Now, two of these values are directly representable in Purescript because of the way in which types are represented in Erlang under the hood.

data LegacyMsg 
  = Err Binary
  | Data Binary

We can let the untagged unions library know that underlying data maps onto this with an instance of RuntimeType that describes it

instance legacyMsg_runtimeType ::
  RuntimeType
    LegacyMsg
    (RTOption (RTTuple2 (RTLiteralAtom "err") RTBinary)
               RTTuple2 (RTLiteralAtom "data" RTBinary))

Come to think of it, that 'err' is horrible, we're in Purescript and there is no reason to have such an ugly constructor - how about renaming it during mapping?

data LegacyMsg 
  = Error Binary
  | Data Binary

instance legacyMsg_runtimeType ::
  RuntimeType
    LegacyMsg
    (RTOption (RTTuple2 (RTLiteralAtomConvert "err" "error") RTBinary)
               RTTuple2 (RTLiteralAtom "data" RTBinary))

We still have that atom all by itself however, the atom 'fin' isn't representable as a Purescript type, it is only an Atom but we'd like still like to be able to receive it!

Well, we can build a type that represents our incoming messages, including that atom.

type Msg = Union |$| AtomSymbol.Atom "fin" |+| LegacyMsg |+| Nil

And then all that is left is to use this in a process that can receive these messages

workerLoop :: ProcessM LegacyApi.Msg Unit
workerLoop = do
  msg <- receive
  ( case_ 
      # on (\(m' :: LegacyMsg) ->
          case m' of 
            Err err -> ...
            Data bin -> ...
        )
      # on (\(_ :: AtomSymbol.Atom "fin") ->
            ...
        )
  ) msg

startWorker :: ProcessM LegacyApi.Msg Unit
startWorker = do
  LegacyApi.subscribe 
  workerLoop

This needs to be exhaustive or we will get a compile error which is super cool.

All of this works without an additional process or a fumbly mapping layer. It's not perfect and relies on actually getting the type description correct but this at least presents a way of doing it without writing error-prone Erlang, which can be useful when dealing with APIs that have a whole array of random structures.

A standard procedure in Erlang, is to invoke an API that then proceeds to send messages back to the invoking process.

some_api:subscribe()

receive 
  Msg -> ..

More often or not, an overload will be provided allowing a Pid to be passed in, in case you didn't want it sending to self.

Pid = spawn_worker(),
some_api:subscribe(Pid)

We could write a function that looks like this in Purescript (using the typeclasses explored in the previous blog entry).

module SomeApi where

data Msg = HelloWorld | Goodbye

subscribe :: forall process. 
  HasProcess Msg process => 
  process -> Effect Unit
subscribe = ...

In this, we are saying "Given a process that accepts Msg, then the subscription will send msg to you.

workerLoop :: ProcessM SomeApi.Msg Unit
workerLoop = do
  msg <- receive
  case msg of 
    HelloWorld -> ...
    Goodbye -> ...

main :: Effect Unit
main = do 
  process <- spawnLink workerLoop
  SomeApi.subscribe process

This isn't great, typically we don't write processes this way - processes usually not only want to subscribe to messages from external sources, but also send themselves messages, typically they'll achieve that by lifting messages into a process specific data type.

data ProcessMsg 
  = ApiMsg SomeApi.Msg
  | Tick

workerLoop :: ProcessM ProcessMsg Unit
workerLoop = ...

Calling SomeApi.subscribe with this process will result in a compiler error because SomeApi.Msg is the wrong type.

Emitters

The pattern we've been using up until now to side-step this is for subscription calls to take in a blank cheque in the form of msg -> Effect Unit, allowing the caller to decide what to do with messages on that subscription.

module SomeApi where

data Msg = HelloWorld | Goodbye

subscribe :: (Msg -> Effect Unit) -> Effect Unit
subscribe = ...

Subscription could then look like this

data ProcessMsg 
  = ApiMsg SomeApi.Msg
  | Tick

main :: Effect Unit
main = do 
  process <- spawnLink workerLoop
  SomeApi.subscribe $ send process <<< ApiMsg

More commonly the use of some form of self would be used inside the process itself, as it makes little sense for one process to subscribe on behalf of another.

workerLoop :: ProcessM SomeApi.Msg Unit
workerLoop = do
  msg <- receive
  case msg of 
    HelloWorld -> ...
    Goodbye -> ...

startWorker :: ProcessM SomeApi.Msg Unit
startWorker = do
  me <- self
  liftEffect $ SomeApi.subscribe $ send me <<< ApiMsg
  workerLoop

main :: Effect Unit
main = do 
  _ <- spawnLink workerLoop

This works and is elegant - and indeed for about two years this has been The Way. Our codebase is/was littered with send me <<< Msg, but in hindsight this isn't great.

Why not? The problem with Effect Unit is that it can do anything. ANYTHING. The problem with passing an Effect Unit is that it can do that "anything" within the context of a completely different process. This can cause problems.

Errors: What happens if the Effect Unit causes an exception? Who crashes? The remote process? Who does that affect? Anybody else who is currently subscribed? How do they find out? Are they in the right supervision tree for this not to be a problem? Can we just swallow the exception? How does the caller find out that it crashed? Oh dear.

Interference: What if this is in a message loop doing other things? What if the Effect Unit takes up precious time? What happens to the backlog? What happens to anything awaiting a message with a timeout?

While the reality is that while nearly all our callbacks are implemented as send me <<< Msg, as our codebases grow and we do more and more of these things, somebody is eventually going to cause problems with the above and we'll be looking at hilarious (and needless) debug sessions as a result. If not us - then somebody else using Purerl (and our numbers are growing...).

HasSelf to the rescue

HasSelf was added fairly recently and makes it possible to write code that operates inside any monad provided it has an implementation of 'self'.

saferSubscribe ::
  forall m msg.
  MonadEffect m =>
  HasSelf m msg =>
  (Msg -> msg) -> m Unit
saferSubscribe f = do
  me :: (Process msg) <- self
  liftEffect $ subscribe $ send me <<< f
  pure unit

Adding onto that, we need to assert that we can evaluate effects in this monad (MonadEffect m =>) in order to actually issue the subscription and suddenly our call to this becomes

workerLoop :: ProcessM SomeApi.Msg Unit
workerLoop = do
  msg <- receive
  case msg of 
    HelloWorld -> ...
    Goodbye -> ...

startWorker :: ProcessM SomeApi.Msg Unit
startWorker = do
  SomeApi.saferSubscribe ApiMsg
  workerLoop

main :: Effect Unit
main = do 
  _ <- spawnLink workerLoop

Note: The liftEffect is gone, as is the call to 'self', the API looks a lot more like the original Erlang and is safer while we're at it.

Because it's no longer a blank cheque, we should probably provide a second method (just like in Erlang) that takes a process to send messages to instead of simply relying on 'self'.

saferSubscribeTo ::
  forall msg.
  Process msg -> (Msg -> msg) -> Effect Unit
saferSubscribeTo p f = do subscribe $ send p <<< f

This is how most of our subscriptions now look across Pinto/our own code/etc. It's safer for everybody concerned and fits well with the ethos of 'trying to look like the Erlang APIs most of this stuff is built on top of'.

Quite a lot of code has been written against those projects internally by colleagues and myself and over time:

• Various improvements/patterns have been discovered within core libraries such as purescript-erl-process.
• A heap of Erlang specific packages have been written and released
• Another mountain of packages have been ported across from Purescript
• Pinto and Stetson have been upgraded repeatedly by various colleagues as our understanding of what we need them to do has evolved.

The purerl cookbook has been updated for these latest releases, as has the demo-ps but felt it worthwhile highlighting some of the changes in a few (of my increasingly rare) blog posts.

In this post we'll be looking at Processes and the concept of 'self'

Processes

A common practise in Erlang codebases is to spawn a new process and use its Pid for communication.

example() ->
  Pid = spawn_link(fun receive_message/0)
  Pid ! hi.

receive_message() ->
  receive
    SomeMessage -> 
      ?PRINT("Got a message"),
      ok
   end.

In Purerl, the type Pid lives in Erl.Process.Raw and is just an imported foreign type

foreign import data Pid :: Type

This typically isn't used to any great amount except in some specific FFI cases, it being far better generally to use the types found in Erl.Process which have the phantom type msg floating around meaning all the sends and receives are limited to the types of message that that process has declared it will receive. Process msg itself is just a newtype around Raw.Pid of course.

newtype Process (a :: Type)
  = Process Raw.Pid

To create a typed process, one could call spawnLink and provide a callback which will be executed in the process created in that underlying call to spawn_link in Erlang. A change made fairly early on in development was to change this callback from something that took some context, to something that operated inside ProcessM msg r, with the context being provided by that monad - this will become important later on in this blog entry so I'll demonstrate this here.

The old way

"Given a callback that accepts a SpawnLinkContext typed around msg, run that callback inside a new process and return that new process, also typed around msg"

spawnLink :: forall msg. (SpawnLinkContext msg -> Effect Unit) -> Effect (Process msg)

That context then provided the means of receiving messages, being defined as something like

type SpawnLinkContext msg = 
  { receive :: Effect msg 
  , receiveWithTimeout :: Timeout -> Effect (Maybe msg)
  }

Thus, the Erlang example, re-written in Purerl would have looked something like this

example :: Effect Unit
example = do
  pid <- spawnLink receiveMessage
  pid ! "hi"

receiveMessage :: SpawnLinkContext String -> Effect Unit
receiveMessage c = do 
  msg <- c.receive
  log "Got a message"

The new way

"Evaluate the given code in the context of a ProcessM typed around msg"

spawnLink :: forall msg. ProcessM msg Unit -> Effect (Process msg)

And quite simply, any calls to receive/etc are defined as functions that operate inside ProcessM, again all typed around msg

receive :: forall msg. ProcessM msg msg
receive = ProcessM Raw.receive

This cuts down on the cruft somewhat, as instead of having to pass a context everywhere, one can simply write functions in the context of ProcessM, re-writing that initial example now looks like this (Note that the log call needs lifting into ProcessM because it is written as an Effect)

example :: Effect Unit
example = do
  pid <- spawnLink receiveMessage
  pid ! "hi"

receiveMessage :: ProcessM String Unit
receiveMessage = do 
  msg <- receive
  liftEffect $ log "Got a message"

Extending this model

It turns out that this is quite a nice pattern for representing the different types of process available in Erlang (OTP, Cowboy and indeed our own application code), consider:

• Cowboy Loop handlers
• Cowboy Websocket handlers
• OTP gen_server
• OTP supervisor
• OTP gen_statem

All of these could be ran as ProcessM, except they have more types associated with them and various functions available designed for use in those specific contexts, for example in the simple case of cowboy..

type WebSocketInfoHandler msg state
  = msg -> state -> WebSocketResult msg (WebSocketCallResult state)

We have the types msg and state in our type because the callbacks involved tend to take state and there is a callback (info) for messages received by the loop handler typed around msg

In the more complicated case of an OTP GenServer, this looks like this

type CallFn reply cont stop msg state
  = From reply -> state -> ResultT cont stop msg state (CallResult reply cont stop state)

Most operations take place inside that ResultT which encodes the cont, stop, msg, and state types for use with our operations. (cont being the message that can be received by handle_continue, stop being a custom stop reason, msg being messages received by handle_info and state being the state of the gen server).

We also end up with our own contexts in our own codebases for specific process types around common units of business logic.

Common Functionality

In most of these cases, just like with Process.spawnLink, something gets returned that represents the started process - for example the GenServer.

startLink :: forall cont stop msg state. 
  (ServerSpec cont stop msg state) 
  -> Effect (StartLinkResult (ServerPid cont stop msg state))

Here we have a ServerPid cont stop msg state returned to the caller - again keeping a lot of useful information around to help us make calls into a GenServer but the type we're interested in here is msg. Most generic APIs will be written around the concept of a Process msg and what we have here is a ServerPid cont stop msg state

The logical step here is to expose

toProcess :: ServerPid cont stop msg state -> Process msg

It turns out that this is a very common operation and so a typeclass is born and added to Erl.Process for everybody to implement when writing this kind of code.

class HasProcess b a where
  getProcess :: a -> Process b

In fact, two typeclasses are born because some APIs only need a Pid after all.

class HasPid a where
  getPid :: a -> Pid

Most functions that take a Process msg therefore don't actually care about it being a Process msg, but only that a Process msg can be gotten from the type

callMe :: forall p msg. 
  HasProcess msg p => 
  p -> Effect Unit

And now all of those custom types can be used with a whole suite of APIs without having to unpack a convoluted structure of newtypes.

Self

There is one more common operation that has been ignored so far, and that is the concept of self. An incredibly common thing in Erlang is to invoke self to get the Pid of the current process.

Self = self(),
some_api:call_me(Self)

For a while, we started having self methods on every module that exported some monad in which process logic could be evalulated, and this would return the full type of the process (complete with cont, stop, state, etc). There were a lot of self functions being exported and imported and in 99.99% of all cases they were immediately followed by a call to getProcess using the HasProcess typeclass implementation for that system.

(me :: Process MyMsg) <- getProcess <$> GenServer.self

What does self mean then? The correct answer is as written above, but the correct answer isn't always the nicest answer - it was very rare that we would need anything from self other than the current Process msg and we were running into issues in modules that had code for more than one of these contexts in them, whose self are we using anyway?

The answer was to be pragmatic and create a typeclass for 'any m' that allowed that 'm' to export a Process msg

class HasSelf (m :: Type -> Type) msg | m -> msg where
  self :: m (Process msg)

For the case of anything running inside a ProcessM, this isn't any more complicated than calling Raw.self to get the current pid and wrapping it up with the relevant newtypes

instance selfProcessM :: HasSelf (ProcessM a) a where
  self :: forall a. ProcessM a (Process a)
  self = ProcessM $ Process <$> Raw.self

Similar implementations then exist for Pinto/Stetson contexts, allowing code to simply call Process.self from practically anywhere to get a typed Process msg valid for the current context.

SomeApi.callMe <<< self

All typed, all safe, nobody sent messages they can't handle - living the dream.

I've been working in Erlang now for nearly a decade and for the last year or two we have been investigating what it would take to start writing a lot more of that Erlang in Purescript instead because in theory working in a sensibly typed language means fewer bugs means fewer support calls means more time spent working on features which can be sold and therefore this is good for profit margins.

Well let me tell you - it blows, we have lost weeks trying to write typeclasses around row types to confirm that 'this chunk of binary is good to go into this piece of code that just wants a chunk of binary' and months pontificating over whether an attribute should be declared at the type-level or merely at runtime and whilst progress has been made, it has been a slow and painful exercise.

There has to be a better way

I worked with one of my colleagues in a previous job many years ago - it was an honest living. We wrote VB.NET together in the heyday of .NETs 1 and 2 and by golly we were very productive indeed. It has types but it also has freedom, It has generics but it also reads really cleanly because so much of its syntax is just words - in other words it is almost the perfect language for writing expressive code. Since departing the world of dotnet however I have noticed its younger brother, C#, appearing in environments like Unity and other game engines as a scripting utility and that got me thinking - what if we could use host .NET in a similar way and use C# as a real language and build real software with it instead? (Leaving us to use VB.NET at our leisure whilst appealing to the masses that prefer C#).

Catching up with dotnet

It has been a while since I touched base with dotnet, probably again about a decade. I largely quit because the default tooling was very mouse heavy and my wrists were starting to get sore (true story), not to mention most enterprise shops using the technology being heavy with the derp. (Some) things have apparently changed since then, tooling-wise everything seems to revolve around the dotnet CLI and with the language-server happenings being commonplace across most platforms, the support for writing .NET languages in Vim is not half bad out of the gate. Huzzah.

.NET Core 5.0.0 seemed the obvious candidate seeing as that seems to be a unification point for all the complicated versioning stories that Microsoft is so fond of creating. Version 5.0.0, not just one zero - but two, that's how you know that software is ready for use!

The solution

We have a lot of legacy Erlang so if we're to start writing dotnet code to replace it we'll first need to find a way to interoperate - the obvious solution is to launch the dotnet code inside the existing Erlang Application side by side so they can talk to each other.

If we write the following app in dotnet (spin up a gen server and return the pid)

  public class MyEnterpriseGradeApp : IApp {
    public Object Start() {
      return GenServer.StartLink(() => new MyCoolGenServer() );
    }
  }

We can embed it in a standard Erlang Gen Supervisor like so:

  init([]) ->
    {ok, { #{ strategy => one_for_one }, 
           #{ start => { dotnet_shim
                       , start_link
                       , [ "priv/acme.dll", "Acme.MyEnterpriseGradeApp" ]
                       }
            , id => acme_app
            , type => worker
            }
       ]}}.

And this just works - Those of you who know both Erlang and dotnet at this point will hopefully be picking your jaw back off the floor - what on earth is going on here?

The simplest example

At its heart, an IApp is simply a class that has a Start method that'll return something back up to Erlang, these can be launched manually from Erlang to execute arbitrary code - for example, here is an app written in dotnet that returns a fixed string to us...

  public class HelloApp : IApp {
    public Object Start() {
      return "Hello Joe";
    }
  }

Can be executed from Erlang like so

  { ok, Result } = dotnet:run_app_from_assembly("priv/acme.dll", "Acme.HelloApp"),
  ?LOG(Result). %% Hello Joe

You will notice that the result from dotnet was automatically wrapped in an {ok, Result} tuple, as most Erlang APIs will expect that, if our dotnet code throws an exception then this will be automatically returned back as an { error, Reason } instead - this seemed to be the cleanest way of expressing error conditions (and a special Exception (TermException) class is provided for explicitly providing that Reason to the Erlang host.

Passing values in

A generic implementation of IApp exists, IApp, for which an input can be requested by the Start method, for example adding two numbers together

  public class AddApp : IApp<Tuple<Int,Int>> {
    public Object Start((int x, int y)) {
      return x + y;
    }
  }

  { ok, Result } = dotnet:run_app_from_assembly("priv/acme.dll", "Acme.AddApp", { 5, 5 }),
  ?LOG(Result). %% 10

The observant reader will notice that we're using a dotnet Tuple<Int, Int> here and passing in an erlang tuple { integer(), integer } and it just maps across automatically, how does this work? Ah well, go read the source code to find out or wait for the blog posts that I'll be writing on that subject, for now just accept that you can stick pretty much anything in your type signature and the equivalent Erlang types will get converted across. How about using a record instead of the tuple for example?

  public record AddArgs (int X, int Y);

  public class AddApp : IApp<AddArgs> {
    public Object Start(AddArgs args) {
      return args.X + args.Y;
    }
  }

  { ok, Result } = dotnet:run_app_from_assembly("priv/acme.dll", "Acme.HelloApp", #{ x => 5, y => 5 }),
  ?LOG(Result). %% 10

Not a problem. The observant reader will notice that not only is the casing automatically adjusted (Erlang is typically snake_case, and dotnet is typically PascalCase) but return type of Start is just a plain ol' object, and the same thing applies there, return anything you like and it'll get converted into something that Erlang will understand. Records on records? Not a problem.

  public record ComplexReturnValue {
    public AnotherComplexValue Why { get; init; }
    public int Y { get; init; }
    public String Rye { get; init; }
  }

  public record AnotherComplexValue ( string Because );

  public class ComplexApp : IApp {
    public Object Start() {
      return new ComplexReturnValue {
        Why = new () { 
          Because = "We can"
        },
        Y = 1337,
        Rye = "Rittenhouse"
      };
    }
  }

  { ok, #{ why := #{ because := Reason }
         , y := Number
         , rye := Whisky
  }} = dotnet:run_app_from_assembly("priv/acme.dll", "Acme.ComplexApp"),

  %% Why: "We can", y: 1337, rye: "Rittenhouse"
  ?LOG("Why: ~p, y: ~p, rye: ~p ~n", [Reason, Number, Whisky ]). 

Back to that Dotnet GenServer then

We've established that we can invoke arbitary dotnet code from Erlang, which is already probably one of the most useful things I've created this year - the Erlang eco system is almost non-existent and being able to access all of that open source technology in Nuget is going to save us a lot of future development time. Going back to the start of this blog post our very first example was one of spinning up an actual genserver in dotnet - that is to say, code that is going to execute in its own Erlang process and in turn be able to execute its own arbitrary logic outside of the single function call that kicked that off.

That example again:

  public class MyEnterpriseGradeApp : IApp {
    public Object Start() {
      return GenServer.StartLink(() => new MyCoolGenServer() );
    }
  }

A standard gen server in Erlang would look something like this

start_link(Arg1, Arg2) ->
  %% Executed in host process
  gen_server:start_link({local, ?MODULE}, [Arg1, Arg2]).

init([Arg1, Arg2]) ->
  %% Executed in new gen process
  #state { one = Arg1, two = Arg2 }

In the dotnet above, initial StartLink call takes place in the host process, and the callback is what is invoked in the init call and is expected to return 'some state'. At its simplest, that state could just be an empty object:

  public class MyCoolGenServer {}

In Erlang, if you want to invoke some code in the process that kicked off these shenanigans, you would invoke that code via a gen_server method (call/cast) or send a message via the pid using the bang (!) operator, and so long as the relevant callback was implemented that would result in something happening.

start_link() ->
  gen_server:start_link({local, ?MODULE}, []).

init([]) ->
  #state{}

handle_info(Msg, State) ->
  ?LOG(Msg),
  {noreply, State}.

Used as thus

  { ok, Pid } = my_cool_genserver:start_link(),

  %% Hello Robert
  Pid ! "Hello Robert".

The same thing works in dotnet (woah), we just need to implement an interface on the object that gets returned in that init call

public class MyCoolGenServer : IHandleInfo<String> {
  public HandleInfoResult HandleInfo(HandleInfoContext ctx, String msg) {
    Console.WriteLine(msg);
    return ctx.NoReply();
  }
}

public class MyEnterpriseGradeApp : IApp {
  public Object Start() { 
    return GenServer.StartLink(() => new MyCoolGenServer());
  }
}

  { ok, Pid } = dotnet_shim:start_link("priv/acme.dll", "Acme.MyEnterpriseGradeApp" ),

  %% Hello Robert
  Pid ! "Hello Robert".

Yes, this just works - arbitrary messages sent from Erlang to a process started (and running!) in dotnet arrive and get translated accordingly. How about doing something with state then?

public class AdditionServer : IHandleCall<int> {
  private int total = 0;

  public HandleCall(HandleCallContext ctx, int msg) {
    this.total += msg;
    return ctx.Reply(total);
  }
}

public class AdditionApp : IApp {
  public Object Start() { 
    return GenServer.StartLink(() => new AdditionServer());
  }
}

  { ok, Pid } = dotnet_shim:start_link("priv/acme.dll", "Acme.AdditionApp" ),

  %% 1
  ?LOG(gen_server:call(Pid, 1)),
  %% 5
  ?LOG(gen_server:call(Pid, 4)),
  %% 11
  ?LOG(gen_server:call(Pid, 6)),

In Erlang we'd have a state record that needs returning as part of the handle_info callback, but that is because Erlang doesn't have mutation. Dotnet is somewhat fuller featured in this regard and we can just store our state on a member variable and forgo the extra steps required to dance around immutable structures.

Pattern Matching in Dotnet

Now you'll notice that our IHandleInfo/IHandleCall interfaces are generic and take the input shape expected, but it's really common in Erlang for a handle_info call to take a multitude of shapes and perform pattern matching based on those shapes to perform the correct block of logic.

Consider this code written in Erlang

handle_info({tell_me, Pid}, State = #state { value = Value }) -> 
  Pid ! Value,
  {noreply, State};
handle_info({op, {add, X}}), State = #state { value = Value }) -> 
  {noreply, State#state { value = Value + X }};
handle_info({op, {mul, X}}), State = #state { value = Value }) -> 
  {noreply, State#state { value = Value * X }}.

This code either receives a { atom(), pid() }, or { atom(), { atom(), integer() }}, we don't have union types in dotnet so can't map this across cleanly - or can we. C# has some semblance of pattern matching these days as any good scripting language should - what if we just request 'Object' as our message type instead?

public class MyCoolGenServer : IHandleInfo<Object> {
  private int value = 0;

  public HandleInfoResult HandleInfo(HandleInfoContext ctx, Object msg) {
    switch(msg) {
      case Tuple<Atom, Pid> t when t.Item1 == "tell_me":
        Erlang.Send(t.Item2, this.value);
        break;
      case Tuple<Atom, Tuple<Atom, Int>> when t.Item1 == "op" 
                                         && t.Item2.Item1 == "add":
        this.value += t.Item2.Item2;
        break;
      case Tuple<Atom, Tuple<Atom, Int>> when t.Item1 == "op" 
                                         && t.Item2.Item1 == "mul":
        this.value *= t.Item2.Item2;
        break;
      default:
        throw new TermException("Unsupported message received");
    }
    return ctx.NoReply();
  }
}

Okay it's not the nicest example in the world but it works. Atom and Pid by the way are C# types that wrap the concept of an Atom (just a string really) and an instance of a Pid respectively. An Erlang static class exists for doing things like sending data to an arbitrary pid and that's what is invoked in that above code.

Now that's interesting, Erlang.Send? That smells a lot like we're calling Erlang from dotnet and you know why? It's because we're calling Erlang from dotnet.

Using Dynamic to invoke arbitrary Erlang code from Dotnet

There is absolutely no point in writing gen servers in dotnet if we then go and use the dotnet File or Sockets API (or perform any other IO for that matter). One of the more glorious aspects of writing Erlang is that when you're opening handles to various IO, you're actually spinning up processes that are linked to an owner and a pile of excellent behaviour is there by default to ensure that if the host crashes that the supervision tree will handle that, restart appropriate children and those handles will get closed cleanly. This is baked right into OTP and is one of the reasons why implementions of actor models in platforms such as Java/Scala (Akka) are missing 100% of the shots they're taking. (This also goes double for implementing these patterns using core.async in Clojure but I digress)

It makes sense therefore to just allow the execution of arbitrary Erlang code from dotnet and therefore open access to these APIs.

Consider file:write_file for example

  result = Erlang.Modules.File.WriteFile("foo.txt", Encoding.ASCII.GetBytes("please write me to that file));
  switch(result) {
    case Atom a when a == "ok": 
      return;
    default: 
      throw new TermException"That didn't work");
  }

Works out of the box. How about file:open, file:write and file:close?

Like most of the Erlang APIs, file:open effectively returns a pid which is linked to the parent process. This also just works in Erlang.NET.

  // Open the file handle and stash the pid for future use
  Tuple <Atom, Pid> success = (Tuple<Atom,Pid>)Erlang.Modules.File.Open("file.txt", new object[] { new Atom("write") } );
  this.pid = success.Item2;

  // And then do some of this
  Erlang.Modules.File.Write(this.pid, data);
  Erlang.Modules.File.Write(this.pid, data);
  Erlang.Modules.File.Write(this.pid, data);

  // And at some point
  Erlang.Modules.File.Close(this.pid);
  this.pid = ErlNifPid.Zero;

In a similar vein, that also means that APIs that involve messages being sent back to the parent process work as well - this can either be implemented in the IHandleInfo as above, or we can spin up an arbitrary process in dotnet to handle those specific messages. Doing this and wrapping this functionality in a dotnet class is a far more pleasant way of hiding the low level dynamic call taking place and exporting a nicer API to the dotnet client.

  public static class GenUdp {

  public static Pid Open(Tuple<Int, Int, Int, Int> ip, int port) {
     Pid parent = Erlang.Self();
     return Process.Spawn((Process ctx) => {
      Tuple<Atom, Pid> success = (Tuple<Atom, Pid>)Erlang.Modules.GenUdp.Open(port, 
      new object[] { Tuple.Create(new Atom("ip"), ip)
                   , Tuple.Create(new Atom("active"), true)
                   }
      });
      return ReceiveLoop(parent, process);
    });
  }


  private static ProcessResult ReceiveLoop(Pid parent, Process process) {
    process.Receive((Process process, Object msg) => {
      switch(msg) {
        case Tuple<Atom, Pid, Tuple<Int,Int,Int,Int>, Int, Byte[]> data: 
          Erlang.send(parent, data.Item5);
          return ReceiveLoop(parent, process);
        default:
          return process.Finish(new Atom("ok"));
      }
    })
  }

  public static void Stop(Pid pid) {
    Erlang.Send(pid, new Atom("stop"));
  }

And this would allow us to write a gen server which connects to a udp socket and writes everything it receives to disk. (Assuming we've wrapped file:open/etc the same way as we just did with UDP)

  public class UdpToFileServer : IHandleInfo<Byte[]>, IHandleTerminate {
    private Pid udp;
    private Pid file;

    public UdpToFileServer(String filename, Tuple<Int,Int,Int,Int> ip, int port) {
      this.udp = GenUdp.Open(Tuple.Create(ip, port));
      this.file = ErlangFile.Open(filename, new object[] { new Atom("write") } );
    }

    public HandleInfoResult HandleInfo(HandleInfoContext ctx, Byte[] bin) {
      ErlangFile.Write(this.file, bin);
    }

    // Not necessary, but be kind
    public void Terminate() {
      GenUdp.Stop(udp);
      ErlangFile.Close(file);
    }
  }

Building more of the application in dotnet itself

So far we've demonstrated how you can invoke arbitary code written in dotnet from Erlang, spin up processes written in dotnet (either as standalone processes or as gen servers), and invoke arbitrary Erlang code from dotnet as well as all the magic type back and forth nonsense we've managed to pack into this library.

What if we want to build more complex functionality in dotnet? It's one thing spinning up a single process which then spins up ad-hoc processes itself, but to build a proper reliable application in dotnet what we need is the ability to define supervision trees!

In Erlang, typically the application will spin up a top level supervision tree - which is effectively a list of ids, and then modules/args to invoke for those ids in order to get a process. If one of these processes crash then depending on how that supervision tree is configured, either the whole tree will be restarted, or just the child that crashed (and then repeated crashes might then cause the supervisor itself to restart). Some of these children can themselves be supervisors and thus with a little careful reasoning about how your application needs to interact, a tree can be built that maximises robustness whilst minimising any potential downtime.

In dotnet... Well we can just do all of this in a single file - here is a supervision tree from one of the tests written in the Erlang.NET project itself.

    public class SimpleSupApp : IApp
    {
      public Object Start() {
        return Supervisor.StartLink("primary-sup", 
                  () => new SupervisorConfig(
                    SupervisionStrategy.OneForAll,
                    new SupervisorChild [] { 
                       new SupervisorWorker("c1", () => GenServer.StartLink("genserver-one", () => new WorkerGenServer1()))
                     , new SupervisorWorker("c2", () => GenServer.StartLink("genserver-two", () => new WorkerGenServer1()))
                     , new SupervisorWorker("c3", () => GenServer.StartLink("genserver-three", () => new WorkerGenServer1()))
                     , new Supervisor("s1", () => 
                         Supervisor.StartLink("secondary-sup", 
                         () => new SupervisorConfig(SupervisionStrategy.OneForOne, 
                           new [] { new SupervisorWorker("sc1", () => GenServer.StartLink("nested-one", () => new WorkerGenServer1()))
                                 ,  new SupervisorWorker("sc2", () => GenServer.StartLink("nested-two", () => new WorkerGenServer1()))
                                 ,  new SupervisorWorker("sc3", () => GenServer.StartLink("nested-three", () => new WorkerGenServer1()))
                                 })))
                     }));
      }
    }

The resultant tree is along the lines of:

• PrimarySup
  • c1 : WorkerGenServer1 (called genserver-one)
  • c2 : WorkerGenServer1 (called genserver-two)
  • c3 : WorkerGenServer1 (called genserver-three)
  • s1 : Supervisor (called secondary-sup)
    • sc1 : WorkerGenServer1 (called nested-one)
    • sc2 : WorkerGenServer1 (called nested-two)
    • sc3 : WorkerGenServer1 (called nested-three)

And then we have the following behaviours

• Invoking sys:terminate(Pid, 'arse') on c1,c2,c3 or even s1 will result in every single process in the tree being restarted
• Invoking sys:terminate(Pid, 'arse') on sc1, sc2, sc3 will only result in that single process being restarted

By restarted, we mean that callback above (the stubby lambda ()=>) being invoked once again. Args can be captured and passed in here if required, it's just closures all the way down.

Conclusion

Now while all the examples in this blog entry are written in C# (as that appears to be the prevailing language in that ecosystem for the moment, yes I know there are F# developers too waves at all five of them), all of this results in our original aim which was to write our gen servers in Visual Basic .NET.

    Public Class MyGenServerVB
        Inherits IHandleInfo(Of Msg)
        Public Sub New()
        End Sub
        Public Function HandleInfo(ByVal ctx As HandleInfoContext, ByVal msg As Msg) As HandleInfoResult
          If msg.Item1 = "hello bob" Then
            Erlang.Send(msg.Item2, "hello joe")
          Else
            Erlang.Send(msg.Item2, "weeee")
          End If
          Return ctx.NoReply()
        End Function
    End Class

That's all it took, I'll be following this blog entry up with how any of this works at all - so subscribe to the RSS (lol) if you want to read that, that'll be all the technical stuff that you don't know you needed to know in order to appreciate the 'it just works' nature of this integration if that's all you wanted.

Take a look over at the Github for the actual Getting Started notes, there will no doubt be tweaks required but it should be enough for us to make some progress at least.

Also no guarantees it works on Windows, pull requests are open.

Well I'm glad you asked, Erlang.NET does indeed work with Elixir and to prove it, I've built a skeleton application over on Github.

What does it look like?

Well, first of all let me say that I've never worked with Elixir before, I don't generally like to get my hands dirty with legacy techologies, and I was never a huge fan of Ruby, so the idea of writing my Erlang with Ruby syntax never appealed either, but for science...

Let's say I've got an application in Elixir

    { ok, result } = add_these_numbers(5,3)
    IO.puts "What a result: #{result}"

And I'm not satisfied with the way in which Elixirs adds these numbers together, but I do have some C# that does this much better?

Well, first off we add the Erlang.NET dependency to our mix.exs

  defp deps do
    [
      {:dotnet, git: "https://github.com/robashton/erlang.net.git", branch: "master"}
    ]

And make sure that the dotnet application is going to start along with our own

  def application do
      [
        extra_applications: [:logger, :dotnet],
        mod: {Eg, []},
      ]
  end

Rather than use Nuget and get Yet Another Package Manager on my machine, I chose to use Mix for this project to get my dependencies and just write some MSBuild because it's a really powerful way of defining builds in Dotnet and can do pretty much anything we ask of it. I created a dotnet project in the 'cs' directory of my Alixer project that looks like this.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
    <EnableDynamicLoading>true</EnableDynamicLoading>
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="../deps/dotnet/cslib/Erlang.csproj" />
    <Content Include="../deps/dotnet/priv/liberldotnet.so">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </Content>
  </ItemGroup>

</Project>

This means I need to configure the Erlang dotnet application to use the assemblies from the built location, no worries

import Config

config :dotnet,
  runtime_config: to_charlist("cs/bin/Debug/net5.0/Eg.runtimeconfig.json"),
  runtime_dll: to_charlist("cs/bin/Debug/net5.0/Erlang.dll")

Whew, Elixering sure is hard work I can see why you'd want to move away from it! Almost there though.

Anyway, this means I can use my far better function in dotnet by writing a class

    public class App : IApp<Tuple<int, int>>
    {
      public Object Start(Tuple<int, int> t) {
        return t.Item1 + t.Item2;
      }
    }

And call this from Elixir like so

    { :ok, result } = :dotnet.run_app_from_assembly(to_charlist("cs/bin/Debug/net5.0/Eg.dll"), to_charlist("Eg.App"), { 5, 3 })
    IO.puts "What a result: #{result}"

Far better than the original, although for some reason Elixir likes to use byte arrays for its strings when dotnet wants char lists so we have to do some conversion to get from the legacy elixir.

This of course allows us to then use gen servers written in dotnet, and over time we can move away from alikser altogether.

 :dotnet.run_app_from_assembly(to_charlist("cs/bin/Debug/net5.0/Eg.dll"), to_charlist("Eg.Gen"), %{ :foo => 1, :bar => 2 })

    public record GenArgs {
        public int Foo { get; init; }
        public int Bar { get; init; }
    }

    public class MyGen : IHandleCall<Atom> { 
      GenArgs args;

      public MyGen(GenArgs args) {
        this.args = args;
      }

      public HandleCallResult HandleCall(HandleCallContext ctx, Atom picker) {
        switch(picker) {
          case "foo": 
            return ctx.Reply(this.args.Foo);
          case "bar": 
            return ctx.Reply(this.args.Bar);
          default:
            return ctx.Reply(new Atom("nope"));
        }
      }
    }


    public class Gen : IApp<GenArgs>
    {
      public Object Start(GenArgs args) {
        return GenServer.StartLink(() => new MyGen(args));
      }
    }

I hope this answers the question and helps with people struggling to move away from their legacy elixir projects, always happy to help.

I covered my decoding test before having a look at context management, so now it's time to look at re-encoding the h264 and demonstrating a round trip through the nvidia hardware.

What we have already

We had this workflow for testing

  #workflow {
    generator = #read_from_ts { name = source, filename  = <<"foo.ts">> },
    processors = [
      #nvidia_decoder { name = decode, from = { source, ?video_frames_with_stream_id(256) } },
      #x264_encoder { name = encode, from = decode },
      #ts_writer { name = write, from = encode, filename  = <<"out.ts">> }
    ]
  }

and what I want is

  #workflow {
    generator = #read_from_ts { name = source, filename  = <<"foo.ts">> },
    processors = [
      #nvidia_decoder { name = decode, from = { source, ?video_frames_with_stream_id(256) } },
      #nvidia_encoder { name = encode, from = decode },
      #ts_writer { name = write, from = encode, filename  = <<"out.ts">> }
    ]
  }

In the decode entry we had a series of pointers to surfaces that we'd pulled out of the decoder along with the corresponding timestamps, these are in NV12 planar format and byte-aligned (and I'm storing all that information alongside the pointer to that surface in a struct so now we just need to see what the API looks like to turn this surface into h264 once again...)

This looks something like this (It actually looks nothing like this as most of this info is stored in a linked list tied to a central pool which contains the shared information), but this is at least representative of what we have.

  typedef struct _nvidia_allocated_surface  {
    uint8_t* data;
    int bpp;
    int byte_width;
    int byte_height;
    size_t pitch;
  } nvidia_allocated_surface;

This surface is allocated on the GPU via CUDA, and the GPU is where we want to do our encode so hopefully we can pretty much use this data directly.

The API

The encode API looks nothing like the decode API in that it's sat in its own header file (which is again expected to be included as part of the repo, but the library we're going to be loading is definitely expected to exist on the runtime as part of the driver install.

• Include/nvEncodeAPI.h contains all definitions/enums/functions/etc

and somewhere on the host OS installed as part of the driver package (for me, /run/opengl-driver/lib/)

• libnvidia-encode.so

Creating the API

The functions that we end up using are loaded as a struct of function pointers and we actually only link a single function directly from that library to get hold of that struct. As a consequence of this, every struct being passed into a function in this API tends to be versioned indepedendently, such is life.

  uint32_t version = 0;
  uint32_t currentVersion = (NVENCAPI_MAJOR_VERSION << 4) | NVENCAPI_MINOR_VERSION;
  NV_ENCODE_API_FUNCTION_LIST nvenc = {NV_ENCODE_API_FUNCTION_LIST_VER};

  NvEncodeAPIGetMaxSupportedVersion(&version);
  if (currentVersion > version)
  {
    return -1; // or whatever
  }

  NvEncodeAPICreateInstance(&nvenc);

  // use the functions on nvenc

This doesn't do anything other than a dlopen of the library exporting the functions and copy the pointers to those functions onto the nvenc struct so they can be called. Invoke it once, cache the results somewhere and then we can get to the business of talking to the GPU.

Creating an encoder

Unlike the decode stack, the encoder doesn't seem to be built directly on top of CUDA, it having the capability of instantiating on top of OpenGL/DirectX/Cuda as options (and then utilise surfaces from those systems). Because we're using the decoder (and planning on using CUDA to run upscaling algorithms and such), it makes sense to use CUDA for the encoder too.

Because of this, we'll want to use our already-created CUDA context and pass this into the open session call.

  NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS encodeSessionExParams = { NV_ENC_OPEN_ENCODE_SESSION_EX_PARAMS_VER };
  encodeSessionExParams.device = ctx;
  encodeSessionExParams.deviceType = NV_ENC_DEVICE_TYPE_CUDA;
  encodeSessionExParams.apiVersion = NVENCAPI_VERSION;
  void* encoder = NULL;

  nvenc.nvEncOpenEncodeSessionEx(&encodeSessionExParams, &encoder);

Now, this doesn't do very much other than give us an encode session that isn't initialise - and to initialise it we need to tell it what form our desired output needs to take. It is entirely possible to spend a day or two wondering why this call gives you back 'NV_ENC_ERR_INVALID_PARAM' if it's not set up correctly (and reading the sample code doesn't help because the configuration of the encoder happens across multiple files because of the way that the samples are written), the below seems to be the the minimum required..

In the Real World (tm) we have an input surface with a known width/height from the decoder, here we'll hard code it along with the frame rate (I know my source is 25fps). In the Real World a load of these args will be passed in from Erlang, not C.

  NV_ENC_INITIALIZE_PARAMS initializeParams = { NV_ENC_INITIALIZE_PARAMS_VER };
  NV_ENC_CONFIG encodeConfig = { NV_ENC_CONFIG_VER };
  NV_ENC_PRESET_CONFIG presetConfig = { NV_ENC_PRESET_CONFIG_VER, { NV_ENC_CONFIG_VER } };

  initializeParams.encodeConfig = &encodeConfig;

  // The essentials
  initializeParams->encodeGUID = NV_ENC_CODEC_H264_GUID;
  initializeParams->presetGUID = NV_ENC_PRESET_P4_GUID;

  // These need to line up with the input surface dimensions
  initializeParams->encodeWidth = 576;
  initializeParams->encodeHeight = 720;

  // Ditto these
  initializeParams->frameRateNum = 25;
  initializeParams->frameRateDen = 1;

  // I seemed to need these as well
  initializeParams.enablePTD = 1;
  initializeParams.encodeConfig->frameIntervalP = 3;
  initializeParams.encodeConfig->gopLength = 50;
  initializeParams.tuningInfo = NV_ENC_TUNING_INFO_HIGH_QUALITY;

Once this initialize params basics is set up, we can ask the encoder to populate the actual details for us based on the preset and codec specified in the 'essentials'

  nvenc.nvEncGetEncodePresetConfigEx(encode_session->encoder, initializeParams.encodeGUID, initializeParams.presetGUID, tuningInfo, &presetConfig);
  memcpy(initializeParams.encodeConfig, &presetConfig.presetCfg, sizeof(NV_ENC_CONFIG));

I'm not entirely sure why we don't just invoke nvEncGetEncoderPresetConfigEx on the struct already contained on the initializeParams, but the samples all end up doing this and the above code is precarious enough already (honestly, the number of ways this can go wrong with a single error is infuriating!). So we stick with the samples way of doing stuff!

I also needed to then go and set up the IDR frequency to match our requested gop length (the samples do this too) and not doing this resulted in NV_ENC_ERR_INVALID_PARAM so..

    initializeParams.encodeConfig->encodeCodecConfig.h264Config.idrPeriod = initializeParams.encodeConfig->gopLength;

With all of the above done correctly, a call to the init fn will return a success and if not I'm really sorry you're on your own because I've served my time already.