In this episode, we talk to José Valim, author of Elixir.
The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.
You can send feedback about the show to podcast@cognitect.com, or leave a comment here on the blog. Thanks for listening!
EPISODE COVER ART
PRODUCER
Our theme music for this episode is Thumbs Up (for Rock N' Roll) by killthenoise with Feed Me which was used under a Creative Commons License.
In this episode, we talk to José Valim, author of Elixir.
RUSS: Hello, and welcome to Episode 120 of The Cognicast, a podcast by Cognitect, Inc. about software and the people who create it.
This week marks the inaugural outing of Cognitect developer, architect, and core.async coauthor Timothy Baldridge as host. Timothy will be talking to the creator of the Elixir programming language, Jose Valim.
But before we get started, we do have a few announcements. There are still some tickets available for Clojure/west, which is happening on March 30th and 31st in Portland, Oregon. Go to 2017.clojurewest.org for more information. There’s a ClojureBridge happening in Seoul, Korea, on March 18th and another one happening in Stockholm, Sweden, on May 6th. In case you don’t know, ClojureBridge is dedicated to increasing diversity within the programming community by offering free beginner-friendly Clojure programming workshops to people from underrepresented groups. I can tell you from personal experience that ClojureBridge workshops a lot of fun as well.
If you have a Clojure related event you’d like us to mention, please drop us a line at podcast@cognitect.com. That’s about it, so on to Timothy and Jose in Episode 120 of The Cognicast.
[Music "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
TIMOTHY: Okay. One last time. This time is for real. Welcome, everyone. Today is February 17th, 2017, and this is The Cognicast. I’m Timothy Baldridge, and today I’m with Jose Valim, creator of Elixir and cofounder of Plataformatec. Jose, welcome to the show.
JOSE: Thank you. I’m glad to be here.
TIMOTHY: We asked, before we started the show, for you to come with something in mind of an experience of art, be it a good experience, a bad experience, nothing you had to create or anything, but just some experience you’ve had of art. Why don’t you start us off with that?
JOSE: Awesome. I live in Krakow in Poland, and I think there is a beautiful experience you have when you’re visiting Krakow. Assuming you arrive by train to the central train station, either that or actually walking to the center, you pass under a tunnel. Then after that tunnel like the whole city center is guarded by walls and gardens, so you have this nice impression, like it has been built throughout centuries of the garden, right? Then there is one street, which is my favorite street in the whole city, which is called Florianska Street that is just beautiful because you are just passed through the gardens and then you pass through the wall and you have the view through a big street that’s it’s like its longest street that ends in the city center with a basilica at the end.
Now I live here, right? But I remember the first time I was there and I had the experience of getting at that point, the exact spot where I can see everything for the first time, and I remember thinking to myself, “Wow. I want to live here. This is just beautiful. It’s phenomenal.” Every time someone comes to Krakow, I try to give them the same experience, which it has been there for a really long, long time, and I’m quite sure I’m not the only one to experience that.
TIMOTHY: That’s fantastic. Yeah, I’ve never been to Poland, although I’d love to. I’ve been through several parts of Europe, though. That’s the thing that’s always struck me as well while there is that here in the U.S. a lot of our buildings – an old building is 100 years, 120 years old. The whole city I live in of Denver didn’t exist 100 years ago. It’s amazing to see the old architecture. That’s fantastic.
JOSE: Yeah. I remember getting here and then the university. I don’t think it was the engineer. It was the general university, it was celebrating 700 years.
TIMOTHY: Wow.
JOSE: That puts things in perspective.
TIMOTHY: It absolutely does. It absolutely does. All right, so let’s dive in. I think we’re going to spend most of the podcast today talking about Elixir. Elixir is a programming language that you wrote, and so why don’t you start us off with what exactly is Elixir?
JOSE: Okay. Beautiful. If you go to the website, you’re going to see that Elixir is a program language designed for building scalable and maintainable applications and if you try to build a little bit beyond that. One of the main reasons for Elixir to exist is the Erlang virtual machine. It’s about exposing everything that the Erlang virtual machine has provided for decades in a different way and to other developers that would not have experienced it in other ways.
In a very summarized version, it’s that. I want to stop here because otherwise I speak about 20 minutes about the virtual machine and everything. In a very summarized way it’s that, a dynamic, functional programming language for building distributed – for building scalable, maintainable, even distributed applications. If you want to know, should I use it? My answer is, if you have anything that’s running on top of a socket, be it a distributed service, a Web app, an API, the answer is yes.
TIMOTHY: Excellent. Yeah, so what is your background then? I mean obviously Elixir is how many years old at this point?
JOSE: It’s five years old, and we are probably two years and a half since 1.0.
TIMOTHY: Okay. Excellent. Yeah, so what was your background before Elixir? Obviously you didn’t go from never programming to, hey, let’s write a language. What other languages did you work in before writing Elixir?
JOSE: Yeah, I’m not going to tell the whole story, but I was mostly a Ruby developer. Before I found Ruby, I did a little bit of C, a little bit of Java, a little of even ActionScript because I wanted to do a website for my band a long, long, long, long time ago.
TIMOTHY: Yes.
JOSE: A little bit there, but my main experience, I would say, before Elixir was Ruby.
TIMOTHY: Okay, and so you went from programming in Ruby to writing a language on the Erlang platform and the BEAM VM. What was the draw to Erlang that you saw?
JOSE: Yeah, so what happened is that it was the old concurrency story in which the Clojure audience is going to be really, really familiar. I’ve learned a lot also from Clojure because, at the time I was thinking about Elixir, Clojure was already around. I like to say it’s one of the top three influences in Elixir, but anyway it tells this whole story about concurrency, right?
Concurrency is becoming more and more important. Our CPUs are not getting any faster or it starts to happen that our machines start to have more and more…. And then that was already obvious because that was like 2010, 2011. It was clear at that moment. It was like, well, if these things become important, we need to have good tools, and I was already feeling the pain writing root software that had to leverage concurrency. I was like we need to have good tools for solving this problem in Ruby.
Then I started to research looking into other communities, other program languages to see how they were solving that problem, right? Then I was slowly moving away because I thought about the ideas behind functional programming and that resonated really, really well. I was like, well, if I was using those concepts to build the software that I want, like half of the concurrency issues I’m having right now, they would simply not be there, right? Then I was like, well, this is good. From this perspective of concurrency, this is great.
Then throughout my research I found out about the Erlang virtual machine, and there was one book that helped me solidify it really, really well, which is the Seven Languages in Seven Weeks book because it goes and explore. I don’t remember the languages any more, but I believe it has Clojure, Haskell, Erlang. After reading the book, it kind of laid all the options for me. Then I looked at Erlang virtual machine. I was like, you know, this is the virtual machine I want to – this is the platform that I want to write my software on because of many reasons, so first, the design decisions they have made.
The Erlang virtual machine was created by Ericsson, which is a telecommunication company.
TIMOTHY: Right.
JOSE: When they designed it, they wanted to build it for building like telephone switches and building their telecommunication infrastructure, which today when you look at how you’re building systems that run on the Web, it’s very, very similar, right? You are talking to a bunch of other machines. If you are building an application, you are talking to a bunch of devices be it a browser, a mobile phone. There are a lot of talks about Internet of Things where we are connecting to devices that are on the road and exchanging data all the time.
Then I was like, if it was good for telecommunication, it’s definitely going to be good for, like I said at the beginning, everything that runs on top of a socket because it was designed for that, right? That’s really interesting, this whole aspect of being designed for that. There are a bunch of interesting properties from the fact the focus on telecommunication that we can leverage in the way we write software, the way we think about software. That was really cool.
What was also really interesting was that while – one of the themes of the book Seven Languages in Seven Weeks was exactly concurrency, but I could see that while all of those languages and even languages that have come up recently, they’re all thinking about concurrency. Erlang was not about concurrency at all. Erlang was about distribution, so when they built the virtual machine nobody was thinking about multi-core concurrency. They were thinking about concurrency, but not multi-core concurrency.
TIMOTHY: Mm-hmm.
JOSE: Right? Because you didn’t have multi-cores back then. When I look at it it’s like, well, I can see that everyone is talking about concurrency, but what happens when you’re running your software in the machine with 40 cores and you cannot get a machine with 80 cores? Right? Now we need to have two machines to help you solve that problem for you to continue moving forward.
To me they had solved the long-term problem, right? When you go to study Erlang or Elixir, we are going to say that we have this abstraction called processes, and processes, it’s what we use both for concurrency distribution, so you can support multiple processes that are all running at the same time, and they’re exchanging messages every time they need to coordinate, and it’s in a way that it doesn’t matter much if they are running the same machines or you separate the machine. It can still exchange messages because the VM abstracts that for us. Those were the main aspects that got me really interested in Erlang virtual machine and led me to explore it further.
TIMOTHY: Excellent. I guess the natural question to come out of that then, and I’m going to phrase this in a nice way–I don’t mean it any way other than a nice way–why not just write in Erlang then? What was the problem, if you want to put it that way, the thing that made you go and write Elixir instead of just saying, well, we’ll just go write in Erlang?
JOSE: Yeah, so there are a couple. Today it’s clearer what I wanted. Elixir, I said that Elixir has five years, but before the Elixir we had today, I spent a year building a prototype that was really, really bad, right? But good it was a prototype.
The things I wanted to have in that prototype would be better abstractions for extensibility and productivity. For example, I wanted to have meta programming to be first class. I know it’s not on everyone’s cup, but on my cup it’s important. I want to have that to be first class. That already pushed me towards a certain direction.
Then I wanted more extensible mechanisms. In Erlang, we have – I’m going to do a comparison with Clojure, for example. In Clojure we have protocols. Protocols, they are a mechanism for ad hoc polymorphism. We can define a contract, let’s say. Then you can implement that contract for any data structure any moment you want, right?
Erlang only has this mechanism in a closed shape because, with Clojure protocols, any time I want I can define a new protocol and implement it for whatever data structure that we are interested in. But in Erlang, that’s closed. The only form I could do polymorphism is with like pattern matching in function clauses.
TIMOTHY: Mm-hmm.
JOSE: Which means that after I define that code in a file and I compile that code, I cannot change it any more, right? That’s not good for software extensibility because, for example, if what I wanted to do is that I wanted to say, well, I have this protocol that converts data structures to JSON. Then when I’m implementing that, I need to know all of these structures that I want to support upfront. Then if later someone has come up with a new data structure and they want to make it handle it, it’s not straightforward. There is not a straightforward mechanism. You may have to go change the library to maybe support passing functions around or even send a pull request to support a data structure that is not part of the Erlang distribution. There’s no extensibility mechanism for example coming from Clojure, but it was really to the lack of having protocols in the language also pushed me to other directions, right?
Then it was a debate. It was a big debate at some point because I can try to extend Erlang to make meta programming more first class, and there are tools in the Erlang community that does that, right? Oh, I can try to implement something like protocols in Erlang as well, but I think there is a huge difference in having that as part of the language and as part of something that we want people to rely on as a building block opposite to something that only part of the community we would eventually use as a separate project.
TIMOTHY: Right. Yeah, and I think you’ve kind of answered a question here that I had, and that is as an outside viewer of Elixir. This really is about more than just we’re going to give Erlang a different syntax, right? There’s some pretty integrated new features, it sounds like, that are hard to do in the Erlang syntax.
JOSE: Yeah. Yeah, so the syntax is interesting because it’s a discussion that comes a lot. The way I like to sum it up is the reason why Elixir ended up borrowing a lot from Ruby is the same reason that Erlang ended up borrowing a lot from Prolog. It’s just that in previous experience from whoever was creating the language ended up affecting the design. Both languages, they ended up refining that throughout the years, so Erlang lost a lot of the Prolog-isms, let’s say with time, and Elixir also lost a lot of the Ruby-isms with time.
TIMOTHY: Mm-hmm.
JOSE: I have considered for some time to implement actually a Lisp for the Erlang virtual machine, but by the time I was exploring that we already had two or three Lisps.
TIMOTHY: Yes.
JOSE: There is one that is still around today. I think none of those are maintained anymore with the exception of LFE, which is maintained by Robert Virding, which is also one of the creators of Erlang. Because there were already two or three, I felt like, okay, I don’t want to do that. The idea and what I wanted to do was to have a known Lisp syntax, but still be able to have “mechanisms” and have a straightforward translation from the syntax, so the abstract syntax tree. If you go back to the McCarthy’s papers about Lisp, he talks about S-Lisp and M-Lisp.
TIMOTHY: Mm-hmm.
JOSE: I wanted to have the M-Lisp, right, and then have the S-Lisp behind the scenes.
TIMOTHY: Excellent. Yeah, and somewhat, if I remember correctly from some times I’ve looked into it, I believe Ruby is somewhat the same way, right? There’s some S-expression type things behind the scenes in Ruby as well.
JOSE: Yeah, so they all end up having that because it’s often necessary for you to do your compiler passes.
TIMOTHY: Right.
JOSE: You need to tokenize the language. You need to parse it. You need to build an internal representation so the parser works. Maybe some languages, they are not going to expose that, and I think that’s why, in Ruby, for some time it was a separate package because the implementation of Ruby is a C.
TIMOTHY: Right.
JOSE: I’m not sure what I’m saying here. I may be wrong regarding…. Okay?
TIMOTHY: Yeah. It’s been several years for me too since I’ve looked at that, so yeah.
JOSE: Yeah. Yeah, but Erlang, for example, has an Erlang abstract format where you convert things. What I wanted for Elixir is to make the conversion as straightforward as possible and have as few rules as possible so we can implement Elixir, Elixir itself without needing to add keywords and special constructs. If you are going to try out Elixir, everything like defining a new module if Ks, defining new functions, all those things they are implemented with macros that just work on the Elixir AST and instead of being special grammar constructs, instead of being keywords in a language. I think that’s the difference, right?
TIMOTHY: Mm-hmm.
JOSE: In Ruby we still have a lot of keywords that give you special construct and special rules for those keywords in particular like class, switch, and things like that. In Elixir, no, it’s all unified. The reason I like it is because then the mechanism that I have used to implement the language, they are there for developers that want to extend the language, and that’s the big difference, right? I am not cheating. As a language developer, I am not cheating.
TIMOTHY: Right. Right. I want to come back to, in a little bit, more details about the internals of Elixir. But before we do that, I wanted to ask how has the uptake been in the Erlang community for Elixir? From an outsider’s perspective, it sounds like it’s been fairly well accepted.
JOSE: Well, at the beginning it was a little bit raw. There were mistakes. There were mistakes on my part as well on how to introduce that and ensure everything is going, let’s say, calmly.
TIMOTHY: Right.
JOSE: But it has been very well accepted in terms of Erlang events. They are welcoming to Elixir talks, Elixir events. They are welcoming to Erlang talks. There are a bunch of meet-ups that they ended doing Erlang or Elixir or whatever, like whatever people wanted to talk about, to be at the same meet-up together. And so at which time everyone could see, well, this exists for a purpose. This other thing exists for a purpose, and that’s how this is going to be, so let’s live peacefully.
It also helped when Elixir started to do more interesting contributions. For example, we have the Hex package manager, which became the package manager for the whole Erlang VM system and not Elixir any more, so there is a build tool called Rebar3 for Erlang, and they use Hex, so that was really nice because it was something that came from the Elixir community and now it’s kind of like giving back because it’s supporting the Erlang ecosystem too.
We were able to get improvements to the Erlang source code itself, so sometimes there are things to improve like sometimes I see the compiler could be faster and we share a bunch of the compiler pieces with Erlang, so we’re going. We speed up the Erlang compiler and then everyone can be happier, right? One of the things that I was also very – I talked a lot about Elixir, and it was all the pain points with Erlang since the beginning was support for Unicode.
TIMOTHY: Yes.
JOSE: For someone who has a name with an accent, I need to have good Unicode support. Now for the next Erlang version, which is Erlang 20, we are increasing. So after Elixir started, they already improved a lot on the Unicode support, and we are increasing it to other parts of the virtual machine like we are supporting atoms, which is kind of like Ruby symbols. I think they have atoms in Clojure as well.
TIMOTHY: Yeah, they’re keywords they’re called, yes.
JOSE: Yes.
TIMOTHY: But, yes, the same sort of thing. Yeah.
JOSE: Yeah, so now Erlang supports Unicode as well. There is someone, which I believe is also…. He’s adding a new module to Erlang that is able to do Unicode manipulations on the strings like being able to split Erlang themes, uppercase, lowercase, and this kind of stuff, which is what we had in Elixir since the beginning. It kind of pushed everyone to improve because we’re like parity, right?
This is important. We need to have that. Then eventually people were like, yes, that is important. We need to have that, and it ends up like – it’s a win/win situation, right? One is pushing the other like, oh, that’s important too, so we need to support that as well. I think it’s a very healthy relationship where it’s kind of like everyone is pushing towards the good of the platform and the ecosystem.
TIMOTHY: Excellent. How about outside the Erlang community? I seem to hear a fair amount of Ruby people that expressed interest in Elixir. Any other trends that you see?
JOSE: Yeah, so the main – there is – Josh Adams, he runs a pull every year. The first year it was very Ruby centric. Now we are getting more diluted, which is very good, so I think the main three were Ruby, Python, and JavaScript, which is interesting. It’s interesting because it’s kind of what you envision that would happen to a good extent because the properties of the language being a dynamic language, but with a very strong focus on concurrency and potential distribution, which those languages I have mentioned they are lacking. It’s kind of nice to see that thing becoming more concrete.
I would say those are the top three, but you end up having people from everywhere coming and going. We have people coming from C# as well and a little bit of people that are doing Java, and they decided to migrate away from the Java virtual machine and they wanted to–
TIMOTHY: Absolutely.
JOSE: –experiment with another platform and so on. Yeah. But what’s really interesting is that, so we were talking about Erlang and then Elixir was never designed for Erlang developers on purpose because the reason for Elixir to exist is the Erlang virtual machine. A good … Elixir is the Erlang virtual machine, right? The Erlang developers, they already have the Erlang virtual machine, so the amount of leverage we have to bring them over, it’s much lower. It was not something that I was ever worried about and it was always more centered on those, on people using … today that are not ready to handle the concurrency, fault tolerance, and distribution challenges that are so important nowadays.
TIMOTHY: Right. Right. How similar is Elixir to Erlang in the sense of the interop sort of things? Are integers still integers, if I create a struct in Ruby, Erlang will understand it, that sort of thing?
JOSE: Yeah. Yeah, so that’s a very good question. So Elixir, there is no untimely cost between calling Elixir from Erlang and vice versa. There is no bridging, right? What I said is true. A time point in Elixir is a time point in Erlang. An integer in Erlang is an integer in Elixir.
Also, a lot of the words like what is a process, what is a reference, the terms, they are the same. The way of thinking, they are the same. There is a lot of similarity.
Some people ask, like, do I need to learn Erlang before I learn Elixir? Our experience is, seeing people in the community, the answer is no. What has happened is that because the words are the same, eventually – one thing that’s nice about Elixir and Clojure … is that there is an existing ecosystem that we can rely on, right? We’re not starting everything from scratch.
Eventually you want to call things that are from Erlang libraries or from the Erlang standard library. You need to know a little bit of Erlang. But because the terms, the constructs like the abstractions, they are all the same, most people, they end up catching, you know, like they are doing Elixir for six months and then a switch flips in their brain. Then they go to the Erlang documentation. They’re like, wow, I can read this. I know what this is talking about. There are syntax differences, of course. But even those we tried to minimize as much as we could. Yeah, so there is a lot of Erlang in there for sure. Yeah.
TIMOTHY: Excellent. Yeah, excellent. We talked about this a little bit, a little bit ago, but my only experience with macros in a language has been in a Lisp. You mentioned meta programming. I assume you’re meaning macros here. There could be other meta programming concepts in Elixir. What do you get when you create a macro? Are they a set of data structures or are they just lists of symbols? How does that macro look and work? I know we don’t have text for you to show us, but I guess describe it.
JOSE: Yeah. Yeah, so when you have a macro, you are going to receive ASD, which we call coded expressions in Elixir. The main representation for a coded expression in Elixir, it’s a tuple with three elements. The first is the function name as an atom, so like add. The second element, it’s metadata about data nodes, so you’re going to find line information and other kind of metadata that is important to that thing. Then the third argument is a list with – sorry. The third element of the tuple is the list with the arguments for that function call. If I’m calling add with one and two, then I’m going to have a list with two elements: one and two.
That’s the main construct. It’s this tuple with three elements, which is quite straightforward to traverse and everything. Then we have quoted literals, which are elements that, when quoted, they return themselves, right? For example if I quote one, it’s going to be one ASD. If I quote an atom, it’s going to be an atom in the ASD. In this case there is no conversion. That’s the main building block. That’s all there is to it.
Then everything ends up converting to this building block from the syntax, right? For example, operators. We have, is it, infix operators when you have the number, the operator, and then the other number?
TIMOTHY: Yes. Yes.
JOSE: Yes. I know the terms in Portuguese. Sometimes I get confused in the translation.
TIMOTHY: No. That’s fine.
JOSE: Anyway, yeah, so we have infix operators. We don’t have the list operator where it’s there at the beginning as any other function call, right? But when it becomes ASD, it’s the same representation as the other, as any other function call. That’s kind of the idea behind the macros.
TIMOTHY: Excellent. Excellent.
JOSE: We still have quote and unquote, which is, for me, one of the most important parts, so being able to – one part is being able to traverse the ASD, and that’s why the representation needs to be simple. But the other important part is to be able to build ASD chunks also from the syntax itself and not having to call functions passing the arguments that build the ASD for me or creating visual patterns or doing all the things by hand, right? That would be very painful. Having quote and unquote mechanisms is also really important for the meta programming
TIMOTHY: Yeah, and for people who aren’t familiar with that, in Clojure I like to think of those as almost like templates. They’re kind of – you create the structure and then you kind of say this slot here gets filled in with this variable.
JOSE: Yeah.
TIMOTHY: It makes it easier than trying to construct the syntax by hand, like you said. Yeah.
JOSE: Yeah. I don’t know who said it, but it’s like string interpolation but for code.
TIMOTHY: Hmm. Yes.
JOSE: Right? You define the code as if it was a string, and then you can interpolate code with code and the thing knows how to do that.
TIMOTHY: Excellent. You mentioned earlier. I think you said there were three major influences, language influences that you kind of drew from when writing Elixir. I forgot to ask about this earlier. You mentioned Clojure. What were the other influences?
JOSE: Yeah, yeah. Yeah, that’s very good. I said that I did a prototype before and that prototype was – when I was in the prototype, I figured out what I wanted to solve, like, oh, I wanted to have meta programming and, like, oh, I wanted to have like polymorphism, which is what protocols bring us, for example.
TIMOTHY: Mm-hmm.
JOSE: And accessibility behind that. But the only answer I knew for those things, I was like, oh, how do I know polymorphism? Object, right?
TIMOTHY: Yeah.
JOSE: Then I was like – I tried to implement an object oriented language so it had a prototype based object oriented language, and then I tried to do meta programming very similar to Ruby, but it didn’t work out. We didn’t have ASD and, when I wanted to do something fancy, I had to define strings and then evaluate the strings. The whole thing was messy because I knew, you know, I had a final vision where I wanted to go, but I was not sure how to get there.
TIMOTHY: Mm-hmm.
JOSE: For some time I thought, okay, I kind of know what I want. I wanted meta programming. I want this accessibility. But I don’t know how to get that. Then I went to study all those languages that I kind touched briefly before in that, so Haskell, Clojure, even more established languages like Python to go a little bit out of my box, right?
TIMOTHY: Sure.
JOSE: And see other ways to solve that problem. Then in 2012 was when we started to work on the Elixir that we have today. The main, the top three influences are Erlang, Ruby, and Clojure.
For example, Elixir has protocols. The reason it ended up being a little bit similar to Go interface, a little bit similar to Go, to Haskell type classes because they all solve the problem of open, ad hoc polymorphism. I was like, no, but I’m going to call them protocols because there are a lot of similarities between Clojure and Elixir in terms of the being dynamic languages and in terms of the macro system. I was like, okay, I’m going to call the protocols because the closest thing we have today to what I want is Clojure.
TIMOTHY: Right.
JOSE: That’s why we’re going to get that. Then right now I don’t remember any more, but there are many things that we ended up getting from Clojure. For example, like even protocols itself, like Clojure has a strong instance of what – like protocols, they do not have the full implementations because it should be only the definitional thing. That’s something that the other languages that provide similar mechanisms, they don’t agree. But the reasoning of why Clojure has that resonated really strong with me and that’s what ended up happening in Elixir. Things for working with nested data structures as well, so I believe it’s getting, putting….
TIMOTHY: Right.
JOSE: –it’s similar in Clojure. I looked at that really strongly, and we ended up having something similar in Elixir. Clojure has agents, if I remember correctly, as well.
TIMOTHY: Yes. Mm-hmm.
JOSE: Yes, so agents was really interesting. In Elixir, if you want to have state, or if you want to have concurrency, we use processes for that. But you know a process, it’s like pulling a very lightweight thread. There is nothing in that process that says, hey, you need to keep state.
The way we implement state is that you need to start a new process, and then that process needs to wait for messages that tell it how to update the state or how to return the state. There is a – if all I want is to keep state, there is a bunch of boilerplate in that, right?
TIMOTHY: Mm-hmm.
JOSE: Then I would look at Clojure where I would say, hey, do you want to keep state? Look. Do agent things. You start an agent. If you want to update this, just do those two lines of code or one line of code, and then it’s like feature envy, right?
TIMOTHY: Yeah.
JOSE: That’s something that bothered me for quite a while. The bothering was coming from Clojure, so on what it was possible to do in Clojure because people were coming to the language. They would say, well, at the time the application boots I need to load something from the file system and keep it in memory. Then we would go and say, hey, open up this file. Put those ten lines of boilerplate, and that’s how you keep your state. Then people would run away, right? They’re like, well, if I need all that to just have state, that’s too much for me.
TIMOTHY: Right.
JOSE: We ended up introducing abstractions called – one is called Agent, which is a process that it only about keeping state. We have another abstraction called Desk, which is about concurrent processing.
TIMOTHY: Mm-hmm.
JOSE: Added that to Elixir, and that task came from .NET, and we got some inspiration from that. Anyway, the idea is that we have this process, which are about concurrency. Then the question was what if we specialized this to have processes that are only about state and something that’s only about concurrency? We ended up having better abstractions because of that. It was also a little bit of Clojure influence too.
TIMOTHY: Excellent. Yeah, so one of the things I’ve found interesting, and I mean as a thought experiment I’ve sat down at times and thought, you know, what would it be like to write Clojure on the Erlang VM. The one thing that I’ve always kind of been hit by, and maybe you’ve experienced this too, is that when it comes down to it, Erlang is very immutable. It shuns mutability at the VM level. It’s not like Erlang is some syntax that doesn’t allow you to mutate things. It seems like the BEAM VM itself is pretty immutable.
So I mean if we go back to Clojure, Clojure has some weird things where, when you’re not looking in some places or when it doesn’t matter, it will do some mutation. It’ll cache maybe a hash number here or what’s the other example I was thinking of? Things like transients would be another one or data collections where I know only one thread is touching this, so I’m going to do some mutability. Then when I’m all done, I pretty it up and say, yeah, here’s this immutable collection.
Have you experienced anything or I might just have the wrong view of the Erlang VM on this?
JOSE: No. You’re right. The Erlang virtual machine was designed to run Erlang. It’s not a general purpose Erlang virtual machine. If you want to write a language that runs on the Erlang virtual machine and you don’t want it to be a horrible prototype as mine, you need to stick with the rules that are in the virtual machine and then immutability, you know, there are mechanisms that go for immutability. Even there are some – okay, story time. Something that we have in Elixir, we have – Clojure has reducers, right?
TIMOTHY: Right.
JOSE: In Elixir, we have – if I remember correctly, when you are playing with sequences and when you are playing with reducers, you kind of need to go from one to the other, right?
TIMOTHY: Right.
JOSE: They have your different APIs and maybe even different protocols. That’s correct, right?
TIMOTHY: Yeah.
JOSE: They use different protocols behind these things, yes. In Elixir, I didn’t want to have that. I wanted to have one protocol and then, if you wanted to be eager or lazy, you could choose between eager and lazy and have all those things running on a single protocol. Then I went to look at reducers, and I was like, okay, I think I can implement that. Then there is, if I remember correctly, reduce take where it uses a mutable thing. I don’t know if it’s an agent or if it was just something very small. It completely shattered my dreams because I was like, no, I cannot build that. I cannot just cheat and have this thing be mutable. I think it was using that to keep the counter of how many things you still have to take or something like that.
TIMOTHY: Yeah, exactly. Yeah.
JOSE: Right?
TIMOTHY: I can fill in there a little bit. Yeah, so in Clojure with transducers, yeah, there are some stateful transducers like partition. I’m going to put a collection in. I’m going to get them chopped up in groups of three. Yeah, there’s an accumulator inside this thing that looks functional, but it’s accumulating three things, and then it writes one, and then it accumulates three more things and writes one. Yeah, exactly. Inside there there’s a little cell that’s mutable inside this thing that’s mostly immutable.
JOSE: Yes. Yeah, exactly. I wanted to have a unified view, and maybe I can do this unified view on top of reducers. Then when I look at it, like, oh, no, I cannot do that. Then it took me some time to actually be able to solve that problem. I went to a talk from Jessica Kerr where she was talking about the solutions they have in Scala and then that led me to something called iteratees, which is something from Haskell, but implementing iteratees would be very expensive, so I did a merge of reducers and iteratees, which I called reducees, and there is a blog post if you search for this word, which probably nobody ever used besides Elixir. If you look at “reducees,” we tell exactly this story. It’s about solving this problem where I cannot just go and make the thing mutable. We need to be able to reduce, keeping the accumulator around and passing that accumulator around because it cannot mutate something.
TIMOTHY: Excellent.
JOSE: Yeah. I want to take a look at that. I remember when transducers came out, and there was on Twitter you had started puzzling over some of this. I followed the chain. I think we even talked for a little bit, and then I forgot about it. And so I got thinking, oh, I should figure out whatever became of that.
JOSE: Yeah.
TIMOTHY: I’m going to read that.
JOSE: Yeah, it took a long time, but we eventually figured it out. It has been a while. Do you remember when reducers came out?
TIMOTHY: Yeah, so Clojure has two things, by the way. There’s reducers, which are kind of parallel transformations of collections.
JOSE: Yes.
TIMOTHY: Then transducers, which are these kind of pipeline type things. Wow, it’s two years, two and a half years, somewhere in there.
JOSE: Yeah. No, it has to be more because, in Elixir in – so 1.0 came in 2014.
TIMOTHY: Okay.
JOSE: We already had – we already – yeah, we already had these frames, which is what we use. Anyway, yeah. But yeah, it’s also a little bit of a story that is related with everything. But, yeah, we cannot cheat that easily. In this case, we could actually cheat. There are ways that would allow us to cheat, so every process has – actually, people will kill me if I say this, but anyway. Every process has a dictionary, which is like, I think, threads in Java. They have a storage that’s….
TIMOTHY: Right.
JOSE: –to threads.
TIMOTHY: Right.
JOSE: It’s the thinking, right? Every process has a dictionary, and you can create unique references, so if I wanted to cheat my way out of this, I could generate a unique reference and use that as a key in my own process dictionary to keep that around. That would very likely work, but it’s very frowned upon, right?
TIMOTHY: Right.
JOSE: Nobody would be happy. Yeah.
TIMOTHY: Yeah. Excellent. Yeah, we were talking before we started recording today that you and I met, have met before back in 2013 at a Code Mesh in London, which was a fantastic conference. It was great.
I wanted to ask. We’ve talked a little bit today about what’s changed, like that got up to the point of maybe 1.0. What’s changed more recently in the past couple years? What are some new features that have been added to the language?
JOSE: Oh, that’s a very good question. That’s way too hard. Okay.
TIMOTHY: Okay.
JOSE: Okay. I’ll go back a little bit in time. When we launched 1.0, I was just like there are two kinds of 1.0, right? There’s the 1.0 like we are getting started, and I think languages like Rest did this kind of 1.0 because I feel like now they are 1.30-something, and there is still a lot of developing happening in the language.
TIMOTHY: Right.
JOSE: With Elixir it was more like this is 1.0, and from now on it’s small improvements to the language. There were some versions that included a good amount of changes after 1.0, but the whole idea was that it’s there, and we will continue to improve things and maybe add new abstractions, but they are going to be small and not a revolutionary way that is going to drastically change how Elixir writes code.
In a way, the answer is like what has happened since then is like not much. But we have had, like, you know, the compiler is always getting better. We are able to optimize more stuff. We can be more assertive of bugs that we can find in your code while we are compiling. So I think one of the big things that came in Elixir 1.3 was the ability to do cross reference checks between modules and applications to find, like, oh, you are calling this thing that never exists because, as a dynamic language, especially a dynamic language that runs in their Erlang virtual machine that allows you to load code at a moment in any time that is strong straightforward to do, there is particular moments, but that’s something that we added.
In 1.4 we added something called a registry, so we have processes in Elixir, right? They are all running at the same time. Sometimes we want to give those things names, so you know where that processor is because the easiest way to talk to a process is to have a reference to that process itself. But I need to pass that reference around, so there actually basically is a way where I could say, well, if you are looking at somebody with this name, that’s where it’s running.
TIMOTHY: Mm-hmm.
JOSE: That’s something that we added in Elixir that’s really, really important. It’s also nice because – so there is this Web framework in Elixir called Phoenix.
TIMOTHY: Yes.
JOSE: One of the things they did is to run some benchmark some time ago where they were able to get – they wanted to reproduce the Whatsapp scenario where they got more than two million connections on a single node.
TIMOTHY: Mm-hmm.
JOSE: They had something like a registry, so that’s the thing we added to Elixir 1.4. It’s really nice because we added something that was like highly scalable. We ran benchmarks on machines in the community. I prepared the benchmarks and then a bunch of people started to run it on really, like, badass machines that I would love to have for development.
TIMOTHY: Right.
JOSE: They’re not here, but a machine with 48 cores, and we could see that you would reduce – you would see all the 48 cores concurrently and you would have no performance impact on that. That was something that I think suits well in the language of having, like, good abstractions that are scalable that we have added recently. But again, they are building blocks, right? We are trying to make your life better bit by bit instead of having a revolutionary way.
TIMOTHY: Excellent. We’re getting towards the end here of our time, but if I wanted to learn Elixir, where should I start? Do you have any good reference? I could go to the website, obviously, and look that up. Any other recommendations?
JOSE: Oh, yeah. The main recommendation is the website because we – something that we’re very – since the beginning was exactly about the getting started experience because if we are going to be realistic, like, people need to learn. They are coming to the language, and it’s not an imperative language. It’s not an OO language, so they need to learn about push-out programming, pattern matching, immutability, right? Then concurrency is important for us, so it’s one of the big reasons, so you need to learn about concurrency and the way you build applications in Elixir, which has like some particular properties.
Because it can be a quite daunting task, we always thought about what is the – how is going to be the experience of developers wanting to try this? So we take a lot of care of the website with love, so we keep the guide up to date that’s going to guide you through the main construction of the language. We also have an advanced guide on the website, which is really cool because it’s about building a distributed key value store over TCP. It’s really nice because rich language is teaching you how to build a distributed key value store. We’re participating in their guides on the website, right? It shows like how accessible those constructs are. That’s all there. But if you prefer something more structured or with a different pace, there is a learning section on the website as well that covers different books, screen casts, and other things you would use to learn. Yeah.
TIMOTHY: Excellent. I guess is there anything else you would like to discuss about Elixir or programming in general or anything?
JOSE: There are definitely, but I won’t be able to remember them at the time right now.
TIMOTHY: That’s fine. That’s fine. Okay. Excellent. Well, this has been great. I’ve enjoyed this, and so let me ask you our traditional final question and that is, do you have any advice for us as programmers, as people, as listeners of this podcast?
JOSE: Yeah, so this may be – so I’m going to give an advice, which is – it’s something that I see every day happening, which is about error messages. I believe, like – so I’m going to give this advice. I’m only meant to give one advice. I’m going to give two.
TIMOTHY: Okay.
JOSE: In the shape of one to different audiences, right? One is if you are writing a library or a language or whatever, you have the responsibility of giving good error messages and good stack traces. It’s your responsibility. It’s your job, right? …oh, something went wrong. Well, something went wrong. What else can you tell about it? What are the possible ways that I can – what is the extra information that I can tell about this? Is this a common error that people are running into? How can I make the road clearer for people from going from this error to a better experience, right?
It’s a potential for errors when you are raising them, right? Don’t raise something that’s wrong. Raise got number, right? Tell me what expected and other important information, so that’s one.
Then on the other side, if one side plans to write good errors, the other side has to plan to read the errors carefully and stack traces because that’s something – so that’s what I was saying. That’s something we see a lot, right? People say, well, something went wrong. Then what went wrong, it’s there in the error message. It’s there in the stack trace, right? I think everyone needs to come together for this care to error messages to ensure a good experience with everyone. One side needs to have the responsibility of having good error messages because then the other side can start to trust those error messages for a better development experience. That would be my advice because it’s something that I see every day on both sides, and we need more of that.
TIMOTHY: Yes. I think that is important. That’s something that I’ve noticed too is that I can often, as a user, figure out what was meant by the library when I compare the error message and the stack trace, and I go and open up the code. Hopefully it’s open source, and I go looking at it and see, ah, this expected a string here and I passed in an int, or whatever the error is. But, yeah, that all hinges on me having access to the source. It’s a lot easier if the library just says, “Hey, you gave me the wrong type of data, and this is what I expected.”
JOSE: Yeah.
TIMOTHY: Yeah. Yeah, it is both ways. That’s great. All right. I think we’re going to wrap it up for there today. Thanks so much for taking the time to talk to us about Elixir. This has been great. I haven’t been able to take too much of a look at it since we talked several years ago, so this has been great. It sounds like the language has got some really interesting stuff, and I’m going to go read some blog posts after this.
JOSE: Oh, awesome.
TIMOTHY: Excellent. Thanks to everyone who spent the time listening to us talk for the past hour or so. This has been The Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
RUSS: You have been listening to The Cognicast. The Cognicast is a production of Cognitect, Inc. Cognitect are the makers of Datomic, and we provide consulting services around it, Clojure, and a host of other technologies to businesses ranging from the smallest startups to the Fortune 50. You can find us on the Web at cognitect.com and on Twitter at @Cognitect. You can subscribe to The Cognicast, listen to past episodes, and view cover art, show notes, and episode transcripts at our home on the Web, cognitect.com/cognicast. You can contact the show by tweeting @Cognicast or by emailing us at podcast@cognitect.com.
Our guest today was Jose Valim, the creator of the Elixir programming language. You can find Jose on Twitter @JoseValim. Our host today was Timothy Baldridge who is @TimBaldridge on Twitter.
Episode cover art is by Michael Parenteau. Audio production is by Russ Olsen. The Cognicast is produced by Kim Foster. Our theme music is Thumbs Up (for Rock N' Roll) by Kill the Noise with Feed Me. I'm Russ Olsen. Thanks for listening.