In this episode, we talk to Luke VanderHart about his new open source project, Arachne.
The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.
You can send feedback about the show to podcast@cognitect.com, or leave a comment here on the blog. Thanks for listening!
EPISODE COVER ART
AUDIO PRODUCTION
PRODUCER
Our theme music for this episode is Thumbs Up (for Rock N' Roll) by killthenoise with Feed Me which was used under a Creative Commons License.
In this episode, we talk to Luke VanderHart about his new open source project, Arachne.
CRAIG: Hello, and welcome to Episode 101 of The Cognicast, a podcast by Cognitect, Inc. about software and the people who create it. I'm your host, Craig Andera.
Well, as always, we want to mention a few things that might be of interest to those of you out in the community. First of all, there's a ClojureBridge happening in Tampere, Finland. That is happening May 20th and 21st in 2016, and you can find out more about that, as always, at the ClojureBridge website at clojurebridge.org.
We also want to talk about EuroClojure. This is our European conference happening this year, 2016, in Bratislava, Slovakia. That'll be happening October 25th and 26th. You can find out more about that conference at euroclojure.org. And, I just want to point out that we are currently seeking sponsors for the conference. So, if your organization is located in Europe, or in the U.S. for that matter, or anywhere in the world, check out euroclojure.org and look at the sponsorship opportunities.
Another conference of note–and I always struggle with the pronunciation of this one, and I hope that the organizers are amused by my attempts rather than annoyed–I'm talking about ClojuTRE. Maybe it's pronounced something like – anyway, they know what I'm talking about. 2016, you can find out more about that conference. It's a free conference organized by Metosin. It's got one track. It's got short talks, so it's like a 20-minute talk, 5-minute QA, and kind of a funky after party. It's got draft beer and fun discussions and everything. This is all a description from their website. I have not, unfortunately, been able to attend yet. Maybe some day. But, they welcome newbies and seasoned Clojurists, and their website is at clojutre.org/2016 for this year's conference.
Speaking of conferences still, early bird tickets are now available for the Clojure eXchange. That's in London. It's going to be in December. You can search for that online. The website is at skillsmatter.com, but the URL is a little bit long, so I suggest you just look for Clojure eXchange 2016. You're certain to find out about that. It's a two-day conference, again held in London.
Lots of awesome stuff going on in 2016. Kind of hard to believe we've gotten to the point where there could be so many really just high quality conferences every year, well attended, but that's where we are. It's a good thing for the community.
That's all I have for you today, so we will go ahead and go on to Episode 101 of The Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
CRAIG: Okay, so
CRAIG: I will echo that. Thanks for listening. This has been the Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
CRAIG: Anything you want to talk about before we get started?
LUKE: No. Ready to jump in.
CRAIG: We've done this before, right?
LUKE: Yeah.
CRAIG: Cool. All right. Hello, everyone. Welcome to The Cognicast. Today is Friday, March 18th, in the year 2016, and I am pleased today to welcome back to the show a developer and a coworker of mine here at Cognitect, author of a couple books you might have heard of. I'm talking about Luke VanderHart. Welcome to the show, Luke.
LUKE: Thanks, Craig. Glad to be here.
CRAIG: It's really nice to have you. We actually have something kind of interesting that we're going to talk about today. But, before I get into that, and people are like, "Why are you teasing me?" But, before I get into that, we will get your answer to the question we start every show with, which is – we ask our guest. That's you Luke. You're our guest.
LUKE: Yes.
CRAIG: Sorry. It's late on a Friday. I might be a little bit punchy here. But, we ask our guest to share with us an experience of art, whatever that means to them, so a book you read, a painting that you enjoyed, you got really high and went paragliding one time. Whatever it is, it means to you.
LUKE: I wish.
CRAIG: Yeah, so go for it.
LUKE: Sure. The thing that most immediately comes to mind, is at the top of my mind right now, is actually a book that I was reading last night on writing. The book is called Several Short Sentences About Writing by Verlyn Klinkenborg, which is quite the name. But, it's a really good little book. I recommend it for anyone who is interested in writing.
** The reason this relates to art or artistic expression for me is because it got me to think about writing in a new way. I've always considered pieces to be works of art**: a play, a novel, a short story, a poem, what have you as far as writing goes. But, he really dives down to the sentence level and talks very carefully about crafting sentences and the way each sentence appears, says its piece, and then disappears, and how you can build a cadence to individual sentences and work with the structure at a very, very low level. It's really a more low level way of thinking about writing than I have before and really kind of illuminated me to some of the beauty that can be found there. There's nothing earthshaking, but it's what I've been thinking about today after reading over that last night.
CRAIG: Very cool. Having read that, do you think back on anything you've read or maybe even anything you've written, and have you been able to sort of retroactively discover that sort of beauty in the things that you've read or written?
LUKE: I wish. To some extent, yes. It's very interesting to get an appreciation for authors that I already appreciate. I appreciate their works and I'm eager to actually go back to some of my favorite authors and do appreciate their sentences.
As for my own writing, well, first, I really don't like reading my own writing. It's like listening to your own voice. It just sounds odd. But, for writing that I'm currently working on, it's definitely something I'm trying to apply.
CRAIG: Cool. Awesome. That's a really cool thing. I'll have to remember to add that book to my list, so thanks for the recommendation.
But, we are going to move on from that question, as interesting as your answer was. We actually have another topic we want to cover today. This is a new thing that you are bringing into the world. I don't really know if I could do it justice in introducing it, especially because you've been thinking a lot about how to do exactly that. Rather than puts words into your mouth, I'm just going to let you take the mic and tell us about this thing you've been working on.
LUKE: Right, so this thing is something I'm working on now. It will be publicly announced by the time this podcast is released. It's not at the time Craig and I are talking. But really, what this is, it's an open source project and admittedly ambitious. Its goal is to provide a real, honest to God, Web framework for Clojure that is usable and will provide power and speed of development on a scale equivalent to some of the existing big names in the Web development world. Think about Rails. Think about Django.
Obviously we can't release with feature parity or complexity with things that have been around for years, but the goal is to put something into the Clojure space that actually has a meaningful value add based on Clojure, as an awesome language, and it lets Clojure developers have that kind of rapid, iterative, development experience based on known best practices when building Web applications in Clojure. There are a few reasons for this, which I'd love to talk about. But really, that's the goal is to build a full-scale, awesome, Web framework in Clojure and to build a community around it if at all possible.
It's a big ambition, but I think it could become a reality. I have some justification for that belief as well.
CRAIG: This project has a name.
LUKE: It does.
CRAIG: Is it final? I know we've talked about it.
LUKE: Yes.
CRAIG: Go ahead and share it with us.
LUKE: It is. Sorry. I should have said that straight up. The project is called Arachne. Arachne is a figure in ancient Greek mythology who was a weaver. She was able to weave more beautifully than anyone else, in fact even more than the gods themselves. Athena became jealous of her and turned her into a spider, so she was doomed to weave webs for the rest of eternity for her presumption of being better at it than the gods.
** It's a vocative story, and clearly with the whole Web connection, the whole creating things connection, I think it's a good name for this project, so Arachne**: a framework for weaving Web applications.
CRAIG: Cool. So, where to begin. This is a really interesting idea. I think Clojure has been applied quite successfully to Web programming. I haven't pulled up the stats, but if you look at the State of Clojure survey, it's pretty clear that people are using Clojure to create Web applications of various stripes, everything from standard page oriented websites to different architectures like we see the single page application. I think Clojure is particularly good at that.
LUKE: Right.
CRAIG: Is Arachne aimed at one particular metaphor? What's your objective there?
LUKE: Yeah, not generally. It's based on all kinds of Web development, so both single page Web development and more traditional content serving, like you'd see in a more traditional content-based website. Single page applications are usually more application feeling than Web page feeling. Arachne is designed for both.
Really, it's going to be useful for any system that needs to serve up HTTP content, it needs to handle multiple requests and responses in a session between a user across a network and some cluster of servers. It's also going to have easy integration points to run backend services that just need to be running continuously and serving requests. It includes things that are not directly Web facing. If you need to build a REST API running over HTTP, Arachne will hopefully be a great choice for that as well.
CRAIG: Okay. That gives me a good sense. We'll dig into a bunch more of this shortly here.
LUKE: Yeah.
CRAIG: I'd actually maybe like to rewind a little bit because I want to get a sense of the motivation.
LUKE: Yes.
CRAIG: Let you explain a bit more why because, on the face of it, I think people might be a little surprised to hear that we're talking about creating or that you and I are discussing your efforts to create a Web framework in Clojure. People are like, "Well, there are already some of those, and that's kind of an unusual thing to decide to do in this day and age." Maybe you could kind of take us through how you got to the point where you decided to do that.
LUKE: Yeah, absolutely. There are three very concrete motivations for building this and building it in Clojure, particularly. The first one is that currently in the Clojure ecosystem at large, particularly in respect to Web development, the story for beginners to jump in is actually not that great.
The current reigning paradigm for building Web services, building websites, building single page applications in Clojure, is to assemble a bunch of libraries. There are a lot of great libraries out there that do a lot of really cool things, but it's kind of the programmer's job, when they're starting up a new project, to figure out which ones they want to use, pull them all into a project, and write the glue code that holds them all together. While this is certainly something that a Clojure developer can do and may even find very useful to have this wide menu of available choices, that is a really hard thing to ask a beginner to do.
In order to do this effectively, you need to know the pros and cons of all the different libraries. You need to know the pitfalls. You need to know the high level structure of an application and how each piece fits into the larger hole. Each one of those things is something that is really not reasonable to expect a beginner to the language to have.
Contrast this with an experience of, say, Ruby on Rails is just another one that I happen to have a lot of familiarity with. The user can create an application, have it up and running before they have any idea what's going on, and then that provides a safe environment for them to grow and expand their skills. They can learn how to program in the context of something that already more or less works as opposed to what Clojure currently asks that you do and frontloads all the hard questions in the architectural level and the more difficult part.
The goal is to invert that for Clojure as well and to give beginners a way to get started quickly and easily, and a safe space to learn how to program without shooting themselves in the foot because, to be honest, for most beginners, even if you pull together something that works, it's not going to be something that's going to scale and continue to operate once you reach higher levels of complexity and performance requirements. So, we want to provide something that will let people build really interesting, cool, useful things and learn on the job while they're doing it. That's motivation number one is a path for beginners.
CRAIG: Thinking about that a little bit, I guess–and maybe you'll come to this in your other two motivations–is there anything about what you're considering doing, what you're planning to do? I look at it like, okay, well, you talked about making some choices for beginners and saying, okay, you've got a bunch of things to sort through. Well, one of the things you could do is write a tutorial that says you should make these choices, or a Leinigen template where you just spit out using existing stuff. But obviously that's not sufficient, in your opinion, so maybe we'll get there via the next couple things you're going to say, but that's the one that comes to my mind when I heard you talk about getting beginners and helping them making choices.
LUKE: Yeah. There are a lot of good tutorials and getting guides for really simple things. There's even some beginner stuff. But, again, it's hard to put yourself in the shoes of a beginner. It seems to you or me, like, the response that we default to is, well, why can't you just do X or Y?
The fact is that I've actually gone through and interviewed a bunch of beginners. I've interviewed people that are currently learning Clojure, and Web applications are one of the harder things. Even to say, well, just write a tutorial. Well, you go online and you search for tutorial for Clojure, and there are going to be 100 of them that come up, and half of them deal with libraries that are now deprecated.
Really getting the weight and momentum behind something that will be consistently supported and consistently a good idea, I think, has a lot of value. Only the beginners will be able to tell me whether we're successful with that or not once we have something out there. But, we're definitely going to be working and using focus groups and using beginners to try to come up with an experience that people love. That is not something people currently say about the Clojure community, unfortunately, is I loved getting started and it was really, really welcoming and easy to get started with.
They say they love the people. They say they love the community. They say they love the language once they get it. I don't hear a lot of people say, "I just love the getting started experience with Clojure," and we hope to change that.
CRAIG: All right, great. You said there were two more.
LUKE: Yes, there are. The second reason relates not just to beginners, but to everyone, and that is in the time and effort required to bootstrap a new Clojure Web project. Here at Cognitect, I'm a consultant. I jump into a company, build a new product. Sometimes it's a Web oriented thing. Sometimes it's not. Sometimes it's an existing system, sometimes not. But, almost everything has a Web component.
** What I've observed is, by the time I get in there and start building this, and I add all the things that any website has to have for the most part**: login, authentication, metrics, routing, HTTP generation, all those common things that everything needs and I would hope would solve problems. Solutions exist in Clojure. Don't get me wrong. But, by the time I figure out what I want to do and put them all together, I'm usually a month into the project. While that's not so bad if you're working on a two-year long project, if you're starting a new project every few months, that very quickly begins to wear on you.
I got tired of building the same code over and over, so I want to build something that lets me build new Clojure Web applications and get started instantly. I want to have something up and running in five minutes. It'll be a lot of boilerplate and a lot of–not boilerplate, but it'll be a lot of–conventions to start out with, but then it'll be easy for me to get in there and start adding my own stuff. But, the key there is that I'm going to be working on my own stuff the same day I start a new Web application.
Right now, if I want to start a new Web application in Clojure, I've got to spend weeks just setting up the infrastructure. The goal is to decrease that time, build all the infrastructure for you so that you can focus on the stuff that matters, immediately, and get prototypes out there quickly and start iterating on your prototypes very quickly. That's something that I don't see in the Clojure world right now is the ability to talk with a client about a website they need and get the first version delivered tomorrow. That is the experience I want to have here.
CRAIG: That makes sense. They both sound like things I want. Yeah. Okay. Go ahead.
LUKE: The third one, this one is where I think we can provide a unique value proposition for the software community at large, not just within the Clojure space. That is a smooth, linear scaling from a small, simple, fast app to a big, complex, enterprise application.
If you look at existing rapid development solutions across the industry, there are a lot of good ones, but the general idea is that there's always an inflection point where the framework that used to be really helpful and gave you a lot of benefit, and gave you a lot of speed in the early days of your project, used to be your friend, starts becoming your enemy. It starts hampering what kind of things you can do. It starts limiting the complexity of the code you can write. It starts limiting the scale and performance that you can operate at.
We already know that Clojure is a great solution for the high-end of scale and complexity. We have people running really large, really complex systems. Clojure is an amazing platform to tackle that on. There's nothing about Clojure, the language, that says it can't also be a good solution for the easy, rapid, iterative space, I guess, at the lower end of the size and complexity spectrum.
** And so, the question is**: Can we build something that will allow you to smoothly scale from fast, small, and simple to large, high availability, high grade, industrial strength applications without any discontinuities along the way? This is an open question. I firmly believe it's possible that we can do this, and I think Clojure is the right platform to build such a tool on. That's the third real goal of Arachne is to provide a solution that is a good, viable platform at every point along the project size curve, if you will.
CRAIG: That one seems like the hard one. Good beginner experience: okay, that makes a lot of sense and I can see where–I mean you're Luke–you're going to be able to do pretty much anything you set your mind too, but that seems achievable. Getting people off and running quickly with a set of, you know, like getting all of the infrastructure stuff out of the way. Okay. Also seems very achievable. Not a trivial problem by any stretch, but very achievable.
Then the linear scaling one, I'm interested there because you said you have some reason to believe that it's possible. I certainly would like to believe that it's possible, and I have an intuition that Clojure, the things that Clojure provides are going to be helpful in doing that. But, I guess I'd love to hear – and I know you said it's an open question, but I'd love to hear what's gone through your head that gives you hope that you're going to find a way through to a solution, if you see what I'm saying.
LUKE: Right. First, let me frame the question a tiny bit more. You know there's a very important essay for software development, the No Silver Bullet essay, which is: There's no way to make things magically happen. I am not selling a silver bullet here. This is: If you were writing a big, complex system, you're going to still have to have a lot of developers thinking really hard and writing a fair amount of code.
The goal here is just to have something that will not get in your way, that you can start with all the speed and agility you want, and then will not get in your way as you put in this effort to grow and make your system more complex. To some extent, you already answered the question. If you think I can build the fast and easy thing, that'll be awesome because we know Clojure can already do the big, complex thing. We've seen it on several projects and big Clojure systems, big enterprise Java systems. There is already existence proofs that Clojure can operate at that end of the spectrum.
** The question is**: Can a Clojure system operate at the lower end of the spectrum without adding chains that will hamper you as you get towards the high end again? I think so. Arachne is built – you know, we can get more into the tech, well, now or later.
CRAIG: Sure, yeah. Whenever you think makes sense. We'll definitely want to talk about it, but it doesn't have to be right now.
LUKE: Right, but most of what Arachne provides is at a project organization and configuration level. At run time, Arachne restricts you in very few ways. Really, once you write your components, if you're familiar with Stuart Sierra's Component library, Arachne is based on a very similar thing where actually I have plans to incorporate the Component library wholesale.
Once you write your components and they're running, and Arachne has wired everything up, it's done your dependency injection, you can write whatever you want in those. You're really not hampered by that. In fact, those are widely considered to be a fundamental tool for building larger, more complicated systems.
At that point, there are no restrictions on what you can build. You can write arbitrary Clojure code that does arbitrary things. Arachne will help you with the low level things. It'll help you write your routing table. It will help you serve up your HTML. But, if you want to serve additional HTML, you can write a route that terminates in a raw handler function that does anything you want it to, exactly the same as you would in Clojure today. The only difference being that you have this suite of tools that's available to you and maybe got you bootstrapped.
But, there is no real situation. Unlike with Rails, if you want to build a more complex endpoint in your application, sometimes you find yourself fighting the framework. Whereas, Arachne is going to be designed with this goal from the bottom up, which is, you can bail at any time you want and just be in your own, write your own code for any components of it. Really, the only level you're required to participate in the tools that Arachne provides will be at like the application start level, starting your components.
Beyond that, we're going to try to architect things in such a way that Arachne is purely an additive value. Hopefully, that will let those existence proofs of things that are already big, complex, heavyweight Clojure applications be written with Arachne in the mix as well. We're just providing some support on the small, easy, getting started side.
CRAIG: Okay. It makes sense, and I think I have a decent understanding of how you've come to the point where you think this thing needs to exist.
LUKE: Yeah.
CRAIG: That was past. My next question is about the present. Where are you at? What's done or what's your current state of development? You know what I mean? Where are you at, at this point?
LUKE: Right. I've been working on this, in one form or another, for about the past six months. This has been on my own time. We have some open source Fridays. I put a few of those into playing around with ideas and, certainly in my personal time, I've been developing this. It's become apparent to me that to build something like this that really has the reach and has the scope that I envision that actually can become a real community standard, it's going to need more time and investment.
And so, the hope is that I'm actually going to suspend my full-time work as a consultant and focus on this project full time coming up this summer. Obviously I need to fund myself while I do so, so I'll actually be launching–probably before this podcast goes live, I will have launched–a Kickstarter project, a Kickstarter campaign to raise funds so that I can afford to work on this full time in conjunction with other people who will donate their time and energy to this. By having at least three, but hopefully as many as six months of full-time work on this, we can build out the basic core system and reach a tipping point where the project will be launched and we can continue building it from there.
Regarding what's already done, I have the very core system done, and I have one or two modules. We can talk about the technology. Arachne is very heavily modularized and pluggable. I've got the core that supports the plugin infrastructure itself. The project configuration infrastructure itself is all done.
** The path forward seems very clear. At this point it's a matter of being able to focus the time into building out the horizontal array of things that a basic webframer needs to do, all the kind of functionalities we need it to have**: authentication, dealing with your databases, dealing with HTML rendering, dealing with the tools to hook in Figwheel, Om, Reagent, and all the other great things to build single page applications. Yeah, just need to go horizontal and build out all those features because it's hard to sell a solution as being the easy solution to anything if it doesn't have all the pieces yet, so we need to build those pieces.
CRAIG: Okay. It's interesting you mention that you're doing a Kickstarter. I think, if I were a listener to this podcast, I would be saying to myself–maybe some people at least would say–"Well, Luke already works for Cognitect. They are stewards of Clojure. Obviously have a big stake in Clojure existing and flourishing in all these various ways." Although, clearly Clojure has grown well beyond the days when you and I started when the number of places where you could do Clojure, you could count on one hand, and Relevance, at the time, was arguably the premier one.
LUKE: Right.
CRAIG: In any event, you're doing this Kickstarter, right? I think someone might ask, "Well, why isn't this your day job?"
LUKE: Right. Well, that is the goal for it to eventually become my day job. Yeah, so Cognitect has its goals and its priorities. We're already funding a substantial amount of development for Clojure core. We have a product team focusing on our Datomic product, which is awesome. You should check it out if you haven't already. I certainly don't want to use any other database at this point.
Really, it's a matter of allocation of resources. As a consultancy, we fund what we do through consulting, and I am on the consulting arm of the project. It's very much a consideration of where the company wants to put its limited resources behind. Cognitect has been extremely supportive of this project. They're very optimistic about it, but we can't afford to fund every single effort that every Cognitect is interested in pursuing. This is a way for me, together with Cognitect, and certainly has through all the people here that I've had conversations with are providing amazing support for this, but also be able to fund this in a way that is sustainable for everyone involved and doesn't require Cognitect to sacrifice its other product development efforts.
CRAIG: In terms of this Kickstarter, what does it look like? The Kickstarters always have levels and rewards and everything. You're going to go and raise, I don't know, hopefully quite a substantial amount of money and go off and work on this for whatever that allows you to. Take me through what you're hoping to achieve with the Kickstarter and where it gets you.
LUKE: Yeah. The base goal of the Kickstarter will provide funding for me to work on this project exclusively for three months of full-time work. Full time being whatever I would normally work in a workweek and then whatever I was putting into Arachne beyond standard 40 hours a week, if you want to think about it that way as well. What that should let me do in that three months is really build out the core and have something that is totally usable and totally worth selling to people as this is something you should use.
At the end of the three months, I want everyone to say, when they're starting a new Clojure application, clearly go with Arachne. That's going to be – I'm going to have something that is feature complete enough and sufficient enough to be the default choice for Web application development in Clojure as of that point. That is the goal, the base goal of the Kickstarter is three months.
I do have a number of stretch goals built on top of that depending on how many people are willing to put their money into this effort and want to see it succeed. I have a very clear vision about it, and a lot of other people I've talked to as well. If we can get enough contributions and enough corporate support, from that point there are a number of stretch goals for building out really rich, additional features and components, modules–technically speaking–things you can plug into Arachne directly to do interesting things. These would be single page applications, really get topnotch, dynamic development experience for the React world, which is quite popular in Clojure. That's the first stretch goal. Certain deployment things, dev-ops style things, that let you build containers and other systems that define whole clusters of machines, deploy to Amazon. We have plans for modules like that.
There are certain features that enterprises will find very useful that exist on some of the higher tier stretch goals, so distributed metrics, distributed logging, corporate single sign on. Then, beyond that, I would be totally thrilled if we got those funding levels complete. But, if we were to even get more, we'd just spend more time building out, horizontally building new features that the community is interested in. Investing more time in – it's going to ship with good documentation, but investing even more time in building up resources for those who are wanting to get started with it and really building the community and providing mentorship for people to get started using this in meaningful ways.
I should mention that this is not a solely me driven task. Several people at Cognitect and elsewhere have graciously agreed to be a sort of steering group or steering committee to set priorities for Arachne. As we hit our initial goals–we get the core done, we get the HTTP, and the asset and HTML generation, the authentication, the database access–from there and as people start to use it and we start building up this dynamic feedback cycle with the community, I believe we'll be able to get some more insight into where we should go from there.
CRAIG: Are you at a stage right now where you can share any of the names on the steering committee?
LUKE: Yes, absolutely.
Tim Ewald is one of them. He's a core member of our product team here at Cognitect.
Russ Olsen, who also works for Cognitect right now.
David Nolen, you may have heard of, he's the author of Om Next.
I'm also working with Jamie Kite who is currently not doing a lot of Clojure work, but has amazing insight into user interface and usability concerns.
Then also Mike Nygard, who is an amazing architect working with Cognitect. He's lending his insights as well.
CRAIG: I think there are people out there that would literally murder to work with that team, so that's pretty impressive, Luke.
LUKE: Yeah. Well, they're where the success is going to come from. I can develop, but they've provided insight and architectural support all along the way.
CRAIG: I think you're being quite modest. They are an amazing group, but you can count yourself as a peer among them.
I wanted to ask you a little bit about some of the technology. Specifically, I guess, I'm curious about some of the other technologies that Cognitect and Cognitects have developed. Obviously there are things like Pedestal and Om Next. I know that you developed Quiescent.
LUKE: Right.
CRAIG: There's a suite of things that are out there already. I don't think you are planning to start from scratch and build yet another thing on top of Jetty or whatever.
LUKE: Yeah, certainly not. Arachne has a number of core philosophies. One of those is pragmatism. We do not want to reinvent anything. We do not want to create something new and cool with this project. We want it to be a platform for people to build new and cool things on.
But, we want a stable set of known solutions that are known to work and that we can provide to people in an easy way that's fast to get started with, and that means reusing existing libraries. The Arachne module system is very lightweight. It should be very low effort to incorporate any of these existing libraries as an Arachne module, or not even both with that and just use it wholesale from your code base. Arachne intends to be fully compatible or be able to be made fully compatible very easily with the vast majority of existing systems out there.
CRAIG: Obviously modules are a big part of the story here. Do you have a more technical description of what it means to be a module? Is it just a protocol or one of a set of protocols? Is it something like Stuarts's component? What exactly does that mean, a module?
LUKE: I'll explain at a functional level first. This has been sort of implicit, but a module is kind of the unit of delivery of a feature. It's a plugin. It's analogous to something like a Ruby Gem or a Node package. It's something that you can drop into your program and just take off with that functionality. I want to use Hiccup instead of Enlive, drop in the Hiccup module for HTML generation, and off I go.
At a technical level, they are based upon Arachne's configuration system. Now, Arachne has a very, very rich configuration story. An Arachne configuration is not simply a text file or even an EDN file or an XML file.
It's a full database with schema of everything that defines an application. This doesn't mean just ports or just external URIs like you'd normally find in any config file. It is anything about the system that can be represented as data, is represented as data within the configuration. This includes things like routing tables. If you're using a relational database, database migrations. All of this exists in the Arachne configuration database.
The database has a schema. What modules do is they extend that schema and allow you to add new kinds of data into the database. Also, when you initialize, each module bootstraps in dependency order, and it can look at the application configure database and say, "Okay. Here's something interesting. I know how to annotate that feature." They can transact additional data to the configuration database that adds in new elements. Take the cast of HTTP routing.
You could have a module that said, "Oh, interesting. I'm going to add in these interceptors or these middlewares at every step of routing because I am a metrics module. So, whenever you call, I'm going to read my configuration that says, 'Add metrics to these routes.' Then I'm going to go and actually look at the entities for those routes in the configuration database and transact them."
The beauty of this is it's very transparent to the user. The module authors know how to interact with the application config database. The users only need to tell the modules what to do. The integration points are well known. Each module knows which modules it depends on and it knows what schema it can rely on being in place based upon that dependency information.
That's a lot of information, but really the story is that this is very, very data driven and that there is a rich schema for that data and all the integration points are well defined so that you can add modules that participate in that configuration in a variety of ways.
Then, at runtime I'm sure the question comes, well, that's great at configuration time when you're building up the definition of the application, but how do they operate at runtime? Well, modules can define in the configuration that certain lifecycle objects exist.
This comes back to Stuart Sierra's idea of lifecycle objects that are part of a system. The configuration database actually contains information about what runable lifecycle objects exist in the system. Of course, modules can define additional ones of those.
If I include my HTTP server, I could define it to start up an HTTP server that can be started and stopped. Then if I add an in-memory database module, the module will handle defining that component that can be started and stopped. The application config is the sum of the inputs of all the modules working together, based upon the information they know about each other through a dependency tree, to build together a single, complete system.
CRAIG: It sounds like a really powerful approach. Is that something that you've run into before?
LUKE: There are certainly echoes of it in various other systems. I hate to say it, but the old Java Spring Library had certain elements of this. Unfortunately, the negatives of that approach far outweighed the positives. We're definitely focusing on agility and introspectability and self-documentation and clarity that Spring did not have. But, there are certain elements of defining a system through data that are echoes of what have come before, certainly. We're not operating in a vacuum.
CRAIG: That's interesting. I'll have to think on that idea some more. It seems like it should be far more universally applicable than configuring Arachne.
LUKE: That's the thing. You have to set your goals somewhere. If this approach is wildly successful with Web apps and Arachne, I could see the Arachne core configuration and module system being used elsewhere in other kinds of applications that have nothing to do with the Web. But, Goals 1 and 2, the beginner experience and being able to rapidly get started with Web apps in Clojure, this is what can enable that kind of experience. And so, I think Web apps are a great place to get started with this kind of an approach.
CRAIG: Yeah, I think you're right not to attempt to boil the ocean. It's going to come out better if you make it work for your situation and then go, "Okay. Here's what we learned," and specific to general.
I had a question. You said that this is going to be an open source project. Is any of this out there now that people could go and take a look at already, or is it not quite there yet?
LUKE: If you define "now" as when you and I are talking, no. But, if you define "now" as when people will be listening to this podcast, then yes.
CRAIG: Awesome.
LUKE: It will be hosted on GitHub under the Arachne-framework project. Just go there and check it out, or Arachne-frameowrk.com and visit the homepage.
CRAIG: Oh, you got the dot-com.
LUKE: I did.
CRAIG: Very nice. Excellent. Okay. We talked a little bit about past, about present, about future. I guess I'd like to peer a little bit farther ahead. You are as successful as you want to be.
LUKE: Mm-hmm.
CRAIG: Everything that you hope for this project comes true. A year from now, what has happened? Is there a quantum shift, or is it just more evolutionary? What's true a year or whatever timeframe you care to talk about from now if you are able to do what you hope you can?
LUKE: I would say my ideal goal for a year from now is that I am back to consulting and every project is a month or two faster because I can get started immediately on any Web component. I would also like to see an influx of new people into the Clojure community.
I think Web applications are where a lot of people want to start and where a lot of people expect to start when they encounter a new language. It gives something very tangible that you can work on, tangible and also very useful in our current climate. Everything is a Web app. I'd like to see a lot of people that have not previously been heavily engaged with Clojure, seeing the value of that, and kind of start pouring into our community.
CRAIG: I know you're working with Jamie Kite, as you said. I know that she has definitely been involved with ClojureBridge. Have you given any thought to whether ClojureBridge – I mean a lot of people have spent a lot of time on the curriculum. From what I hear, it's really good. I haven't taken the class myself or gone to TA, as I probably ought.
LUKE: Yeah.
CRAIG: Is there any thought or discussion around whether something like Arachne might not be helpful in the context of something like ClojureBridge?
LUKE: Yes, absolutely, I think it could be. I haven't talked with the ClojureBridge folks yet. I know I sit right next to one of them. I've been focused on getting Arachne set up.
The goals of Arachne and ClojureBridge are very much aligned in providing a safe space for beginners to learn, but also learning good things and building powerful, interesting things in a very intuitive, easy to get started way. I could easily see Arachne becoming part of the ClojureBridge curriculum. Once we have something up and running, I will definitely be working with them and participating in writing that bit of the curriculum.
CRAIG: You've been at Cognitect and, before that, Relevance, for quite a while, just like me. One of the great practices that we have brought into – we didn't invent it, but one of the things that we bring to organizations that oftentimes haven't seen it before is the risk assessment.
LUKE: Yes.
CRAIG: Would you be willing to do a little risk assessment with me right now on Arachne?
LUKE: Absolutely.
CRAIG: I think it would be interesting.
LUKE: Clearly it's a very risky project, and I have very lofty goals.
CRAIG: You do have lofty goals. What do you see as the major–? A year from now, things didn't work out the way you wanted them to.
LUKE: Right.
CRAIG: That was because of X.
LUKE: Right. I'll just enumerate things in the severity that they appear to me. One is that people will not think that this is something worth having or a goal worth striving for in the Clojure space. I see this as a possibility, but fairly low risk based upon: I've had a lot of conversations with a lot of people and a lot of people who I really respect and trust.
It's not just me that feels the absence of something like that quite keenly. I think this is something that I regularly hear people ask for. If you search for Clojure Web framework, you come up with five to ten closed Stack Overflow questions because they're too subjective asking, "What's the best Web framework for Clojure?" and all the answers are along the lines of, "Well, there isn't really one. You kind of put one together from your libraries."
There are some that will help you get started. We probably should spend some time talking about the existing Web frameworks for Clojure and how Arachne is different and why I think it'll have a little bit broader uptake than they do. That's another question.
But, I think demand exists. That's actually another point of demand is that these new Web frameworks for Clojure keep showing up. I do have reasons to believe Arachne will be successful where they are not. But, I think there's just a lot of demand for it.
Another risk is that it's a great thing to have, but it just can't be built to the level we want to build it, or I'm not the right guy to build it, perhaps. That's why I have assembled a team of advisors and the steering group that I have. I have some level of confidence in my own abilities, certainly. I have a lot of experience in this domain. But, I've tried to assemble the best people to get on my side with this and to help give me advice. That's why I'm mitigating that risk. That is a real risk that what we want to build is just too hard, but I don't think that will come to pass simply because of the team that I've assembled.
Other risks are that we build it, it's great, but not a lot of people use it, and we're unable to get it up to the level of uptake that will allow it to really get the – you know, one of the big things about Web frameworks is that they all have big communities that are very active. If we can't get critical mass and the community behind it, it's going to be difficult for it. Again, I don't think that'll happen. We're certainly going to mitigate that any way we can by trying to build a very welcoming community.
We are going to accept pull requests via GitHub. We're going to try to make it easy for people to contribute. One of the main goals of the module system is so that people can build real meaningful things that are very easy to drop into an Arachne application and use without having to have any commit rights to the system or even submit any kind of patch or pull request.
People can just build things that participate in Arachne without anyone's permission, and we can get a thriving ecosystem of things going that way. That's the goal for community building. Any one of these things we could fail to execute on, but we're really trying to anticipate those points and account for them. If you can see any more points or if you, the listener, think of anything, please contact me. We want to head any of those things off early if we possibly can.
CRAIG: Well, I think that's a good segue, actually, to talking about the existing framework. You mentioned that because I think one of the risks is simply discoverability, right?
LUKE: Right.
CRAIG: There's this whole – there's maybe two I can think of that you didn't mention. That would be one: It's awesome, and the people that try it love it, but people can't find the tree through the forest, if you will.
LUKE: Right.
CRAIG: Then another one that's sort of related is this tendency of Clojure frameworks to disintegrate into libraries, right?
LUKE: Right.
CRAIG: You look at Noir. That was a framework and it turned into libraries. I think there are other examples we could think of as well.
LUKE: Right.
CRAIG: I wonder if those had been on your radar at all. Actually, also, please use that as a platform for talking a bit about the existing frameworks and how you see Arachne relating to them.
LUKE: Right. The existing frameworks largely fall into – actually, they don't fall into two categories. All of them are collections of libraries that the authors of these frameworks have assembled. They've built some glue code to pull these libraries together and then blessed and said this is a set of libraries that you should use together. We have a Leinigen template or something that will let you get started and to make it easy to do so.
Unfortunately, what none of these frameworks have really accounted for is being able to swap out those components without sewing up the seams with more coding. None of them have the experience that you have in some other Web frameworks of simply dropping in new functionality and then start using it. Most of them, like, say a library chooses Hiccup as its HTML templating system. Well, if you want to use anything except for Hiccup, you've got to go remove the Hiccup library, replace all of the Hiccup templates, add in a new library, say Enlive, and add in the code to call the Enlive handlers and merge the data from whatever data system you're using and render that to a string and serve it up to the user.
Those are all coding activities. In essence, to modify any of the existing frameworks, if any of the frameworks have anything that you don't like, it becomes a coding activity to change that. With Arachne, we're really aiming to build certain interfaces via this configuration system so that you can provide drop in handlers. We're going to have an explicit asset pipeline that is fully codified in data, so a number of transform functions that things can go through and rules for routing assets through the transform functions.
We haven't designed this part in great detail yet with specific respect to HTML templating, but in Arachne the system for building an HTTP module would simply be to drop in the Hiccup or drop in the Enlive module and have some way of detecting or where Enlive templates were. It would transact that to your asset management chain, and everything would start working. You could start writing Enlive templates. You could tell it which Clojure files contain templates. There'd be a defined format for how your data was, probably some simple map or record-based format, and it could just plug in and go.
Being very conscious from the very beginning, and we're not sewing things together with code. We're sewing things together with data and with well-known data schemas that the different components can code against. That really is a core differentiator between Arachne and these other frameworks is we're incorporating a bunch of libraries, but the interfaces are all data interfaces, not code interfaces. And, we're designing it from the ground up to be useful on a variety of platforms.
As of this point, one of the most mature, most useful Clojure Web frameworks is Immutant, but Immutant is unashamedly tied to the JBoss stack. And, if you're on the JBoss stack and you want to do things in a JBoss way, Immutant is amazing. You should absolutely go use it.
But, if you don't want to use JBoss, well, Immutant probably isn't the tool for you because you'd have to do a lot of recoding to get it to work meaningfully on other types of platforms and you diminish the utility of that. Arachne's goal is to provide modules, if you do want to deploy in JBoss, to pop in there and have it work and not actually build Immutant modules to bridge that functionality, but to work on a variety of platforms and allow individuals to publish the adapters and the unique things themselves and not require any change of code or modification of the core platform to do so.
The other change is why I'm writing the Kickstarter. Most of the Clojure Web frameworks, if you simply Google that phrase "Clojure Web framework" are clearly people's part-time projects. I have nothing against part-time projects. I've run very, very many of them myself. But, I think, for something of this magnitude, we really need to get started with some full-time effort, and that's where the Kickstarter comes in.
CRAIG: You're looking for both individuals to help out, but also you're open to corporate sponsorship as well.
LUKE: Absolutely.
CRAIG: Yeah.
LUKE: Yeah, and I encourage you to go look at that Kickstarter page. It'll tell you all the ways in which you can help out if that sounds like something interesting to you.
CRAIG: We will definitely link to that in the show notes, but I think you said it's going to be Arachne-framework.com.
LUKE: That's correct.
CRAIG: People can check that out. Cool. Well, you know, this is very exciting, Luke. I mean quite a vision, and I think it's funny, actually. We've been recording the 100th episode of the podcast. For the last several weeks, we've been going around and talking to everybody. I spoke with you earlier, and I've been asking people a series of questions. This episode will come out after that one, so I'm not giving anything away here.
** You know one of the questions has been to give advice. I think it's actually the same question we use at the end of every one of our other episodes as well, and I've been asking everybody that one. A couple of times people have said words to the effect of**: Don't be afraid to tackle a big problem. Obviously you've taken that to heart.
I think there are lots of people in this community, and by this community I mean Clojure, in general, Cognitects specifically, perhaps, who have successfully tackled big problems. I think about all sorts of things, really. It'd be really, really great if you achieve what you're setting out to achieve, and I have every reason to believe that if anyone can do it, it's the set of people you've named.
LUKE: Right.
CRAIG: That's awesome.
LUKE: Yeah.
CRAIG: Anyway, all of which is kind of to say that's a great summary. We do have some time left if you have anything else that you think is important to talk about at this stage. I think clearly this is an ongoing project. I think it'll make sense to have you back on again at some point in the future to talk more about how the project is going and has gone.
LUKE: Yeah.
CRAIG: But, here at the beginning, if there's anything else you think we should cover to let people know how you're taking off from here, we can do that. Was there anything else you think we should talk about today?
LUKE: I think we covered it. I'm just looking over my outline of talking points for it. I think the only one we really didn't hit on at length, and I kind of wish I had talked about this earlier, up front, before people had a chance to get bored with me rambling on about tech, is that Arachne really is a project about people. Software is great, and it's really important to have solid algorithms and rigorous design. But, ultimately, software is written by people and it's written for people.
Arachne is squarely in the middle of that. It's a tool that we, human developers, are going to use to write applications. The people that are going to use those Web applications are other humans. This is really kind of a driving philosophy for us. More so than, for example, a language or databases that technical rigor needs to be the important thing there.
We are going to be technically rigorous. But, we also really care about usability. We care about design. We care about experience. I'm not saying you have to have fun using Arachne, but it should be something that doesn't frustrate you. It should leave you feeling happy with what you accomplished when you're done using it.
It means that everything it produces should be attractive or useful by default. We're going to do our best to work with designers to make the experience of building beautiful things easy. It means that we're really going to try to care about user experience as a first-class concern. That's something that certainly I can't fault the Clojure community for producing what they have.
I love Clojure. I love the community. I love most things that have been produced. But, I think there's also room for a little bit of a human element. I think a Web application framework is the perfect place for that because it is so focused on building something that people use and engage with. That is really a core value of Arachne and something I'm very passionate about personally. In addition to all that cool tech stuff, we really do want this to be a project about people.
CRAIG: Well, our listeners are probably bored by this time of hearing me say software is about people, but obviously I've said it before and I clearly agree with you, so I'm thrilled to hear you say it as well. I'm not surprised, of course. I know you. But, nonetheless, I think that's going to be – I certainly would bet on that being one of the key elements of your eventual success, so it's cool to hear you say that.
LUKE: Great.
CRAIG: Awesome. Anything else, dude?
LUKE: Not that I can think of right now. Yes, there is innumerable things to talk about.
CRAIG: Of course.
LUKE: But, nothing that I think makes sense to try to tackle now. I do look forward to talking with you in the future about this stuff. If any of you listeners have any questions about this stuff, I'd be more than happy to answer them. Check out the website. Check out the Kickstarter campaign. Contact me. Email me. I want to build support for this, and I want to convince you that it's going to be awesome, if you don't already believe that. And, I want to know how I can make it better, so I want to talk with you.
CRAIG: Very cool. People should definitely do that. Luke is a really nice guy, everybody. He'd be happy to talk with you. He's a very, very nice guy. You'll enjoy the conversation, as I have.
We are here at the end of the conversation, and that means it's time for our final question. Our final question is, as always, to ask you, Luke, to provide us with a piece of advice. The way I've been putting this lately is – I've actually mentioned that we've been asking everybody this, so I've had a change to ask the question a lot recently. As we always say, it can be advice that you've been given or that you like to give, or maybe even that you've overhead.
I've also added the caveat that – well, it's not a caveat, but the extra point that while we usually assume people are going to hand out good advice, you're also perfectly free to hand out bad advice. It's your call. Anything that you want to give to our listeners in the form of advice would be welcome at this point.
LUKE: Am I allowed to say that you should all go donate to my Kickstarter campaign? Does that constitute advice?
CRAIG: I think you just did, and absolutely. Everyone – I'll say it. You're a modest guy, but I'll say it. Everyone should go donate to Luke's Kickstarter campaign. I certain am going to.
LUKE: Awesome. I appreciate that, Craig.
CRAIG: All right. Excellent. All right, Luke. Thanks again so much for coming on today and taking the time to talk to us. I think I feel pretty safe in saying that we're all very, very excited to see you tackle this project and looking forward with great pleasure to seeing how it evolves and that we all have a lot of confidence in your ability to lead the project and to tackle these problems, some of which are very difficult, but which are important, so thanks both for taking it on and for taking the time to come and talk to us today.
LUKE: Thank you, Craig. I really enjoyed talking to you. Appreciate it.
CRAIG: All right. Likewise. We will also thank your audience for listening. This has been The Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
CRAIG: You have been listening to The Cognicast. The Cognicast is a production of Cognitect, Inc. Cognitect are the makers of Datomic, and we provide consulting services around it, Clojure, and a host of other technologies to businesses ranging from the smallest startups to the Fortune 50. You can find us on the Web at cognitect.com and on Twitter, @Cognitect. You can subscribe to The Cognicast, listen to past episodes, and view cover art, show notes, and episode transcripts at our home on the Web, cognitect.com/cognicast. You can contact the show by tweeting @Cognicast or by emailing us at podcast@cognitect.com.
Our guest today was Luke VanderHart, on Twitter @LeVanderHart. The Kickstarter that we discussed today is, as of this recording, already funded at more than $25,000 of Luke's $35,000 goal with about a week remaining, so really within striking distance. Head over to Arachne-Framework.org to read more about the project and, even better, to pledge your support.
Episode cover art for The Cognicast is by Michael Parenteau, audio production by Russ Olsen and Daemian Mack. The Cognicast is produced by Kim Foster. Our theme music is Thumbs Up (for Rock N' Roll) by Kill the Noise with Feed Me. I'm your host, Craig Andera. Thanks for listening.