In this episode, we talk to Claire Alvis about data types, macros and city builder games.
Twitch plays Pokemon - emulation of Pokemon Red
Twitch plays Pokemon subreddit
Racket macro facilities
The Reasoned Racketeer - a RacketCon talk by Claire Alvis
Syntax-parse
Spec-tacular
Sparkfund
Core.typed
ORM
Typed Racket
Cities: Sky Lines
WildStar Lua
Curse
Building Designer Patreon Campaigns
EPISODE COVER ART
AUDIO PRODUCTION
PRODUCER
Our theme music for this episode is Thumbs Up (for Rock N' Roll) by killthenoise with Feed Me which was used under a Creative Commons License.
In this episode, we talk about data types, macros and city builder games.
CRAIG: Hello, and welcome to Episode 99 of The Cognicast, a podcast by Cognitect, Inc. about software and the people who create it. I'm your host, Craig Andera.
Before we get to announcements today, I would like to read a post that Justin recently put up on our website. Now, this has to do with some legislation that was recently passed in North Carolina, which is the home of Cognitect. We're a distributed company, but two of our founders live there. Our headquarters is there, so we do have some grounding there. I'll read it out verbatim.
"Cognitect strongly values the rights of all citizens to equal treatment under the law. We find it disheartening that the North Carolina General Assembly would pass, and our Governor would sign, a law that singles out LGBTQ persons for unequal treatment. The law is heartless on its face; discriminatory and mean-spirited. We share with other companies, organizations and individuals our dismay that North Carolina would be actively going backwards on issues of fairness and inclusion. At Cognitect, we believe that no one should be discriminated against based on gender identity or sexual orientation whatsoever, and we will continue to act on those values regardless of what the NCGA does."
That's the end of the statement. I just felt strongly that I wanted that statement to be included in this podcast. I would encourage you to go and research the legislation. I'm not going to take time to discuss it here. But, it's certainly easy enough to find a description of what's involved in that particular bill online.
All right, so moving on to news. Clojure/west is coming up. Tickets are still available. That's going to be held April 15th and the 16th. You can find out all sorts of information about Clojure/west at Clojurewest.org - a fun conference. There are, like I said, tickets still available, but the last I heard are going pretty quick, so you definitely are going to want to go and snag one if you are planning to go. Don't delay.
There's also still room in the Datomic training that precedes the conference. That'll be on Thursday, April 14th. If you're interested in learning more about Datomic, go and take a class with one of our experts and attend the conference right afterwards. That'd be fun too.
What else? I think one more thing to mention, I will mention that the amazingly named Seajure, which is the Seattle Clojure meet-up group, will be getting together on the first Thursday of each month at 7: 00 p.m., so keep your eye out for that. As of this recording, I guess the next meeting would be April 7th. You can certainly find out more about that group and about meeting times and places at Seajure.github.io.
I think that's it for announcements, so we will go ahead and go on to Episode–believe it or not–99 of the Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
CRAIG: Hello, everyone, and welcome to the Cognicast. Today is Wednesday, February 10th, 2016. With us today is a software engineer at SparkFund, Claire Alvis. Welcome to the show, Claire.
CLAIRE: Thank you. Thanks for having me.
CRAIG: I am thrilled to have you on. We have met a few times at various conferences, and had a number of really interesting conversations about the work you've been doing and other things, and so it was a perfect time to have you on.
Actually, you were suggested by someone else at the company who said, "Oh, we should have this person Claire on. She looks really interesting." I'm like, "I know her. We should definitely have her on," so that was great. It was a good conjunction.
But, before we get too much further, I've got to ask you the question that we always ask and that I warned you about. Here at the beginning of the show, we like to ask our guests to share some sort of experience of art, whatever that means to them. You've had a chance to think about this a little bit, and I wonder what you would like to share with us.
CLAIRE: Yep. I would like to talk a little bit about Twitch plays Pokemon. I don't know if you're familiar with that.
CRAIG: Those two things individually, yes. Together, not so much.
CLAIRE: Together, no?
CRAIG: Yeah.
CLAIRE: Okay, well, Twitch is kind of just a website that hosts streaming content. Gosh, I guess it was about a year or two ago someone hosted basically an emulation of Pokemon Red on Twitch. Basically, they had this game up on their stream, and the chat that goes along with the stream was basically inputting moves into this emulation of Pokemon Red. It was this chat just like screaming, like, "Go. Make the character go up," or, "Make the character go left," or, "Interact with something."
The emulation would just pick one of these things at random and execute it in the simulation, and you get to see what happens to the character over time. That was really fascinating. It was fascinating that we actually ended up finishing Pokemon Red and a lot of other very interesting things happened.
But, what was most interesting to me was all of the artwork that came out of it because there was a subreddit dedicated to Twitch plays Pokemon, and every time you went there, there would be like 100 new pieces of artwork that people had done based on what happened on the stream yesterday. People were just basically creating these religious figures and making political statements about this basically emulation of Pokemon Red. You can Google some of the artwork that came out of that, and it was so organic. It was really amazing.
CRAIG: Huh. I actually remember hearing about this a little bit, now that you mention it. It seemed interesting, certainly. I actually was wondering for a little while when you were talking whether it was the game itself that you considered art. Obviously you chose to highlight the art that people created based on it.
CLAIRE: Yeah.
CRAIG: Is there an aspect? I just wonder. Do you think there's an aspect of art to the game itself in any way?
CLAIRE: Maybe. Obviously Pokemon Red itself was a work of art, but there's also kind of a reinterpretation as this emulation that just was randomly being created. I think that certainly is art as well.
CRAIG: Huh.
CLAIRE: In a way.
CRAIG: You played, it sounds like, when you said "we."
CLAIRE: I did. Yes, I did say "we" intentionally. I was certainly one of the people screaming commands at my computer screen trying to get it, you know, not to release that pokemon or to not fall down over that ledge that we just spent three hours trying to get up to.
CRAIG: The art aspect is certainly interesting, but I have to say I feel like I need to dwell on the game itself. The game finished, so obviously it was–
CLAIRE: Yeah, we won.
CRAIG: Right, and so there was enough cooperation. It reminds me a bit of a chat room in general, right, which is you wonder whether one troll can spoil the whole thing.
CLAIRE: It turns out it can, actually.
CRAIG: Hmm.
CLAIRE: Part of the way through, the creator of the stream introduced basically a new form of deciding the input to the emulator. The original method was just anarchy, like anyone could say whatever they want. Just every once in a while, about every 30 seconds or so, maybe more often than that, it would just pick a random direction from the chat, so anyone could just intentionally try to release your pokemon or try to intentionally make you lose the game. Then, halfway through, they introduced democracy, so basically Twitch chat would have to agree on the next input to the simulation.
CRAIG: Huh.
CLAIRE: That was certainly interesting.
CRAIG: I'm sure there's a distributed systems analogy in there somewhere.
CLAIRE: I'm sure there is.
CRAIG: I'm sure we could explore it, but I think there are other things that we could talk about and will talk about that are even more interesting, although that's pretty interesting. But anyway, one of the things, for instance, that I'd like you to talk about, and I want to make sure that we don't miss, is your work on a library called Spec-tacular.
CLAIRE: Yeah.
CRAIG: Maybe before we jump into that, it is interesting, before we jump into that, we could talk a little bit about your background. You actually have a pretty, in my opinion, interesting background.
A lot of people come to Clojure because you are a Clojure developer via Java or Ruby. But, we do get a few people from other language backgrounds, not to pigeonhole you. I don't know. I think you're one of a minority in certain respects. Maybe you can fill in what I'm alluding to here.
CLAIRE: Yeah. I definitely came from, I guess, a different background from Racket and, more generally, just Scheme and Lisp-like languages. As I saw in the recent State of Clojure survey, not a lot of people actually came in from existing Lisps, especially from Racket. I believe I actually had to write in Racket on that survey. If you saw a write in Racket, that was me.
Yeah, it definitely gives me a different perspective because things that I was expecting from Lisp weren't necessarily things that I went on to experience in Clojure. I think some of those are actually good and some of them are lacking from my experience with Racket.
CRAIG: Give us a tour. We have a great relationship with the Racket folks. They have awesome ideas. We've had Matthew Flatt on the show a couple times, and we love the Racket communities. I don't think there's any harm in just getting your perspective. Feel free to say what you like and what you miss. I think that would be great to hear.
CLAIRE: Sure. Basically, as you alluded, I have a library that's pretty much a DSL in Clojure. I think, were I able to write that in Racket, I would have had a much better experience just because of the facilities in Racket for macro writing and being able to implement libraries, in general, is much better in Racket just because of the more established macros, more principle macro system that they have.
But, at the same time, I very much like Clojure, mostly because of the data. I think it treats the data better than Racket does. I don't think that it's a matter of one thing in specific. It's just like, especially like the reader macros or the reader syntax for sets, maps, and stuff in Clojure. It makes it a lot easier to write in, understand, and have faith in your data. I think those are the two biggest differences I see between the two languages.
CRAIG: Mm-hmm. Yeah, that's been my broad experience, although the Racket one for me is from the outside. I did find it interesting at RacketCon. You were there, was it this year?
CLAIRE: Yeah.
CRAIG: I forget the researcher's name. She presented. She's been working on some of the immutable – well, actually quite a substantial portion of the immutable data types that Clojure has, persistent immutable collections, but for Racket. I'm trying to remember her name.
CLAIRE: I think I might have left. I didn't get to stay for that entire RacketCon. I don't think I saw that talk.
CRAIG: Okay. In any event, so good ideas are flowing in both directions.
CLAIRE: Yeah.
CRAIG: And the persistent. You know Rich is the first to admit he didn't invent the persistent immutable collections, but we certainly have a good implementation of them. But, it looks like other languages are are doing that as well. This is a question that I've asked Matthew Flatt a few times, and every time I get a little bit closer to understanding, but it's certainly something that comes up again and again, namely that the macro system in Racket is really great. It's a huge win, and it's a thing that's a huge asset to the language.
CLAIRE: Yeah.
CRAIG: You just mentioned it yourself, so I wonder if you could explain to somebody who hasn't done Racket what it is about the macro system in Racket that is so great and what maybe even what you think it would look like if we had that in Clojure.
CLAIRE: I think my favorite part about the Racket macro system is the things that you don't have to do and not the things that it actually enables you to do. There's a form called syntax-parse, which that would be like death macro, an improvement on death macro in Clojure, where syntax-parse kind of just let's you just say what your syntax is and let's you just say what it expands to, and it kind of handles or at least lets you express in a more clean way what should happen if things were to fail. If your macro failed to parse the syntax or something, it makes it very easy to say what exactly should be parsed and what should be output to the user if it doesn't kind of expand very cleanly.
That's probably the biggest thing that I miss from Racket's macros, whereas in Clojure it basically just treats it like a function, right? You get the syntax in as if it was just another piece of data, and you basically have to do all the manipulations, parsing, matching, error checking, and everything. You have to do that all yourself inside the code. Clojure doesn't give you anything, any tools for that other than the tools from the general language itself.
CRAIG: Mm-hmm.
CLAIRE: That's probably what I miss the most.
CRAIG: Yeah, and you mentioned error handling, and that of course is one of the big consequences of that. Unless you do a bunch of work yourself in Clojure, you wind up with, because of the way macros expand, run, compile time and execute later, you can wind up with errors that don't really relate directly to what the developer wrote down, right?
CLAIRE: Yep. Yeah.
CRAIG: Relative to the expansion. Yeah, yeah. This is stuff that Chris Houser gave a talk–oh, gosh–I think it was almost three years ago now at Clojure/conj. Actually, similar, I think, to what Colin Fleming talked about just this most recent one with his work around macros.
CLAIRE: Yeah.
CRAIG: Huh. Okay. Interesting.
CLAIRE: I actually went up to him afterwards and talked about syntax-parse.
CRAIG: Mm-hmm. Yeah, he's a nice guy, and I'm sure you had good input for him because every time I've talked to you, you've always said smart things.
This actually leads very well, I think, to talking about spec-tacular. Obviously you have a background in Racket. It sounds like you were using that experience to help you write this library. I'll let you say what it's about - spec-tacular.
CLAIRE: Yeah. The intention of spec-tacular is to sit in between Clojure and Datomic. Should I explain what Datomic is? Would your viewers know?
CRAIG: I think we'll assume that they do.
CLAIRE: Okay.
CRAIG: We have a lovely website that they can check out if not. You have so many interesting things to say, we will leave space for them to explore that on their own should they need to.
CLAIRE: Okay. Yeah, so it sits in between, and the intention is to kind of give you a safer–and this is the catch-line–a safer and more convenient place to live. It kind of gives you better interactions between Clojure and Datomic. And so, that's two-fold. I want to kind of add an extra specification over Datomic's schema that let's you say more interesting things about the relationships between the datums, mostly typed, making sure that the actual fields inside Datomic entities actually make sense relative to some kind of grander schema. Then, if you do that specification, if you take the time to write down these types for your Datomic entities, then it'll give you a lot of really nice API functions to create those, to update those, to query them kind of in a more succinct way than just raw Datomic lets you do.
CRAIG: Yeah, so raw Datomic, as you say, the schema is "just" the specification of the attributes, right?
CLAIRE: Yes.
CRAIG: That's kind of the moral equivalent. Not an exact analogy, but the moral equivalent of column in a relational database.
CLAIRE: Mm-hmm.
CRAIG: And so, there's really no – I think what you've done, in part, is to do the thing people always want to do, which is to say, "Well, I have this notion of an entity type, like a person that has this collection of attributes," whereas Datomic would let you say, "Well, you can talk about name, and you can talk about address, but we don't really give you a way to say that there's a thing in the database that has both a name and an address and cannot have a shoe size, or whatever.
CLAIRE: Yep.
CRAIG: Is that a fair assessment of part of what you've done?
CLAIRE: Yeah. It let's you basically define – it's called defspec, and it let's you define. You could say, "Define your person," and it's well understood what fields a person has or what attributes of a person you want to express. You would say, "Well, it can have your name and your address," as you say.
Then, if you had a separate entity type, like a car or something like that, and you say, "Well, my cars can have wheels." If you interact with your database with spec-tacular, it will basically prevent you from ever putting a wheel directly onto one of your person entities or putting an arm or something from your person entity onto your car.
CRAIG: Obviously you wrote this for a reason. We mentioned briefly that you work at SparkFund. I actually don't think I've talked to you about this history of this thing so much, so what was the genesis of this? What was it about your work where you were just like, "We really need to have this thing"?
CLAIRE: SparkFund, we have an engine that handles a bunch of financial transactions, so we're interested in modeling kind of complex business agreements and relationships, and then all of the actual movements of money between those entities. We kind of wanted an extra layer of sanity that makes sure that all of the things that we're storing about these relationships and actual movements of money are typed according to this kind of very strict type schema. That was kind of the motivation behind the library was wanting to add an extra level of confidence when we interacted with the database that's holding all these very sensitive and critical pieces of information.
CRAIG: When you say type, you actually are making use. There are a couple things there, right?
CLAIRE: Yeah.
CRAIG: I think you know at least the two that I'm thinking of, so maybe you could go into that a bit.
CLAIRE: Yeah. spec-tacular basically gives you a core.types right off the bat, so if you define what your person is, and it also gives you a person that is a core.types type that you can use for static type checking. Then it also gives you a predicate that you can use for schematic schema. So, if you are more of a runtime checking person, then you can use those predicates for schematic schema. Now matter what your relationship is with types, we hope we have an answer for that.
CRAIG: Mm-hmm. You're using this in production at SparkFund?
CLAIRE: Yep. Yep. To great success, I think. It's given us a lot of confidence in refactoring our engine code that we wouldn't really have if we didn't have this extra layer of typed interaction.
CRAIG: I've got to ask. One of the things I think people could look at this and say, "Oh, this is an ORM," and I think that's always a tricky classification to make on any library like this, but it is an interface layer between database and the code that talks to the database.
CLAIRE: Mm-hmm.
CRAIG: Historically, there have been. I haven't used spec-tacular, but my experience of other ORMs has generally not been good. I find that the big issue is that I lose control of the queries that I'm making, right?
CLAIRE: Mm-hmm.
CRAIG: I lose control of the interaction and that it matters. For instance, really, really silly SQL will get created for me in something like – oh, gosh. I'd have to bring my mind back to 15 years ago when I was still using C# and doing this sort of thing. How have you avoided those problems? Maybe you don't have that same experience. Maybe you're like, no, I really do want that abstraction. What's your answer to that observation?
CLAIRE: I think my answer, in general, is that spec-tacular itself right now probably isn't all that great for probably exactly those reasons. What I've come to decide over the last couple of months is that it has exactly that problem where, if it doesn't fit exactly into this query language that I've created, I have no other option except to then drop all the way down back to Datomic. Now, I've given up a lot of the freedom that I would have had if I had used just Datomic schema and actually went to create a Datomic schema that I would have created not thinking about the types. Then things just become a lot harder.
CRAIG: Is this a problem that you've run into, or it's just when you're looking at it thinking about the design. You're like, "You know what? I could see that being an issue."
CLAIRE: Yeah, actually both. There have been a lot more times recently where I've deliberately chosen to use just the Datomic API and not spec-tacular's API, like when I'm writing in production and also when I'm thinking about the design of spec-tacular. I think I would express it as saying there's no little increment between the Datomic API and spec-tacular's API. It's either it fits into this very, very strict kind of syntax for like the queries, for example, or you have to drop all the way back down to just doing Datomic. I think there's something in between that kind of could fill this gap where, yeah, it doesn't fit into the very succinct, cute, syntactic sugar kind of thing, but you also still get a lot of the benefits that you should get because you wrote down. Like, you wrote down the spec, so you shouldn't lose everything.
CRAIG: Right, so obviously I've seen that because I mentioned it.
CLAIRE: Mm-hmm.
CRAIG: One of the problems that you had in the old style ORMs or I guess the ones that I used to use, which I suppose makes them old style given when I was doing this, was that the query, the result of this mapping is essentially a string. The thing that you send to the database is a string. Effectively, it's opaque.
Now, you can do things like parse it. But, if you're working with SQL, at least, it doesn't really have or at least the libraries that I was using didn't expose. I shouldn't say it doesn't have. It didn't expose any kind of a regular structure that I could work with. Whereas with Datomic, the query, the native form after the query is in fact data.
CLAIRE: Mm-hmm.
CRAIG: And so, I wonder whether, when you think about how to solve that problem, you're like, "Well, the fact that it's data is something that's going to enable me to maybe, hopefully find a way to pull these two things apart and still get all of the utility of each of them independently or pull them together and get the power of combinatorial explosion, if you will.
CLAIRE: Yeah, absolutely. That's actually exactly where I was planning on drawing the solution from because the query right now, the query that spec-tacular gives you is just a giant macro, right? And so, the actual query syntax is not data. It's syntax, and I basically create a query that is data at compile time, but you, as the user, have no access to that. You could get access to it via a macro expansion, but it's basically you don't have control over it.
Whereas, I think what I kind of lost when I moved from Datomic's query up to the spec-tacular was exactly that, the data. And so, I think there is a way. There's obviously a way. I just haven't written it yet or I haven't thought of it yet, to get all the benefits of spec-tacular, but still have that query be data that the user can then have in their hand, and then it can be composable, and they can inspect it, and they can optimize it if they feel like it if spec-tacular has done something silly like created an "or," a giant "or" that it doesn't really need. Then the user can see that and make optimizations if they want.
CRAIG: Mm-hmm. A phrase that we've thrown around before is data function macro, right?
CLAIRE: Mm-hmm.
CRAIG: There's this sort of – I don't know. It's not really a law. We've talked before about flipping it on its head is sometimes the right thing, but certainly there's this sort of path or hierarchy, something where we say, "Okay, we prefer data to functions, functions to macros."
I'm just thinking about this problem and going, well, you know, you went with macros. Obviously you had some reason for doing that, and now you're thinking about playing with the idea of data. I guess I'm just wondering. Do you think, if you make that refactoring, will you find yourself removing or lessoning your reliance on macros somehow? Does that make any sense?
CLAIRE: Probably. Basically, the main motivation for that query to be a macro right now is the type. If you write a spec-tacular query, it will actually do all the correct casts to actually type that in core.typed. So, you could have a query, which is obviously an improvement over Datomic where, yeah, you get back data, but you're not really sure what it is. Whereas, you can actually just write it directly in core.typed and have that query be statically checked alongside the surrounding code.
But, the fact is, you don't actually need most of the query in order to determine that type. You basically only need the find variables to find out the actual return type. There still is going to have to be some macro aspect of it if we want to retain that behavior, but the rest of it can just be a function. As soon as you figure out what the types should be, then you can hand it off to a function that will do all the good things that functions do rather than have it all be a giant macro.
CRAIG: Hmm. Mm-hmm. Okay. It makes sense. It makes sense. Interesting. What were the challenges that you had when you were developing this? Obviously you mentioned that you were used to a different and arguably more powerful macro system. What else was hard about making this thing?
CLAIRE: I was coming to this language when I first started without having a good understanding of Datomic as a whole. I had a very good understanding of what we were using Datomic for and what uses we, in our production engine, wanted out of this library. And so, the initial versions of spec-tacular that I released were very much geared towards our own use cases.
Over time, I had to kind of broaden the use cases of spec-tacular by doing all the research into, okay, Datomic is this thing that I'm working with now. Why do people like Datomic? I had to kind of not only understand all these reasons why people liked Datomic, but also how this library can help them and still keep the things that they like. Obviously, like I've just been talking about over the last ten minutes, I think I kind of missed that point. And so, now it's very difficult to kind of reinterpret that or look at all that information again and come up with a different design that is hopefully more in line with why people like and why Datomic is so good.
CRAIG: Just to make sure I understand, the thing you're referring to is the query syntax, or is there some other feature of Datomic that you feel you've hidden from people?
CLAIRE: For example, we didn't make any use of pull syntax. The way that I allow you to do updates with those cute little update functions is not very composable. It doesn't let you do any kind of batch operation, so you basically have no choice, but to kind of use a low level function to get the data out and then do the composition yourself. There's no way to specify that I want all 10,000 of these things to happen on the same transaction, for example.
CRAIG: Mm-hmm. Maybe you haven't figured it out yet, but do you feel like there's a good way to do that?
CLAIRE: Yes.
CRAIG: You do? Okay. Good, because I think one of the things that happens sometimes is that we get attached to an idea. In this case, I could say one of the ideas, and I think there are a couple in here, is providing core.typed and, to some extent, schema, pro schematic schema information around my entities in Datomic and preserving that as I move the data back and forth between my data processing layer and the database.
CLAIRE: Mm-hmm.
CRAIG: I feel like sometimes we kind of get all into the idea, and then we do more and more things to preserve that idea. I'm not saying this is the case. It sounds like you've thought of a way. It's not, but I feel like certainly I do this where I'm like, okay, well, I've got to keep that even when, by keeping it, I'm giving up other things to the point where, at some point, the costs outweigh the benefits.
CLAIRE: Uh-huh.
CRAIG: Hopefully you understand what I'm saying.
CLAIRE: Yeah, absolutely.
CRAIG: Anyway, it sounds like you feel like that's not the case. I'm curious to hear what you have in mind.
CLAIRE: Yeah. I haven't had a lot of time to think about this, but, in the time that I have spent thinking about this, I don' think there's anything fundamental that we've lost that was a result of that decision to make things well typed or at least a little bit more typed than they were before. I think all of the things that I've talked about here about the loss functionality in between, or kind of lost by using spec-tacular instead of Datomic, is not a result of that decision of making things typed. It's just more of a kind of API, like the API is just not quite right. It's not because of anything that has to do with making these things typed. I think we can have both. I can have my cake and eat it too.
CRAIG: Yeah. I'm inclined to agree with you. I haven't thought about it anywhere near as much as you have, but it seems to me like what you've done is said, "Well, look. Datomic certainly has a notion of entities, but it doesn't have a notion of entity type, and ignoring things like polymorphism." If you just say, "Look, this entity is going to have this set of required attributes and this set of optional attributes. And, when I pull it back over here, I'm going to preserve that," that does not seem to me to be a fundamentally really strict requirement.
CLAIRE: Exactly.
CRAIG: If your data is shaped that way anyway. Now, if you have more flexible data, if you have a data schema where you're like, "Look. I really do have a pile of attributes, and they can in fact apply very loosely to entities," then I could see maybe that an approach like that wouldn't be particularly – the impedance mismatch would be high.
CLAIRE: Uh-huh.
CRAIG: But, I think a lot of programs are not that way.
CLAIRE: That's my hope and my understanding. Basically, what that would look like is a single spec with 100 fields.
CRAIG: Right.
CLAIRE: That's not really going to help you. I'm not sure why you made your database like that, but that's certainly not the kind of database that I want to support or I have in mind of supporting. But, yeah, I agree. I think most or at least the databases that I've had experience with, would fit into the category of databases that I would like to support with spec-tacular.
I think, if your database looks like the kind of database that I can support with spec-tacular, then it should be easy to interact with it using spec-tacular's API. I think that's the part of it that's broken down right now. Yes, we can represent your database. Yes, there's a spec for it. Yes. But, if you want to go and query it, just use plain old Datomic is basically the answer that we have for you right now, which it shouldn't be. It shouldn't be the answer. It should be you can use spec-tacular to do this.
CRAIG: Yeah, or maybe, like you were saying. I think I'm understanding better what you were saying earlier where there's got to be some sort of nice intersection between, "Look. I still have everything that Datalog gives me. But, when I'm done, what I have is maybe, at least optionally, a pile of values that I can treat as these instances of these types that I've defined. Right? I'm querying for people, so let me write the data log. Then, at the end, I've got things that I can interact with in the same way that I interact with people when I create them and do the other things that I do programmatically with the types I've defined through the defspecs.
CLAIRE: Yep.
CRAIG: Yeah, yeah. It's funny though. I just said, "Okay, yeah, I get how most applications are like that." But then, at the same time, I'm thinking to myself, "Well, but then why do we like Clojure's maps so much?" There's another really open data type. Obviously core.typed has syntax. Schema has syntax for saying, "Yes, this is a map, but we're going to talk about what keys must be in it, can be in it, cannot be in it," that type of thing.
CLAIRE: Uh-huh.
CRAIG: But, at the same time, I think one could make an observation and say, "Well, in Clojure, we have this great data type to map. It's open. It's essentially flexible. It's very untyped, and we're happy with it. We like it." In some sense, it's kind of that other type of program. In other words, we're frequently writing that other type of program where we don't really care what the data is, or it's very, very open. It's like we joke, right? If you write Java the way that you write Clojure, you type everything as object, right?
CLAIRE: Uh-huh.
CRAIG: I know that's pushing the analogy very, very hard, but do you see what I'm saying?
CLAIRE: I do.
CRAIG: We actually like this flexibility, and so, to some extent, I'm like, "Well, are we moving into a regime where we're going back away from the thing that we demonstrate our affinity for every day in Clojure by working with these maps?" Do you know what I mean?
CLAIRE: Yeah. No, I haven't thought of that before, but you make a very good point. Of course I use maps. I use maps everywhere, and I'm trying to think of why. Why do I use these kind of unstructured maps?
CRAIG: I don't have an answer either, by the way.
CLAIRE: Yeah.
CRAIG: It's just an interesting question that came to my mind while we were talking about this.
CLAIRE: I can make them, then I can use them, and then I can just throw them away. I think that's what I like about it. If I suddenly just need, for efficiency or for ease of expression, if I just suddenly need a mapping from friendly identifiers to numbers, or something like that then, yeah, I just make a map. Then I use "get" to get the information. Then it's gone. I think if I ever wanted to store that, or if I wanted that to persist in some way, I would make a record instead of a map because I would want to have – or, if not a record, at least some sort of predicate or core types that would say that this map is actually as I expected if I wanted it to persist in some way or have some sort of level of confidence about what's actually in this map.
CRAIG: Mm-hmm. I wonder. This is something that I've been thinking about a little bit. It was a discussion I had with–I'm forgetting his name–someone at RacketCon.
CLAIRE: Mm-hmm.
CRAIG: They were a big fan of Typed Racket, which, as people that have listened to the show are aware, was the big inspiration for Ambrose writing core.typed.
CLAIRE: yeah.
CRAIG: But, we were speculating, given that he's a fan of dynamic languages, but he still likes the typing that Typed Racket provides. We were wondering whether there's not some mode of programming wherein you use dynamic programming to very quickly and easily explore the problem space.
CLAIRE: Mm-hmm.
CRAIG: But then you codify the constructs that you discover that are–once they kind of settle down a bit–through types. I wonder sometimes. I've always meant to explore what would it mean to write a program where I start out with no types and then add them in over time. I know lots of people have done this, but I haven't really.
I've either said, "I'm going to try core.typed," and gone in and written the program type from the beginning, or said, "Forget it. I'm going to use none of that and written my programs the way I normally write a Clojure program," which is without any type information. But, I've never actually tried to create a cycle over the long-term where I introduce types as concepts, make themselves evident as something that's going to be long-term. You used the word "persistent" in this case, meaning database persistence.
CLAIRE: Mm-hmm.
CRAIG: I wonder whether there's some sort of analogy to be had to that concept there.
CLAIRE: I guess the way that I've been taught or the way that I did teach people to kind of approach the empty buffer problem was to think about your data first. Don't just start writing. That's, of course, what How to Design Programs teaches, which we taught at Northeastern, to think about your data first.
And so, the first thing that I would write, if I was faced with the empty buffer problem, wouldn't necessarily be a type in the way that we understand types, like a typed annotation or something like that. It might just be a comment that says what my data looks like, like what keys my maps have, for example. Then I would design all the functions inside of that with that very good understanding of what my data looks like.
I'm definitely not saying that you start writing typed annotations because that's succinctly what I don't want to do. But then, as time passes and those functions change, you want to have something that isn't written in a comment so that you remember to go back and update what you said your data was if your data changes. That's where types really come into it for me.
Yes, I don't want them. I want them to be written in a comment, as I'm doing the fast prototyping that people love dynamic languages for and stuff. But then, once the code sits there for a while, and maybe that's what I mean by persistence was the code sits there for a while itself. You want to be able to say, "Hand that artifact off to someone else," and have them be able to understand the types of data that this code is working on. And so, if you still have your typed declaration in a comment, nothing is keeping that up to date.
Maybe that's still the first. Maybe you haven't changed that since you had your empty buffer and, since then, your data has evolved, and it's changed a little bit as you've written these functions. Now you're handing this code to someone that is looking at this comment and is like, "This is not the data that you're using anymore." If you had that written down in a type that was statically checked that you couldn't hand this code off to someone if it didn't type check, then you'd go back and you'd update your understanding. You'd change the type annotation to be more correct.
CRAIG: Mm-hmm.
CLAIRE: That's pretty much what I use types for, I think.
CRAIG: Does that extend? Is that analogy exact with respect to spec-tacular? You know, whenever we cross system boundaries.
CLAIRE: Yeah.
CRAIG: Certainly, going from your data processing code into the database is crossing a system boundary. Is there anything about that idea that, in your mind, breaks down or changes?
CLAIRE: That's really interesting. spec-tacular certainly wouldn't let you do that. You would basically have to have your specification ahead of time. Any time that you tried to interact with the database in a way that didn't agree with that specification would basically prevent you. It will entirely prevent you from being able to interact with the database if, say, you have the wrong keyword in a map somewhere that goes against the specification. I wonder what it would look like to kind of have a node check, if you will, equivalent for spec-tacular, like core.typed has.
CRAIG: Yeah. It's a little different, though, right? For people that aren't familiar, node check just means, "Hey, I know I said this thing is a foo, but don't keep me honest," right?
CLAIRE: Don't make me prove it.
CRAIG: Yeah, don't make me prove it. And so, it's a little bit different because, of course, core.typed is entirely static, right?
CLAIRE: Mm-hmm.
CRAIG: It's just going to look at your program and go, "Yeah, I think you're good." Presumably spec-tacular is not doing code analysis at all. It's like Runtime, right?
CLAIRE: Except for the little bits of a query that do expand and compile time, yeah, it's absolutely Runtime.
CRAIG: Mm-hmm. Yeah, so that's a great question. I'm going to be fascinated to hear what you come up with on that one.
CLAIRE: Yeah. What would that look like? I don't know. I guess it would just have to let you interact with the database, and it would tell you, "Maybe your spec is a little out of date, but I'm going to let you do it anyway."
CRAIG: Huh, and that's interesting, especially in a database like Datomic where we keep everything, so it's going to let you make a mistake. It depends on your definition of mistake, I suppose, but it's going to let you say something nonsensical.
CLAIRE: Uh-huh.
CRAIG: At least with respect to your spec, and put it in the database. Then there it is, right?
CLAIRE: Yeah.
CRAIG: Now that can have huge value, and it's actually very much intentional. At Cognitect, we frequently talk about the benefit of recording your mistakes. People are always like, "Well, how do I get rid of these things? I wrote some bad code and it put it in the database." We're like, "You know, you don't actually want to cover your tracks. You don't want to shred the documents just because there was a typo. Put a new document in, but don't burn down the building because somebody spilled some ink," right?
CLAIRE: Yeah.
CRAIG: Yeah.
CLAIRE: That's really interesting.
CRAIG: Cool. spec-tacular is super interesting, and I can tell that you've put a lot of thought into this.
CLAIRE: Yeah.
CRAIG: Like any really good library, like every time I'm talked to an author – maybe not every time. There are rare occasions, I suppose, where one works on a problem and it just beautifully lays down. You find the one solution. It's like, okay, great. The hallmark of that usually is the person stops working on it because it's done, right?
CLAIRE: Mm-hmm.
CRAIG: Which, in the open source world, of course, is a sign of something being low quality if you stop working on it.
CLAIRE: Yeah.
CRAIG: But of course in this case it's like, "No, no. I figured it out. Here it is. It's a gem. It's done." But, of course, there are lots of other libraries where people are like, "I'm still–" You know, these good ideas take years, usually, to sort out. I'm fascinated to hear that your thinking is evolving, it's already useful, but that you can see places where you can improve it.
I don't want to spend our entire time together talking about spec-tacular. We can, but I want to make sure that I ask you what else is up because I suspect you're one of those people who have several irons in the fire and that even the ones I haven't heard of are as fascinating as the rest. I don't know. I guess it makes it unfair expectations, but I'm wondering what else you've been up to.
CLAIRE: Oh, gees.
CRAIG: What's caught your interest lately?
CLAIRE: I can't really talk more about the software. I admit that I am not working as much on spec-tacular as I would like to be. I've kind of switched focus mainly towards our engine at SparkFund and our end of the year reporting. What happened in 2015? That's probably where most of my focus is right now.
CRAIG: Sure. Well, what do you do for fun outside of work?
CLAIRE: Let's see. I'm a big gamer, honestly.
CRAIG: Well, I could tell when you said – what was the phrase I think you used? It was, "Well, Pokemon Red is a classic, obviously."
CLAIRE: Obviously.
CRAIG: "Obviously" was the keyword. Yeah, yeah, what do you like?
CLAIRE: I play MMOs mostly, but also city builders. What I've been playing most recently is a game called Cities: Skylines, which is kind of the spiritual successor to kind of what we all wanted SimCity to be. I've been playing SimCity since, I think, like SimCity 2000, I think was my first game. I've had a long, long experience, many hours put in just building cities in the Sim world.
CRAIG: That's cool. What do you like about it? I've done a few of those things. I won't bore you with the details. I'm into flight simulators, so there's an aspect of simulation in my current interest in gaming. What is it that you like about the Cities Builders?
CLAIRE: I think it's mostly the problem solving, honestly. You basically have the empty buffer problem, right? You have this empty city that's kind of connected to a highway in Cities: Skylines. Immediately, people want to live there, obviously, so you zone a bunch of residential areas. Then obviously they need some place to work, so you zone industrial. Then they need a place to sell their goods, so you zone a bunch of commercial.
Then, obviously, you need power and water, and so you have this, it's basically just like an interconnected ecosystem of different constraints and demands of your population. The problem is being able to scale that up to a point where your city covers the entire map, but everything is still kind of well balanced in that you have enough power, but not too much power. You have enough water, but not too much water because it's expensive. Your people can actually get to the different places that they need to get to, but traffic isn't so bad. You don't want everyone to be going all to the same area at the same time or else you'll have this giant traffic jam.
Of course, as I'm sitting here explaining it, I'm like this is computer science. This is computer science in a different form.
CRAIG: Sure. I'm curious. I've actually gotten into flight sims enough where I can have conversations with people that hold a pilot's license and I'm not totally ignorant. Do you find yourself with an appreciation for people that do city planning for real?
CLAIRE: Absolutely. Absolutely. I find myself evaluating on ramps. I'm like, this on ramp isn't long enough, and it's causing traffic to back up onto this road. If they were to reposition this on ramp so that I had more time to gather speed before getting on the highway, it wouldn't back up onto this. I've had these conversations in my brain with myself because I've played Cities: Skylines.
CRAIG: I'm curious. When you do that, are you also thinking, "But it would cost $3 million, so there are all these constraints"?
CLAIRE: Yes! Exactly. Exactly.
CRAIG: Yeah, it's fascinating. It's really interesting. Have you ever run into somebody that does that sort of work for a living?
CLAIRE: I have not. I would love to. I would love to. If anyone is listening and knows about city planning, I would love to talk to you.
CRAIG: Cool. We'll have them tweet at you. That's very, very cool.
I know what I was going to ask you. Have you done the thing that I think a lot of us who have the ability to create software do, which is take in this hobby, or maybe one of your other hobbies, and made software for it? Like a SimCity plugin or maybe just a website where you computer, or even spreadsheets or anything like that? I find myself doing this sort of thing.
CLAIRE: Yeah. I absolutely have done that. The only one that comes to mind right now is I played an MMO called WildStar a few years ago that ended up not being the amazing success we all thought it would be. But, for the months that I did play it, I actually made an add-on in Lua.
I saw that there was no add-on that anyone had already created to basically buy and sell commodities on the auction house, and I wanted to make a lot of–I think the currency was called–platinum, so that I could pay for my game time for free. These add-ons didn't exist, so I actually wrote one and released it on Curse, which is the way that we share these add-ons, so yes. Yes, I have done that.
CRAIG: Did you make a lot of money?
CLAIRE: I did make a lot of in-game currency, and I got one $5 donation for my time.
CRAIG: Sounds about right. Cool. Actually, I think that might be changing a little bit, I feel like. Maybe it's just me. I don't know. I feel like people are a little bit more willing to give money for software that they find useful.
CLAIRE: Yeah.
CRAIG: And there's more of a culture around donations and a willingness to give even small amounts, or maybe especially small amounts.
CLAIRE: Yeah. You see that a lot on Twitch, which I talked about. There you can subscribe to channels with streamers whose content you like, and I think it's a $5 donation every month, basically. I've seen lots of people willing to just give $5 to that cause.
There was one success story also in Cities: Skylines where one of the people who designed buildings for the original SimCity games–I forget his name. That's horrible of me–he set up a Patreon that every time he came out with a new building or something like that, people would agree to give him money. That also worked very well for him, if I understand correctly. Yeah, people definitely are willing to at least part with some small amount of money for these good content creators. I love to see that.
CRAIG: Yeah, that's very cool, actually. We love it when somebody makes something for something you love in what feels like maybe a small community, and then they get rewarded. That's always kind of gratifying somehow.
CLAIRE: Yeah.
CRAIG: Well, we are kind of starting to draw near the end of our time. We should probably wind down and let you get back to work. I want to make sure we leave a little time if there's anything else that you'd like to talk about. If not, that's cool. I think we definitely have to have you back. I think your work, just spec-tacular alone, it sounds like you've got at least three or four interesting ideas you're going to explore, and I'd love to hear how they come along over the next months or whatever. Anyway, was there anything else that we should hit before we start to wind down?
CLAIRE: I think we covered all the things that I wanted to talk about.
CRAIG: It was fascinating. It was fascinating. I do have one more question for you, though, as you've been forewarned.
CLAIRE: Yeah.
CRAIG: We always ask our guests to give a bit of advice. Actually, not necessarily to give a bit of advice. I like to say share because it might not be advice that they're necessarily handing out to the audience, although often it is, and usually very good at that. But, maybe it's something that someone gave you some advice once and it was specific to you, but it was great. Anyway, just would love to ask you to share a little bit of advice of whatever kind.
CLAIRE: Yeah. I think what I really want to say, and of course I didn't come up with this, but don't ever be afraid to say you don't know something. That is the single best piece of advice that I think I've gotten my entire career. I can't tell you who told me it first or who told me it last, but every single day I find a reason to say, "I don't understand what you just said. I don't understand what that acronym stands for. I don't understand what you're talking about. Can you please explain it to me?"
The follow up to that would be: Surround yourself with people that don't look down on you when you say that because you want people around you that are also willing to explain these things to you. I think those two things together, just please do that.
CRAIG: Mm-hmm. I think that's great advice. I love it. Well, absolutely fascinating conversation with you, unsurprising to me as I have been fortunate enough to have several such conversations with you previously, but I really appreciate you taking the time out of your day today to come on and speak with me where I can share it with the rest of our listeners. Thank you so much for coming on today.
CLAIRE: No problem.
CRAIG: It's been great, and I look forward to speaking with you again some time soon.
CLAIRE: I'd love to.
CRAIG: Yeah, yeah. It'd be great, and we'll keep our eye on spec-tacular. Of course we'll make the link available to everybody. But, I'm sure if they search for spec-tacular Datomic, they'll go straight to it.
CLAIRE: Yep.
CRAIG: Anyway, I think we will go ahead and close it down there. This has been the Cognicast.
[Music: "Thumbs Up (for Rock N' Roll)" by Kill the Noise and Feed Me]
CRAIG: You have been listening to the Cognicast. The Cognicast is a production of Cognitect, Inc., whom you can find on the Web at Cognitect.com and on Twitter, @Cognitect. You can subscribe to the Cognicast, listen to past episodes, and view cover art and show notes at our home on the Web, Cognitect.com/cognicast. You can contact us by tweeting @Cognicast or by emailing us at Podcast@Cognitect.com.
Our guest today was Claire Alvis, on Twitter @chckadee. Episode cover art is by Michael Parenteau. Audio production by Russ Olsen and Daemian Mack. The Cognicast is produced by Kim Foster. Our theme music is Thumbs Up (for Rock N' Roll) by Kill the Noise with Feed Me. I'm your host, Craig Andera. Thanks for listening.