This week's book giveaway is in the OCMJEA forum. We're giving away four copies of OCM Java EE 6 Enterprise Architect Exam Guide and have Paul Allen & Joseph Bambara on-line! See this thread for details.
To keep things rolling here, I'd like to hear how people are using Clojure.
Are you experimenting on a personal project? Are you introducing it at work? Have you taken it to production?
I'll start by telling my story. I work for an Internet dating company. Our legacy platform was developed over ten years (before I joined!) and we then spent about eighteen months rebuilding the platform from scratch. The complexity of the platform is such that concurrent execution is a big issue so one of the things I wanted to introduce was cleaner code and simpler concurrency - functional programming with immutable data is a perfect fit for that. I initially introduced Scala for some low level infrastructure, mostly immutable, heavily concurrent with actors. The problems were that Scala was a compile-deploy-debug-edit type of language so development was slow (faster than with Java but still compile, deploy, restart, test, rinse, repeat) and also that the type system - which brings Scala's power to bear - is also very dense for intermediate programmers. Ultimately, Scala was not a good fit for our team so I looked for another functional language on the JVM. Clojure has turned out to be a perfect fit: it's dynamically typed, has a REPL for exploration, is easily callable from Java, offers immutable data for safe concurrency while still offering mutable data in a safe space (STM).
We replaced our lowest level environment control code with Clojure so that we could leverage that across multiple languages. Then we built a small, simple, pseudo-ORM in Clojure to replace some of the libraries we were using - so that we can mix'n'match between RDBMS and noSQL datastores by using plain maps for data ad persisting them transparently. We took the Clojure code to production a few weeks back and it's proven rock solid (any bugs have been our own code!). We just added connection pooling (a few lines of Clojure).
Over time we expect our use of Clojure to continue growing. I don't expect our use of Scala to grow. Our experience so far has been close to a 10x factor of simplification in terms of code - clean, simple, testable code that is much easier to compose and reuse.
Sean Corfield wrote:To keep things rolling here, I'd like to hear how people are using Clojure.
Are you experimenting on a personal project? Are you introducing it at work? Have you taken it to production?
What's your story?
I recently learned of Clojure and since I have no prior lisp experience I'm still learning to program in the language. Once I become more comfortable with Clojure, I plan on re-producing small size applications I've previously produced using some other language. After completing the re-productions, I'll re-evaluate the situation and go from there.
Unfortunately I'm not using Clojure for any useful thing, not in production neither for personal projects.
I really want to participate in open source project but I didn't find any thing that ignites my motivation. Basically all open source Clojure projects till this day are libraries or wrappers for other libraries.
Currently I'm considering creating a web framework for Clojure (I'm writing down the features) but time is a luxury that I don't enjoy these days.
In the race between Scala and Clojure, Scala has the upper hand IMHO.
I work as a research scientist in a big IT company, and I am using Clojure for all my R&D projects started since last year: mostly text analytics (information extraction, sentiment analysis, etc) at the backend and visualization for the Web front end. If we are very successful in these projects and the management plays the politics right, one of these projects has some chances of getting into products. For now, it is too soon to tell...
I will start by saying that I have been retired from working and programming for five years now.
I am a mechanical engineer that started to use Autocad for my job in the mid 80's, learned how to program in lisp, developed a cad program and live on maintaining it for the last 25 year of my productive life.
Information technology has evolved a lot since I retired and I think that this could be the time to put in motion the idea for a revolutionary engineering application that I have had in mind for a while now.
I have chosen Clojure, first because is a lisp dialec, that I am sort of familiar with, also because it offers super portability with its interaction with Java, because it is fast and I think it can do what I want to do faster.
I realize that I am pretty rusty after all these years and probably will need a lot of help but, I am going to try anyway !!!
It's been several months since I started this thread and talked about how we're using Clojure at World Singles so I thought I'd give folks an update on where we are today (especially since we just pushed a new release to production this morning)...
Sean Corfield wrote:We replaced our lowest level environment control code with Clojure so that we could leverage that across multiple languages. Then we built a small, simple, pseudo-ORM in Clojure to replace some of the libraries we were using - so that we can mix'n'match between RDBMS and noSQL datastores by using plain maps for data ad persisting them transparently. We took the Clojure code to production a few weeks back and it's proven rock solid (any bugs have been our own code!). We just added connection pooling (a few lines of Clojure).
We have MongoDB in production as well as MySQL and have been migrating data sets over to MongoDB with only minimal changes to our application (the ORM wrapper takes care of where the data is stored, not the application). We consider that a big win (for abstraction).
We have switched all our logging over to Clojure (using clojure.tools.logging and Log4J). We've rewritten our search engine code in Clojure. We've written log file processing code in Clojure (processing Power MTA logs which are ~200MB a day containing ~500,000 lines) and we manage all of our email status tracking that way.
Sean Corfield wrote:Over time we expect our use of Clojure to continue growing. I don't expect our use of Scala to grow.
Indeed, we ended up replacing our Scala code with Clojure which has resulted in a smaller, faster, more robust "publisher" process (that exports member profiles from MySQL to our search engine). I posted that anecdote on the Clojure mailing list and someone posted it to Hacker News which meant an entertaining 48 hours of infamy attempting to justify the anecdote(!).
We just rewrote our email template handling (using the Enlive library) and, in today's production build, we rolled out our new continuous profile matching process for our members, all in Clojure. That involved unraveling and replacing quite a bit of code across our platform, including our i18n subsystem (which is now less than 40 lines of Clojure). We now have just over 2,300 lines of production Clojure code and just over 500 lines of unit tests. We're finding with the bottom-up style of exploratory development using the REPL, we're not writing as many tests as we used to (but we're looking at our workflow to recast the REPL explorations into unit tests so we have better regression coverage).
Since a lot of our business logic is "do some complex query; filter out a subset of those results; apply some business function across that subset", Clojure is proving a very good fit since we can use lazy sequences to pipeline processes together and we don't have to worry about excessive memory usage from passing around large data sets, nor about thread safety, nor "off by one" errors. We're also getting faster at producing concise, elegant solutions to problems that - in the past - we've either struggled with or have solved in more complex, less maintainable ways...
John Todd wrote:How do you rate/review the maintainability of the source code base?
I can only posit that because it's much less code and in general much simpler, it will be easier to maintain. I've just started publishing a series of blog posts, entitled "Real World Clojure", which will talk about how we're using Clojure at World Singles and showing some of the code (just little pieces).
I am brand new to Clojure. I'm currently a doctoral student at Pace University as well as a full time employee (although this will probably end soon, but that's another story). My dissertation is in the area of keystroke biometrics. We have some software tools developed at the University and one of those tools was developed in Clojure. I'm trying to learn enough so I can modify this particular tool. Coming from a procedural background with using languages like C, C#, Java, etc. I'm finding Clojure quite challenging.
One thing that would be nice is the ability to step through code, display variables, etc. Not sure if you can do this the way you can with Eclipse and Java or the .Net languages with Visual Studio, but it certainly would be nice. I am currently using the Clojure plugin (counter clockwise) with Eclipse.
Ned Bakelman wrote:One thing that would be nice is the ability to step through code, display variables, etc. Not sure if you can do this the way you can with Eclipse and Java or the .Net languages with Visual Studio, but it certainly would be nice. I am currently using the Clojure plugin (counter clockwise) with Eclipse.
Since Clojure compiles to Java bytecode, you can do this but the difficulties include mapping line numbers from Clojure to bytecode and a phenomenon known as local clearing. Clojure debugging is definitely behind Java in terms of tooling but if you take a TDD approach and evolve your code bottom-up in the REPL, you should find you don't need step debugging anywhere near as much in Clojure as you might in Java. That said, Google CDT for Clojure for an interesting debugging approach.
Great topic and thanks for sharing your blog Sean! I've recently dipped my toes into the world of Clojure, started a development blog (http://blog.milehighcode.com) and have had my excitement and interest in programming reawakened through Clojure.
Let me rewind to how I got started programming. I was a curious kid, a tinkerer, who liked to take things apart and see how they work. I grew up in an 8-bit world in the 80's and began dabbling in BASIC at the end of that decade on an Apple IIgs. My youthful forays into the world of code were not disciplined and jumping ahead I was also unfocused in college, so I ended up with a liberal arts degree although I still carried the dream of becoming a developer. In 1998 I began a career as a system administrator, learning all that I could about Windows, Linux, and Solaris systems. The rise of Linux in the workplace kept things interesting, but there was something fundamentally lacking in what I was doing. There just weren’t enough outlets for my creativity and although I got into scripting and taught myself Perl, AppleScript, and some VBScript, it was primarily to automate repetitive systems management tasks and I craved to learn more about the fundamentals of development. Oh, how I wished I would have applied myself more in college and achieved that CS degree, I would lament. I just wasn't able to discipline myself to balance my career and personal life, while also delving deeper trying to become a "real programmer".
The end of 2007 changed all this, when I was offered a real life development position. I was a "web engineer" contractor for a fortune 500 company at the time and asked if I would consider a position as a developer, supporting some of the applications I had worked on. I didn’t consider myself a developer, more a scripter if that even, CVS confused me, Perl references were a mystery to me, but my quick ability to pick things up and learn on my own showed through as well as what I brought to the table with my nine years of systems experience. The fact that I truly cared about the applications I was working on as a web engineer also helped. So I was hired on as developer support for a couple of legacy apps in Perl and Java. This was a great position to be in, since it meant there wasn't a lot of coding required of me to start with, and I was on a team that was growing where there would be limitless opportunities to further my coding skills. Career-wise this was the most exciting turn of events I could have dreamed for and I took full advantage and jumped with two feet in.
The first few months had a steep learning curve, it is hard to explain to someone who understands version control management, branching, and merging how confusing this all was to me. I was in a new world. No longer was I overrun by mission critical system failures and help tickets, now I was in the land of research and development. Well, I added the research part, but having a little room to breathe gave me enough time to research and learn as well as be a productive employee. I also made the switch from Vim to Eclipse and I remember when I first looked at this new fangled graphical editor, I thought, "Wow, that’s a lot of buttons and stuff!".
I began to really dig into Perl that first year in 2008 and read the Learning Perl, Intermediate Perl and Programming Perl series from O'Reilly. It was so much fun, but then suddenly I hit this wall when I picked up Damien Conway's, Object Oriented Perl. The first few chapters weren't bad, in fact I loved the complexity of the subject, but then somewhere along the first quarter of the book, I became lost and my procedural brain, just wasn't able to wrap itself around OO programming, especially not in Perl.
At about that same time I was becoming lost with Perl OO, I was put on a product development team charged to develop a multi-threaded Java application, which took data feeds and loaded them into databases. Talk about steep learning curve! It took me the better part of a year to feel comfortable with Java and the concepts of OO programming, at least to the point where I could intelligently look at code and see what was going on. Going through all of these learning curves, I've begun to develop a thick skin to things that are completely foreign and being in a state of non-comprehension. I know that if I stick with them long enough, the new concepts do eventually get soaked up by my synapses, like what happened with version control and Perl references, which were by now my good friends.
The O'Reilly Head First Java series really helped me along the Java learning curve, as well as Mehran Samahi's Programming Methodology lectures from Stanford (http://see.stanford.edu/see/courseinfo.aspx?coll=824a47e1-135f-4508-a5aa-866adcae1111). Yet it was still slow going and particularly difficult trying to ramp up on a team that was developing a complicated multi-threaded piece of software, which made heavy use of databases, stored procedures, persistence, XML, on and on the libraries went.
I began to latch onto this idea that I needed to learn more about OO design patterns, to help me understand what people where talking about in these architecture and design meetings. So again I became lost as I tried to understand design patterns and had not yet fully grasped OO design in its own right. I was reading the Head First Design Patterns book, and not yet having a strong understanding of OO principles I gave it up. It was mind blowing and enlightening, I was even able to apply what I learned to parts of my code and better understand things my peers were implementing, but somehow I also felt horribly lost and more confused than ever before. I had to take a step back from how I was trying to learn design patterns and realized I wanted to learn patterns by discovering them, not through rote memorization and that would give me this deeper understanding, which I so desperately sought.
That's when I came across the Structure and Interpretation of Computer Programs lectures and began reading about how people raved about this course, if you could stomach the math involved and difficulty of the topic as well as learning a new language, which you likely wouldn’t use in the real world. I started the Abelson and Sussman lectures and also discovered Brian Harvey’s CS61A lectures from Berkeley, which seemed a little more accessible to my already muddled brain, but it didn’t last. Shortly after I hit the A&S lecture on the wonderful lambda, and not having much incentive to use scheme in my day to day work, I lost my drive and gave up.
I took me the better part of a year before I came back to SICP. I was prepared to discipline myself and get through these SICP lectures and the exercises over the course of a year. And that is what brought me to Clojure recently. I really liked Scheme, but I wasn't sure which version to use, there were so many variants. A member of my team had been coding his unit tests in Scala earlier this year, so I began to see what other languages people were using to do the SICP exercises and if Scala would work. Maybe I could find something more modern that would still allow me to gain the knowledge of the wizard book and be practical. Enter Clojure! The more I read about Clojure the more excited I became. Wow, a Lisp dialect that runs on the JVM like Scala, has a great community following and the eclipse plug-in was more mature than what I had seen in earlier tests I’d done with Scala. Why hadn't I known about this language before?! I began to read more about Clojure and I really liked the elliptical syntax, it made sense to me, unlike Scala, which reminded me of when I first saw came across Perl. Clojure had a little syntactic sugar here and there, but fundamentally it was a Lisp and I found out I could "dumb it down" to its roots and use it as a tool to go through the SICP lectures. Plus other people were doing the same thing!
So that is where I'm at with Clojure for the moment. I'm reading Stuart Halloway's, “Programming Clojure”, after having starting Fogus and Houser's, “Joy of Clojure”, and learning it was a little beyond my current grasp of functional programming. I've also recently begun a blog (http://blog.milehighcode.com) to chronicle my brain on development, which was born from my excitement on discovering Clojure. I’ve written a few introductory articles on my experiences using Clojure with Brian Harvey's CS61A SICP class along with other miscellaneous development topics.
I'm not itching to use Clojure in the real world yet, so it is academic for me at the moment, but I am excited that it will become something I will use in the real world. I wouldn't be able to say that if I were learning Scheme. I'm a slow learner and I'm in this for the long haul, but I've already seen benefits learning functional programming in my Java and OO practices. Concepts that were difficult before are becoming easier to digest. (Perhaps anything is once you go down the functional path.)
I'm excited to learn more about programming and OO from the bottom up using a functional language and Clojure is a fine supplement to my current Java work. The Clojure community also is active and exciting, because there are so many people, who are well...excited! Just last night I discovered Anthony Grimes blog (http://blog.raynes.me/) and can't wait for his Meet Clojure book to come out from No Starch Press. He is a great writer and an inspiration. The Meet Clojure book will be available online for free once it is released and is stated to be similar to the popular, “Learn you a Haskell”, book. It’s a big new polyglot world out there and I’m taking it all in, one step at a time.
I definitely think that you'll find advanced OO topics easier to "grok" once you've got a solid functional programming background under your belt - I really do wish more universities would teach Scheme / Lisp / etc as a first language rather than Java. It'll be interesting to see how the MIT introductory programming course works now that they're teaching Python instead of Scheme. Prof. Sussman has talked about that shift (perhaps best paraphrased on Chas Emerick's blog post about MIT's shift). Sussman spoke at Strange Loop 2011, BTW, and it was a fascinating talk about the nature of computing problems (you can see the list of recorded presentations from Strange Loop including Sussman's keynote, as well as Rich Hickey's "Simple Made Easy" which is worth watching several times!).
Anthony Grimes spoke at Clojure/conj 2011 and his session was brilliant - he's a great speaker for one so young (just 17 years old - his mom accompanied him to the conj!) and a very smart cookie. He's responsible for the sandbox machinery behind both http://tryclj.com/ and http://www.4clojure.com/ as well as lazybot on #clojure (irc.freenode.net). He's also a very nice, modest guy. "Meet Clojure" should be an interesting book.