We've got Clojure projects running against the most stringent demands.
IG is the world's #1 retail provider for spread betting, CFDs and physical shares trading with support for SIPP and ISA accounts. The platform is massively used servicing a large variety of workflows, meaning that the software has to be incredibly robust.
The company recently celebrated its 40 year anniversary. In the early days all business flows were conducted via the use of telephones. Now it has all been automated using a mix of Java, C, C++, and recently some Clojure.
Jon Pither: Where did the Clojure journey start?
Daniel Lebrero: We had a manager who encouraged us to seek out new technologies that we could potentially try out at IG. I was sick of Java and we were looking for something that we could use in the office that was JVM based. We tried Groovy but it didn't take hold, and I was learning Scala when my manager pointed at an example of someone using Clojure successfully at a bank and found it productive, and so I had a look.
I began by learning Clojure from the 'Programming Clojure' book, and I soon found that all the pain points I had with Java just seemed to disappear, it was amazing. I found myself being more way more productive than with Java.
Jon Pither: Why didn't you go for Scala?
Daniel Lebrero: Scala is too difficult for me. I had already learned Erlang and so Scala and Akka should have been easy to pick up, but Scala just seemed to bounce off. There are too many rules and difficult little things to remember. Clojure is a simpler language where you are able to achieve more straight away. I do think that Scala is a good language, but I just think that Clojure is simpler and more practical.
I think I'm more open-minded now after picking up a Lisp - I didn't think I'd ever find myself using a dynamic Lisp. It's been a revelation for me - now I'm talking at events, blogging; I'm passionate about it.
Jon Pither: How did IG start using Clojure?
Daniel Lebrero: We wanted to use Clojure but first we had to de-risk it. We had to ensure that we could deploy a Clojure application into the IG production environment. We picked a low priority project and ran it through the standard deployment pipeline of Maven and Tomcat etc that is widely used here. We also proved out interactions with familiar libraries such as EhCache, Terracotta and Quartz.
Clojure is a simpler language where you are able to achieve more straight away.
Neil Laurance: The first big, important service we built was a 'Market Screener' - basically a tool to filter market data based on some user supplied criteria.
Daniel Lebrero: One team estimated 10 weeks and about 4 devs to do it in Java. We did it in a couple of days using Clojure.
Neil Laurance: The problem was inherently functional, it was about filtering and mapping. We were filtering data in 2-3 lines as opposed to what would have been a hundred or so in Java.
This 'market screener' project sowed the seed. We had shown a much quicker way to develop.
Jon Pither: What does the Clojure landscape look like now?
Neil Laurance: Now we've got a number of Clojure microservices. A microservice architecture provides for better resilience and continuous delivery practices, and allows teams some autonomy in the frameworks and languages they use.
Daniel Lebrero: We're also using Clojure in the big data space with Kafka, Hadoop and Spark. From a Clojure point of view, de-serializing messages from a Kafka topic and doing a little bit of munging before pushing data to another queue is very simple. Spark is also functional and so it's a good match.
Jon Pither: How many devs are writing Clojure?
Neil Laurance: We've got about fifteen developers writing Clojure. We've had graduates that we washed though Clojure and they have come out the other side very keen. They can pick up Clojure and start contributing to the project team within a month which shows us that Clojure is not crazy or difficult.
I would say that IG is a micro-climate of the technology adoption curve - we've got our innovators and early adopters, but there's some way to go to fully cross the chasm. It'll take time, and we've found the best way of getting devs up to speed is to have informal tech sessions. We're getting some traction there.
Jon Pither: Could you tell me more about the tech sessions?
Daniel Lebrero: We have a weekly two hour 'Clojure Club'. Anyone is invited and we try out various advanced things (for example Datomic, core.async, and core.logic). This is a good session for people who know some basic Clojure.
Now we also do a 'Munching Clojure' club; a lunch club that is more targeted at the basics; encouraging people of any level to come and try out Clojure. We work through 4Clojure exercises where there is a range of easy and difficult problems. There's a couple of experts there so that anyone can come and ask questions.
Neil Laurance: We also do brown bag lunch sessions. Interestingly the other non-Clojure teams are starting to watch more Clojure related videos. There is an interest in Rich Hickey's videos such as The Value of Values, Hammock Driven Development, and Simple Made Easy. People are interested in his philosophy and not just Clojure itself.
The people at IG are motivated and enthusiastic technologists, we have a lot of passionate people who want to look at interesting tech.
Jon Pither: What IDE do you use for development here?
Daniel Lebrero: Cursive is great because developers are already using IntelliJ and so it's the simplest way of getting people started.
Jon Pither: What about the Church of Emacs?
Neil Laurance: Some of us are dusting off our old copies of Learning Emacs. I expect Emacs will become more popular as we move to a more mobile working environment.
Daniel Lebrero: The learning curve for both Clojure and Emacs is big. Cursive is awesome and Emacs might be something of a barrier. I use Emacs at home where I am learning it but I use Cursive at the office. It's important to have something that people are used to.
Hiring and Training
Jon Pither: How have you found the hiring process?
Daniel Lebrero: We always look for Java devs as we've still got lots of Java, but we add that Clojure knowledge is a plus. This is one of the reasons we get a lot of candidates, and now we've started asking Clojure questions.
We've found that Clojure has a real positive influence on retention as people have the opportunity to learn it and to code it. This applies to myself also, as I want to code Clojure!
State of Clojure
Jon Pither: What's your view on the current state of Clojure?
Neil Laurance: clojure.spec is out. Spec is a loose typing tool which is nice, but you have to remember to apply the conformations! It's still early days, we shall see.
Daniel Lebrero: The lack of static types is the only drawback or the possibly disagreeable part. It makes it a little harder to sell some times.
Jon Pither: What do you think the regional state of Clojure is? For you Daniel in Spain, and Neil in London?
Daniel Lebrero: There is an active Clojure group in Barcelona. There are people in the Clojurians-spain slack group. There's a conference call Lambda World.
Neil Laurance: The networking side of the meet-ups is extremely useful, to get a sense of where Clojure is happening. There are many success stories right now.
Jon Pither: Any parting thoughts?
Neil Laurance: I want to emphasis Clojure is not a hobby language. IG have extremely high up-time requirements. We've got teams of support engineers to ensure our apps stay up, and we've got Clojure projects running against the most stringent demands.
Daniel Lebrero: Running Clojure in a Java environment is no issue. I was profiling a Clojure app the other day with the YourKit Java profiler. It just works, I easily found a bottleneck and improved it.
Neil Laurance: Sometimes you just need to find some data and massage. Some people spend hours sticking it into Excel and transforming it, and they will often just give up. We had one example of a team manually grabbing data via Postman and sticking into Excel to work with it. We made a quick Clojure equivalent that delivered immediate value. Don't underestimate the power of the REPL!
Check out the IG tech blog.