More CL libs on Github

Recently I made available some GP code I had in Common Lisp (see previous post). Today I also put on my github account the library I did for Ant Colony Optimzation (cl-aco) and a few others like parsers for MKP and QAP, but also a very basic CFFI bindings for libLBFGS. Like before, these are mostly not to get lost since I’m not using/developing  them anymore. But if someone finds this useful in any way, that’s great!

GP code on Github

Some time ago I got an email asking for the code of one of my old papers. Unfortunately the code was lost. But that reminded me that I still had some other code that could actually be made available. Essentially, some Genetic Programming (GP) libraries that I did around seven years ago (which in turn started on some other old code that I had done before).

Fast foward some weeks and it’s now available on my github account these two GP libraries:

  • mini-gp: a minimalistic library that only allows simple tree-based GP
  • core-gp: a larger library that does GP, STGP, GE and GAs.

The code is released as is and it’s not even re-tested. Basically it’s there not to be lost in some old external hard drive. Still, it might be useful to someone who has some interested in GP and Common Lisp. The code it’s not particulary nice in some aspects and it even contains some over-engineering parts, but it’s was fun to do it and back then was very useful to me.

PS: I still have another library for Ant Colony Optimization that I would like to put it on Github too but it still requires a little clean-up. Hopefully in the near future it’s done.

Evolution and Deep Neural Networks

In the last few weeks, a few papers containing evolutionary techniques applied in the context of deep neural networks have been published. For someone with a background on evolutionary computing and interested in everything that is bio-inspired, these are great news! Recently we’ve seen: Evolving Deep Neural Networks , Genetic CNN, Large-Scale Evolution of Image Classifiers and PathNet: Evolution Channels Gradient Descent in Super Neural Networks.

These recent papers are not the first ones on the topic (and won’t be the last) since many different applications of evolutionary techniques to neural networks, including deep ones, have been published in the past. However, it “feels” that finally the field is catching up and paying attention to the very fast developments in neural networks. Especially when organizations like DeepMind and the Google Brain Team are investing in the topic.

The research and development of evolutionary techniques for deep nets is, in my opinion,  very important. These methods have achieved many “human competitive results” and thus have the potential to present innovative solutions and at the same time, reduce human intervention in the process of design and optimization of a deep model. It can also be used to produce new insights that can latter be used to develop new methods, by looking and analyzing the proposed solutions. Some people may simply criticize these approaches by claiming it’s playing LEGOs, “forgetting” that humans are already playing it. Or for the amount of resources required. This is a valid point which just means that more understanding and development is required.

Since there are no free lunches, you need to understand when it makes sense to apply this type of methods as well as how to design them. Unfortunately, it’s very common to see direct applications of concepts that are already outdated. There’s no value in using the standard genetic algorithm from John Holland or the Koza-style genetic programming. Even though they are easy to apply and understand, they are outdated! Another example is not knowing how to analyze an evolutionary algorithm through their representation properties. This means that most likely you will have an inefficient approach.

Recently, the inspiration for most of the new advances in deep learning are coming from the math/game theory/etc., side. And not so much from the biological side, e.g., neuroscience. I would expect more coming from this area of inspiration (an attempt can be found here for example: Towards an integration of deep learning and neuroscience) since ultimately, the human brain is used as the main example of what kind of AI we want/would like to build. However, the brain is not a final product, and it was not designed in a single step. It’s the product of a long evolutionary process (which still goes on)! It means that we need to study and understand more these two so that we can effectively use them for the artificial variants. Deep Neuroevolution should be a path to pursue.

Reading List for F# Learning

There are several books and other resources to learn F# and when someone wants to start looking at the language, one of the first questions is which book to use. I don’t think there is a straight answer mostly because of two main reasons. First, there isn’t a single book that outshines all the others. Although there are some really good books, they actually complement more each other. Second, people have very different backgrounds, motivations and learning styles. And this actually influences how someone gets the most out of a book. For these reasons, I don’t think there is a single book to learn F# from but probably several. At least that’s what it worked best for me and I’m still discovering the language. From all the F# books I managed to get my hands on here’s my view on them and how they can be used to learn the language.

The Must Haves

To start I recommend Programming F# 3.0 by Chris Smith. The book is very well written and as a first introduction to F# it does the job very nicely. It is composed of three parts. The first part deals with all the fundamentals and then proceeds to teach the language in three main programming paradigms (functional, imperative and object-oriented) and ends with a chapter on .NET programming. This gives you a solid foundation on the language. The second part of the book builds on the first one, i.e., advanced topics on functional and object-oriented programming but also other important topics like Asynchronous and Parallel Programming, and scripting using F#. The last part deals with the truly advanced features of the language, such as Type Providers, Computation Expressions and Quotations. However, although the book is very good to learn the basics and the intermediate parts, the advanced stuff seems light.

And this is the reason I believe a good complement to this book is Expert F# 3.0 by Don Syme, Adam Granicz, Antonio Cisternino. In other words, after you finish Part I/II of Chris Smith book, it’s good to start looking at Expert F# in parallel since it contains a much larger selection of topics and goes more in depth in many things. This book offers much more practical examples of F# (for example, web and mobile development, numerical programming) and it clearly shows how to use the advanced features of the language. Moreover, it also contains chapters on topics which are also important, for example, how to organize your code, libraries and inter operating with other languages, packaging, debugging, designing libraries, etc. I find this book a must in your library. Personally, the minus of this book lies exactly with the initial chapters. I believe Programming F# is much better at the basics and offers a better overall overview of the language than Expert F#. These two books complement each other very well and should be more than enough to start with F# from total beginner to advanced user. It also fits well to people with different backgrounds and level of knowledge (e.g., functional programming).

In short, these two books are more than enough to learn the language and get to an intermediate level quickly.

Extra Reading

Another book that I like a lot is F# for C# Developers by Tao Liu. Don’t let the name fool you and assume this book is more suitable if you’re a C# programmer. The book is also divided into three parts and the first one is about learning the language. As the title implies, it’s directed towards someone who has a C# background. Which is fair enough. However, the next two parts of the book are the real thing. Part II dedicates itself mostly to Type Providers and other advanced features of F#. The coverage of type providers is great! The third part concerns with Real-World applications of F#: how to create an F# Portable Library and develop a Windows Store Application, web development using WebSharper, Cloud and Service programming and GPGPU. Interestingly, this book has some sort of hidden gem. Strating at page 392 and ending in page 455, the book contains code in F# of several algorithms. From sorting, tree structures and common algorithms, string operations, graphs and CSP. All of this is not even mentioned in the table of contents!

A small book but which was a nice surprise is F# for Quantitative Finance by Johan Astborg. The book is small but it contains a very nice quick introduction to the language. Again, the book is worth for the follow up material. In this case, using the main topic of Quantitative Finance the author explores topics which might not be covered so well in other books. I found it very nice to include Math.NET numerics, data visualization and how to implement lots of algorithms in F#.

Another small book that could be of interest is Building Web, Cloud, & Mobile Solutions with F# by Daniel Mohl. I personally have mixed feelings about this book. First, the book is not suitable for F# beginners: you must already be comfortable with F#. The title and topics are interesting especially for someone who is a beginner in any of them. Still, the book advances and presents the topics in such a way that it feels that you must know very well all the topics which at some point you wonder how useful the book is. Still, I can still some value in it and since it touches several non-standard topics, it’s worth taking a look.

Advanced Reading

Finally, two books which I believe are also a must have but fall into the advance material section. The first one is Real-World Functional Programming by Tomas Petricek with Jon Skeet. This was actually the first book on F# that I bought but I must confess it was a mistake to use it as a first book to learn. Although the book is very good, I consider it advanced material. It explains lots of concepts and shows them in F# and C#. It’s very good to know more the ins and outs of the language and how to do functional programming in .NET. The problem is that this is not a good approach to learn a language. I can imagine some people might like this style but I personally don’t. I started to appreciate this book after some coding under my fingers.

The second book, I must confess, I didn’t read it yet since it just came out. It’s F# Deep Dives by edited by Tomas Petricek and Phillip Trelford. I only read the sample chapters but from the table of contents and samples, it looks good and another required advanced reading. Sergey Tihon wrote a review of the book.

Other Books

There are more books about F# but everything above is what I think it’s good to read to learn the language and have a good knowledge of what it can be done (and how!). There are some other books that are already outdated and others that don’t add much. For example, The Book of F# by Dave Fancher has an interesting format but I don’t think it adds much to the previous ones.

Finally, the best way to learn a language is to use it :-)

ELS2014: Proceedings, Videos and Slides

The organizers of this year’s European Lisp Symposium made available all the videos of the talks with the respective slides, as well as the PDF containing all the accepted papers.

This is really great and everyone involved in making this happening should be congratulated! For everyone that didn’t attend (like me) it’s a good opportunity to see what happened at ELS. Even though the talks, discussions and networking between participants is the most valuable thing while attending a conference, this is very nice for everyone who was not able to be there.

Once again, kudos to the organizers!

Simple Parallel Array Filtering in F#

Recently I was doing some data processing and since it fit nicely in a parallel setup, I changed my pipeline to use the Array.Parallel module. An easy and simple change that can speed up your task especially if it’s time consuming.

Nonetheless, I also needed Array.filter and to my surprise it wasn’t available in the Parallel module. After some searching, I realized that perhaps the common way to do what I wanted was to use the PSet module (see for example here).

However, I found this nice little example in F# Snippets. In short, a very simple extension of the Parallel module is done to include the filter operation and then it can be used like all others. The relevant code is summarized below:

To me this shows the elegance of F# in two simple aspects: 1) we are able to extended quite easily a standard module; 2) The use of Options. Initially, one could think that special care would be necessary to handle the case where no results are returned from collect. Using Some and None not only makes the coder shorter but also easy to understand and rather elegant.

Articles about using F# to implement Lisp

I was looking for examples of Lisp implementations using F# and found these 3 interesting series of blog posts. You can find other posts about implementing Lisp or Scheme using F# but these were the ones I found more comprehensive and different enough to compare different implementations. Although these Lisp implementations are learning experiments and not for production use, it’s fun to see how they are implemented. The series are the following ones:

Tim Robinson’s Lisp Compiler in F#:

  1. Introduction
  2. Parsing with fslex and fsyacc
  3. Expression trees and .NET methods
  5. What’s next?

Ashley Nathan Feniello’s FScheme series:

  1. Scheme in F#
  2. Just ‘let’ Me Be Already!
  3. Lambda the Ultimate!
  4. Rinse and Recurse
  5. What ‘letrec’ Can’t Do
  6. What’s Lisp Without Lists?!
  7. No Wait, Macro the Ultimate!
  8. Oh, The Humanity!
  9. Language vs. Library
  10. Turning Your Brain Inside Out with Continuations
  11. Playing Dice with the Universe
  12. Functional I/O (or at least “O”)
  13. Functional I/O (including “I” this time)
  14. Functional I/O (Historical Debugging)

Luca Bolognese’s Write Yourself a Scheme in 48 Hours in F#:

  1. Part I
  2. Part II
  3. Part III
  4. Part IV
  5. Part V
  6. Part VI
  7. Part VII