Evolution and Deep Neural Networks

In the last few weeks, a few papers containing evolutionary techniques applied in the context of deep neural networks have been published. For someone with a background on evolutionary computing and interested in everything that is bio-inspired, these are great news! Recently we’ve seen: Evolving Deep Neural Networks , Genetic CNN, Large-Scale Evolution of Image Classifiers and PathNet: Evolution Channels Gradient Descent in Super Neural Networks.

These recent papers are not the first ones on the topic (and won’t be the last) since many different applications of evolutionary techniques to neural networks, including deep ones, have been published in the past. However, it “feels” that finally the field is catching up and paying attention to the very fast developments in neural networks. Especially when organizations like DeepMind and the Google Brain Team are investing in the topic.

The research and development of evolutionary techniques for deep nets is, in my opinion,  very important. These methods have achieved many “human competitive results” and thus have the potential to present innovative solutions and at the same time, reduce human intervention in the process of design and optimization of a deep model. It can also be used to produce new insights that can latter be used to develop new methods, by looking and analyzing the proposed solutions. Some people may simply criticize these approaches by claiming it’s playing LEGOs, “forgetting” that humans are already playing it. Or for the amount of resources required. This is a valid point which just means that more understanding and development is required.

Since there are no free lunches, you need to understand when it makes sense to apply this type of methods as well as how to design them. Unfortunately, it’s very common to see direct applications of concepts that are already outdated. There’s no value in using the standard genetic algorithm from John Holland or the Koza-style genetic programming. Even though they are easy to apply and understand, they are outdated! Another example is not knowing how to analyze an evolutionary algorithm through their representation properties. This means that most likely you will have an inefficient approach.

Recently, the inspiration for most of the new advances in deep learning are coming from the math/game theory/etc., side. And not so much from the biological side, e.g., neuroscience. I would expect more coming from this area of inspiration (an attempt can be found here for example: Towards an integration of deep learning and neuroscience) since ultimately, the human brain is used as the main example of what kind of AI we want/would like to build. However, the brain is not a final product, and it was not designed in a single step. It’s the product of a long evolutionary process (which still goes on)! It means that we need to study and understand more these two so that we can effectively use them for the artificial variants. Deep Neuroevolution should be a path to pursue.

Reading List for F# Learning

There are several books and other resources to learn F# and when someone wants to start looking at the language, one of the first questions is which book to use. I don’t think there is a straight answer mostly because of two main reasons. First, there isn’t a single book that outshines all the others. Although there are some really good books, they actually complement more each other. Second, people have very different backgrounds, motivations and learning styles. And this actually influences how someone gets the most out of a book. For these reasons, I don’t think there is a single book to learn F# from but probably several. At least that’s what it worked best for me and I’m still discovering the language. From all the F# books I managed to get my hands on here’s my view on them and how they can be used to learn the language.

The Must Haves

To start I recommend Programming F# 3.0 by Chris Smith. The book is very well written and as a first introduction to F# it does the job very nicely. It is composed of three parts. The first part deals with all the fundamentals and then proceeds to teach the language in three main programming paradigms (functional, imperative and object-oriented) and ends with a chapter on .NET programming. This gives you a solid foundation on the language. The second part of the book builds on the first one, i.e., advanced topics on functional and object-oriented programming but also other important topics like Asynchronous and Parallel Programming, and scripting using F#. The last part deals with the truly advanced features of the language, such as Type Providers, Computation Expressions and Quotations. However, although the book is very good to learn the basics and the intermediate parts, the advanced stuff seems light.

And this is the reason I believe a good complement to this book is Expert F# 3.0 by Don Syme, Adam Granicz, Antonio Cisternino. In other words, after you finish Part I/II of Chris Smith book, it’s good to start looking at Expert F# in parallel since it contains a much larger selection of topics and goes more in depth in many things. This book offers much more practical examples of F# (for example, web and mobile development, numerical programming) and it clearly shows how to use the advanced features of the language. Moreover, it also contains chapters on topics which are also important, for example, how to organize your code, libraries and inter operating with other languages, packaging, debugging, designing libraries, etc. I find this book a must in your library. Personally, the minus of this book lies exactly with the initial chapters. I believe Programming F# is much better at the basics and offers a better overall overview of the language than Expert F#. These two books complement each other very well and should be more than enough to start with F# from total beginner to advanced user. It also fits well to people with different backgrounds and level of knowledge (e.g., functional programming).

In short, these two books are more than enough to learn the language and get to an intermediate level quickly.

Extra Reading

Another book that I like a lot is F# for C# Developers by Tao Liu. Don’t let the name fool you and assume this book is more suitable if you’re a C# programmer. The book is also divided into three parts and the first one is about learning the language. As the title implies, it’s directed towards someone who has a C# background. Which is fair enough. However, the next two parts of the book are the real thing. Part II dedicates itself mostly to Type Providers and other advanced features of F#. The coverage of type providers is great! The third part concerns with Real-World applications of F#: how to create an F# Portable Library and develop a Windows Store Application, web development using WebSharper, Cloud and Service programming and GPGPU. Interestingly, this book has some sort of hidden gem. Strating at page 392 and ending in page 455, the book contains code in F# of several algorithms. From sorting, tree structures and common algorithms, string operations, graphs and CSP. All of this is not even mentioned in the table of contents!

A small book but which was a nice surprise is F# for Quantitative Finance by Johan Astborg. The book is small but it contains a very nice quick introduction to the language. Again, the book is worth for the follow up material. In this case, using the main topic of Quantitative Finance the author explores topics which might not be covered so well in other books. I found it very nice to include Math.NET numerics, data visualization and how to implement lots of algorithms in F#.

Another small book that could be of interest is Building Web, Cloud, & Mobile Solutions with F# by Daniel Mohl. I personally have mixed feelings about this book. First, the book is not suitable for F# beginners: you must already be comfortable with F#. The title and topics are interesting especially for someone who is a beginner in any of them. Still, the book advances and presents the topics in such a way that it feels that you must know very well all the topics which at some point you wonder how useful the book is. Still, I can still some value in it and since it touches several non-standard topics, it’s worth taking a look.

Advanced Reading

Finally, two books which I believe are also a must have but fall into the advance material section. The first one is Real-World Functional Programming by Tomas Petricek with Jon Skeet. This was actually the first book on F# that I bought but I must confess it was a mistake to use it as a first book to learn. Although the book is very good, I consider it advanced material. It explains lots of concepts and shows them in F# and C#. It’s very good to know more the ins and outs of the language and how to do functional programming in .NET. The problem is that this is not a good approach to learn a language. I can imagine some people might like this style but I personally don’t. I started to appreciate this book after some coding under my fingers.

The second book, I must confess, I didn’t read it yet since it just came out. It’s F# Deep Dives by edited by Tomas Petricek and Phillip Trelford. I only read the sample chapters but from the table of contents and samples, it looks good and another required advanced reading. Sergey Tihon wrote a review of the book.

Other Books

There are more books about F# but everything above is what I think it’s good to read to learn the language and have a good knowledge of what it can be done (and how!). There are some other books that are already outdated and others that don’t add much. For example, The Book of F# by Dave Fancher has an interesting format but I don’t think it adds much to the previous ones.

Finally, the best way to learn a language is to use it :-)

ELS2014: Proceedings, Videos and Slides

The organizers of this year’s European Lisp Symposium made available all the videos of the talks with the respective slides, as well as the PDF containing all the accepted papers.

This is really great and everyone involved in making this happening should be congratulated! For everyone that didn’t attend (like me) it’s a good opportunity to see what happened at ELS. Even though the talks, discussions and networking between participants is the most valuable thing while attending a conference, this is very nice for everyone who was not able to be there.

Once again, kudos to the organizers!

Simple Parallel Array Filtering in F#

Recently I was doing some data processing and since it fit nicely in a parallel setup, I changed my pipeline to use the Array.Parallel module. An easy and simple change that can speed up your task especially if it’s time consuming.

Nonetheless, I also needed Array.filter and to my surprise it wasn’t available in the Parallel module. After some searching, I realized that perhaps the common way to do what I wanted was to use the PSet module (see for example here).

However, I found this nice little example in F# Snippets. In short, a very simple extension of the Parallel module is done to include the filter operation and then it can be used like all others. The relevant code is summarized below:

To me this shows the elegance of F# in two simple aspects: 1) we are able to extended quite easily a standard module; 2) The use of Options. Initially, one could think that special care would be necessary to handle the case where no results are returned from collect. Using Some and None not only makes the coder shorter but also easy to understand and rather elegant.

Articles about using F# to implement Lisp

I was looking for examples of Lisp implementations using F# and found these 3 interesting series of blog posts. You can find other posts about implementing Lisp or Scheme using F# but these were the ones I found more comprehensive and different enough to compare different implementations. Although these Lisp implementations are learning experiments and not for production use, it’s fun to see how they are implemented. The series are the following ones:

Tim Robinson’s Lisp Compiler in F#:

  1. Introduction
  2. Parsing with fslex and fsyacc
  3. Expression trees and .NET methods
  4. http://www.partario.com/blog/2009/06/lisp-compiler-in-f-il-generation.html
  5. What’s next?

Ashley Nathan Feniello’s FScheme series:

  1. Scheme in F#
  2. Just ‘let’ Me Be Already!
  3. Lambda the Ultimate!
  4. Rinse and Recurse
  5. What ‘letrec’ Can’t Do
  6. What’s Lisp Without Lists?!
  7. No Wait, Macro the Ultimate!
  8. Oh, The Humanity!
  9. Language vs. Library
  10. Turning Your Brain Inside Out with Continuations
  11. Playing Dice with the Universe
  12. Functional I/O (or at least “O”)
  13. Functional I/O (including “I” this time)
  14. Functional I/O (Historical Debugging)

Luca Bolognese’s Write Yourself a Scheme in 48 Hours in F#:

  1. Part I
  2. Part II
  3. Part III
  4. Part IV
  5. Part V
  6. Part VI
  7. Part VII

F# Impressions

Last December I decided to start to learn properly F#. Before that I only had done the usual explore and play to have an idea of the language. Anyway, I started to read Programming F# 3.0 by Chris Smith complemented by Expert F# 3.0 by Don Syme, Adam Granicz and Antonio Cisternino. Two excellent books without a doubt.

However, nothing beats actual practice and after I felt a bit comfortable with F#, I started my usual “Hello World” project: an Evolutionary Algorithm library/framework. Personally, this is my ideal beginner project since I can only focus in the learning the language and the task has a good deal of complexity that allows to explore the programming language in many ways. Until now I have completed three iterations, starting with a simple Genetic Algorithm using only the more basic constructs and focusing more in the functional approach, to something more generic and library-like exploring the multi-paradigm functionality of F#.

The impressions so far have been very positive. It mixes very well the functional and object-oriented approaches and the support to switch to parallel programing is very good (for example, you can just change Array.map to Array.Parallel.map). The thing I miss most so far it’s macros like in Common Lisp. I’ve already faced two/three situations which were “screaming” for a macro but I assume this is also a sign that I still need to program more in F#. I haven’t used F# so much in a script way like many people do but that’s something that I need to explore more. I feel it’s a nice language and should be worth knowing it better, especially if you need to work with .NET.

Common Lisp on .NET ?

For the past months I have spent most of my time on .NET using C# but I also would have liked to use Common Lisp on it. I do not know of a CL implementation for .NET and I read some time ago that probably doing one is not that easy, as this answer in Stack Overflow states. I wonder if it is related to floating-point exceptions (if someone knowledgeable in this area would like to comment, I thank you in advance!).

Anyway, I decided to look more carefully to see if I could find something and discovered four projects. Initially I was a bit excited that probably there is something after all. But after a more careful observation, they are far from what you would expect or want. For future reference:

The first project, ClearLisp, looked promising but it’s not even CL. Looking at the documentation you realize some differences between it and CL. Also, the project shows no sign of life since 2008 and after playing with it a little bit, it’s far from a modern CL implementation. The second one is not a proper implementation since it’s using ABCL and IKVM. The third one seems to be a way to connect ECL to .NET and uses it as a scripting language. The fourth one also looks more like an attempt than a real implementation (I didn’t explore it).

In the end, there is really no CL implementation for .NET or even one that compiles to bytecode. There is RDNZL which lets you call .NET libraries from a Common Lisp application. However, it’s not the same as having a real implementation.

Furthermore, the state of Lisp-languages on .NET is not much better since you can find lots of half-baked, abandoned languages/implementations or just without real followers (for example, L#, Yarr, dotlisp, Common Larceny, ayaka and others). The major exceptions are probably IronScheme and Clojure. And even the CLR Clojure implementation is just playing catch-up with the Java one.

The set of programmers interested in CL (or even just Lisp) and .NET is probably extremely small and thus, the current state of affairs. However, I feel that this can still be a good area of opportunity if done right.

Edit 1: zvrba on reddit explains the floats issue, which I put here:

The problem with CLR floating-point is that the virtual machine supports only one floating-point type, double (64-bit). (32-bit) float is a storage format only, so in some cases double rounding occurs, leading to results that differ from “proper” float calculations by 1 ulp.

Edit 2: clausb also on reddit pointed out to a list of Lisp projects on .NET he compiled before.

Disclaimer: the views and opinions expressed on this blog are my own and not those of my employer.