SimplicityandNetflowProduct

…Simplifications have had a much greater long-range scientific impact than individual feats of ingenuity. The opportunity for simplification is very encouraging, because in all examples that come to mind the simple and elegant systems tend to be easier and faster to design and get right, more efficient in execution, and much more reliable than the more contrived contraptions that have to be debugged into some degree of acceptability….Simplicity and elegance are unpopular because they require hard work and discipline to achieve and education to be appreciated. – Edsger W. Dijkstra

Last week, I remembered the quote from Dijkstra when evaluating a "netflow" oriented product. What do I mean by a "netflow" oriented product ? a software capable of collecting Netflow flows, storing/aggregating them and providing a nice interface (UI or command line) to display them in a efficient way. These requirements look like pretty obvious and building a software to meet the criteria looks very feasible for any proprietary or free software (remark is valid for the both world) vendor or author. During the evaluation of the software, we started by discovering a marketing interface to manage the flow processing and collecting. You know, a very nice looking java interface where you can move nice icons and create link between node. But It doesn't work, with the help of an engineer (without kidding), we were unable to create a Netflow collection on a specific UDP port for netflow version 9. The interface is the "Where is Waldo" configuration interface… Impossible to understand the logic behind and impossible to create what we want. Just a collector on a specific UDP port. We got the brilliant idea to look at the beast from behind and looks for configuration files. That was the second mistake of the day after accepting the evaluation of that product. Running some recursive grep, we were expecting to find some UNIX-like configuration or a nice vty. What a joke… The "Where is Waldo" comic is very organized compare to the location of the various configuration in that product. We found a configuration that could match we tried to describe in the baroque (not in the sense of a artwork) Java interface. Now we asked the fundamental question ? What process to restart in order to take into consideration the new configuration file. The third mistake of that day was to look the various process and how they operate ? What a mix. Too much processes, strange dependencies and when you have a process to monitor the other processes. We are in a kind of trap… We started to forget the basic goal to collect the netflow and the lost engineer started to divert the fundamental question to a dark area. At the end of the day, the "netflow" product is able to generate useless report from a pcap based interface. The worst part is not that one, the web interface is the reflect of the backend but the classical marketing style. Lot of stuff, useless to real analysis. I won't name the product as it's not a recommended product. A very nice example of a very complex product where simplicity was not taken into consideration. Dijkstra quote is so universal… Simplicity is very difficult and difficult to sell. In the netflow world, we really like two netflow products playing in the simplicity and genius field : nfsen/nfdump and Arbor Peakflow.