FP and design flexibility [for Pierre-Yves Saumont]

[Simon Roberts]

One of the things I've long held regarding software design starts with the idea that some aspects of a problem domain are more stable than others. For example, in a typical business system, business domain entities like Customer, Invoice, Product, have significantly more stability than business processes such as how to process a sale. Now, if one arranges that the fundamental building blocks of the software are the more stable parts, then there's a better chance the software will have sufficient flexibility to allow growth and change over time. This notion suggests to me that OO, carefully applied (which is a lot rarer in the real world than one might hope!), makes good sense for business systems, but that procedural design might make more sense for software that models systems where the "business processes" are intrinsically stable (such as computational problems in physics, where presumably the process--the laws of physics--are totally stable).

Assuming you accept a useful degree of merit in that position (which is of course very briefly stated, and not "black vs. white" anyway, so you might reasonably not), then would you care to comment, with that as a backdrop, on how you see the flexibility of functional software designs? Keep in mind, if you would, that while I'm completely comfortable with some ideas of FP (specifically functions as first-class elements of a language, higher order functions, partial application, currying, immutable data, and in particular, the notion of functions that take functions as arguments and return modified functions as results) and I was writing LISP 30+ years ago, I do not claim to be an FP practitioner. Consequently, it might be helpful if you could include some level of examples; what you would do to achieve flexibility, alongside why you see any given technique or approach as providing flexibility.

Hope that rather rambling question makes some sense, I look forward to any insight you can share.

[Junilu Lacar]

Simon Roberts wrote:This notion suggests to me that OO, carefully applied (which is a lot rarer in the real world than one might hope!), makes good sense for business systems

Therein lies the rub.

The question of flexibility is a good one but I'd like to add to it, if I may.

I almost had to inherit a Scala app that was far worse than the worst procedural Java application I have ever had the misfortune to inherit. Luckily, I dodged that bullet but not before I was forced to review the Scala code and argue vigorously with its developers about its lack of organization and clarity. That project, thankfully, got sh*t-canned and I went back to nursing my old Struts 1.0 web app from IBM WAS hell that just can/will not die until 2020.

My point being—and forgive me Simon for interjecting with my misery—that if it's rare in the real world to find OO carefully applied, I would hazard a guess that FP properly applied quite possibly is even more rare. YMMV but it seems the best one can do is to hope and pray that some of the developers on your team know enough to get things to a state of "workable" and "sustainable" for the useful lifetime of the system.

Given that there are far fewer developers who can create good, flexible designs than those who do otherwise, how much more difficult/easy is it to salvage/refactor a poor FP design than it is a poor OO design? Is there anything in FP that makes it more/less amenable to refactoring/redesign? What are some of the major contributors to FP design rigidity that you have had to deal with? With OO designs, the biggest obstacles for me have been lack of layering and modularization and lack of testability (both mainly due to very tight coupling + rampant duplication).

[Sean Corfield]

I'm very interested in hearing Pierre-Yves's response to this question but in the mean time I'd like to offer some thoughts of my own. I studied FP back in the early 80's, my early jobs were assembler, COBOL, and C programming. I picked up C++ in '92, Java in '97, and then over the last decade I've worked with Groovy, then Scala, and for the last nearly six years I've been doing Clojure in production. Along the way I'd been keeping my fingers in the FP pool, always hoping it would go mainstream (in the early 90's I hoped Haskell would the "One True Language"... sigh...).

OOP's great promise to the industry was reusability. We were supposed to be able to build models of things in our businesses and then reuse them across multiple programs. We were supposed to get flexible, reusable abstractions. I think we all know the reality of that? We've seen a lot of "No True Scotsman" arguments around OOP -- "if it doesn't work, you're not doing it right" -- but I view this more as Stockhausen syndrome: we've invested so much as an industry that it's really hard to admit we've made a big, expensive mistake. Tools, training, patterns, consultants -- a huge, unstoppable machine based on helping you get OOP "right". Not all of OOP is bad of course. It's really good for modeling certain things. But if you look at what Alan Kay had in mind, it's not what we got. He had in mind a coarse-grained model of objects, that communicated via "messages". It wasn't really about reusability, it was more about autonomy.

Simon posits that OOP should be good for systems where we have well-defined "entities" and perhaps less well-defined "processes". When I look at that, I see data and functions. In OOP, those are blended together. If your process changes, you need to modify your objects. If your data changes, you need to modify your objects. But with either kind of change, you're changing your fundamental building blocks in a way that is intertwined.

With FP, your data and your functions are separated. Alan Perlis said "It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures." (Epigrams on Programming, 1982). In Clojure, you have a number of abstractions and, for each abstraction, you have a lot of functions that are applicable. Mostly your data is a basic hash map and you can apply any number of generic hash map functions on it. You can have a sequence of your data elements, and apply any number of generic sequence functions on them. 100 functions, one data structure. One of the problems with OOP is that, in general, each entity has a specific class type and therefore a specific set of functions that can operate on it. It becomes hard to compose those functions because they are specific, rather than generic: they aren't abstract enough.

My experience, spanning both FP and OOP, is that you can create extremely flexible systems with FP because functions are composable and you have unlimited "plug'n'play" if you have the right level of data abstractions. Your design can also be flexible at the data level if you ensure your entities are accretive: either you just add new fields or you add composable functions that present a compatible API for your data. Since we switched from an OOP approach to an FP approach at work, we've found our ability to pivot and incorporate new requirements easily has improved dramatically. We can extend our data model easily too (at least accretively -- removing things takes more work).

[Simon Roberts]

Sean Corfield wrote:If your process changes, you need to modify your objects. If your data changes, you need to modify your objects. But with either kind of change, you're changing your fundamental building blocks in a way that is intertwined.

While hoping to keep this on the topic of FP, I have to say that seems to reflect what I see as one of the most common errors in typical OO implementations. Mixing _business_ process with domain entities. Very wrong, but fantastically common. Heck, even the "FP for the rest of us" (or something like that) article that's cited in a useful post in this forum blatantly makes that error when the author suggests that the only reason that adapter and facade are different patterns might be to fulfill a page-count requirement in the contract! Failure to factor out the unrelated (how the business uses customers, accounts, etc, is a totally different concern from "being an account") is nearly universal. Unfortunately, I see that our teaching institutions--from universities on down to the shop-floor--fail to teach "why" we do things in particular ways in OO, only teaching "what to do" which is a near useless recipe. I also think that the reusability thing was surely overblown. Maintainability is the promise that is more relevant, and should be achievable if we'd only approach it right.

But, that's not what I'm hoping to get a discussion about--rather I'm still hoping our visiting author will find time (amongst what are surely a great many questions demanding long, complex, answers!) to put his $0.03 in on the original question.

(oops, and on that basis, I'm still parsing the rest of your comments. I suspect this might not be the kind of question that can be addressed without several weeks of examples!)

[Sean Corfield]

You misunderstood my comment, Simon, and I'm happy to take up the discussion elsewhere but, like you, I'm definitely interested in hearing Pierre-Yves's thoughts on this topic.

[Junilu Lacar]

Simon Roberts wrote:Unfortunately, I see that our teaching institutions--from universities on down to the shop-floor--fail to teach "why" we do things in particular ways in OO, only teaching "what to do" which is a near useless recipe. I also think that the reusability thing was surely overblown. Maintainability is the promise that is more relevant, and should be achievable if we'd only approach it right.

Simon, this is something we've discussed to some extent here. There a still semi-active thread about it in the Rattlesnake Pit: https://coderanch.com/t/672942/requirements-academia-industry

I couldn't agree with you more about maintainability being more relevant.

Cows to both you and Sean for your great insights.  

[Pierre-Yves Saumont]

Sorry for being a bit late to the party. I'll try to answer both the initial question and some comments. First of all, I would say that functional programming is not tied to a specific language. One can write fully functional programs in Java although never using specifically functional constructs. Applying referential transparency, favoring immutability, clearly separating functions from effects, all these techniques can be employed with any language. Some more specific techniques will require more functional friendly languages. Java became a bit more functional friendly with the introduction of lambdas and method references, but higher order functions could already be implemented in Java before Java 8. It was just much more cumbersome. On the opposite side, a (much) more functionally oriented language like Scala is not a guarantee to write better, or even simply more functional, programs. As I already said in another thread, writing good programs is difficult whatever the paradigm. But writing bad programs is easier using the imperative paradigm, which makes FP appear more difficult to beginners. But to benefit from this, one has either to have a very strict discipline, or to use a language that does not allow non functional constructs (which is never the case, even for Haskell).

Now to come back to the initial question, I agree that FP generally appears to be better at solving mathematical problems than business problems, although I don't think business problems are less stable by nature. They are generally much less formalized. They are often so badly stated that we invented a method to handle such problems. We called it "agile" which, in my experience, is a tool supposed to allow programmers to solve problems that can't be clearly expressed before they are solved. I never heard about any programmer needing to be "agile" to solve a mathematical problem.

I use to push this reasoning even further. There is a big difference between how imperative and functional programmers consider programs. Imperative programs are seen as a machines that crunch data to produce a result. They do this by mutating components state, testing the result for some conditions and branching to some other computations. It makes so much sense that imperative programmers make heavy use of logging and debuggers to see what is happening in their programs. (Obviously, this means seeing how different it is from what was expected.) Functional programmers make little use of logging and debuggers. One reason might be that they are confident that if components work in isolation, their composition will produce a predictable result. One way to obtain this is referential transparency. And this is much easier to obtain if the problem is clearly specified, like most mathematical problems generally are. But at a more abstract level, functional programmers do not see programs as a machine producing a solution when it is executed. The program is the solution. This is due to the fact that functional programs are composed of expressions rather that instructions. It is like a mathematical demonstration. Once it is written, you don't have to execute it to produce the solution. It simply is the solution.

But a huge difference between solving business problems and solving mathematical problems is about sharing mutable state. Sharing mutable state is generally a requirement in business programming. Generally, it is much less crucial, if ever needed, when solving mathematical problems. Sharing mutable state is generally solved by "sequentiallizing" access to shared data. In imperative programming, this is probably the most difficult thing to get right. But FP programmers are used to push abstraction as far as possible. So FP oriented languages often offer a way to abstract shared mutable state. One example is the actor model, that is available in Erlang or Scala. But the reality is that the abstracted part is not functional at all. It uses the same techniques as imperative solutions, but this is only implementation detail. This is always the case with FP. A good FP library may be implemented in FP or through imperative techniques (often for performance optimization). This does not matter. What is important is that it should be functional when seen from the user perspective. FP fundamentalists might disagree with this, but if we think about it, all functional programs end in imperative form, the only form that can be executed by a computer.